Scholarly Tweets: Measuring Research Impact via Altmetrics

FEATURE / MANCHETTE

Scholarly Tweets: Measuring Research Impact via Altmetrics

Robin M. Featherstone

Robin M. Featherstone1, Alberta Research Centre for Health Evidence, Department of Pediatrics, University of Alberta, Edmonton Clinic Health Academy, 11405 - 87 Avenue, Edmonton, AB T6G 1C9.

1Corresponding author. (e-mail: feathers@ualberta.ca)

JCHLA / JABSC 35: 60–63 (2014) doi: 10.5596/c14-015

An anecdotal introduction to altmetrics

Last spring a faculty member asked me to find their h-index, collect the Impact Factor (IF) for all the journals in which they were published, and compile the number of citations each of their articles had received. When I presented the results, they insisted their work was more influential than what the metrics suggested. I explained why IFs for specialized journals were lower than more generalized publications such as Science or Nature, and why citation counts for recent articles need time to grow. I introduced the topic of altmetrics, or alternative metrics, and how they could help this faculty member build a more complete picture of their research impact.

Altmetrics measure research impact by including references outside of traditional scholarly publishing [1]. These social web metrics were first proposed in 2010 as a response to scholars moving their work online [2]. Altmetrics analyze tweets, blogs, presentations, news articles, comments, or any social commentary about a diverse group of scholarly activities that are captured on the web [2].

As librarians have been explaining for years, there are limits to what IF and h-index figures can tell us. A junior faculty member may have created and shared hundreds of captivating lectures online but only published a few articles. That teaching is not reflected in their h-index. They may author a widely followed blog in which they engage with an audience of academic peers, but there is no IF for the blog. As numbers of Twitter followers or Facebook friends quantify social media activity, altmetrics measure and rank researcher output, impact, and influence from the social web.

Before continuing, I should declare any possible conflicts of interest. I have been a tester for Altmetric.com since September 2013, and I have enjoyed free access to the Altmetric.com Explorer. Aside from that association, I am a health librarian working in an academic research centre without any financial interest in altmetric products. I am writing from the perspective of an information specialist who uses altmetrics to track research publications, to assess the impact of my centre's social media communication strategy, and to plan future knowledge dissemination projects.

I first heard about altmetrics at the Medical Library Association conference in 2013. The topic of this intriguing new cross-section of social media and article metrics was heard at numerous sessions. Altmetric start-ups were pushing products that measured research in a different way, and librarians were excited to learn more.

The obvious caveat about altmetrics is that they are only valid and valuable for the most recent publications [3, 4]. A 2013 publication in Annals of Internal Medicine on lifestyle interventions for diabetes [5] received a score of 148 by the Altmetric.com Explorer. The score was calculated based on 128 Twitter users sharing the publication, two bloggers citing the article, and seven Facebook users mentioning it. In context, the article was amongst the highest ever scored in Annals of Internal Medicine, (ranked 37 out of 2470). In contrast, a 2005 publication from the same journal on coronary artery disease [6] received no score. Social media mentions are rare for articles published prior to 2011 [4, 7], and altmetric products often exclude older datasets in their analysis [8].

Some altmetric tools, such as Altmetric.com in the aforementioned example, tell us about individual articles and others tell us about researchers. Researcher-focused products, such as ImpactStory and ResearchGate, resemble familiar social networking sites in that they rely on contributors creating and maintaining personal profiles. The evolution from Facebook to LinkedIn to ImpactStory makes logical sense. User-contributed profiles became online résumés and then dynamic curricula vitae with embedded metrics for research products. For researcher-focused altmetric tools, older publications, presentations, and products can be manually added. These products that tell us about researchers are more likely to include contributions prior to 2011, and for that reason they are better for analyzing research output over time than article-focused altmetric tools.

Because different altmetric tools tell different stories, the landscape is full of start-ups positioning themselves as investment-worthy knowledge providers. The following is an overview of a few key players that have persevered past infancy.

Altmetric.com scores articles with embeddable, donut-shaped badges. Subscription costs vary and the company has been generous to librarians in offering free accounts during their start-up phase. Their application programming interface (API) is available for free to any web developer who wants to embed Altmetric.com badges on their site. The company has published extensively on the subject of altmetrics [2, 911] and their datasets are contributing to bibliometric research [4, 12, 13]. Altmetric.com collects and analyzes mentions on social media sites, particularly Twitter and Facebook. The Altmetric.com Explorer (available by subscription) searches datasets by keywords and subject headings, but it works best with PubMed Identifiers (PMIDs), International Standard Serial Numbers (ISSNs), or Digital Object Identifiers (DOIs).

ImpactStory.org is an open-source product that connects PMIDs, DOIs, GoogleScholar citations, ORCID identifiers (unique researcher identifiers), and SlideShare profiles to count “Research Products.” ImpactStory.org creates a free public profile for the individual researcher that includes their Wikis and blogs and praises their Open Access publications with a medal ranking. ImpactStory helps scholars create and disseminate online resumes in a similar way to LinkedIn.

PLOS Article-Level Metrics examines the overall “performance and reach” of articles, and is available for every article published by PLOS (Public Library of Science) [14]. PLOS Article-Level Metrics aggregate usage data (i.e., downloads), citations, ratings, social networking mentions, blogs, and media mentions. Like Altmetric.com, PLOS distributes a free API to share their article metrics on third-party websites.

Plum Analytics.com, which was acquired by EBSCO Information Services in January 2014, gathers online metrics of research output [15]. Plum Analytics’ “artifacts” are collected in the PlumX database and include articles, blog posts, book chapters, conference papers, datasets, patents, even source code. Artifacts are aggregated and presented in graphs that summarize the output of an author or a group (lab, department, etc.).

ResearchGate.net also claims to measure impact in a new way and ranks “scientific reputation” through their RG Score [16]. ResearchGate hosts an open platform for researchers to share and discuss their work. Products from researcher profiles contribute to RG Scores, as do evaluations of those products by ResearchGate peers. Aggregated RG Scores are also presented for institutions based on member contributions. For example, at the time of writing, the University of Alberta had a Total RG Score of 45,512.24 from 4,706 members; the top three ranked institutions were the Chinese Academy of Sciences (106,070.02), the University of São Paulo (91,088.50), and the French National Centre for Scientific Research (80,190.02).

The future use of such large-scale, quantitative measurements of research output will be of particular interest to funding agencies. In an environment of accountability, researchers will require assistance disseminating “products” that will be counted by a new generation of assessment instruments. Librarians who understand altmetrics will be available to advise on targeted social media dissemination strategies that link products with profiles and contribute to individual and institutional rankings.

Challenges to measuring social media output

As expert searchers of grey literature will recognize, it is a nightmare task to capture every social media mention, tweet, blog comment, SlideShare upload, etc., for a particular researcher or publication. As pointed out by Hausten, et al. [12], an article may be tweeted in such a way that it is not easily identified (e.g., “Great article by @dparton in the latest Lancet”). The lack of standardization in social media communication results in questionable data accuracy by altmetric providers [17], and bibliographic analyses using altmetrics include lengthy discussions of the limitations of their data sources.

Altmetric product development is occurring within a larger movement to utilize available web data [3]. Webometrics require number-crunching computing power to be coupled with clean web design and the principles of information organization to facilitate the union. It is no surprise that librarians, such as Mike Buschman with Plum Analytics, are contributing to the development of altmetric tools. To achieve accurate records of scholarly output, altmetric products rely on PMIDs, DOIs, and ORCID identifiers. One lesson to take away from this column is that smart self-promoters include a PMID or DOI when they tweet or blog about their research publications. Including unique identifiers is the best way to ensure that social media output is counted by almetric products.

Gaming altmetrics

No one blames the researcher who wants to give their publication a little boost with a tweet or a Facebook mention, but the methods for collecting and displaying evidence of researcher output ought to make such activities transparent [18]. Self-citing or self-tweeting does not qualify as unethical; however, those self-tweets should have lesser value or be regarded in a separate context from legitimate sharing by unbiased experts. In addition, data collector spamming needs to be detected and eliminated by altmetric proviers through transparent methods. Automated programs that “game” the analysis and inflate rankings hamper producers. Anyone using altmetrics should recognize this gaming phenomenon and the potential for research rankings to be artificially inflated.

In addition to catching incidents of gaming, another future challenge for altmetric start-ups will be standardizing methods of counting the online artifacts of research output. An analogy can be made between the need to have COUNTER compliant statistics for journal usage. Without standardization in altmetrics, these tools are just comparing apples to oranges. A researcher could have over 1000 “products” in one altmetric measurement but a low “rank” and fewer “artifacts” in another. When an altmetric product achieves the equivalent recognition of the gold-standard IF academic institutions can then review their tenure dossiers using these values. However, at this early stage in product development, the contribution of almetrics to the process of scholarly promotion is limited.

Altmetrics in context

PLOS maintains a collection of scholarly altmetrics publications to help monitor this rapidly growing area of research [19]. Numerous investigations have focused on the relationship between altmetrics, citation counts, and IF scores [4, 10, 12, 13, 20, 21]. Findings from these studies suggest a positive but weak correlation between altmetrics and traditional impact measures [4]. Owing to the variety of altmetrics rankings and the different methods of comparison, these analyses are limited. Although proponents of altmetrics compare new tools against the standard citation measures, the evidence does not support replacing traditional citation metrics with altmetrics. However, information scientists agree citations and altmetrics measure different types of impact [4, 12]. In a society where social media is pervasive, it means something for a scholarly article mention to receive a million likes on Facebook, independent and regardless of any eventual citation count.

A more fruitful approach to webometrics research will combine IF and other available datasets with altmetrics to examine social media use and knowledge dissemination strategies. For example, using altmetrics with publisher data could tell us how many times an article was tweeted compared with how many times the full-text was downloaded. We can compare downloads for high and low altmetrics scores or between and across fields for different social media sites. Altmetrics offer assessment opportunities for granting agencies, research centres, universities, and libraries. Alternative metrics also provide insight into the results of social media engagement strategies and deserve to be integrated into knowledge translation assessment.

Altmetrics in health

Costas et al. found that the highest share of publications with altmetrics scores came from the biomedical and health sciences field [4]. An analysis of Twitter use in biomedicine found that 20.4% of the articles indexed in PubMed in 2012 received at least one tweet [12]. This percentage had increased from 2.4% in 2010 and 10.9% in 2011 [12]. Researchers, particularly in biomedicine, are increasingly using social media for health promotion [22, 23], education [24], and to facilitate communication and improve knowledge [25].

Despite evidence of the value of social media in healthcare, I often hear academic clinicians dismiss tools like Twitter. They created an account, used it for a couple of weeks, and then lost interest. Because my own experience with social media had a similarly rocky start, I understand these reservations about using Twitter, Facebook, Pinterest, Slideshare, and the myriad other social networking sites. I have grown to recognize their value from working in an academic research centre. In healthcare, Twitter is being used to share clinical guidelines, webinars that teach research methods, or to facilitate online journal clubs; SlideShare disseminates conference proceedings; wikis are used for teaching; and blogs generate new ideas. The social web is a complex landscape full of everything from the Centre for Reviews and Dissemination sharing reporting discrepancies for clinical trial registries to Will Smith fan selfies and thinspo (so-called inspirational images for dieters). Some aspects of social media activity are worth avoiding, but many others are worth the interaction.

Social media presences are commonplace for research groups, granting agencies, libraries, and universities. If academics are investing energy in social media activities, then we should learn what we are gaining. Through altmetrics, I can find out how many people saw our research centre's tweets about clinical guidelines. I can also learn where in the world those people are located and gain insight into future audiences for health information products. If you invest in social media activities for research, then altmetrics can measure your return on that investment.

Future directions

Publishers, like PLOS, are among the most ardent early adopters of altmetrics. The APIs of altmetrics are now a common feature on the table of contents pages in many journals. Social media metrics are becoming more pervasive in the information landscape, but to retain permanence altmetric products must focus on building trust with the researcher community. We trust IF citation metrics because of the transparency in their data collection and analysis. Altmetric tools need to acknowledge and guard against incidents of gaming and to present self-promotion activities separately in their analysis. As information specialists who teach and utilize IFs and citation metrics, we should also understand how altmetrics are calculated and how they can benefit our institutions and researchers.

References

1.Baynes G. Scientometrics, bibliometrics, altmetrics: Some introductory advice for the lost and bemused. Insights. 2012;25(3):311–5. doi: 10.1629/2048-7754.25.3.311.
2. Priem J, Taraborelli D, Groth P, Neylon C. Altmetrics: A manifesto 2010. Available from: http://altmetrics.org/manifesto/.
3. Galligan F, Dyas-Correia S. Altmetrics: Rethinking the Way We Measure. Serials Review. 2013;39(1):56–61. doi: 10.1016/j.serrev.2013.01.003.
4. Costas R, Zahedi Z, Wouters P. Do altmetrics correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective. arXiv preprint arXiv:14014321. 2014.
5. Schellenberg ES, Dryden DM, Vandermeer B, Ha C, Korownyk C. Lifestyle Interventions for Patients With and at Risk for Type 2 Diabetes: A Systematic Review and Meta-analysis. Annals of Internal Medicine. 2013;159(8):543–51. doi: 10.7326/0003-4819-159-8-201310150-00007.
6. Clark AM, Hartling L, Vandermeer B, McAlister FA. Meta-Analysis: Secondary Prevention Programs for Patients with Coronary Artery Disease. Annals of Internal Medicine. 2005;143(9):659–72. doi: 10.7326/0003-4819-143-9-200511010-00010.
7. Haustein S, Peters I, Sugimoto CR, Thelwall M, Larivière V. Tweeting biomedicine: An analysis of tweets and citations in the biomedical literature. Journal of the Association for Information Science and Technology. 2013;65(4):656–69.
8. Altmetric.com. What does Altmetric Do? Discover what do we do and how. 2014 [25 April]. Available from: http://www.altmetric.com/whatwedo.php.
9. Priem J, Groth P, Taraborelli D. The Altmetrics Collection. PLoS ONE. 2012;7(11). doi: 10.1371/journal.pone.0048753.
10. Priem J, Piwowar HA, Hemminger BM. Altmetrics in the wild: Using social media to explore scholarly impact. arXiv preprint arXiv:12034745. 2012.
11. Adie E, Roe W. Altmetric: enriching scholarly content with article-level discussion and metrics. Learned Publishing. 2013;26(1):11–7. doi: 10.1087/20130103.
12. Haustein S, Peters I, Sugimoto CR, Thelwall M, Larivière V. Tweeting biomedicine: An analysis of tweets and citations in the biomedical literature. Journal of the Association for Information Science and Technology. 2013.
13. Thelwall M, Haustein S, Larivière V, Sugimoto CR. Do Altmetrics Work? Twitter and Ten Other Social Web Services. PLoS ONE. 2013;8(5). doi: 10.1371/journal.pone.0064841.
14. PLOS (Public Library of Science). PLOS Article-Level Metrics: Overview. 2014 [25 April]. Available from: http://article-level-metrics.plos.org/alm-info/.
15. News Release: PlumTM Analytics Becomes Part of EBSCO Information Services Ipswich. MA: EBSCO Information Services; 2014 [1 April]. Available from: http://www.ebscohost.com/newsroom/stories/plum-analytics-becomes-part-of-ebsco-information-services.
16. ResearchGate.net. RG Score: A new way to measure scientific reputation. 2014 [25 April]. Available from: http://www.researchgate.net/publicprofile.RGScoreFAQ.html.
17. Wouters P, Costas R. Users, Narcissism and Control: Tracking the Impact of Scholarly Publications in the 21st Century. Centre for Science and Technology Studies, Leiden University; 2012. SURFfoundation, The Netherlands.
18. Lin J. A case study in anti-gaming mechanisms for altmetrics: PLOS ALMs and DataTrust. 2012. Available from: http://altmetrics.org/altmetrics12/lin/.
19. PLOS (Public Library of Science). Table of Contents: Altmetrics Collection. 2014 [25 April]. Available from: http://www.ploscollections.org/altmetrics.
20. Waltman L, Costas R. F1000 Recommendations as a Potential New Data Source for Research Evaluation: A Comparison With Citations. 2013. Centre for Science and Technology Studies, Leiden University, The Netherlands. Available from: https://openaccess.leidenuniv.nl/handle/1887/20766.
21. Zahedi Z, Costas R, Wouters P. How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications. Scientometrics. 2013; March:1–23. doi: 10.1007/s11192-014-1264-0.
22. Williams G, Hamm MP, Shulhan J, Vandermeer B, Hartling L. Social media interventions for diet and exercise behaviours: a systematic review and meta-analysis of randomised controlled trials. BMJ Open. 2014;4(2). doi: 10.1136/bmjopen-2013-003926.
23. Hamm MP, Chisholm A, Shulhan J, Milne A, Scott SD, Given LM, et al. Social media use among patients and caregivers: a scoping review. BMJ Open. 2013;3(5). doi: 10.1136/bmjopen-2013-002819.
24. Hamm MP, Klassen TP, Scott SD, Moher D, Hartling L. Education in Health Research Methodology: Use of a Wiki for Knowledge Translation. PLoS ONE. 2013;8(5):e64922. doi: 10.1371/journal.pone.0064922.
25. Hamm MP, Chisholm A, Shulhan J, Milne A, Scott SD, Klassen TP, et al. Social media use by health care professionals and trainees: a scoping review. Academic Medicine: Journal of the Association of American Medical Colleges. 2013;88(9):1376–83. doi: 10.1097/ACM.0b013e31829eb91c.

Refbacks

  • There are currently no refbacks.




 

JCHLA/JABSC | CHLA/ABSC on Twitter