This presentation explains how research impact measurement has changed with the advent of the internet, and provides examples of how impact can be measurement using several online tools.

1. Kinga Hosszu
Outreach Director, Faculty of 1000
Kinga.hosszu@f1000.com
@KingaNYC
http://f1000.com
MEASURING RESEARCH IMPACT ON THE WEB

2. THE FACULTY OF 1000
The Faculty of 1000 is the publisher of unique scientific
literature services that support and inform the work of life
scientists and clinicians worldwide.
http://f1000.com/prime
http://f1000.com/posters
http://f1000research.com

3. “[Journal Impact Factor is] a poor
indicator of citations to specific
papers or of the future performance
of individual researchers”
Nature Materials 12, 89 (2013)
http://bit.ly/1jN7Y6a
“I am sick of impact factors and so
is science.”
Prof. Stephen Curry, Imperial
College, Aug 2012
http://bit.ly/1onlWjj
“The widely held notion that high-impact
publications determine who gets
academic jobs, grants and tenure is
wrong.”
Dr. Michael Eisen, UC Berkeley, Feb
2012
http://bit.ly/1kACAwz
“Citations are heavily gamed and are
painfully slow to accumulate, and overlook
increasingly important societal and clinical
impacts.”
Priem et al., PLoS One 2012
http://bit.ly/1hsIi4c

4. IMPACT FACTORS AND CITATION METRICS
Advantages Disadvantages
Reproducible Slow – delay of up to 2 years
Transparent calculation (kind of) Data not publicly available
Predictor of journal quality Poor predictor of paper and researcher
quality

5. I, Cawi 2001 [CC-BY-SA-2.5 (http://creativecommons.org/licenses/by-sa/2.5)], via Wikimedia Commons
http://commons.wikimedia.org/wiki/File:Internet-Sign.jpg

6. TOOLS FOR SHARING RESEARCH CONTRIBUTIONS...
Sum of total impact > Sum of research citations

7. Themes of DORA recommendations:
1.Eliminate the use of journal-based metrics in funding, appointment, and promotions
2.Assess research on its own merits not the journal
3.Capitalize on opportunities provided by online publication such as exploring new
indicators of significance and impact
DORA has been signed by >9000 individuals and nearly 400 organizations, including:
American Association for the Advancement of Science (AAAS)
American Society for Cell Biology
Howard Hughes Medical Institute
Faculty of 1000
Public Library of Science
http://am.ascb.org/dora/

8. ALTMETRICS AND ARTICLE-LEVEL METRICS – DISAMBIGUATION
Article level metrics (ALMs): established metrics that measure the
overall performance and reach of published research articles
(bibliometrics + altmetrics)
Altmetrics: A measure of web-based article usage and reader interaction.
Example: how many data and knowledge bases refer to the work, article
views, downloads, or mentions in social media and news

9. ALTMETRICS: WHY ARE THEY IMPORTANT? - FUNDING
“For all new grant applications from 14 January, the US
National Science Foundation (NSF) asks a principal
investigator to list his or her research “products” rather than
“publications” in the biographical sketch section. This means
that, according to the NSF, a scientist's worth is not
dependent solely on publications. Data sets, software and
other non-traditional research products will count too.”
Heather Piwowar, UBC, in Nature Jan 2013
http://www.nature.com/nature/journal/v493/n7431/full/49315
9a.html

10. ALTMETRICS: WHY ARE THEY IMPORTANT? - PROMOTIONS
“Metrics --- such as the number of article citations, your h-index or
others (such as those available at ImpactStory.org) --- can be
useful in making the case that the publication or scholarly work
was significant. The Faculty Promotions Committee discourages
the use of journal-based metrics (such as journal impact factors),
since it is the quality and importance of the research contribution
itself that is the key.”
Excerpt from UC Denver promotions handbook guide to dossier preparation

11. Scholarly Public
Recommendations F1000 Popular Press
Citations Traditional citations Wikipedia
Discussions Scholarly blogs Blogs, Twitter
Saved Mendeley, CiteULike Google Scholar
Views/Downloads PDF views HTML views
CITATIONS CAN MEASURE…

12. Scholarly Public
Recommendations F1000 Popular Press
Citations Traditional citations Wikipedia
Discussions Scholarly blogs Blogs, Twitter
Saved Mendeley, CiteULike Google Scholar
Views/Downloads PDF views HTML views
ALTMETRICS CAN MEASURE…

13. SOME TOOLS TO INTERACT WITH RESEARCH CONTRIBUTIONS

16. Source: http://impactstory.org/collection/v7b25c

17. ALTMETRICS TOOLS
Source: http://www.altmetric.com/details.php?citation_id=938211

20. HOW F1000PRIME WORKS
• Faculty of over 5000 peer-nominated scientists and clinical researchers
• Faculty Members select, rate and comment on the top research
• Assigns one of three positive ratings: Exceptional/ Very Good/ Good
• Text also serves as a short, expert, recommendation written for a global
readership
• Publishes about 1500 recommendations per month (>150,000 published to
date)

21. F1000, ALTMETRICS AND RESEARCH IMPACT
Collection of studies of F1000Prime: http://bit.ly/F1000studies
• Published study by the Welcome Trust: F1000Prime highlights
important papers that bibliometric indicators alone would miss
• A study by the Medical Research Council: F1000Prime
recommendations indicate future citation impact
Studies have shown that tweets and number of readers on Mendeley
predict citation impact

22. (ALT)METRICS – ADVANTAGES AND DISADVANTAGES
Advantages Disadvantages
Fast – data available immediately Heterogeneity across tools
All research products tracked Emergence and disappearance of
tools
Lots of open data and tools
available
Can be manipulated
Much broader picture of impact Can lack context/meaning

23. SOME PERSPECTIVE. WHAT IS IMPACT?
http://blog.f1000.com/2013/05/20/what-is-impact/

24. Sutherland WJ, Goulson D, Potts SG, Dicks LV. PLoS ONE 2011; 6(11):e27537

25. IMPACT - DOWNLOADS
Source: http://www.jmedicalcasereports.com/mostviewed/alltime

26. IMPACT – LIVES SAVED
Research finds that antibiotic
drug almost halves AIDS-
related death in children

27. SUMMARY
• All metrics – citation and non-citation metrics – have advantages and
disadvantages
• Impact factors are not good for assessing individual papers and
scientists
• The internet has given us a wealth of new data and tools for a broader
picture on the impact
• Still trying to understand the meaning of new metrics – we need data
with context
• Some “alternative” metrics are beginning to influence research
assessment

28. Kinga Hosszu
Outreach Director, Faculty of 1000
http://f1000.com