This study analyzed the relationship between social media and traditional bibliometric metrics through two case studies. The first examined tweets of biomedical papers and found Twitter coverage increasing but still low, with weak correlations between tweets and citations. The second analyzed astrophysicists on Twitter and also found negative or weak correlations between tweeting and publication/citation metrics. While tweet counts vary across fields and papers, tweets alone cannot replace citations as impact measures. More research is needed on altmetric meanings and differences across fields through user surveys.

1. Empirical analyses of scientific
papers and researchers on Twitter:
Results of two studies
Stefanie Haustein, Timothy D. Bowman, Kim Holmberg,
Vincent Larivière, Isabella Peters, Cassidy R. Sugimoto, & Mike Thelwall

2. Background
• when Garfield created SCI, sociologists of science
analyzed meaning of publications and citations
(Merton, Zuckerman, Cole & Cole, etc.)
• sociological research
• What is it to publish a paper?
• What are the reasons to cite?
• empirical bibliometric research
• disciplinary differences in publication
and citation behavior
• delay and obsolescence patterns

3. Background
• empirical studies helped sociologists to understand
structure and norms of science
• for bibliometricians, studies provided a theoretical
framework and legitimation to use citation analysis
in research evaluation
• knowledge about disciplinary differences and
obsolescence patterns helped to normalize statistics
and create more appropriate indicators

4. Background
• recently social-media metrics have become
important in the scholarly world
• suggestions to complement (or even replace)
citation analysis by so-called ”altmetrics“
• broader audience (not just citing authors)
• more timely
• however, similar to bibliometrics in the 1960s,
little is known about the actual meaning of various
social-media counts

5. Research questions
• What is the relationship between social-media and
citation counts?
• How do various social-media metrics differ?
• Why are papers tweeted, bookmarked, liked…?
• Who tweets (bookmarks, likes…) scientific papers?
• How do these aspects differ across scientific disciplines?
Two case studies on Twitter
• large-scale analysis of tweets of biomedical papers
• in-depth analysis of astrophysicists on Twitter

6. Aim of the study
• large-scale analysis of tweets of biomedical papers
• Twitter coverage
• Twitter citation rates (tweets per paper)
• correlation with citations
• discovering differences between:
• documents
• journals
• disciplines & specialties
! providing empirical framework to understand the extent
to which biomedical journal articles are tweeted
Study I: Tweeting biomedicine
Haustein, S., Peters, I., Sugimoto, C.R., Thelwall, M., & Larivière, V. (in press). Tweeting biomedicine: an analysis of tweets and citations
in the biomedical literature. Journal of the American Society for Information Science and Technology, http://arxiv.org/abs/1308.1838.

7. Data sets & methods
• 1.4 million PubMed papers covered by WoS
• publication years: 2010-2012
• document types: articles & reviews
• matching of WoS and PubMed
• tweet counts collected by Altmetric.com
• collection based on PMID, DOI, URL
• matching WoS via PMID
• journal-based matching of NSF classification
• tweets per article, Twitter coverage and correlation
with citations for:
• journals
• NSF disciplines and specialties
Study I: Tweeting biomedicine

8. Data sets & methods: framework
Study I: Tweeting biomedicine

9. Data sets & methods: correlations
Study I: Tweeting biomedicine
PY=2010 PY=2011 PY=2012

10. Results: documents
Study I: Tweeting biomedicine
Publication
year
Twitter
coverage
Papers
(T≥1)
Spearman's ρ Mean Median Maximum
T2010
2.4% 13,763 .104**
2.1 1 237
C2010 18.3 7 3,922
T2011
10.9% 63,801 .183**
2.8 1 963
C2011 5.7 2 2,300
T2012
20.4% 57,365 .110**
2.3 1 477
C2012 1.3 0 234
T2010-2012
9.4% 134,929 .114**
2.5 1 963
C2010-2012 5.1 1 3,922
• Twitter coverage is quite low but increasing
• correlation between tweets and citations is very low

11. Results: documents
Study I: Tweeting biomedicine
Article Journal C T
Hess et al. (2011). Gain of chromosome band 7q11 in papillary thyroid carcinomas of young patients
is associated with exposure to low-dose irradiation
PNAS 9 963
Yasunari et al. (2011). Cesium-137 deposition and contamination of Japanese soils due to the
Fukushima nuclear accident
PNAS 30 639
Sparrow et al. (2011). Google Effects on Memory: Cognitive Consequences of Having Information at
Our Fingertips
Science 11 558
Onuma et al. (2011). Rebirth of a Dead Belousov–Zhabotinsky Oscillator
Journal of Physical
Chemistry A
-- 549
Silverberg (2012). Whey protein precipitating moderate to severe acne flares in 5 teenaged athletes Cutis -- 477
Wen et al. (2011). Minimum amount of physical activity for reduced mortality and extended life
expectancy: a prospective cohort study
Lancet 51 419
Kramer (2011). Penile Fracture Seems More Likely During Sex Under Stressful Situations
Journal of Sexual
Medicine
-- 392
Newman & Feldman (2011). Copyright and Open Access at the Bedside
New England
Journal of Medicine
3 332
Reaves et al. (2012). Absence of Detectable Arsenate in DNA from Arsenate-Grown GFAJ-1 Cells Science 5 323
Bravo et al. (2011). Ingestion of Lactobacillus strain regulates emotional behavior and central GABA
receptor expression in a mouse via the vagus nerve
PNAS 31 297
Top 10 tweeted documents: catastrophe & topical / web & social media / curious story
scientific discovery / health implication / scholarly community

12. Results: journals
• 97.7% of 3,812
journals at least
tweeted once
• two-thirds of
journals have
coverage below
20% and Twitter
citation rate < 2.0
• high Twitter citation
rates often caused
by few papers
• high coverage and
Twitter citation rates
for general journals
Study I: Tweeting biomedicine

13. Results: disciplines
Study I: Tweeting biomedicine

14. Results: specialties
Study I: Tweeting biomedicine
• specialties differ in
terms of coverage,
Twitter citation rate
and correlations
with citations
• 47 of 61 specialties
show low positive,
3 negative and 13
no correlation
bubblesize=Twittercitationrate

15. Aim of the study
• in-depth analysis of astrophysicists on Twitter
• number of tweets, followers, retweets
• characteristics of tweets: RTs, @messages,
#hashtags, URLs
• comparison with scientific output
• publications
• citations
• comparison of tweet and publication content
! provide evidence in how far astrophysicists on Twitter
use Twitter for scholarly communiation
Study II: Astrophysicists on Twitter
Haustein, S., Bowman, T.D., Holmberg, K., Larivière, V., & Peters, I., (submitted). Astrophysicists on Twitter: An in-depth analysis of
tweeting and scientific publication behavior. Aslib Proceedings.

16. Data sets & methods
• 37 astrophysicists on Twitter identified by
Holmberg & Thelwall (2013)
• web searches to identify person behind account
• publications in WoS journals
• publication years: 2008-2012
• author disambiguation
• Twitter account information
• 68,232 of 289,368 tweets downloaded and analyzed:
• number of RTs per tweet
• % of tweets that are RTs
• % of tweets containing #hashtags, @usernames, URLs
Study II: Astrophysicists on Twitter
Holmberg, K., & Thelwall, M. (2013). Disciplinary differences in Twitter scholarly communication. In: Proceedings of ISSI 2013 –
14th International Conference of the International Society for Scientometrics and Informetrics, Vienna, Austria (Vol. 1, pp. 567-582).

17. Data sets & methods
• grouping astrophysicists according to tweeting and
publication behavior
• analyzing differences of tweeting characteristics
between user groups
Study II: Astrophysicists on Twitter
Selected
astrophysicists
(N=37)!
tweet rarely
(0.0-0.1 tweets
per day)!
tweet
occasionally
(0.1-0.9)!
tweet
regularly
(1.2-2.9)!
tweet
frequently
(3.7-58.2)!
total
(publishing activity)!
do not publish
(0 publications 2008-2012)!
--! --! 1! 5! 6!
publish occasionally
(1-9)!
4! 3! 4! 2! 13!
publish regularly
(14-37)!
--! 5! 5! 3! 13!
publish frequently
(46-112)!
1! 3! 1! --! 5!
total
(tweeting activity)!
5! 11! 11! 10! 37!

18. Data sets & methods
• comparison of tweet and publication content
• extraction of noun phrases from tweets and abstracts
• limited to 18 most frequently publishing astrophysicists
to ensure certain number of abstracts
• analyzing overlap of character strings
• calculating similarity with cosine per person and overall
Study II: Astrophysicists on Twitter
Selected
astrophysicists
(N=37)!
tweet rarely
(0.0-0.1 tweets
per day)!
tweet
occasionally
(0.1-0.9)!
tweet
regularly
(1.2-2.9)!
tweet
frequently
(3.7-58.2)!
total
(publishing activity)!
publish regularly
(14-37)!
--! 5! 5! 3! 13!
publish frequently
(46-112)!
1! 3! 1! --! 5!
total
(tweeting activity)!
1! 8! 6! 3! 18!

19. Results: correlations
• comparison of Twitter and publication activity and impact
• publications and tweets per day: ρ=−0.339*
• citation rate and tweets per day: ρ=−0.457**
• citation rate and RT rate: ρ=0.077
Study II: Astrophysicists on Twitter

20. Results: characteristics
Study II: Astrophysicists on Twitter
Mean share of tweets containing at least one user name or
URL per person per group

21. Results: content similarity
Study II: Astrophysicists on Twitter
• overall similarity between abstracts and tweets is low
• cosine=0.081
• 4.1% of 50,854 tweet NPs in abstracts
• 16.0% of 12,970 abstract NPs in tweets
• Twitter coverage among most frequent abstract terms is
high, although this differs between users
• 97,1% of 104 most frequent noun phrases on Twitter

22. Conclusions
• Twitter coverage of biomedical papers is low but increasing
• number of tweets per paper varies between journals,
disciplines, specialties and from year to year
! tweet counts need to be normalized accordingly
• correlations between tweet and citation counts are low
(biomedical papers) or even moderately negative
(astrophysicists)
! tweets cannot replace citations as measures of
scientific impact
! challenge is to differentiate between high tweet counts
because of value (to scientists and/or the general public)
and curiosity

23. Outlook
• user surveys and qualitative research to investigate who is
using scholarly content on social media and why
• empirical large-scale studies on other metrics

24. Haustein, S., Peters, I., Sugimoto, C.R., Thelwall, M., & Larivière, V. (in press). Tweeting
biomedicine: an analysis of tweets and citations in the biomedical literature. Journal of the
American Society for Information Science and Technology.
Haustein, S., Bowman, T.D., Holmberg, K., Larivière, V., & Peters, I., (submitted). Astrophysicists
on Twitter: An in-depth analysis of tweeting and scientific publication behavior. Aslib Proceedings.
Stefanie Haustein
Thank you for your attention!
Questions?
stefanie.haustein@umontreal.ca
@stefhaustein