Presentation at the ATL Conference held in Atlanta, Oct 11 2017.

1. Do altmetrics capture
societal engagement?
A comparison between survey data
and social media data

2. Do altmetrics capture societal engagement? A
comparison between survey data and social
media data
N. Robinson-Garcia1, I. Ramos-Vielba2, R.
Costas3 P. D’Este2 and I. Rafols2,3
1School of Public Policy, Georgia Institute of Technology, Atlanta (United States)
2INGENIO (CSIC-UPV), Universitat Politècnica de València, Valencia (Spain)
3Centre for Studies of Science and Technology (CWTS), Leiden University, Leiden
(The Netherlands)

3. Intro Broadening to other audiences
beyond academia through social media
• Scholars are increasingly using social media – Haustein et al.,
2014
• Social media is a good venue for interacting and engaging with
stakeholders – Grande et al., 2014
• Researchers have mixed feelings towards the benefits of social
media – Jordan, 2014

4. Motivation What do we mean by
altmetrics?
Bibliometric approach Altmetric approach

5. Motivation What do we mean by
altmetrics?

6. Motivation Social media as a proxy for
societal impact
• Social media as a potential channel to accelerate transition from
sci literature to practice – Grande et al., 2014
• Altmetrics as informative tools with the potential to be used in
research evaluation – Bornmann, 2014
• Altmetrics indicators signal a variety of phenomena, from
scientific impact to popularity or buzz – Haustein et al., 2016
• Maybe talking about impact is too ambitious, but we might
capture engagement – Robinson-Garcia et al., 2017

7. Ultimate goal Do altmetrics reflect offline
societal engagement?
Academics
SMEs, Non-profits,
Hospital,
Gov. Agencies,
Associations, Large
firms
SocietyAltmetrics

8. Main goal Do altmetrics reflect offline
societal engagement?
• Relation between type and variety of stakeholders and altmetric
mentions
• Differences by broad fields
• Differences by type of altmetric source

9. Project Excellence and knowledge transfer
• Survey aimed at understanding the interactions with non-academic
stakeholders in knowledge transfer processes
• Population: 57,406 researchers affiliated to public Spanish institutions
with at least one publication in WoS during the 2012-14 period
• Scholars were surveyed between June and July, 2016
• 21% response rate (11,992 respondents)

10. Material & methods Survey data
• Variety and type of interacting stakeholders
• Formal interactions with stakeholders (contracts, consultancy,
training…)
• Period asked: 2013-2015

11. Material & Methods Types of stakeholders
Bussiness
• SMEs
• Large firms
Public service
related
• Gov. agencies
• Hospitals
• International orgs.
Non-profits
• NGOs
• Associations
• Foundations

12. Material & Methods Altmetric data
• Altmetric.com -> Main provider (Nature Springer Group)

13. Material & Methods Altmetric data
• Altmetric.com -> Main provider (Nature Springer Group)
The unbearable emptiness of tweeting –
About journal articles

14. Material & Methods Altmetric data
• Altmetric.com -> Main provider (Nature Springer Group)
Chinese social media platform (dealing
with Spanish output)

15. Material & Methods Altmetric data
• Altmetric.com -> Main provider (Nature Springer Group)
Blurred mixture of sources with unclear
meaning

16. Material & Methods Altmetric data
• Altmetric.com -> Main provider (Nature Springer Group)

17. Material & Methods Final dataset
10802 respondents
61056 pubs
• 2013-2015 period
• Articles and reviews only
• Documents with DOI number only

18. Material & Methods Final dataset
Fields # Researchers # Publications % pubs in
Altmetric.com
LIFE & HEALTH SCIENCES 3320 21951 51.9%
SOCIAL SCI & HUMANITIES 1439 4119 40.6%
STEM FIELDS 4791 30794 28.6%
MULTIDISCIPLINARY 1252 4192 37.3%
TOTAL 10802 61056 38.4%

19. Material & Methods % out of papers
covered in Altmetric.com

20. Preliminary results

21. Preliminary results

22. Discussion and further steps Towards a
nuanced framework of on/offline engagement
• FB mentions cover the highest share of publications
• Policy document mentions very rare (even in pub. service
related)
• FB and news show higher average # mentions by pub
• News mentions seem to show a highly skewed distribution
• We observe different patterns by altmetric source
• Higher variety of stakeholders seems to show more FB mentions
• No other relation found for the rest of metrics
• No clear pattern by type of stakeholder

23. Discussion and further steps Towards a
nuanced framework of on/offline engagement
Do altmetrics capture social engagement?
• Maybe, but not necessarily related to formal engagement
• Other factors could be influencing
• More fine-grained analysis
• Selection of altmetric sources
• Better defined disciplines
• In-Depth multivariate analysis
• Opens window to more mixed methods analyses

24. Discussion and further steps Towards a
nuanced framework of on/offline engagement
Descriptive results promising but insufficient:
• Influence by academic rank?
• Other altmetrics?
• Which disciplines?
• Types and frequency of interactions?
• Relation between altmetrics and researchers’ attitudes towards
social media?
• Social media as mediator or facilitator of knowledge transfer?

25. Do altmetrics capture societal engagement? A
comparison between survey data and social
media data
THANK YOU!
@nrobinsongarcia, @ireneravi, @RodrigoCostas1,
@PabloDEste and @IsmaelRafols