1. Citations count!
Bibliometrics in the
Repository and for REF
at Cardiff University
Kate Bradbury
Senior Consultant, Research Support
BradburyK1@cardiff.ac.uk

2. Outline
1. Bibliometrics – introduction
2. Bibliometrics in ORCA (Cardiff’s repository)
3. Bibliometrics for REF
4. New developments

3. 1. Bibliometrics -
introduction

4. 1.1 What is bibliometric data?
Data resulting from quantitatively
analysing academic publications.

5. http://www.wordle.net/

6. 1.2 Some popular bibliometric
measures:
• Citations: The number of times a journal article has been cited by other
authors.
• Impact Factor (Thomson): The journal impact factor is the average number
of times articles from the journal published in the past two years have been
cited in the Journal Citation Reports year.
• SJR (SCImago Journal Rank)/SNIP (Source Normalized Impact per Paper):
used by Scopus Analytics for journal rankings.
• H-score or H-index: a scholar with an index of h has published h papers
each of which has been cited by others at least h times. Variations include
the g-score (more weight given to highly cited publications) and m-score
(length of time publishing taken into account).

7. 1.3 New measures - altmetrics
• Alternative metrics.
• Measure of attention not quality.
• Measures “data trail” or “digital exhaust”.
• Counts may include downloads, page views, mentions in
news, social media and blog mentions, readers in
reference manager sites.
• Not an alternative to using academic citations but an
alternative to only using academic citations.

8. 1.4 Some uses for bibliometric data
• Publicising work/profile & in CVs.
• Finding journals to publish in.
• Finding partners for collaborative and interdisciplinary working.
• Determining wider impact from Altmetrics.
• REF - Citation data & contextual data is being used for all of
panel A sub-panels and some panel B sub-panels.
• Requests from grant funders for citation data.
• Major component of League Tables eg QS Top Universities
(Scopus), Times World University rankings (Web of Science),
Shanghai Index (Web of Science).

9. 1.5 Some limitations of bibliometric data
• Comparing different disciplines – need benchmarks and
normalisation.
• Publication types not all evenly covered.
• May be working in a very small, specialised field not well
covered by citation data.
• Bias in favour of English language publications.
• Early-career researchers – takes time for citations to
accumulate.
• Analysis of new areas of research may be limited.
• Negative citations.
• Self-citations.
Therefore should be used cautiously & should be
complementary to other analysis of research quality & impact.

10. 2. Bibliometrics in
ORCA

11. 2.1 Why add bibliometrics to the repository?
• Information for readers – showcasing Cardiff’s
research.
• Information for authors – indicating to authors
the reach of their research, both in social media
and academic citations.
• Information for analysis – download citation data
to analyse it, for example for a school or
department.
• Easy linking to citing works.
• Livens up the text!

13. ORCA Institutional Repository:
citation data

17. ORCA Institutional Repository –
article altmetrics

19. 3. Bibliometrics
for REF

20. 3.1 Background
• 2008: HEFCE proposed basing the next REF mainly on bibliometric data & ran
a consultation exercise 2007-2008.
• Commissioned a report from Leiden University to analyse the 2001 RAE
exercise data.
• Commissioned a pilot exercise & expert advisory groups.
• “The pilot exercise concluded that citation information is not sufficiently robust
to be used formulaically or as a primary indicator of quality in the REF; but
there is scope for such data to inform and enhance the process of expert
review..”
• Used citation data for outputs plus contextual data for Panel A and some Panel
B Units of Assessment.
• Now running an independent review of the role of metrics in research
assessment commissioned by David Willetts & consulting again.
• New Universities minister - Greg Clark - new policy?

21. 3.2 REF2014 panels using citation data

22. 3.3 Contextual data for REF2014
• Provides context for the REF panels to judge the citation
count for an individual article, in relation to the field of
research and a publication’s age.
• Each journal in the database is assigned to one or more
subject classifications, using their ‘All Science Journal
Classification’ (ASJC) codes.
• The citation count for the article can be judged alongside
the contextual data for the subject where the journal is
classified. ie how many citations would place it in the top
1%, 5%, 10%, 25% of other articles for the same subject
published in the same year.

23. 3.4 REF 2014 support for bibliometrics at
Cardiff
• Training & support for use of bibliometric data.
• Checked all REF journal outputs for the relevant panels against
Scopus to ensure that it was returning a citation count.
• If not, checked why. Reasons include:
– Incorrect data
– Missing issue
– Missing articles
– Journal not indexed on Scopus
– Article too recent
• Reported to Scopus if required & corrected.

24. 4. New
developments

25. 4.1 Cardiff - SciVal subscription for 6
months trial
• Assessment of research performance based on publication counts and
citation data measured alongside benchmarked data for the subject.
• All research universities.
• Groups – country, Russell group, self-created groups (eg. Research
groups).
• Subjects – provided by SciVal or self-created subject/journal groups.
• Collaboration data.

26. 4.2 Cardiff – new CRIS system
• We are just finalising our selection of a new Current
Research Information System (CRIS).
• Currently exploring potential with bibliometrics & altmetrics.
• Issues:
– Do not want to lose ORCA functionality – citations and
altmetric data.
– Importing data from Scopus/potentially SciVal as well as
Thomson data.

27. 4.3 Some recent developments & trends
• HEFCE review to “consider the robustness of metrics across
different disciplines, and assess their potential contribution to
the development of research excellence and impact.”
• Increase in the number of UK university subscriptions to Scival
(Elsevier) and InCites (Thomson) and corresponding increase in
library jobs for bibliometricians/research data/research
information analysts. Increased opportunities for collaboration
with colleagues from planning, research office, business
intelligence sections.
• Intensified university interest in League Tables & how the
position of individual universities can be improved.