1. The use of
bibliometric indicators
in research assessment:
A critical overview
Henk F. Moed
Senior scientific advisor,
Elsevier, Amsterdam, Netherlands
2. Contents
1 Beyond journal impact factor and H-index
2 University rankings have a limited value
3 Combine indicators and peer review
4 Social sciences deserve special attention
5 Indicators can be manipulated
6 Explore usage-based indicators
3. Contents
1 Beyond journal impact factor and H-index
2 University rankings have a limited value
3 Combine indicators and peer review
4 Social sciences deserve special attention
5 Indicators can be manipulated
6 Explore usage-based indicators
4. Journal impact measures
are no good predictors
of an individual paper’s
actual citation impact
Partly based on International Mathematical
Union’s Report ‘Citation Statistics’ (2008)
6. What is the probability that .......
a randomly selected boy
is at least as tall as a
randomly selected adult?
Av. Length: Boys 85 cm; Adults: 185 cm
Almost zero
62 %a randomly selected PAMS paper
is cited at least as often as a
randomly selected TAMS paper?
JIF: PAMS: 0.43; TAMS: 0.85
13. A journal’s ‘Raw’ Citation Impact
‘Topicality’ of its subject field
SNIP: Base concept
SNIP =
14. How is a field’s ‘topicality’ measured?
Topicality
Citation potential
Length of cited
reference lists
15. Differences in citation potential between fields
Molecular Biology Mathematics
Number of received citations
%
P
a
p
e
r
s
Refe-
rence
lists
16. A journal’s raw impact per paper
Citation potential in its subject field
SNIP =
Journal
scope,
focus
Database
coverage
peer
reviewed
papers only
A field’s
frequency &
immediacy
of citation
Measured
relative to
database
median
18. Example 1 : Molec Biol vs. Mathematics
Journal JIF Cit Pot SNIP
(= JIF/
Cit Pot)
INVENT MATH
1.5
MOLEC CELL
13.0
3.80.4
3.2 4.0
19. Example 2 : Within Mathematics
Journal JIF Cit Pot SNIP
(= JIF/
Cit Pot)
Int J Nonlinear
Sci & Num Sim
4.2
Commun Partial
Different Equat
1.1
2.12.0
0.5 2.1
20. Example 3 : Social Sci vs. Biol & Med Sci
Journal JIF Cit Pot SNIP
J GERONTOL - A
(Biol & Med Sci) 3.7
J GERONTOL - B
(Psych & Soc Sci) 2.7
2.0 1.8
2.31.2
21. Strong points of SNIP
• Takes into account a journal’s scope
• Allows cross-subject comparisons
• Is independent of an a priori subject categorization
• Can be calculated for general journals
• Less potential for gaming
• Accounts for differences between and within journal
subject ‘categories’
26. Bibliometric indicators more and more....
Feature Example
Embody ways to put
numbers in context
Field-normalized citation
measures
Take into account
“who” is citing
Citations weighted with impact
of citing source
Take into account
relationship citing-
cited author
Impact outside the own niche;
multi-disciplinarity;
bridging paradigms
Combine various
types of indicators
HR data on personnel (gender,
age, funding, ...)
27. Contents
1 Beyond journal impact factor and H-index
2 University rankings have a limited value
3 Combine indicators and peer review
4 Social sciences deserve special attention
5 Indicators can be manipulated
6 Explore usage-based indicators
29. Research assessment methodologies must take
into account… [EC AUBR Expert Group]
1. Inclusive definition of research / output
2. Different types of research and its impacts
3. Differences among research fields
4. Type and mission of institution
5. Proper units of assessment
6. Policy context, purpose and user needs
7. The European dimension
8. Need to be valid, fair and practically feasible
30. Types of outputs (SSH)
Impacts Publication/text Non-publication
Scientific-
scholarly
Journal paper; book
chapter; monograph
Research data file;
video of experiment
Educational Teaching course
book; syllabus
Skilled researchers
Economic Patent Product; process;
device; design; image
Cultural Newspaper article; Interviews; events;
Performances; exhibits
31. In institutional research assessment
bottom-up approaches must include
data verification by evaluated authors
32. Top-down institutional analysis
Select an institution’s papers using
author affiliations (incl. verification)
Categorize articles into
research fields
Calculate indicators
Compare with benchmarks
33. Bottom-up institutional analysis
Compile a list of researchers
Compile a list of publications per
researcher (incl. verification)
Aggregate researchers into groups,
departments, fields, etc.
Calculate indicators;
compare with benchmarks
35. Gini index of disciplinary specialization
Gini =
0.0
Gini =
0.27
Gini =
0.52
Gini =
0.70
Data for a general,
a poly-technical
and a specialized
university
36. Relative Citation Rate (RCR)
The average citation rate of a unit’s papers
÷
world citation average in the subfields in
which the unit is active
Corrects for differences in
citation practices among fields,
publication years and type of article
37. Specialized universities perform in their fields of
specialization less well than general institutions do
Data: Scopus /
Scimagoir (n=1,500)
Data: Scopus /
Scimagoir (n=1,500)
Specialized
General
High
Low
38. No linear correlation between a country’s
institutional concentration and its citation impact
Data:Scopus/
Scimago
39. Contents
1 Beyond journal impact factor and H-index
2 University rankings have a limited value
3 Combine indicators and peer review
4 Social sciences deserve special attention
5 Indicators can be manipulated
6 Explore usage-based indicators
41. Case study: A national Research Council
• Proposals evaluated by committees covering
a discipline
• Reports from external referees
• Committee members among applicants
42. Affinity applicants – Committee
0 Applicants are/were not member of any
Committee
1 Co-applicant is/was member of a Committee,
but not of the one evaluating
2 First applicant is/was member of a Committee,
but not of the one evaluating
3 Co-applicant is member of the Committee(s)
evaluating the proposal
4 First applicant is member of the Committee(s)
evaluating the proposal
43. For 15 % of applications an applicant is a member of the
evaluating Committee (Affinity=3, 4)
0
10
20
30
40
50
60
70
%APPLICATIONS
AFFINITY APPLICANTS-COMMITTEE
Projects 63.2 10.2 11.5 5.9 9.1
0 1 2 3 4
44. Probability to be granted increases with
increasing affinity applicants-Committee
30
40
50
60
70
80
%GRANTEDAPPLICATONS
AFFINITY APPLICANTS-COMMITTEE
Projects 37.0 46.9 60.1 62.6 74.0
0 1 2 3 4
45. Logistic regression analysis:
Affinity Applicant-Committee has a significant effect
upon the probability to be granted
MAXIMUM-LIKELIHOOD ANALYSIS-OF-VARIANCE TABLE (N=2,499)
Source DF Chi-Square Prob
-------------------------------------------------------------
INTERCEPT 1 18.47 0.0000
Publ Impact applicant 3 26.97 0.0000 **
Rel transdisc impact applicant 1 0.29 0.5926
Affinity applicant-committee 2 112.50 0.0000 **
Sum requested 1 45.47 0.0000 **
Institution applicant 4 25.94 0.0000 **
LIKELIHOOD RATIO 199 230.23 0.0638
46. The future of research assessment
exercises lies in the intelligent
combination of
metrics and peer review
47. Intelligent combination of ‘metrics’ and peer
review
• Policy makers let the type of peer review depend
upon the outcomes of a bibliometric study
• Peer committees use citation analysis for initial
rankings and explicitly justify why their final
judgments deviate
• Metrics are used to assess peer review processes
48. Contents
1 Beyond journal impact factor and H-index
2 University rankings have a limited value
3 Combine indicators and peer review
4 Social sciences deserve special attention
5 Indicators can be manipulated
6 Explore usage-based indicators
52. CI coverage by field
EXCELLENT
(>80%)
GOOD
(60-80%)
FAIR
(40-60%)
MODERATE
(<40%)
Biochem &
Mol Biol
Appl Phys &
Chem
Mathematics Other Soc
Sci
Biol Sci –
Humans
Biol Sci –
Anim & Plants
Economics
Humanities
& Arts
Chemistry
Psychol &
Psychiat
Engineering
Clin Medicine Geosciences
Phys &
Astron
Soc Sci ~
Medicine
Journals
Books,
proceedings
53. Options for creating a
comprehensive database of
research outputs in
social sciences & humanities
54. Option Example/Case
1 Combine existing SSH bibliographies CSA-Illumina
2 Create new SSH databases Iberian Citation Index
3 Expand existing citation indexes WoS, Scopus
4 Explore Google Scholar; Book Search
5 Combine output registration systems MAETIS (NL)
6 Citation index from repositories Book Citation Index
Project
7 Electronic Library Catalogues WorldCat
55. Contents
1 Beyond journal impact factor and H-index
2 University rankings have a limited value
3 Combine indicators and peer review
4 Social sciences deserve special attention
5 Indicators can be manipulated
6 Explore usage-based indicators
57. Effects of editorial self-citations upon journal
impact factors
[Reedijk & Moed, J. Doc., 2008]
• Editorial self-citations: A journal editor cites in his
editorials papers published in his own journal
• Focus on ‘consequences’ rather than ‘motives’
58. Case: ISI/JCR Impact Factor of a Gerontology Journal
(published in the journal itself)
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
2000 2001 2002 2003 2004
IMPACT FACTOR YEAR
CITESPER'CITABLE'DOC
59. Decomposition of the IF of a Gerontology journal
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
2000 2001 2002 2003 2004
IMPACT FACTOR YEAR
CITESPER'CITABLE'DOC
Editorial self citations
Free citations
60. One can identify and correct for the following types of
strategic editorial behavior
• Publish ‘non-citable’ items
• Publish more reviews
• Publish ‘top’ papers in January
• Publish ‘topical’ papers (with high short term
impact)
• Cite your journal in your own editorials
• Excessive journal self-citing
61. Contents
1 Beyond journal impact factor and H-index
2 University rankings have a limited value
3 Combine indicators and peer review
4 Social sciences deserve special attention
5 Indicators can be manipulated
6 Explore usage-based indicators
62. Analogy Model
Formal use Informal use
(Collections of)
publishing authors
(Collections of) users
Citing a document Retrieving the full text
of a document
Article User session
Author’s institutional
affiliation
User’s account name
Number of times cited Number of times
retrieved as full text
63. Age distribution downloads vs. citations
[Tetrahedron Lett, ScienceDirect; Moed, JASIST, 2005]
0
4
8
12
16
20
1
3
5
7
9
11
13
15
17
19
21
23
25
27
29
AGE (MONTHS)
%
SD USES
CITATIONS
Downloads
Citations
%
Age (months)
64. Ageing downloads vs. citations:
Two factor vs. single factor model
0.01
0.1
1
10
100
0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 88
Age (months)
Uses
Downloads
Observed
Downloads
Computed
Downloads
Singular
Points
Citations
Observed
Citations
Computed
%
Age (months)
Downloads
Citations
66. Citations lead to downloads
[Moed, J. Am Soc Inf Sci Techn, 2005]
1
10
100
1000
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
AGE PAPER A (MONTHS)
DOWNLOADS
A
B (B cites A)
C (C cites A and B)
Paper
A
published
Paper B
published;
it cites A
Download of A
increases
Paper C
published;
it cites A and B
67. Downloads and citations
relate to distinct phases in
scientific information processing
.... but (many) more cases must
be studied
