The document discusses using citation metrics as an alternative to peer review for assessing research quality in the UK's Research Excellence Framework (REF). It finds high correlations (0.69-0.86) between citation metrics and peer review ratings at the institutional level across various fields like physics and chemistry. It also models peer review uncertainty and finds citation metrics correlate almost as well as the peer review model for physics. The authors conclude metrics could potentially replace peer review for the REF, pending further analysis.

1. Replacing peer review by metrics in
the UK REF?
CWTS, Leiden University
8 September 2017
V.A. Traag & L. Waltman

2. UK REF
• Research Excellence Framework (REF) is UK’s system “to assess
the quality of research” of institutions.
• Each institution assessed separately per Unit of Assessment
(UoA) e.g. Physics, Chemistry, etc….
• Three profiles are assessed:
1 Research output;
2 Impact beyond academia; and
3 Research environment.
• Each output is awarded 1–4 stars.
4∗
is “world-leading”, 1∗
is “recognised nationally”.
• Publications are awarded 1–4 stars based on peer review.
• Results are divulged at institutional/UoA level (as % of 1–4∗).
2/18

3. Metrics
• Metric Tide (2015) correlate peer review results with various
citation metrics on article level. Find correlations roughly 0.3 –
0.6.
• Myrglod et al. (2015) use departmental h-index, correlations
range from 0.39 (Physics) – 0.71 (Chemistry).
• Elsevier (2015) use proportion highly cited publications. Find
correlations roughly 0.20 (Physics) – 0.75 (Chemistry).
3/18

4. Problems
• Correlation analysis of Metric Tide at wrong level. Should be
institutional/UoA level, not article level.
Possible to have low correlations at lower level but high
correlations at higher level.
• Difference in set of publications. Elsevier & Myrglod based on
coverage in Scopus subject categories of UoA.
• Central problem: when is a correlation high enough? Outliers.
• Peer review also has uncertainties.
4/18

5. Analysis
• Re-do citation analysis of all submitted publications.
• Calculate correlations at institutional level.
• Compare to uncertainty in peer-review.
Unfortunately, peer review results per article are unavailable.
Create model of peer review uncertainty.
Central idea: what if we repeat the REF?
5/18

6. Citation analysis
• Match all publications to WoS using DOI.
• Use citation window until 2014 (publications from 2008–2014).
• Normalise based on CWTS publication classification.
• Calculate PP(top 10%) indicator per institution/UoA.
• Correlations between PP(top 10%) and PP(4∗) per UoA.
6/18

7. Correlation results
10 20 30 40 50 60 70
PP(Top 10%)
0
5
10
15
20
25
30
35PP(4*)
Correlation = 0.86
Unit of Assessment: Physics
7/18

8. Correlation results
0 10 20 30 40 50 60
PP(Top 10%)
10
0
10
20
30
40
50PP(4*)
Correlation = 0.83
Unit of Assessment: Chemistry
8/18

9. Correlation results
10 20 30 40 50 60 70
PP(Top 10%)
10
0
10
20
30
40
50
60PP(4*)
Correlation = 0.69
Unit of Assessment: Biological Sciences
9/18

10. Review
Model
1 Each paper has some value vi.
2 Review yields some perceived value pi = vi .
3 Award 4∗ if perceived value exceeds some threshold pi > p4∗
.
Resampling
1 Sample a value vi given a paper’s 4∗ status.
2 Sample perceived value pi = vi .
3 Award 4∗ if perceived value exceeds some threshold pi > p4∗
.
10/18

11. Probability 4∗
0.0 0.5 1.0 1.5 2.0
Value
0.0
0.2
0.4
0.6
0.8
1.0(*=)
=
= .
= .
11/18

12. Probability of value (σ2
)
0.0 0.5 1.0 1.5 2.0 2.5 3.0
0.00
0.05
0.10
0.15
0.20
0.25
0.30(=*)
=
= .
= .
12/18

13. UK REF Correlations
EconomicsandEconometrics
ClinicalMedicine
Physics
Chemistry
PublicHealth,HealthServicesandPrimaryCare
EarthSystemsandEnvironmentalSciences
Psychology,PsychiatryandNeuroscience
ElectricalandElectronicEngineering,MetallurgyandMaterials
BiologicalSciences
Agriculture,VeterinaryandFoodScience
Geography,EnvironmentalStudiesandArchaeology
BusinessandManagementStudies
PoliticsandInternationalStudies
GeneralEngineering
AlliedHealthProfessions,Dentistry,NursingandPharmacy
ComputerScienceandInformatics
Communication,CulturalandMediaStudies,LibraryandInformationManagement
Philosophy
Education
ArtandDesign:History,PracticeandTheory
Law
History
MathematicalSciences
Aeronautical,Mechanical,ChemicalandManufacturingEngineering
SocialWorkandSocialPolicy
SportandExerciseSciences,LeisureandTourism
TheologyandReligiousStudies
EnglishLanguageandLiterature
Music,Drama,DanceandPerformingArts
Architecture,BuiltEnvironmentandPlanning
CivilandConstructionEngineering
Sociology
ModernLanguagesandLinguistics
Classics
AreaStudies
AnthropologyandDevelopmentStudies
0.0
0.2
0.4
0.6
0.8
1.0Correlation
PP(top 10%) Bootstrap (0.1) Bootstrap (0.5) Bootstrap (1)
trics
icine
ysics
istry
Care
nces
ence
rials
0.0
0.2
0.4
0.6
0.8
1.0
Correlationtrics
icine
ysics
istry
Care
nces
ence
rials
nces
ence
0.0
0.2
0.4
0.6
0.8
1.0
PP(to
13/18

14. Conclusions
Conclusions
• Correlations at article level = correlations at institutional level.
• Correlations on metrics should consider peer review uncertainty.
• For UK REF, physics metrics correlate almost equally well as
(model of) peer review.
Future
• Longer citation window?
• Compare to Scopus correlations.
• Corroborate model of peer review uncertainty.
14/18

15. Thank you!
Questions?
v.a.traag@cwts.leidenuniv.nl @vtraag
15/18

16. Model technicalities
Distributions
• Value distributed as vi ∼ LogN(µv , 1).
• Error distributed as ∼ LogN(−σ2
2 , σ2), so that = 1.
• Perceived value distributed as pi ∼ LogN(µv − σ2
2 , 1 + σ2).
Estimation
• Assume overall value µv = 1.
• Calculate p4∗
such that Pr(pi > p4∗
) = PP(4∗) overall.
• Assume values for institution/uoa is LogN(µv , 1).
• Calculate µv such that Pr(pi > p4∗
) = PP(4∗) per
institution/uoa.
16/18

17. Probability of value (µv)
0 1 2 3 4 5
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45(=*)
= .
=
=
17/18

18. Accuracy
0.0 0.2 0.4 0.6 0.8 1.0
PP(4*)
0.0
0.2
0.4
0.6
0.8
1.0Accuracy
Accuracy (4*) Accuracy (< 4*) Accuracy (overall)
18/18