Talk given by Prof Stephen Curry at the R2R Conference, London, Feb 2017 – a reminder to researchers, in an overly metricised world, to focus on the things that matter in their work. Conference programme: https://r2rconf.com/programme/

1. Zen and the Art of
Research Assessment
Stephen Curry
Imperial College
R2R Conference | 20 Feb 2017

2. An Inquiry into Values

3. Why do people become researchers?
To understand the world
To change the world
To earn a living
To be remembered
Wikimedia Commons (Dona Eidam/USGS)

4. How do people become researchers?
h"p://www.nature.com/news/science-publishing-the-golden-club-1.13951
To understand the world
To change the world
To earn a living
To be remembered
Easier to achieve if
publishing is open and
evaluaHon more broadly
based
Publish in a high impact
factor journal

5. Measurement has its uses…
h"ps://www.premierleague.com/tables?co=1&se=42&mw=-1&ha=-1
English Premier League Table (2015-16)

6. …but where are the limits?

7. The Metric Tide (2015): a call for responsible metrics
• Indicators, not metrics?
• Numbers inform but cannot replace judgement
• Users & providers need to be transparent
The Metric Tide
Report of the Independent Review
of the Role of Metrics in Research
Assessment and Management
July 2015
h"p://www.ascb.org/dora/
h"p://www.hefce.ac.uk/pubs/rereports/Year/2015/metricMde/Title,104463,en.html

8. Swimming against the metric Hde: be careful what you wish for
“Don’t aim at success […] for success, like
happiness, cannot be pursued; it must
ensue, and it only does so as the
unintended side-effect of one’s
dedicaMon to a cause greater than
oneself…”
Viktor Frankl
On your deathbed, will your bibliography or
your h-index be uppermost in your mind?

9. Swimming against the metric Hde: be humble
Fickle nature of human judgement…
Fallacy of What You See Is All There Is
Halo effects
Anchoring effects
Hindsight bias
Loss aversion
“I have yet to meet a successful scientist who
lacks the ability to exaggerate the importance
of what he or she is doing.”
Daniel Kahneman

10. Swimming against the metric Hde: accoun;ng for the unexpected
Apollo 8 Mission Transcript (24 Dec 1968)
Anders: "Oh, my God, look at that picture over there.
There's the Earth coming up. Wow, is that pre`y!"
Anders (to Lovell): "You got a color film, Jim? Hand me
a roll of color, quick, would you?"
Lovell: "Oh, man, that's great! Where is it?"
Anders: "Hurry. Quick."
Lovell: "Down here?"
Anders: "Just grab me a color. A color exterior. Hurry
up. Got one?"
Lovell: "Yeah, I'm lookin' for one. C368."
Anders: "Anything quick."
Anders: "I think we missed it."
Within seconds, Lovell sees the shot again in another
window. He asks for the camera.
Anders: "Wait a minute, just let me get the right
seing here now, just calm down. Calm down, Lovell!"
h"p://www.npr.org/secMons/thetwo-way/2013/12/23/256605845/on-
anniversary-of-apollo-8-how-the-earthrise-photo-was-made

11. Focusing researcher assessment on published outputs is problema;c
h"p://journals.plos.org/plosmedicine/arMcle?id=10.1371/journal.pmed.1001747
h"p://occamstypewriter.org/scurry/2012/08/13/sick-of-impact-factors/
h"p://dx.doi.org/10.1016/j.cub.2007.06.014
h"p://www.palgrave-journals.com/arMcles/palcomms2016105h"p://nuffieldbioethics.org/project/research-culture/

12. Can the openness of our scienHfic heritage help us to do be`er research?
Is the amateur (and open) ethos sMll one of the norms of the academy?

13. Some of the work is technical (and requires collaboraHon…)
Can openness
help to change
behaviour?
Cita%on Distribu%ons
Royal Society Journals
EMBO Journal
PLOS
PNAS
Nature
Nature CommunicaHons
Nature Chemistry
ScienHfic Reports
Acta Cryst. (A-F)
…
No promo;on of JIFs
PLOS
eLife
ASM journals
…

14. Academic leaders and insHtuHons have to do their bit…
Vale, R. D. (2012) Mol Biol Cell 23, 3285–3289.
h"p://www.nature.com/news/fewer-numbers-be"er-science-1.20858
1. Research, publicaHons, grants
2. Managerial & academic duHes
3. Mentoring & teaching
4. Clinical work (if applicable)
5. Entrepreneurship & community outreach
Researcher assessment at UMC Utrecht
h"ps://emckiernan.wordpress.com/pledge/

15. A public good: how open science can be be`er science
Preprints: faster communicaHon
Focus on the content, not the container
Largest possible audience (sharing & scruHny)
Data sharing: re-use & scruHny benefits
Be`er for changing the world

16. A public good: how open science can be more inclusive
CiHzen science teaches researchers about:
new (non-tradiHonal) audiences & scienHsts
CommunicaHon + ParHcipaHon = Public Trust

17. PoliHcal realiHes: we have to be open with the public
“People in this country have had enough of experts.”
Michael Gove, MP
“…we have to do more to show how, and why, universiHes
are one of society’s great achievements and best hopes.”
Chris Husbands (VC, Sheffield-Hallam University)

18. Thank you
AlternaHve Metrics?
CitaHon counts are easy and not uninteresHng, but…
How reliable is your research?
How easy is it for other researcher to access your data?
How many lives has your work saved?
How many people has your research helped to heal?
How many people have been lived from poverty through your
published output?
How much economic growth has your work generated?
How much has your research addressed significant naHonal and
internaHonal challenges?
How much has your research reduced inequality in the world?
How many young minds has your work inspired?
How well has your mentoring helped to nurture the next
generaHon of researchers?