Slides for my talk at Calsberg #cix working in Copenhagen on 30 June 2015. Covers evaluation and questions of identifying "excellence"

1. Excellence, Innovation, Evaluation:
A critical view on how poor questions lead to misleading answers
@cameronneylon
http://orcid.org/0000-0002-0068-716X

2. “How can excellence in innovation
lead to economic growth?”

3. “How can excellence in innovation
lead to economic growth?”

4. What do we mean when we use the word…
“excellence”?

5. Pursuit of excellence:
We strive
for excellence, seeking
to apply the highest
standards to benefit…
University of Queensland

6. The university pursues
academic excellence
through the creation,
preservation…
University of San Diego

7. …to strive
for excellence and
distinction…
Curtin University

8. We prize excellence
Cape Peninsula
University of Technology

9. We demand excellence
University of Westminster

10. …with world-
class excellence
across all of its
departments
Princeton

11. As a world-leading centre of
academic excellence we aim…
University of Edinburgh

12. The university counts the diversity of
its community as […] integral to its
mission of excellence, and it is
committed to diversity and
inclusiveness in both educational…
University of Maryland

13. excellence in…?

14. excellence in professionally
relevant teaching
University of Westminster

15. excellence in research
and education
Indiana University

16. excellence in research,
education and knowledge
exchange
Strathclyde University

17. excellence in teaching,
research, and public
engagement
Cornell University

18. excellence in teaching
and research
Curtin University

19. …leadership, innovation
and excellence in teaching
Curtin University
and research

20. …a center of
excellence for cultural,
athletic and other events
University of Toledo

21. …internationally recognized
excellence in information
technology services and
infrastructure.
Indiana University

23. What do we mean when we use the word…
“excellence”?

24. How can we determine or measure…
“excellence”?

25. [schools] are recognized for
their excellence through
national and international
peer comparisons.
Indiana University

26. Pursuing excellence means being
satisfied with no less than the
highest goals we can envision.
University of Missouri

27. Pursuing excellence involves being
informed by regional, national and
global standards, as well as our
personal expectations. University of Missouri

28. “excellence” is the holy
grail of academic life
Michèle Lamont
How Professors Think

29. excellence is produced and
defined in a multitude of sites
and by an array of actors
Michèle Lamont
How Professors Think

30. “excellence” tells us nothing
about how important the science is
and everything about who decides.
Jack Stilgoe
Against Excellence

31. Using the word
excellence
is a means of avoiding
a difficult conversation

32. Using the word
excellence
is a means of avoiding
talking about values

33. Impact
Research
Economic
Cultural
Education
Health
Environment

34. Impact
Research
Economic
Cultural
Education
Health
Environment
excellence

35. Impact
Research
Economic
Cultural
Education
Health
Environment

36. Impact
Research
Economic
Cultural
Education
Health
Environment
impact

37. Impact
Research
Economic
Cultural
Education
Health
Environment
quality

38. Impact
Research
Economic
Cultural
Education
Health
Environment
innovation

39. Impact
Research
Economic
Cultural
Education
Health
Environment

40. Impact
Research
Economic
Cultural
Education
Health
Environment

41. Impact
Research
Economic
Cultural
Education
Health
Environment

42. Impact
Research
Economic
Cultural
Education
Health
Environment

43. Impact
evaluation

44. …what we may be doing […]
when we make [a] value
judgement of a […] work...
Barbara Herrnstein Smith
Contingencies of Value

45. …articulating an estimate of
how well that work will serve
implicitly defined functions…
Barbara Herrnstein Smith
Contingencies of Value

46. …for an
implicitly defined audience…
Barbara Herrnstein Smith
Contingencies of Value

47. …experiencing the work under
implicitly defined conditions…
Barbara Herrnstein Smith
Contingencies of Value

48. Impact
Research
Economic
Cultural
Education
Health
Environment

49. Impact
There is some good news…

50. http://www.flickr.com/photos/verzo/8020565592/ CC
BY

51. https://www.flickr.com/photos/see-­‐through-­‐the-­‐eye-­‐of-­‐g/5392290809
CC
BY

52. https://www.flickr.com/photos/jenny-­‐pics/3239638494
CC
BY

57. Citation

58. Bookmark
Citation

59. Bookmark
Citation
Social
media
conversation

60. Bookmark
Citation
Previous
co-­‐authorship
Conference
co-­‐attendance
Social
media
conversation

74. 5. * Neylon, C. & Wu, S. (2009). Article-
Level Metrics and the Evolution of Scientific
Impact. PLoS Biol 7, e1000242+
Evidence of Impact: Over 150 citations
according to Google Scholar.…

75. 6. * Neylon, C. The Political Economics of
Open Access Publishing (blog series). Science
in the Open (2015-16).
Evidence of Impact: Over 2500 unique blog
views to date…

77. Impact
“excellence” becomes a performance

78. Impact
…performance of
“excellence”

79. Impact
…in the place of
excellent performance

81. Impact
Change the narrative…

82. Impact
“soundness”

86. 1.85
1.9
1.95
2
2.05
2.1
2.15
2.2
1996 2001 2006 2011
Global
R&D
Expenditure
(%
of
GDP)

87. 0
50000
100000
150000
200000
250000
1996 2001 2006 2011
Researchers
per
million
people

88. evaluation

89. Impact
PLOS ONE (and arXiv)

90. Impact
Infrastructure and capacity

91. Impact
Research
Economic
Cultural
Education
Health
Environment
evaluation

97. Impact
Research
Economic
Cultural
Education
Health
Environment

98. Impact
Answers are easy…

99. Impact
…it’s the questions that are hard

100. @cameronneylon
http://orcid.org/0000-0002-0068-716X