Presentation at APE Berlin, 2010

1. Video of this talk: http://river-valley.tv/measuring-is-knowing-or-is-it/
Measuring is Knowing
Or is it?
Jan Velterop – APE – 20 January MMX

2. NSF Workshop on
Scholarly Evaluation Metrics
Washington DC, December 2009

3. About Citations and Usage
(More about Citiations than Usage)

5. Mixed feelings

6. Mixed feelings
useless

7. Mixed feelings
useless
meaningless

8. Mixed feelings
useless
meaningless
interesting

9. Mixed feelings
useless
meaningless
interesting
just play the game

10. Every ‘usage’ has its metric

11. Every metric has its ‘usage’

13. How science is linked

15. And metrics less so

16. Herbert van de Sompel Johan Bollen
Michael Kurtz Peter Binfield Jevin West
Jorge Hirsch Jan Velterop Julia Lane
Alexis-Michel Mugabushaka

18. growth of informal science communication
re-use of data as impact
metrics for data sets
measure
metrics suitable to take decisions? local metrics vs global ones
Eigenfactor vs Impact Factor web metrics poorly understood
h-bar index refinement of h-
focus on networks
index
multiple author, junior author, senior author
article level metrics relative funding decisions
increasing inputs not always leading to increased
output
how to quantify exposure in lay news
science is non-linear
media?
attributed nano-publications

19. Current knowledge transfer

20. Current knowledge transfer
Signal to noise ratio?

22. Current knowledge transfer – an analogy – needle transport

23. Current knowledge transfer – an analogy – needle transport
Nano-
publications
are the
“needles”

24. Ascoviruses (AVs) induce a unique pathology in their insect host cells causing cleavage of the cells
into virion-containing vesicles. The mechanism by which AVs induce vesicle formation is poorly
understood. It is postulated that the virus initially induces apoptosis leading to cell fragmentation.
The apoptotic bodies are however, rescued by the virus to form the vesicles. Here we show that
Heliothis virescens AV (HvAV-3e) is able to inhibit chemically induced apoptosis from around 16 h
after infection. Analysis of the genome of the virus indicated the presence of a putative inhibitor of
apoptosis (orf28) gene that encodes a protein with an imperfect baculovirus inhibitor of apoptosis
repeat (BIR) and a RING domain. Transiently expressed orf28 did not inhibit chemically induced
apoptosis suggesting that the protein may not serve as an inhibitor of apoptosis. Nevertheless, RNA
interference studies revealed that the gene is probably essential for virus pathology and replication.
Ascoviruses
[16747851] Ascoviruses [16747851]
Induction
[26277419] Induction of Pathology [26277419
Pathology 13817745]
[13817745]
Insect [839958] Cell of Insect [283053 839958]
Cell [283053]
Ascoviruses > induction of pathology > insect cells
[16747851] > [26277419 13817745] > [283053 839958]

25. Ascoviruses (AVs) induce a unique pathology in their insect host cells causing cleavage of the cells
into virion-containing vesicles. The mechanism by which AVs induce vesicle formation is poorly
understood. It is postulated that the virus initially induces apoptosis leading to cell fragmentation.
The apoptotic bodies are however, rescued by the virus to form the vesicles. Here we show that
Heliothis virescens AV (HvAV-3e) is able to inhibit chemically induced apoptosis from around 16 h
after infection. Analysis of the genome of the virus indicated the presence of a putative inhibitor of
apoptosis (orf28) gene that encodes a protein with an imperfect baculovirus inhibitor of apoptosis
repeat (BIR) and a RING domain. Transiently expressed orf28 did not inhibit chemically induced
apoptosis suggesting that the protein may not serve as an inhibitor of apoptosis. Nevertheless, RNA
interference studies revealed that the gene is probably essential for virus pathology and replication.
Ascoviruses
[16747851] Ascoviruses [16747851]
Induction
[26277419] Induction of Pathology [26277419
Pathology 13817745]
[13817745]
Insect [839958] Cell of Insect [283053 839958]
Cell [283053]
Ascoviruses > induction of pathology > insect cells
[16747851] > [26277419 13817745] > [283053 839958]
Semantic codification of assertions

26. Author
<rdf:Description rdf:about=”http://www.nbic.nl/cwa/relation/#26277419#13817745">
<cwa:typeRelation rdf:resource=” http://predicate.conceptwiki.org/index.php/#2121378”/>
<cwa:direction>1,2</cwa:direction>
<cwa:strength>1.0</cwa:strength>
<cwa:author rdf:resource=”http://people.conceptwiki.org/index.php/#85094810”/>
<cwa:provenance rdf:resource=”http://article.conceptwiki.org/index.php/#121646370”/>
<cwa:timestamp>1240641052059</cwa:timestamp>
<cwa:annotated_by rdf:resource=”http://people.conceptwiki.org/index.php/#43065817”/>
<cwa:annotation rdf resource=”http://www.virusdb.org/viruses/av/Heliothis_virescens_insect”>
</rdf:Description>
Provenance
<rdf:Description rdf:about=”http://www.nbic.nl/cwa/relation/#26277419#13817745">
<cwa:typeRelation rdf:resource=” http://predicate.conceptwiki.org/index.php/#2121378”/>
<cwa:direction>1,2</cwa:direction>
<cwa:strength>1.0</cwa:strength>
<cwa:author rdf:resource=”http://people.conceptwiki.org/index.php/#85094810”/>
<cwa:provenance rdf:resource=”http://article.conceptwiki.org/index.php/#121646370”/>
<cwa:timestamp>1240641052059</cwa:timestamp>
<cwa:annotated_by rdf:resource=”http://people.conceptwiki.org/index.php/#43065817”/>
<cwa:annotation rdf resource=”http://www.virusdb.org/viruses/av/Heliothis_virescens_insect”>
</rdf:Description>

27. Author
<rdf:Description rdf:about=”http://www.nbic.nl/cwa/relation/#26277419#13817745">
<cwa:typeRelation rdf:resource=” http://predicate.conceptwiki.org/index.php/#2121378”/>
<cwa:direction>1,2</cwa:direction>
<cwa:strength>1.0</cwa:strength>
<cwa:author rdf:resource=”http://people.conceptwiki.org/index.php/#85094810”/>
<cwa:provenance rdf:resource=”http://article.conceptwiki.org/index.php/#121646370”/>
<cwa:timestamp>1240641052059</cwa:timestamp>
<cwa:annotated_by rdf:resource=”http://people.conceptwiki.org/index.php/#43065817”/>
<cwa:annotation rdf resource=”http://www.virusdb.org/viruses/av/Heliothis_virescens_insect”>
</rdf:Description>
Provenance
<rdf:Description rdf:about=”http://www.nbic.nl/cwa/relation/#26277419#13817745">
<cwa:typeRelation rdf:resource=” http://predicate.conceptwiki.org/index.php/#2121378”/>
<cwa:direction>1,2</cwa:direction>
<cwa:strength>1.0</cwa:strength>
<cwa:author rdf:resource=”http://people.conceptwiki.org/index.php/#85094810”/>
<cwa:provenance rdf:resource=”http://article.conceptwiki.org/index.php/#121646370”/>
<cwa:timestamp>1240641052059</cwa:timestamp>
<cwa:annotated_by rdf:resource=”http://people.conceptwiki.org/index.php/#43065817”/>
<cwa:annotation rdf resource=”http://www.virusdb.org/viruses/av/Heliothis_virescens_insect”>
</rdf:Description>
They are “nano-publications” because they can be cited

28. In essence the NSF workshop
was all about Impact
(Though “Impact” was not defined)

29. Let me help define “Impact”

30. But first…
Why do scientists publish?

31. The record
“keeping the minutes of science”
The
of S
Rec
c
ord
ienc
e

32. The record
“keeping the minutes of science”
e
of Scienc
ord
The Rec

33. Transfer of
information and knowledge

34. Credit in the ego-system; the acknowledge economy

35. Credit in the ego-system; the acknowledge economy
Often seen as
impact

37. “Scientific impact is the sound
you hear if you drop a feather
down the Grand Canyon and
wait for the echo.”
Paraphrasing Don Marquis, 1878-1937

38. “The notion of impact is incoherent, likely
to reward the sensationalist and second-
rate ... and risks turning academics into
door-to-door salesmen for vulgarised
versions of their increasingly market-
oriented products.”
Stefan Collini, professor of intellectual history and English
literature at the University of Cambridge

40. Impact Factor
g-index
h-index
h-b-index
Eigenfactor

41. Journal Impact Factor
Individual g-index
Individual h-index
Topic h-b-index
Journal Eigenfactor

42. Lazy Man’s Credit League Tables?
Journal Impact Factor
Individual g-index
Individual h-index
Topic h-b-index
Journal Eigenfactor

43. Lazy Man’s Credit League Tables?
Journal Impact Factor
Individual g-index
Individual h-index
Topic h-b-index
Journal Eigenfactor
In Defiance of Science?

45. Defiance of Science?

46. Defiance of Science?
Higher precision than needed for
a space rendez-vous

47. Apart from journal level , also article level metrics:

48. Apart from journal level , also article level metrics:
usage
citations
blog posts
bookmarks
comments/ratings
trackbacks

49. The less I understand it, the more
quality it is bound to have

50. The less I understand it, the more
quality it is bound to have
How about that as a metric?

51. Anyway, what does it mean, all these
metrics and the resulting ranking of
McJournals * and McArticles*?

52. Anyway, what does it mean, all these
metrics and the resulting ranking of
McJournals * and McArticles*?
* obviously, in this context ”Mc” just means:
falling within a narrow band of variability

53. not just self-referencing ivory tower influence?

54. not just self-referencing ivory tower influence?

55. Number of career decisions based purely on citation counts of one sort or another

56. Number of career decisions based purely on citation counts of one sort or another

57. Number of career decisions based purely on citation counts of one sort or another
Just in case this graph
makes it to the web – it’s
totally fictional!

58. Number of career decisions based purely on citation counts of one sort or another
Just in case this graph
makes it to the web – it’s
totally fictional!
But believable, too!

59. Akin to
phrenology?

60. Akin to
phrenology?

61. Eigenfactor
Akin to down-
loads g-index
phrenology?
h-index h-b-index

62. Eigenfactor
Akin to down-
loads g-index Impact
phrenology?
Factor
h-index h-b-index

63. Societal Impact

64. 79-page document
‘Citation’ mentioned
exactly twice.
‘Impact Factor’ not
mentioned at all.
‘Quality’ mentioned
50 times.

67. The Scientific Egosystem is an Acknowledge Economy
where ROI means Return on Influence

68. The Scientific Egosystem is an Acknowledge Economy
where ROI means Return on Influence
Shouldn’t that be Societal Influence?

69. “Not everything that can be counted
counts; not everything that counts
can be counted.”
Albert Einstein .

70. Are the most cited articles also the ones worth
citing most?

71. Are the most used articles also the ones worth
using most?

72. Are the most cited and used articles also the
ones worth most to current and future society?

73. the old story of goals and means

74. Julia Lane (NSF)
One thing that bugs me is that people answer the question ‘what
“ impact has your funding had’ with things like ‘I hired 3 postdocs and 2
support staff.’
”
Societal impact?
How has your research – done with our money – made the world a
better place?

75. Can we truly know?

76. Ivory tower
Foundation

77. Thank you!
velterop@conceptweballiance.org