1st LEARN Workshop. Embedding Research Data as part of the research cycle. 29 Jan 2016. Presentation by Peter Murray-Rust, ContentMine.org and University of Cambridge

1. The Culture of Research Data
Peter Murray-Rust,
ContentMine.org and UniversityOfCambridge
LEARN, London, UK 2016-01-29
The technology for Managing Research Data is already here…
…but we need a change of culture
Open Notebook Science
Publishers must be forced to serve us, not tyrranize us

2. Just read the big
letters
He’s got zillions of
slides…

3. My European Heroes
Young People(ContentMine)
NEELIE KROES

4. The Right to Read is the Right to Mine
http://contentmine.org

5. Themes
• Highly domain-dependent (chem, cryst, phylo)
• Requires community and centrality
• University repositories are NOT the solution
• Openness makes it dramatically easier/better
• The publisher-academic complex is a major
problem.
• Infrastructure must be open and under our
control

6. WE pay for scholarly
publications that WE
can’t read
[1] The Military-Industrial-Academic complex (1961)
(Dwight D Eisenhower, US President)
Publishers Academia
Glory+?
$$, MS
review
Taxpayer
Student
Researcher
$$ $$
in-kind
The Publisher-Academic complex[1]

7. Elsevier wants to control Open Data
[asked by Michelle Brook]

8. Some topics
• Github / software mgt informs data mgt
• Open notebook science
• Open source malaria + LabTrove
• Open phylogenetics
• Computational chemistry
• Crystallography
• Early career researchers can change the world, if
we let them.
• Are “publishers” tyrants or servants?

9. Every Research
Data Manager
should be using
me

10. Why I reposit software in GitHub
I WANT TO!!!
BETTER
QUICKER
SECURE
AUDIT, BACKTRACKABLE
EASY
get collaborators
Most early career software creators have repos
How many people have USED Git?

11. Free/Open Software Development
CODE
REPOSITORY
World
community
CODE
rewrite
validate
CODE
fork
CODE
Re-use
CODE
Re-use
Github, BitBucket
StackOverflow,
Apache
inspires
OSI
Example: ContentMine at
http://github.com/ContentMine/quickscrape
BORN-OPEN-SOURCE
NO WALLS

12. GIT housekeeps AUTOMATICALLY, eternally
Daily record of commits and
Merges. Can backtrack to ANY
Previous version

13. Community involvement
https://github.com/ContentMine/quickscrape/pulls
Contributions from
People “outside project”

14. Compile Fail
Inactive
Fail Tests
Pass Tests
Continuous Integration (Jenkins)
Every time I commit a change
50 projects are recompiled
and tested.
Impossible to do this manually!

15. Software management
Is a success!
Research DATA
management
Is a mess.

16. Traditional Research and Publication
“Lab” work paper/th
esis
Write
rewrite
Re-experiment
publish
???
Validation??
DATA
output “belongs”
to publisher
Every process is LOSSY

17. How NOT to publish data
HT Henry Rzepa
From Henry Rzepa:
this article http://doi.org/10.1126/science.aad6252
which provides a 22 Mbyte PDF of data (mostly bitmaps of NMR
spectra) and comes in at 404 pages long. [1]
But this one http://doi.org/10.1021/jacs.5b05902 [comp chem]
is 505 pages long (the current record holder?)
[1] DATA Behind paywall

18. 505 pages PDF, was a
machine-readable log file
that could and
should have been in a repo
Computational
Chemistry

19. MORE of the PDF
DATA Destruction
Blind humans and
Machines cannot
read this

20. ALWAYS put your
(computational,
instrumental,
observational)
data directly into a
repository

21. Let’s see some
visionaries

22. JD Bernal’s 1965 vision
However large an array of facts, however rapidly they
accumulate, it is possible to keep them in order and to
extract from time to time digests containing the most
generally significant information, while indicating how to
find those items of specialized interest. To do so, however,
requires the will and the means. (Bernal, 1965)
Quoted by PMR in http://journals.iucr.org/d/issues/1998/06/01/ba0011/ba0011.pdf

23. PMR’s Tribute
Planned Memorial Meeting
July 14th 2014 Cambridge
OPEN NOTEBOOK SCIENCE

24. https://en.wikipedia.org/wiki/Bermuda_Principles
• Automatic release of sequence assemblies larger than 1
kb (preferably within 24 hours).
• Immediate publication of finished annotated
sequences.
• Aim to make the entire sequence freely available in the
public domain for both research and development in
order to maximise benefits to society.
HUMAN GENOME project used
Open Notebooks

25. Open is FASTER, BETTER, MORE
EFFICIENT

26. Open is FASTER, BETTER,
MORE, MORE EFFICIENT

27. Open Notebook Science, ONS
Jean-Claude Bradley 2006
All data immediately
available to all. NO
INSIDER INFORMATION.

28. TOOLS
Open Notebook Science
Open
engineered
repository
World
community
INSTRUMENT
validate
merge
MODEL
CODE
DATA
DATA
knowledge
calibrate
Problems are solved communally;
Nothing is needlessly duplicated; “publication“ is
continuous ; data are SEMANTIC
Machines
and humans
Working
together

29. Here are three
examples

30. Mat Todd (Sydney) and MANY collaborators
http://opensourcemalaria.org/ (Chrome for interactivity)
Mat Todd, Univ Sydney, runs an Open Notebook community
to create new antimalarials.

31. Notebook managed on Git.

32. Interactive OPEN chemical search tool from cheminfo.org

33. Interactive OPEN molecular display Jmol (Bob Hanson et al)

35. Interactive OPEN chemical search tool from cheminfo.org

36. data is associated with the proposed
scientific endeavour prior to or at the
point of creation rather than by
annotating the data with commentary
after the experiment has taken place
University of Southampton

37. Data thrives on Community

38. Henry Rzepa does Open
Notebook Computational
Chemistry…
http://www.rzepa.net/blog/?p=14272
This is a current open notebook discussion,
http://www.ch.imperial.ac.uk/rzepa/blog/?p=15552 (see comments,
currently 67).
… on his blog

41. COMMUNITY
INVOLVEMENT

42. Crystallography – a model for Data
Management
• Pro-active, friendly international community
• Committed active International Union(IUCr)
• Data publication valued (1960-present)
• Community develops semantics/dictionaries
• Committed volunteer software innovators
• Heavily Open approach
• Massive and valuable re-use of data
• Culture of validation/reproducibility
• Respect and credit for tool development

43. IUCr DICTIONARIES

44. IUCr VALIDATION
CRITERIA/TOOLS

45. DATA

46. PUBLICLY
VALIDATED
TRUSTABLE
SCIENCE

47. Where to reposit published
crystallography?
Proteins -> PDB, Open
BUT
Inorganics -> ICSD Closed
Organics -> Cambridge (CCDC) Closed
SO
The community has built a Crystallography Open
Database

48. Restrictions on Re-use of Crystallographic data
NOTE: The CCDC is based on data contributed by
scientists as part of publication and validation
Crystallographic data from
publications now belongs to CCDC

49. Open Source and Open Data
www.crystallography.net

50. Interactive OPEN crystal search tool

52. Panton Fellows (Early Career Researchers)
Panton Principles of Open Scientific Data 2010
Publish data openly
(CC0) and record
your wishes

53. Sophie Kershaw, Panton Fellow :
Doctoral Training in Oxford

54. Sophie Kershaw, Panton Fellow

55. Rotation-Based Learning (RBL)
Phase 1: Initiator
• No communication
permitted between groups
• Attempt to reproduce
existing literature
• Deliver a coherent research
story by the end of Phase 1
Phase 2: Successor
• Communication between
groups still prohibited
• Validate and develop the
inherited research story
• Critique your predecessors
• Role of research producer vs. research user
• Can this approach help to foster awareness of reproducibility issues?
Throughout Phases 1 & 2:
• Daily lectures on open
science culture & techniques
• First-hand application to own
research work
• Version control using GitHub
• Daily group supervision

56. … third-year graduate
students
So first-year grad
students should be
trained by…

57. https://en.wikipedia.org/wiki/Tree_of_life CC BY-SA

58. Authors don’t deposit data (Ross Mounce)

59. http://www.slideshare.net/rossmounce/the-pluto-project-ievobio-2014

60. And we did it as Open Notebook
Science
all data and code on Github
Discussion on public Discourse Tool

61. https://en.wikipedia.org/wiki/Tree_of_life CC BY-SA

62. 4300 images in Github

63. “Root”
We analysed every pixel

64. Many diagrams had author errors

65. Supertree created from 4300 papers

66. Aves
Apterygidae
Marsupialia
Monotremata
Mammalia
Reptilia
Amphibia
Arthropoda
Myriapodia
Okapia johnstoni
Pyrus
Stuffed Tree of Life

67. Supertree for 924 species
Tree

68. Can we mine for
animals?
YesWith the
Phylogeny
Cmunity [*]
[*] overlaps with “Tree of Life”, “Evolutionary Biology” , “Taxonomy”, “Species”

69. So now we can
legally mine the
whole literature in
the UK
Yes! And we are
starting to do it…
NORMA

70. So why not Git for Data?

71. DAT is Git for Data!!

72. DAT! Queen Mary UL reposits DNA

73. The John S. and James L. Knight Foundation is an American private, non-profit foundation
dedicated to supporting "transformational ideas that promote quality journalism, advance
media innovation, engage communities and foster the arts."[2]
DAT supports public data

74. @Senficon (Julia Reda) :Text & Data mining in times of
#copyright maximalism:
"Elsevier stopped me doing my research"
http://onsnetwork.org/chartgerink/2015/11/16/elsevi
er-stopped-me-doing-my-research/ … #opencon #TDM
Elsevier stopped me doing my research
Chris Hartgerink

75. I am a statistician interested in detecting potentially problematic research such as data fabrication,
which results in unreliable findings and can harm policy-making, confound funding decisions, and
hampers research progress.
To this end, I am content mining results reported in the psychology literature. Content mining the
literature is a valuable avenue of investigating research questions with innovative methods. For
example, our research group has written an automated program to mine research papers for errors in
the reported results and found that 1/8 papers (of 30,000) contains at least one result that could
directly influence the substantive conclusion [1].
In new research, I am trying to extract test results, figures, tables, and other information reported in
papers throughout the majority of the psychology literature. As such, I need the research papers
published in psychology that I can mine for these data. To this end, I started ‘bulk’ downloading research
papers from, for instance, Sciencedirect. I was doing this for scholarly purposes and took into account
potential server load by limiting the amount of papers I downloaded per minute to 9. I had no intention
to redistribute the downloaded materials, had legal access to them because my university pays a
subscription, and I only wanted to extract facts from these papers.
Full disclosure, I downloaded approximately 30GB of data from Sciencedirect in approximately 10 days.
This boils down to a server load of 0.0021GB/[min], 0.125GB/h, 3GB/day.
Approximately two weeks after I started downloading psychology research papers, Elsevier notified my
university that this was a violation of the access contract, that this could be considered stealing of
content, and that they wanted it to stop. My librarian explicitly instructed me to stop downloading
(which I did immediately), otherwise Elsevier would cut all access to Sciencedirect for my university.
I am now not able to mine a substantial part of the literature, and because of this Elsevier is directly
hampering me in my research.
[1] Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L. M., Epskamp, S., & Wicherts, J. M. (2015). The
prevalence of statistical reporting errors in psychology (1985–2013). Behavior Research Methods, 1–22.
doi: 10.3758/s13428-015-0664-2
Chris Hartgerink’s blog post

76. Some Children
of the Digital Enlightenment
• David Carroll & Joe McArthur: OAButton
• Rayna Stamboliyska & Pierre-Carl Langlais
• Jon Tennant
• Ross Mounce
• Jenny Molloy
• Erin McKiernan
• Jack Andraka
• Michelle Brook
• Heather Piwowar
• TheContentMine Team
• Rufus Pollock
• Jonathan Gray
• Sophie Kay
Jean-Claude Bradley [1] a chemist
developed Open notebook science;
making the entire primary record of a
research project publicly available
online as it is recorded. (WP)
J-C promoted these ideas with
UNDERGRADUATE scientists.
[1] Unfortunately J-C died in 2014;
we held a memorial meeting in
Cambridge
Sophie
Kay

79. OPEN CLOSED
Zenodo Figshare
Git
Dat
OpenOffice Word, PPT
LabTrove, cheminfo.org Chemdraw
CrystallographyOpenDB Cambridge Cryst data Centre
WriteLatex / Overleaf
ReadCube, Symplectic,

80. This is a current open notebook discussion, http://www.ch.imperial.ac.uk/rzepa/blog/?p=15552
(see comments, currently 67).
This is an earlier one, http://www.rzepa.net/blog/?p=14272 (with 86 comments) and also
incorporates Jsmol to visualise all the data
This one starts discussion as an open notebook http://www.ch.imperial.ac.uk/rzepa/blog/?p=1211
with the resulting formal publication at 10.1002/jcc.23985
This was the original open notebook post http://www.ch.imperial.ac.uk/rzepa/blog/?p=984 with
the resulting formal publication at 10.1038/NCHEM.596
This one incorporates open data into its citation list
http://www.ch.imperial.ac.uk/rzepa/blog/?p=15505 and is also an open notebook follow up to my
PhD thesis work, formally published in 1975 or so, thus operating in reverse to the above.
This shows some end outcomes: http://www.ch.imperial.ac.uk/rzepa/blog/?p=15313
This shows the principles: http://www.ch.imperial.ac.uk/rzepa/blog/?p=10972
This is an introductory tutorial http://www.ch.imperial.ac.uk/rzepa/blog/?p=14454
This is a critique http://www.ch.imperial.ac.uk/rzepa/blog/?p=13826
This is “convincing case” http://www.ch.imperial.ac.uk/rzepa/blog/?p=13248
This is about metadata http://www.ch.imperial.ac.uk/rzepa/blog/?p=12932
And its use http://www.ch.imperial.ac.uk/rzepa/blog/?p=12526
You have seen this data nightmare before: http://www.ch.imperial.ac.uk/rzepa/blog/?p=12728
This is about ORCID http://www.ch.imperial.ac.uk/rzepa/blog/?p=12513

82. Open Source software inspires Open Science
Jean-Claude Bradley 2006

88. Ross Mounce (Bath), Panton Fellow
• Sharing research data:
http://www.slideshare.net/rossmounce
• How-to figures from PLOS/One [link]:
Ross shows how to bring figures to life:
• PLOSOne at http://bit.ly/PLOStrees
• PLOS at http://bit.ly/phylofigs (demo)