Why we need open science and how to go about it

1. Why we need open science
Dominique Roche
Université de Neuchâtel, Switzerland
@dom_roche

2. Talk outline
• Open science : what? why?
• How do we do open science?

3. What is open science?
Practices that improve the transparency, credibility
and reuse potential of research results.

4. Why do we need open science?

5. We have a problem…
Many scientific disciplines currently face a ‘reproducibility crisis’, whereby a
worryingly large proportion of results cannot be independently replicated.
(Stodden 2011, Ioannidis 2014, Parker and Nakagawa 2014, Parker et al. 2016)

6. In psychology and biomedical science, at least two-thirds of
published findings are estimated to be incorrect.
53 'landmark' studies
in cancer research
Only 6 studies (11%) were
successfully replicated

7. 100 studies in three psychology
journals using high-powered designs
and original materials when available
39% of effects rated to have
replicated the original result
In psychology and biomedical science, at least two-thirds of
published findings are estimated to be incorrect.

8. The reproducibility crisis in science
The case against science is straightforward: much of the scientific
literature, perhaps half, may simply be untrue. Afflicted by studies with
small sample sizes, tiny effects, invalid exploratory analyses, and flagrant
conflicts of interest, together with an obsession for pursuing fashionable
trends of dubious importance, science has taken a turn towards darkness.
(Horton 2015, The Lancet)

9. We don’t really know…
How serious is the problem in Ecology & Evolution?

10. There are only a handful of
replication efforts in E&E
Clint Kelly, UQÀM

11. BUT: they have devastating effects on the public perception of scientists
and promote mistrust and denial on topics such as evolution, climate
change, overfishing, pollution, etc.
Irreproducible results in E&E might not have the same consequences
for human welfare as in medicine and psychology…

13. (Nature 2016, 533: 437)
What does the ‘average’ scientist think?

14. Nature paper and video

16. The public is also taking notice.

17. The reproducibility crisis is also a credibility crisis
At a time of post-truth politics, widespread pollution, climate change,
and an increasing anti-vaccination movement,
science cannot afford a credibility crisis.
(Stodden 2011, Amstat News )

18. Why do we have a crisis?

19. “Publish or perish.”

20. “Selection for high output leads to poorer methods and increasingly
high false discovery rates.”
(Smaldino and McElreath 2016, Roy Soc Open Sci)

21. • publication bias
• confirmation bias
• HARKing
• under-reporting
• selective reporting
• P-hacking
Poor practices that contribute to the replication crisis:
(Parker et al 2016, TREE)

22. • publication bias
• confirmation bias
• HARKing
• under-reporting
• selective reporting
• P-hacking
Poor practices that contribute to the replication crisis:
(Parker et al 2016, TREE)

23. Confirmation bias
= increase in false discoveries
The widespread human tendency to preferentially search for and interpret
observations as consistent with one's beliefs about how the world works.

24. Selective reporting
Reporting only a subset of hypotheses tested.
= increase in false positives

25. P-hacking
A variety of practices that increase the odds of finding a statistically
significant result by, for instance, conducting multiple versions of an analysis
with different covariates, interactions, or subsets of the data.

26. P-hacking
= increase in irreproducible results

27. Finding a meaningful solution to the crisis requires a cultural shift towards
greater openness, transparency and collaboration among researchers.

28. = open science

29. Talk outline
• Open science : what? why?
• How do we do open science?

30. Improving our research workflow
The sequence of activities & procedures at
different stages of a research project.
1. Question, hypotheses, predictions
2. Experimental and statistical design
3. Data collection
4. Data management (organization, cleaning)
5. Statistical analysis (using script)
6. Interpretation of results
7. Publication
preregistration}

31. Preregistration = making (and registering) key decisions about the
study design and data analysis early in the research lifecycle, before
collecting the data.

32. Registered Reports are a publication format where peer review occurs
prior to data collection and analysis.
Accepted papers then are virtually guaranteed publication in the
journal if the authors follow through with the registered methodology.
80 journals publish RRs

33. Does preregistration prevent additional data
collection / analyses?
No : non-preregistered measurements and analyses are allowed
but must be labelled as exploratory (rather than confirmatory) in
the publication.

34. Curbing the reproducibility crisis by improving our
research workflow
= reduce selective reporting and P-hacking

35. Improving our research workflow
The sequence of activities & procedures at
different stages of a research project.
1. Question, hypotheses, predictions
2. Experimental and statistical design
3. Data collection
4. Data management (organization, cleaning)
5. Statistical analysis (using script)
6. Interpretation of results
7. Publication
preregistration}
blinding, proper documentation}

36. Conduct blind studies
subjects unaware of
treatment received

37. Conduct double-blind studies in E&E
control
treatment
Blind protocols are uncommon in
the life sciences and nonblind
studies report higher effect sizes
and more significant p-values.
(Holman et al 2015 PLOS Biol)

38. Does the experimental design exclude the possibility of observer
and experimenter bias (e.g., by double blind protocol)?
Video recording facilitates blinding and is possible in most
well-devised experiments.
(Clark et al 2016 Trends Ecol Evol)

39. Provide detailed materials & methods

40. Curbing the reproducibility crisis by improving our
research workflow
= reduce selective reporting and P-hacking
= reduce confirmation bias, allow replication

41. Improving our research workflow
The sequence of activities & procedures at
different stages of a research project.
1. Question, hypotheses, predictions
2. Experimental and statistical design
3. Data collection
4. Data management (organization, cleaning)
5. Statistical analysis (using script)
6. Interpretation of results
7. Publication
preregistration}
open data & script}
blinding, proper documentation}

42. Editorial open data policies
Representative journals that require data archiving
•The American Naturalist
•Biological Journal of the Linnean Society
•Biology Letters
•BMC Ecology
•BMC Evolutionary Biology
•BMJ
•BMJ Open
•Ecological Applications
•Ecological Monographs
•Ecology
•Ecosphere
•Evolution
•Evolutionary Applications
•Frontiers in Ecology and the Environment
•Functional Ecology
•Genetics
•Heredity
…

43. = reduce selective reporting and P-hacking
= reduce confirmation bias
= lots of benefits (for individuals and society)
Why do it?

44. The sequence of activities & procedures at
different stages of a research project.
1. Question, hypotheses, predictions
2. Experimental and statistical design
3. Data collection
4. Data management (organization, cleaning)
5. Statistical analysis (using script)
6. Interpretation of results
7. Publication
preregistration}
open data & script}
blinding, proper documentation}
Curbing the reproducibility crisis requires
improving our research workflow

45. = reduce selective reporting and P-hacking
increase credibility
= reduce confirmation bias, allow replication
increase reproducibility
= reduce genuine errors and miscondut
increase reusability, accelerate discovery
Open Science

46. Thanks to Sandra Binning, Redouan Bshary, Hanna Kokko, Loeske Kruuk, Michael
Jennions and Rob Lanfear and the eco-ethology lab at UniNE for insightful discussions.
Image / illustration credits: A. Saego, Google Images@dom_roche