The Reproducibility Crisis in Psychological Science: One Year Later

1. The Reproducibility Crisis in Psychological Science Jim Grange www.jimgrange.wordpress.com

2. 2011 – “A Year of Horrors”

4. http://bit.ly/2eNL05d “Derailed”

9. “…the field of psychology currently uses methodological and statistical strategies that are too weak, too malleable, and offer too many opportunities for researchers to befuddle themselves and their peers”

10. Each case raised unique questions about how science is conducted in psychology From how research is planned right through to how data are analysed and published

11. All cases pertain to growing concern over the number of false positives in the literature

15. “How reproducible are psychology findings?”

16. Open Science Collaboration 270+ researchers from across the globe

17. Open Science Collaboration Performed close replications of 100 psychology studies

19. Only 36% of studies replicated!!

21. A Year Later

22. Recommendations 1. Replicate, replicate, replicate… 2. Know your statistics 3. Open your science 4. Incorporate open science practices in teaching 5. Reward open science practices

25. VERIFICATION!!!

26. Devoting resources to verification is irrational if the original findings are valid

27. Devoting resources to verification is rational if the original findings are invalid

28. We have a professional responsibility to ensure the findings we are reporting are robust and replicable

31. 2. Know Your Statistics

32. 2. Know Your Statistics

33. False-positives propagate because most researchers don’t understand the p-value (Cumming, 2012)

34. a) the probability that the results are due to chance b) the probability that the results are not due to chance c) the probability of observing results as extreme (or more) as obtained if there is no effect in reality d) the probability that the results would be replicated if the experiment was conducted a second time

36. True or False? The p-value tells us something about the size of an effect

37. True or False? The p-value tells us something about the importance of an effect

38. True or False? The p-value tells us something about the probability of our hypothesis

40. 2. Know Your Statistics p(D|H)

41. 2. Know Your Statistics p(D|H) p(H|D) Same?

42. p(Dead|Murdered)? = 1 p(Murdered|Dead) ~<.001

52. Science works by verification

53. 27% (!!) What percentage of researchers shared their data?

54. Researchers should NOT keep their data private if they have published from it

59. Nice Bonus (1): It’s going public, so I make sure data & analysis is of high quality

60. Nice Bonus (2): I can find my data easily if asked for it

61. As of January 1, 2017, signatories (as reviewers and/or editors) make open practices a pre-condition for more comprehensive review

63. EXPLORATORY CONFIRMATORY

65. “I appreciate your results were unexpected, but in order to tell a nicer story, you should re- write your introduction as if you expected these results”

66. HARK-ing Hypothesising After the Results are Known

67. HARK-ing 92% of psychology articles report confirmed hypotheses (Fanelli, 2010)

71. www.osf.io

77. Strong Incentives to Pursue New Ideas

78. Strong Incentives to Pursue New Ideas Publications

79. Strong Incentives to Pursue New Ideas Grant Income ($$)

80. Strong Incentives to Pursue New Ideas Employment

81. Strong Incentives to Pursue New Ideas Promotion

82. Strong Incentives to Pursue New Ideas Fame…?

83. 5. Reward Open Science Practices Good for Science: - Truth seeking - Rigour - Quality - Reproducibility Good for Individuals/ Institutions: - Publishable - Quantity - Novelty - Impact

84. “…the solution requires making the incentives for getting it right competitive with the incentives for getting it published” (Nosek et al., 2012)

85. Individual Reputations Are At Stake

86. University Reputations Are At Stake

87. (Utopian) Ideas for Institutions Doing research right takes longer

88. (Utopian) Ideas for Institutions Be tolerant of lower output (if doing it right)

89. (Utopian) Ideas for Institutions Limit the “Publish or Perish” mentality

90. (Utopian) Ideas for Institutions “Rigour or Rot” “Rigour or Perish”

91. (Utopian) Ideas for Institutions Reward those doing it right

93. Universe A & B: • Investigating embodiment of political extremism • Participants (N = 1,979!) from the political left, right, and center • Moderates perceived shades of grey more accurately than left or right (p<.01).

95. Universe A & B: Moderates perceived shades of grey more accurately than left or right (p<.05).

96. Universe A • Moderates perceived shades of grey more accurately than left or right (p<.01).

99. Universe B • Moderates perceived shades of grey more accurately than left or right (p<.01). • Surprised by the effect, so tries to replicate the result before publishing –Uses even larger sample size than Study 1 • Replication fails to reproduce the effect –No publication

101. In which universe will this student most likely receive a lectureship position?

102. There is something wrong with hiring decisions if “getting it published” is rewarded more than “getting it right”

103. (Utopian) Ideas for Hiring Committees Look for evidence of open science practice

104. (Utopian) Ideas for Hiring Committees Have open science practice as a “desired” (or “essential”!) item on job specification

105. (Utopian) Ideas for Hiring Committees Judge publication quality rather than quantity

108. Thank You! www.jimgrange.wordpress.com

The Reproducibility Crisis in Psychological Science: One Year Later

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Destacado

Destacado (18)

Similar a The Reproducibility Crisis in Psychological Science: One Year Later

Similar a The Reproducibility Crisis in Psychological Science: One Year Later (20)

Más de JimGrange

Más de JimGrange (6)

Último

Último (20)

The Reproducibility Crisis in Psychological Science: One Year Later