Talk for the Scripps Howard Institute on the Environment and Science at Florida Atlantic University, May 23, 2012

1. Fraud and Why Studies are Flawed:
Should Journalists Trust Peer Review?
Ivan Oransky, MD
Co-Founder, Retraction Watch
Executive Editor, Reuters Health
Scripps Howard Institute on
the Environment and Science
May 23, 2012

2. Can You Trust Journal Studies?
• How good is peer review?
• Positive publication bias
• Over-reliance on embargoed studies
• How often it turns out to be wrong
• How to get it right

3. How Does Peer Review Work?
• Usually three outside reviewers
• Usually anonymous
• Sometimes in as little as 72 hours

4. How Good Is Peer Review?

5. How Good Is Peer Review?

6. How Good Is Peer Review?
• Bad at detecting fraud
• Slow
• Costly (and unfunded)
• Prone to bias

7. Peer Reviewers: Worse With Experience

8. Positive Publication Bias

9. Positive Publication Bias
Publish a trial that will bring US$100,000 of
profit or meet the end-of-year budget by
firing an editor.
-- Former BMJ editor Richard Smith

10. Embargoes and the Ingelfinger Rule
By the late 20th century, journals needed to compete
not just with each other but with newspapers and other
media…In 1969, the Journal articulated this relationship
in its Ingelfinger Rule, a policy against publishing
anything that had already appeared elsewhere. Other
journals followed suit. This rule, combined with
embargo policies, has led to a carefully choreographed
production in which medical journals and the popular
press work cooperatively and competitively to
influence the news cycle.
-- NEJM, April 19, 2012

11. Even Without Embargoes, We’d Still Have
Ingelfinger

12. How Often Are Studies Wrong?
Ioannidis JPA. PLoS Med 2005; 2(8): e124

13. How Often Are Studies Wrong?

14. Retraction Watch
• http://retractionwatch.com
• Launched August 2010 with Adam Marcus
• Frequently cited in major news outlets, from Nature
to Der Spiegel to New York Times
• 250,000 pageviews per month

15. Retractions on the Rise
-The Wall Street Journal

16. Retractions on the Rise

17. Retractions on the Rise
-Neil Saunders

18. The Unofficial Record Holder

19. Why Do Journals Retract?
-Nature

20. Why Do Journals Retract?
• Error is more common than fraud
• 73.5% of papers retracted for error (or
undisclosed reason) vs 26.6% for fraud
• Most common cause of retraction: scientific
mistake (234 papers; 31.5%)
• Fabrication (including data plagiarism) more
common than text plagiarism
• 67 retractions (9.0%) had multiple causes, but 134
papers (18.1%) retracted for ambiguous reasons
-Journal of Medical Ethics 2010

21. Fraud: Image Manipulation

22. Fraud: Faked Data

23. Is Fraud on the Rise?
Over the years…surveys have asked scientists
directly about their behaviour…. [T]hese studies
have used different methods and asked different
questions, so their results have been deemed
inconclusive and/or difficult to compare. A non-
systematic review based on survey and non-survey
data led to estimate that the frequency of “serious
misconduct”, including plagiarism, is near 1%.
-- Fanelli, PLoS ONE, 2009

24. Is Fraud on the Rise?
A pooled weighted average of 1.97% (N = 7, 95%CI:
0.86–4.45) of scientists admitted to have
fabricated, falsified or modified data or results at
least once – a serious form of misconduct by any
standard – and up to 33.7% admitted other
questionable research practices. In surveys asking
about the behaviour of colleagues, admission rates
were 14.12% (N = 12, 95% CI: 9.91–19.72) for
falsification, and up to 72% for other questionable
research practices.
-- Fanelli, PLoS ONE, 2009

25. Is Fraud on the Rise?
Meta-regression showed that self reports
surveys, surveys using the words “falsification”
or “fabrication”, and mailed surveys yielded
lower percentages of misconduct. When these
factors were controlled for, misconduct was
reported more frequently by
medical/pharmacological researchers than
others.
-- Fanelli, PLoS ONE, 2009

26. Is Fraud on the Rise?
Considering that these surveys ask sensitive
questions and have other limitations, it appears
likely that this is a conservative estimate of the
true prevalence of scientific misconduct.
-- Fanelli, PLoS ONE, 2009

27. This is Transparency?

28. Should Reporters Cover Conferences?

29. Conference Pitfalls
• Conferences select presenters based on < 1000 words
• Urologists at U of Florida & Indiana U studied 126
randomized controlled trials presented in 2002-2003

30. Conference Pitfalls
• RCTs are the “gold standard” of medical evidence
• But the quality of that evidence wasn’t pretty
• No abstract said how trial subjects were randomly
assigned to different treatments or placebos
• None told how the study ensured that neither the
researchers nor their doctors knew which they got
• Only about a quarter said how long researchers
followed the subjects in the trial

31. Just Say No
Sometimes, it’s better not to cover something.
But if you must…

32. Always Read the Study
Writing about a study after reading just a
press release or an abstract
– without reading the entire paper –
is journalistic malpractice

33. How to Get Studies
• www.EurekAlert.org for embargoed material
• Association of Health Care Journalists membership
includes access to Cochrane Library, Health Affairs,
JAMA, and many other journals
www.healthjournalism.org
• ScienceDirect (Elsevier) gives reporters free access to
hundreds of journals www.sciencedirect.com
• Open access journals (e.g., Public Library of Science
www.plos.org)
• Ask press officers, or the authors

34. Who Has an Interest?
• Disclose conflicts
• PharmedOut.org
• Dollars For Docs series
http://projects.propublica.org/docdollars/

35. Don’t Rely Only on Study Authors
• Find outside sources. Here’s how:

36. A Dirty Little Secret
Keep a biostatistician in your back pocket
Photo by Peyri Herrera, on Flickr

37. Acknowledgement/Contact
Thanks: Nancy Lapid, Reuters Health
http://retractionwatch.com
ivan-oransky@erols.com
Twitter: @ivanoransky