With the progress towards open science, scientific communication is facing a new wave of innovations towards more openness and speed of research publication which will deeply affect the way the peer review function is carried out and the overall role of journals in assuring quality and adding value to manuscripts. Several initiatives are promoting the generalized adoption of open access preprints as a formal beginning stage of research publication, which has been common since the 90’s in the physics community. And, in the last decade, new ways to carry out the evaluation of manuscripts have emerged either to replace or to improve the traditional methods, which are widely criticized as being slow and expensive in addition to lacking transparency. Quality nonprofit journals from emerging and developing countries have succeeded to follow the main innovations brought by the Internet. In addition to the technicalities of the digital publishing, there is a wide adoption of Open Access in the international flow of scientific information. The new wave of innovations that affect the peer review function and the changing role of journals pose new challenges to the emerging and developing countries in regard of scientific publishing. The adoption of these innovations is essential for progress of SciELO as a leading open access program to enhance scientific communication. The scope of this workshop aims at an in-depth analysis and discussion of the state of art and main trends of the peer review function, the modalities of carrying it out as well as of the increasing adoption of mechanisms to speed publication such as preprints and how they affect and potentially renew the role of journals. These recommendations will guide SciELO policies on manuscript evaluation and on the adoption of preprint publications.

1. Open peer review
Liz Allen
Director of Strategic Initiatives, F1000
Scielo 20| Sao Paolo| September 2018
(through the lens of F1000’s open research publishing platforms)
@allen_liz

2. F1000’s Director of Strategic Initiatives (2015 – present)
Head of Evaluation at Wellcome (2000 - 2015 )
ORCID Board Director (2010 – 2015)
Co-led development of project CRediT (2010 - present)
Crossref Board Director (2017 – present)
Visiting Senior Research Fellows at Policy Institute @ Kings College London
Love all things research outputs, metrics & ‘science of science’
About me (declarations)

3. Introductory thoughts
A crisis in peer review?
The future is open …
The case of F1000’s post-publication open peer review model
Exploring the benefits and challenges of open peer review
Thoughts? Discussion?
Outline of discussion

4. Source: https://www.editage.com/insights/7-common-types-of-academic-peer-reviewwitter

5. A crisis in peer review?

7. Source: https://www.forbes.com/sites/geoffreykabat/2015/11/23/the-crisis-of-peer-review/#61e5474c463e
“If peer review was a drug, it
would never get on the market."
Geoffrey Kabat
Author & Cancer Epidemiologist (quoted in 2015)

8. Hannes Alfvén (1908-1995)
awarded Nobel Prize for physics 1970
"The peer review system is
satisfactory during quiescent times,
but not during a revolution in a
discipline such as astrophysics, when
the establishment seeks to preserve
the status quo."

9. Source: https://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970 Source: http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(13)62329-6/

10. “There are many details to be resolved, but
the basic principle of independent, expert
peer review seems to be at the absolute core
of scientific [publishing]
Source: Royal Society (2015) Future of Scholarly Scientific Communication
https://royalsociety.org/~/media/events/2015/04/FSSC1/FSSC-Report.pdf
Royal Society (2015)
… we need to retain that basic principle,
however we go about organising it.”

11. “Although frequently criticized, peer review
still plays an important role in validating
research results and advancing discovery.
Source: Digital Science (2016) What might peer review look like in 2030?
https://www.digital-science.com/blog/news/the-future-of-peer-review-new-report-by-biomed-central-and-digital-science-
spotonreport/
We want to start conversations with all
stakeholders to find progressive ways of
improving peer review for researchers
globally and across all disciplines.”

12.  Tailoring & selection precision
 Diversity
 Experimentation – new approaches
 Training & mentoring
 Cross-publisher sharing /portability/
efficiencies
 Recognition, credit & reward for reviewers
 Technology to improve effectiveness
Recommendations
Source: Digital Science (2016) What might peer review look like in 2030?
https://www.digital-science.com/blog/news/the-future-of-peer-review-new-report-by-biomed-central-and-digital-science-
spotonreport/

13.  Tailoring & selection precision
 Diversity
 Experimentation – new approaches
 Training & mentoring
 Cross-publisher sharing /portability/
efficiencies
 Recognition, credit & reward for reviewers
 Technology to improve effectiveness
Recommendations
Source: Digital Science (2016) What might peer review look like in 2030?
https://www.digital-science.com/blog/news/the-future-of-peer-review-new-report-by-biomed-central-and-digital-science-
spotonreport/

14. Source: https://www.editage.com/insights/7-common-types-of-academic-peer-review#.Wb8ag6Uf074.twitter

15. The future is open …

16. Source: https://www.fosteropenscience.eu/content/what-open-science-introduction

18. https://sfdora.org

19. Ross-Hellauer T. What is open peer review? A systematic review F1000Research 2017, 6:588
(doi: 10.12688/f1000research.11369.1)
“ … open peer review (OPR) as an
umbrella term for a number of
overlapping ways that peer review
models can be adapted in line with
the ethos of Open Science …

20. ‘Taxonomy’ of open peer review
Ross-Hellauer T. What is open peer review? A systematic review F1000Research 2017, 6:588
(doi: 10.12688/f1000research.11369.1)

21. ‘Open peer review (OPR) is a
cornerstone of the emergent open
science agenda.
Yet to date no large scale survey of
attitudes towards OPR amongst
academic editors, authors, reviewers
and publishers.’
Survey (2016) sample n=3000+
Ross-Hellauer T, Deppe A, Schmidt B (2017) Survey on open peer review: Attitudes
and experience amongst editors, authors and reviewers. PLOS ONE 12(12):
e0189311. https://doi.org/10.1371/journal.pone.0189311
https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0189311

22. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0189311

23. Ross-Hellauer T, Deppe A, Schmidt B (2017) Survey on open peer review: Attitudes
and experience amongst editors, authors and reviewers. PLOS ONE 12(12):
e0189311. https://doi.org/10.1371/journal.pone.0189311
https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0189311

24. http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0189311

25. HOW WE SHARE &
TALK ABOUT
SCIENCE IS
CHANGING FAST

26. Widening range of research outputs available
Data
Resources
Software
all with a
trackable &
persistent
DOIs

27. Source: 2017 http://asapbio.org/
Growth in rapid publication models

28. Evolving policy & research funding landscape

29. Source: https://www.nature.com/articles/d41586-018-06178-7

30. Introducing F1000’s publishing model

31. Publish. Engage. Accelerate.
• Post-publication peer review
• Open peer review
• Open data
• Open access
• Versioning
• Indexing
• Comprehensive meta-data
• Full usage metrics
• Living figures
• more to come ….

32. F1000’s post-publication, open peer review
Preprint-like stage

33. F1000’s post-publication, open peer review
 invited peer review
 indexing in bibliographic databasesPreprint-like stage

34. 1 …………………
2 …………………
3 …………………
4 …………………
5 …………………
Author
Author
suggestions
AI tool
Memory
Editor
Editorial
verificationAuthor selection
F1000 peer review selection model
Authors control the selection of reviewers suggested for peer review

35. ‘published’ & awaiting peer review
Article immediately
citable with a DOI and
usage indicators
Article status
noted: awaiting
peer review
Article version clear ...

36. ‘published’ & undergoing peer review

37. http://cartoonsbyjosh.co.uk

38. Source: https://www.editage.com/insights/7-common-types-of-academic-peer-review#.Wb8ag6Uf074.twitter

39. “ … undoubtedly an increase in the
use of researchers publishing their
research on alternative platforms
for biomedical sciences …”
Kirkham J and Moher D. Who and why do researchers opt to publish in post-publication peer review
platforms? - findings from a review and survey of F1000 Research F1000Research 2018, 7:920

41. Kirkham J and Moher D. Who and why do researchers opt to publish in post-publication peer review
platforms? - findings from a review and survey of F1000 Research F1000Research 2018, 7:920
Motivation to publish on F1000Research: open access?
84%

42. Kirkham J and Moher D. Who and why do researchers opt to publish in post-publication peer review
platforms? - findings from a review and survey of F1000 Research F1000Research 2018, 7:920
Motivation to publish on F1000Research: open peer review?
74%

43. Kirkham J and Moher D. Who and why do researchers opt to publish in post-publication peer review
platforms? - findings from a review and survey of F1000 Research F1000Research 2018, 7:920
Motivation to publish on F1000Research: speed of publication?
81%

44. Easier adoption in certain research areas?

45. Publication trends F1000Research

46. Debating the benefits & challenges of OPR

47. Response rates in open system? Variation by demographics seniority?
geography? etc
Level and nature of reviews in open system: detail & nature of response (e.g.
constructive?)
Levels of agreement & tendency towards group think?
Time & resource burden: open vs closed models
Ability to get recognition and credit for peer review work
System efficiencies? Reduce resource involved in finding & securing reviews
Outline of discussion: OPR experimentation

48. Response rates in open system? Variation by demographics seniority?
geography? etc
Level and nature of reviews in open system: detail & nature of response (e.g.
constructive?)
Levels of agreement & tendency towards group think?
Time & resource burden: open vs closed models
Ability to get recognition and credit for peer review work
System efficiencies? Reduce resource involved in finding & securing reviews
Outline of discussion: OPR experimentation

49. Levels of agreement between reviewers in open, post-publication model
Results
1,133 articles = 2,266 reviews
Median time between first two reviews = 18 days [interquartile 6-52 days]
Cohen K indicated only a ‘fair agreement’ between reviewers; changing
minimally over time between review
Evidence?
Agreement between reviewers
did not change over time
No evidence that reviewers are
systematically influenced by
seeing the review of a previous
reviewer
But
Need comparators
Monitor over time as new, open
models become more
commonplace
Source: https://f1000research.com/posters/6-1678

50. ‘published’ & undergoing peer review
Transparent
review
history

51. Response rates in open system? Variation by demographics seniority?
geography? etc
Level and nature of reviews in open system: detail & nature of response (e.g.
constructive?)
Levels of agreement & tendency towards group think?
Time & resource burden: open vs closed models
Ability to get recognition and credit for peer review work
System efficiencies? Reduce resource involved in finding & securing reviews
Outline of discussion: OPR experimentation

52. Review activity with trackable &
persistent DOIs & linked to ORCID &
Crossref metadata

53. … near ORCID id full house!

54. Recognition for peer review efforts

55. Response rates in open system? Variation by demographics? seniority?
geography? etc
Level and nature of reviews in open system: detail & nature of response (e.g.
constructive?)
Levels of agreement & tendency towards group think?
Time & resource burden: open vs closed models
Ability to get recognition and credit for peer review work
System efficiencies? Reduce resource involved in finding & securing reviews
Outline of discussion: OPR experimentation

57. Source: https://figshare.com/articles/Diversity_and_inclusion_in_peer_review_and_beyond/7098674/2

58. Gender of submitting authors
58
Gender of referees
Total submitting authors: 258 Total suggested referees: 1908
2%
47%51%
6%
31%
63%
Female
Male
Unknown
Overall gender balance on WOR
Source: https://figshare.com/articles/Diversity_and_inclusion_in_peer_review_and_beyond/7098674/2

59. 59
Gender balance of author-suggested
referees
Gender of author-suggested
referees
Gender of referees
suggested by a female author
Total referees suggested by
authors: 1545
Total referees suggested
by female authors: 749
Female
Male
Unknown
6%
31%
63%
4%
60%
36%
Gender of referees
suggested by a male author
Total referees suggested
by male authors: 743
6%
27%
67%
Source: https://figshare.com/articles/Diversity_and_inclusion_in_peer_review_and_beyond/7098674/2

60. Gender balance of AI and editor
suggested referees
60
5%
33%62%
Gender of editor-suggested referees Gender of referees suggested by the algorithm
Total referees suggested by editors: 183 Total referees suggested by the tool: 329
6%
31%
63%
Female
Male
Unknown
Source: https://figshare.com/articles/Diversity_and_inclusion_in_peer_review_and_beyond/7098674/2

61. Key findings:
• Author selected peer review provides comparable gender balance to other
publisher studies (~25-35% female reviewers).
• There is little difference in gender balance when comparing author
suggestions, editorial suggestions and algorithmic suggestions.
• Male authors suggest a higher % of male reviewers, while female authors
suggest a higher % of female reviewers – comparable to other publisher
studies.
Source: https://figshare.com/articles/Diversity_and_inclusion_in_peer_review_and_beyond/7098674/2

62. What do you think?
Liz Allen
Director of Strategic Initiatives, F1000
Scielo 20| Sao Paolo| September 2018
@allen_liz

63. Right click background and
select Format Background
Upload your
background image
Step 1 Step 2

64. Title option 3 - Photographic
Author name