This document summarizes a presentation about NISO's work developing reproducibility badges and definitions to support open scholarship. It discusses how NISO convened stakeholders who agreed on defining reproducibility and replicability. They also agreed on a series of badges that recognize different stages of reproducible science practices. In early 2021, NISO released its recommended practice on this work but formed a standing committee to continue evolving it based on further community feedback. The committee aims to promote broader adoption by publishers, researchers, and funding agencies.

1. NISO Reproducibility Badging
and Definitions
NISO RP-31-2021
SSP Chicago, June 2, 2022
Session 3B
Community Standards and Recommendations Supporting Open Scholarship: A Host of Benefits for All
Gerry Grenier, STM Publishing Consultant
Affiliated with Code Ocean and Cadmore Media

2. So, What is Reproducibility?
• Reproducibility is part of the broader effort of Open Science
• One of the pathways by which the scientific community confirms the validity
of a scientific discovery is by repeating the research that produced it
• The scholarly research community – from funders to researchers to
publishers are beginning to address reproducibility
• National Academy of Sciences, at the behest of the U.S. Congress, conducted
a major study in 2019 to assess the extent of issues related to reproducibility
and to offer recommendations for improving rigor and transparency in
scientific research
• See full study at: http://sites.nationalacademies.org/sites/reproducibility-in-
science/index.htm
• Reproducibility in publishing is in its *very* early stages

3. What are Reproducibility Badges and Why the
Publishing Community Needs Them
• Badges: A recognition and reward scheme to recognize a researcher’s
participation in reproducibility (and therefore Open Science)
• Why: A number of reproducibility definitions and badging schemes
were under development within the scholarly publishing industry
• Standardization can result in Badges becoming a currency in the academic
rewards culture
• Instant recognition by peers and grant-funding agencies

4. NISO convened a broad stakeholder group

5. Agreement on crucial definitions
• First challenge – definitions used in reproducibility domain
• Quickly recognized and addressed ambiguity in existing definitions of
Reproducibility and Replicability
• Examined the scholarly record and adopted well-documented, supported
definitions
• Reproducibility: obtaining consistent results using the same input data,
computational steps, methods, and code, and conditions of analysis.
• Replicability: an independent study, aimed at answering the same scientific
question, has obtained consistent results leading to the same findings
(potentially using new artifacts or methods).

6. Recognition of several stages of reproducible
practice
• Agreement on a series of badges that recognized the different stages
of reproducible science
• Open Research Objects (included in primary research)
• Research Objects Reviewed
• Results Reproduced (assigned after a published article reproduces the original
results)
• Results Replicated (assigned after a published article reports on replication of
the results)

7. Recommended Practice released in early
2021, but will continue to evolve
• Public comment period resulted in some changes
• However, once the RP was finished the community provided more
feedback 
• Standing committee convened in early 2022 to address:
• Modifications
• Badge icon design
• Uptake by publishers

8. Standing committee going forward
• Broad publisher adoption
• Researcher adoption
• Recognition by the funding agencies