Jacek Zurada, IEEE Life Fellow 2014 TAB Chair (IEEE V-P Technical Activities)

1. Reproducible Research: Best Practices for
Reviewing (and Writing) a Paper
Jacek Zurada, IEEE Life Fellow
Candidate for 2019 IEEE President-Elect (2020 President)
May 20, 2018

2. These Select (non-comprehensive) Slides are Opinions
of the Author and not Necessarily Opinions of IEEE
Summarize the Authors’ editorial experience and related
workshops in his multiple roles in IEEE that include:
Candidate for 2019 IEEE President-Elect (2020 President)
2014 TAB Chair (IEEE V-P Technical Activities)
Editor-in-Chief of IEEE Transactions on Neural Networks
1998-2003
Chair of IEEE TAB Periodicals Committee 2010-11
Chair of IEEE TAB Periodicals Review and Advisory
Committee 2012-13
2

3. Reproducible Research
-Reproducible Research embraces the idea
that the ultimate research product (i.e.,
paper, report, communications, etc.)
contains full disclosure of the environment
used to produce the results such as the
code, data, experimental and laboratory
conditions etc. that are needed to
reproduce the results and possibly expand
it through new work

4. Why Reproducible Research?
-Motivations and pressures come from
•The digitized web-of-science world and the
publication business that has exploded in
volume since 2000
•Authors/researchers/readers
•Reviewers, editors, evaluators
•Sponsors, funding agencies
•Economy
•.... and common sense
4

5. Reproducible Research: report

6. Reproducible Research Across Disciplines
- Theoretical sciences
• Easy
- Experimental sciences
• Physics, chemistry: moderate
• Life sciences: moderate to difficult
- Engineering and computational sciences
• Theoretical: easy
• Practical/computational/experimental:
moderate, but definitely should not be difficult!

7. Who Holds the Keys to Reproducible Research?
▸Authors (all)
▸Sponsors
▸Reviewers
▸Associate/Action Editors
▸Editors-in-Chief
▸AND READERS DEMAND IT

8. How to Review a Paper for Originality?
▸What novel aspects are revealed in the paper?
▸It should be one or more
- new experimental approach
- new phenomenon or material, or properties thereof
- new method, relevant application or model
- new theorem(s)
- new data generated
- new interpretations or insights
These selected best practices are contributed by an experienced editor
and are not official IEEE review guidelines

9. How to Review a Paper for RR? (underlined passages
connect to RR)
▸Experimental paper: needs a hypothesis, provides
experimental validation (equipment, materials, approach,
method need to be discussed)
▸Modeling paper: assumptions, mathematical approach,
tools used, method, illustrations
▸Computational paper: inputs, computational approach,
tools used, method, initial conditions, possibly computing
statistics (if results not directly tractable)
These selected best practices are contributed by an experienced editor
and are not official IEEE review guidelines

10. How to Review a Paper for RR?
▸Work should be very well-referenced and credit given to
previous contributors
▸Results in the paper should be compared with either
benchmark results or other authors’ results
▸Physical and computational experiments must be
repeatable, if deterministic (initial conditions,
assumptions)
▸When computational experiments are non-tractable,
repetitive experiments and their statistics needed,
instead
These selected best practices are contributed by an experienced editor
and are not official IEEE review guidelines

11. How to Review a Paper for RR?
▸Research should be replicable by others in the public domain (data
open publicly)
▸Formulas must be verifiable
▸If a theorem is proven, the proof must be completely tractable:
assumptions, theorems, derivations, proofs
▸All quantities must be clearly defined (i.e. ‘MSEs’ are often vague
and left to interpretations)
▸Tests on benchmark data or problems essential
▸Discussion of alternative solutions
These selected best practices are contributed by an experienced editor
and are not official IEEE review guidelines

12.  IEEE DataPort is a web-based, cloud-services
platform/data repository supporting the RR needs of
authors and readers, https://ieee-dataport.org/
 Incubated through the IEEE Big Data Initiative
 Like IEEE Xplore, but Data-focused
 Full service platform
• Referenceable datasets through Data Object
Identifiers (DOI)
• Linkable to/from published and unpublished
documents through URLs
How IEEE DataPort Helps with RR

13. How IEEE CodeOcean Helps with RR?
IEEE publishes a large volume of articles that reference code
and algorithms
- By using Code Ocean’s services, algorithms in IEEE articles can
now be executed in real time
- IEEE authors can upload, share, and run their code on the Code
Ocean platform
- IEEE Xplore users can discover, view, and execute code in real-
time
- Users will discover and access the bidirectional link between
IEEE Xplore and Code Ocean
- The service is free and accessible through IEEE Xplore
- https://codeocean.com/signup/ieeeData as of 31December2016