Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.
Making computations reproducible
Tokyo.SciPy #6
2014-08-02
1 / 31
Abstract
Scientific computations tend to involve a number of experiments
under different conditions.
It is important to man...
Outline
...1 Introduction
...2 Discipline
Three elements
Three rules
Complements
...3 Practice
...4 Summary
3 / 31
1. Introduction
4 / 31
Background
A lab notebook is indispensable for experimental research in natural
science. One of its role is to make experi...
Common problems
Common problems in computational experiments:
I confused which results is got under which condition.
I ove...
Goal
To archive all results of each experiment with
all information required to reproduce them
so that we can retrieve and...
Note
What is introduced in this talk is not a established methodology,
but a collection of field techniques. Same with word...
2. Discipline
9 / 31
Three elements
We distinguish the following elements which affect reproducibility of
computations:
Algorithm an algorithm ...
Three rules
Give an Identifier to each element and archive them.
Record a machine-readable Recipe
with a human-readable com...
.
Identifier..
......Give an Identifier to each element and archive them.
Algorithm
use version control system
Data
give a n...
.
Recipe
..
......
Record a machine-readable Recipe
with a human-readable comments.
A recipe should include all informatio...
Typically a recipe include the following information:
in which order
which data is processed
by which algorithm
under whic...
.
Mechanize..
......Make every manipulation Mechanized.
Run the whole process of an experiment by a single operation.
No m...
complement: Tentative experiment
Too large archive detracts substantive significant of reproducibility.
For tentative exper...
complement: Reuse of intermediate data
In order to reuse intermediate data, utilize an identifier.
Explicitly specify inter...
3. Practice
18 / 31
Identify Algorithm
Use a version control system to manage source codes
such as Git and Mercurial.
It is easy to record a r...
Identify Data
File
Give appropriate names to directories and files,
then a resolved absolute path can be used as an identifi...
Identify Environment
Python package
Use PyPa tools (virtualenv, setuptools and pip) or Conda/enstaller.
Library
Use HashDi...
HashDist
A tool for developing, building and managing software stacks.
An software stack is described by YAML.
We can crea...
Recipe: configuration file
A configuration in recipe should be of a machine-readable format.
Use ConfigParser, PyYAML or json ...
Recipe: script file
A script in recipe should run the whole process
by a single operation.
There are several alternatives t...
maf
“maf is a waf extension for writing computational experiments.”
Conduct computational experiments as build processes.
...
Recipe: automatic generation
Do it yourself, or use Sumatra.
“Sumatra: automated tracking of scientific computations”
recor...
4. Summary
27 / 31
Summary
We have introduced 3 rules to manage computational experiments
so that their results are reproducible.
However, ou...
Links
PyPa http://python-packaging-user-guide.
readthedocs.org
Conda http://conda.pydata.org
enstaller https://github.com/...
References
[1] G. K. Sandve, A. Nekrutenko, J. Taylor, E. Hovig, “Ten Simple Rules
for Reproducible Computational Research...
fin.
back to outline
Revision: f2b0e97 (2014-08-03)
31 / 31
Próxima SlideShare
Cargando en…5
×

Making Computations Reproducible

1.232 visualizaciones

Publicado el

Publicado en: Ciencias
  • Sé el primero en comentar

Making Computations Reproducible

  1. 1. Making computations reproducible Tokyo.SciPy #6 2014-08-02 1 / 31
  2. 2. Abstract Scientific computations tend to involve a number of experiments under different conditions. It is important to manage computational experiments so that their results are reproducible. In this talk we introduce 3 rules to make computations reproducible. 2 / 31
  3. 3. Outline ...1 Introduction ...2 Discipline Three elements Three rules Complements ...3 Practice ...4 Summary 3 / 31
  4. 4. 1. Introduction 4 / 31
  5. 5. Background A lab notebook is indispensable for experimental research in natural science. One of its role is to make experiments reproducible. Why not for computational research? . ......Lack of reproducibility means lack of reliability. 5 / 31
  6. 6. Common problems Common problems in computational experiments: I confused which results is got under which condition. I overwrote previous results without intent. I used inconsistent data to get invalid results. ... Not a few problems are caused due an inappropriate management of experiments. 6 / 31
  7. 7. Goal To archive all results of each experiment with all information required to reproduce them so that we can retrieve and restore easily in a systematic and costless way. 7 / 31
  8. 8. Note What is introduced in this talk is not a established methodology, but a collection of field techniques. Same with wording. In this talk, we will not deal with distributed computation documentation or test publishing of a paper release of OSS 8 / 31
  9. 9. 2. Discipline 9 / 31
  10. 10. Three elements We distinguish the following elements which affect reproducibility of computations: Algorithm an algorithm coded into a program implemented by yourself, calling external library, ... Data input and output data, intermediate data to reuse Environment software and hardware environment external library, server configuration, platform, ... 10 / 31
  11. 11. Three rules Give an Identifier to each element and archive them. Record a machine-readable Recipe with a human-readable comments. Make every manipulation Mechanized. 11 / 31
  12. 12. . Identifier.. ......Give an Identifier to each element and archive them. Algorithm use version control system Data give a name to distinguish data kind give a version to distinguish concrete content Environment find information of platform find a version (optionally build parameters) of a library Keep in mind to track all elements during the whole process: every code under version control no data without an identifier no temporary environment 12 / 31
  13. 13. . Recipe .. ...... Record a machine-readable Recipe with a human-readable comments. A recipe should include all information required to reproduce the results of an experiment (other than contents of Algorithm, Data and Environment stored in other place.) A recipe should be machine-readable to re-conduct the experiment. A recipe should include a human-readable comment on purpose and/or meanings of the experiment. A recipe should be generated automatically by tracking experiments. 13 / 31
  14. 14. Typically a recipe include the following information: in which order which data is processed by which algorithm under which environment with which Parameter Typically a recipe consists of the followings: a script file to run the whole process a configuration file which specifies parameters and identifiers a text file of comments 14 / 31
  15. 15. . Mechanize.. ......Make every manipulation Mechanized. Run the whole process of an experiment by a single operation. No manual manipulation of data. No manual compilation of source codes. Automated provision of an environment. 15 / 31
  16. 16. complement: Tentative experiment Too large archive detracts substantive significant of reproducibility. For tentative experiments with ephemeral results, it is not necessarily required to record. test of codes trial on tiny data ... If there is a possibility to get a result which might be used, referred or looked up afterward, then it should be recorded. 16 / 31
  17. 17. complement: Reuse of intermediate data In order to reuse intermediate data, utilize an identifier. Explicitly specify intermediate data to reuse by an identifier. Automatically detect available intermediate data based on dependency. ... 17 / 31
  18. 18. 3. Practice 18 / 31
  19. 19. Identify Algorithm Use a version control system to manage source codes such as Git and Mercurial. It is easy to record a revision and uncommitted changes at each experiment. (Learn inside of VCS if you need more flexible management.) 19 / 31
  20. 20. Identify Data File Give appropriate names to directories and files, then a resolved absolute path can be used as an identifier. If no meaningful word is thought up, use time-stamp or hash. DB or other API A pair of URI and query of which results are constant can be used as an identifier. If API behaves randomly, keep the results at hand (w/time-stamp). 20 / 31
  21. 21. Identify Environment Python package Use PyPa tools (virtualenv, setuptools and pip) or Conda/enstaller. Library Use HashDist. It is an alternative to utilize CDE. Platform Use platform, a standard library of Python Server configuration Use Ansible or other configuration management tool, and Vagrant or other provisioning tool. 21 / 31
  22. 22. HashDist A tool for developing, building and managing software stacks. An software stack is described by YAML. We can create, copy, move and remove software stacks. $ git checkout stack.yml $ hit build stack.yaml 22 / 31
  23. 23. Recipe: configuration file A configuration in recipe should be of a machine-readable format. Use ConfigParser, PyYAML or json module to read/write parameters in INI, YAML or JSON format. A receipt should include the followings: command line argument environment variable random seed 23 / 31
  24. 24. Recipe: script file A script in recipe should run the whole process by a single operation. There are several alternatives to realize such a script: utilize a build tool (such as Autotools, Scons, and maf) utilize a job-flow tool (such as Ruffus, Luigi) write a small script by hand (e.g. run.py) 24 / 31
  25. 25. maf “maf is a waf extension for writing computational experiments.” Conduct computational experiments as build processes. Focus on machine learning: list configurations run programs with each configuration aggregate and visualize their results 25 / 31
  26. 26. Recipe: automatic generation Do it yourself, or use Sumatra. “Sumatra: automated tracking of scientific computations” recording information about experiments, linking to data files command line & web interface integration with LATEX/Sphinx $ smt run --executable=python --main=main.py conf.param input.data $ smt comment "..." $ smt info $ smt repeat 26 / 31
  27. 27. 4. Summary 27 / 31
  28. 28. Summary We have introduced 3 rules to manage computational experiments so that their results are reproducible. However, our method is just a makeshift patchwork of field techniques. . ...... We need a tool to manage experiments in more integrated, systematic and sophisticated manner for reproducible computations. 28 / 31
  29. 29. Links PyPa http://python-packaging-user-guide. readthedocs.org Conda http://conda.pydata.org enstaller https://github.com/enthought/enstaller HashDist http://hashdist.github.io CDE http://www.pgbovine.net/cde.html Ansible http://www.ansible.com Vagrant http://www.vagrantup.com Scons http://www.scons.org maf https://github.com/pfi/maf Ruffus http://www.ruffus.org.uk Luigi https://github.com/spotify/luigi Sumatra http://neuralensemble.org/sumatra 29 / 31
  30. 30. References [1] G. K. Sandve, A. Nekrutenko, J. Taylor, E. Hovig, “Ten Simple Rules for Reproducible Computational Research,” PLoS Comput. Biol. 9(10): e1003285 (2013). doi:10.1371/journal.pcbi.1003285 [2] V. Stodden, F. Leisch, R. Peng, “Implementing Reproducible Research,” Open Science Framework (2014). osf.io/s9tya 30 / 31
  31. 31. fin. back to outline Revision: f2b0e97 (2014-08-03) 31 / 31

×