Version Control in AI/Machine Learning by Datmo

@TheNickWalsh +@TheNickWalsh +
Version Control for Machine
Learning and AI
@TheNickWalsh
Workshop
Metis SF

@TheNickWalsh +@TheNickWalsh +@TheNickWalsh
Before we begin:
datmo.com/get-started
Datmo installation:*

Nick Walsh
Developer Evangelist, Datmo
@TheNickWalsh

Workshop Outline
1. Conventional version control
2. The curious case of QoD’s
3. How Datmo bridges the gap
4. Iris dataset model example
5. Reproduce + use the model

What is Version Control?
The management of changes to
documents, computer programs, large
web sites, and other collections of
information.
*AKA `Source Control`
“

https://www.ctl.io/developers/assets/images/blog/scmhistory.png
Version Control Timeline
mercurial

You’ve probably heard of Git.
Git is a version control system for tracking
changes in computer files and
coordinating work on those files among
multiple people. It is primarily used
for source code management in software
development, but it can be used to keep
track of changes in any set of files.

So, GitHub, right?
(Yes, and no.)

Git(Hub) Revolutionized
Software Development

GitHub = SCM + Hosting + Much More

For developers: For enterprises:
• Self-managed SMC servers
became a thing of the past
• Developers could leverage
industry best practices for their
own personal work
• Community of knowledge
built around a known standard
• Collaboration on Open Source
Software
• Advent of continuous
integration / deployment
• Removed need for external
code issue tracking tool
• Consolidation of code storage
and versioning tool 
• Pull Requests, code review,
documentation through
ReadMe

QoD’s == Quantitative Oriented Developers
Artiﬁcial IntelligenceData Science Machine Learning

It’s time to talk about MLOps
https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-
systems.pdf

MLOps: The Elephant in the Room
https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-
systems.pdf

ML systems have a special capacity for incurring
technical debt, because they have all of the
maintenance problems of traditional code plus an
additional set of ML-specific issues. This debt may be
difficult to detect because it exists at the system level.
“
(Sculley et. al, 2015)

Typical methods for paying down code level
technical debt are not sufficient to address
ML-specific technical debt at the system level.
“
(Sculley et. al, 2015)

http://eng.uber.com/wp-content/uploads/2017/09/image8.png
Here’s where traditional SCM falls short

https://eng.uber.com/michelangelo/
https://code.facebook.com/posts/1072626246134461/
introducing-fblearner-flow-facebook-s-ai-backbone/

As for everyone else?

What is Datmo?
Datmo is a CLI workflow tool for ML,
AI, and Data Science developers. It
helps with managing model version
control, easy environment handling,
and reproducing results through the
power of snapshots.

What are Datmo Snapshots?
Code
Environment
Configuration
Files*
Metrics

Why are they important?
Environment
Configuration
Metrics
Datmo Snapshots
Git Commits
Code
Files*

How will it help?
Datmo leverages containers to quickly
spin up perfectly reproducible
developer environments. It tracks this
environment, along with model
metadata inside of snapshots.

From a broad perspective:
Make ML Ops and workflows
manageable and simple, not
completely abstracted away.
Reduce the amount of glue code
so that people can have more
robust pipelines.

GitHub = SCM + Hosting + More
Datmo = Model Versioning +
Environments + Deployment + More

Datmo in today’s example
We’re going to use Datmo to show how we can
quickly iterate on our model and streamline our
workflow.
We’ll go through using snapshots for A/B testing,
saving our tasks, and enabling you all to reproduce
my results/make your own changes to the model.

Problem:
Multiple Classification of Flower Species

Dataset: Fisher’s Iris Flower
http://archive.ics.uci.edu/ml/datasets/Iris

At a glance:
- 4 Features
- 3 Classes
- 150 Rows (50 per class)

Model Experimentation

Live Demo

Workshop Outline
1. Conventional version control
2. The curious case of QoD’s
3. How Datmo bridges the gap
4. MNIST model example
5. Reproduce + use the model

Reproducing the Model
$ datmo setup
One time initial setup:

Fork the model
Fork from Web Platform GUI (top right corner):
https://datmo.com/nmwalsh/workshop-iris-classification

Fetch the model from Datmo
$ datmo clone nmwalsh/workshop-iris-classiﬁcation
Clone the Datmo Model:
$ cd workshop-iris-classiﬁcation
Jump into this directory:

View all model Snapshots
$ datmo snapshot ls

Checkout to a particular snapshot
$ datmo snapshot checkout --id ______

Track Snapshots

Run the Task
$ datmo task run “python3 classiﬁer.py”

Run the Task
$ datmo task run “python3 classiﬁer.py”
We want our Python file to be run
inside of the container. Why?

What just happened?
• Datmo cloned the model from the platform,
bringing all of the necessary resources to local.
• Datmo set your current code to the state of the
desired snapshot.
• Datmo built the environment inside of a
container.
• Datmo executed the task inside of the container,
and logged the results.
datmo clone
datmo snapshot
checkout
Command Result
datmo task run

Make Your Own Snapshot!
1. Change a hyperparameter
2. Save the file
3. $ datmo task run “python3 classifier.py”
4. $ datmo snapshot task --id “<id>” -m “<your message>”
Tip: To see your new snapshot, use $ datmo snapshot ls

1. Traditional Source Control isn’t enough for QoD
Key Takeaways
2. Think about ML Ops before you’re “in too deep”
3. In the same way GitHub revolutionized Software
Engineering, Datmo does the same for QoD’s

Going Forward

Nuts and Bolts of Source Control:
http://ericsink.com/scm/source_control.html

2015 NIPS Paper from Google
https://papers.nips.cc/paper/5656-hidden-
technical-debt-in-machine-learning-systems.pdf

Code Available at:

Thank You!

Version Control in AI/Machine Learning by Datmo

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Version Control in AI/Machine Learning by Datmo

Similar a Version Control in AI/Machine Learning by Datmo (20)

Último

Último (20)

Version Control in AI/Machine Learning by Datmo