Slides for the Verta.ai ModelDB 2.0 webinar #1. Covers: (1) Why model versioning (2) ModelDB 2.0 Architecture (3) Hands-on exercise with ModelDB

1. www.verta.ai Confidential
Model Versioning Done Right:
A ModelDB 2.0 Walkthrough
Manasi Vartak, Ph.D.
CEO, Verta.ai
!1

2. www.verta.ai Confidential
Agenda
• Welcome & Logistics (5 mins)
• Model Versioning Overview (20 mins)
• Hands-on tutorial (20 mins)
• Wrap-up (5 mins)
!2

3. www.verta.ai Confidential
Welcome & Logistics
!3

4. www.verta.ai Confidential
About
• MIT CSAIL Ph.D.
• Thesis work: Infrastructure for
Model Management and Diagnosis
• Built first open-source model
management system: ModelDB
• Used by research labs,
Fortune-500s
• Founded Verta.ai to build ML Infra
!4
• Building the operations stack for production ML
• Model versioning, deployment & ops,
monitoring
• Built for data science; able to run at large scale

5. www.verta.ai Confidential
Logistics: This webinar will be hands-on!
!5
bit.ly/verta-webinar-1-instructions

6. www.verta.ai Confidential
Model Versioning Overview
!6

7. www.verta.ai Confidential
Data Scientists & ML Engineers build 100s of models
!7

8. www.verta.ai Confidential!8
Challenge: Model Development is empirical & ad-hoc
Model 1
Accuracy: 62%

9. www.verta.ai Confidential!9
Challenge: Model Development is empirical & ad-hoc
Model 3
Accuracy: 76%
val udf1: (Int => Int) = (delayed..)
df.withColumn(“timesDelayed”, udf1)
RandomForestClassifier

10. www.verta.ai Confidential!10
Challenge: Model Development is empirical & ad-hoc
Model 5
Accuracy: 68%
val udf1: (Int => Int) = (delayed..)
df.withColumn(“timesDelayed”, udf1)
RandomForestClassifier
credit-default-clean.csv
val lrGrid = new ParamGridBuilder()
.addGrid(rf.maxDepth, Array(5, 10, 15))
.addGrid(rf.numTrees, Array(50, 100))

11. www.verta.ai Confidential!11
Challenge: Model Development is empirical & ad-hoc
Model 50
Accuracy: 82%
val udf1: (Int => Int) = (delayed..)
df.withColumn(“timesDelayed”, udf1)
RandomForestClassifier
credit-default-clean.csv
val lrGrid = new ParamGridBuilder()
.addGrid(rf.maxDepth, Array(5, 10, 15))
.addGrid(rf.numTrees, Array(50, 100))
val labelIndexer1 = new LabelIndexer()
val labelIndexer2 = new LabelIndexer()
…
val udf1: (Int => Int) = (delayed..)
val udf2: (String, Int) = …
df.withColumn(“timesDelayed”, udf1)
.withColumn(“percentPaid”, udf2)
.withColumn(“creditUsed”, udf3)
val scaler = new StandardScaler()
.setInputCol(“features”) …

12. www.verta.ai Confidential
SOTA in Model Versioning
!12

13. www.verta.ai Confidential
Why is this a problem?
Scientific rigor
• “How do I or colleagues reproduce this work / this model?”
Production Safety
• “What version of the model is deployed?”
• “This model is breaking, we need to rollback”
Speed of Delivery
• With the right versioning, you can use CI/CD tools
Insurance
• “If Jane leaves the team, how do I preserve the knowledge?”
Productivity
• “What experiments have I run before?”
!13

14. www.verta.ai Confidential
Model Version Control
• Manage changes to models over time
• Audit, Change logs, Immutability
• Uniquely identify a model and the ability to return to it at any time
• Reproducibility, Governance
• Enable multiple data scientists to contribute and reuse
• Collaboration, Blame
!14

15. www.verta.ai Confidential
The software industry was built on the back of code versioning
• First code version system in 1972 for Linux
• ~5 generations of code versioning systems until now
!15
SCCS
RCS
Perforce

16. www.verta.ai Confidential
So what should we version?
!16
Source Code Binary Container
Software
Code + Other Ingredients
Weights /
Checkpoints
Container
Machine
Learning

17. www.verta.ai Confidential
Ingredients of a model
!17
1. Code

18. www.verta.ai Confidential
Ingredients of a model
!18
2. Config
Same Code, Diff
models

19. www.verta.ai Confidential
Ingredients of a model
!19
3. Data
Data changes
underneath the model

20. www.verta.ai Confidential
Ingredients of a model
!20
4. Environment

21. www.verta.ai Confidential
Versioning vs. Metadata vs. Experiment Management
!21
Reproducibility
• Uniquely identify a model
• Track every change
• Go back in time
Analytics
• Metrics, Performance
• Organizational data
• Searching, sorting
Applications
Experiment
Management
Compliance
Metadata
Versioning
Collaboration
run.metrics[‘accuracy’] = 0.7
run.tags = [‘demo’, ‘nlp’]
run = run_experiment(code,
data, config, environment)

22. www.verta.ai Confidential
Model Versioning in ModelDB 2.0
!22

23. www.verta.ai Confidential
ModelDB
• ModelDB 1.0: Metadata focused
• What did it manage?
• Hyperparameters
• Metrics
• Pipeline metadata
• Tags
!23
• ModelDB 2.0: Versioning focused
• Git-like constructs for versioning
• Commits, Forks, Merges
• Metadata tracking as before
• APIs for applications
• Integration into MLOps
https://github.com/VertaAI/modeldb/

24. www.verta.ai Confidential
Why not Git?
• Git is fantastic for code
• Git doesn’t handle large files very well; DBMSs can’t be stored on Git
• Needs to be accessed as a library vs. command-line
• Git is distributed, each user must download the full repo history (1MM
models)
• All commits & branches (even unsuccessful ones) need to be preserved
!24

25. www.verta.ai Confidential
ModelDB 2.0: A “super repo”
!25
ModelDB Repo

26. www.verta.ai Confidential
ModelDB 2.0: A “super repo”
!26
ModelDB Repo
Config RepoEnv Repo
Data Repo Code Repo
Delegate operations to
specialized versioning
systems
??
?? ??
Where systems don’t
exist, we have created
light-weight systems

27. www.verta.ai Confidential
ModelDB 2.0: A “super repo”
!27
ModelDB Repo
Config RepoEnv Repo
Data Repo Code Repo
Delegate operations to
specialized versioning
systems
Where systems don’t
exist, we have created
light-weight systems
Company
FeatureStore

28. www.verta.ai Confidential
What does a ModelDB Version look like?
!28
census-repo 5y64er mdb://census-repo
data-train 34ae59 s3://census-data
code ke87a5 git://github.com/census
config 1ee3pk config://census-repo
env-train 5le723 env://census-repo
census-repo 5y64er mdb://census-repo
data-train 34ae59 s3://census-data
code ke87a5 git://github.com/census
config 1ee3pk config://census-repo
env-train 5le723 env://census-repo
annotation-data 67er44 s3://census-annotations
annotation-code 8bh651 git://github.com/annotation-code

29. www.verta.ai Confidential
How do you re-constitute a model?
!29
> mdb checkout mdb://census-repo@5y64er
> git checkout git://github.com/census@ke87a5
> docker --build env://census-repo@5le723
> docker run train.py --data s3://census-data@34ae59
--config config://census-repo@1ee3pk
census-repo 5y64er mdb://census-repo
data-train 34ae59 s3://census-data
code ke87a5 git://github.com/census
config 1ee3pk config://census-repo
env-train 5le723 env://census-repo

30. www.verta.ai Confidential
Advantages of a “super repo”
!30
ModelDB Repo
Config RepoEnv Repo
Data Repo Code Repo
Revert
RevertRevert
RevertRevert

31. www.verta.ai Confidential
Advantages of a super repo
!31
ModelDB Repo
Config RepoEnv Repo
Data Repo Code Repo
Merge
Merge
MergeMerge
Merge

32. www.verta.ai Confidential
How do you fork and merge?
!32
census-repo 5y64er mdb://census-repo
data-train 34ae59 s3://census-data
code ke87a5 git://github.com/
censusconfig 1ee3pk config://census-repo
env-train 5le723 env://census-repo
Fork
Fork
census-repo 5y64er
data-train g3e4rr
code ke87a5
config 1ee3pk
env-train 5le723
census-repo 5y64er
data-train 34ae59
code r77fef
config kj7h14
env-train 5le723
census-repo 5y64er
data-train g3e4rr
code r77fef
config kj7h14
env-train 5le723
Merge

33. www.verta.ai Confidential
Hands-on Tutorial
!33

34. www.verta.ai Confidential
You will need…
!34
bit.ly/verta-webinar-1-instructions

35. www.verta.ai Confidential
That’s a wrap!
!35

36. www.verta.ai Confidential
Thanks!
• ModelDB 2.0 is constantly updated
• Use it (Star, fork, etc., let us know about use cases)
• Give us feedback, partner with us
• Contribute to it
• We love this stuff — hit us up if we can do a tech talk / help you get set up!
• If you are looking for a hosted version, drop us a line: modeldb@verta.ai
!36

37. www.verta.ai Confidential
Thanks!
!37
https://github.com/VertaAI/modeldb | Slack: http://bit.ly/modeldb-mlops | https://verta.ai
manasi@verta.ai | @DataCereal | modeldb@verta.ai