Headstart takes reproducibility very seriously. Our system needs to be fully auditable: our “match score” is a crucial element for candidate selection. At any point in time we need to be able to: - Access the models that were being used in production when the match score was computed; - Examine their code (including all upstream ETL/preprocessing pipelines); - Examine the data they were trained on; - Be able to deserialize the models and run diagnostics/tests on them. To support our requirements, we developed our own internal model versioning system using Git, Docker, CircleCI, AWS S3 and Pipenv. This presentation will share the design, implementation and functionalities of our versioning system, with a detailed walkthrough using our skill recommendation engine as a streamlined running example.

1. A practical approach to continuous deployment of Machine Learning pipelines
Machine Learning in Production
Christos Dimitroulas
Fullstack developer & DevOps
Luca Palmieri
Data Scientist & Machine Learning
#rapids2018

2. Prototyping Production
#rapids2018

3. ProductionOperational
requirements
=> Reproducibility!
● Robustness
● Availability
● Monitoring
● Debugging & auditability

4. ReproducibilityIngredients
● Code
● Environment
● Data
Software

5. ReproducibilityData
● Training
● Input
● Serialized dataset
● Database queries
● Output
● Serialized models
● Serialized transformers
● Inference...

6. Use caseSkill recommendation
● Training
● Input
● Users’ skills (MongoDB)
● Output
● Serialized SVD model
● Serialized transformers
● Automated retraining
● Automated deployment on changes

7. AutomationData Science Culture
● Code quality delivers value
● DataOps?
● Data scientist ownership needs to
get closer to production

8. #rapids2018

9. #rapids2018

10. Requirements
● Strict dependency management
● Model performance analysis
● Model versioning
● Infrastructure (as code)
● Data versioning
● Continuous integration/delivery
#rapids2018

11. Dependency Management
Lockfile! Managed virtualenv! Reproducible environments!
#rapids2018

12. Model Performance Analysis
1. Does the model meet the basic
performance requirements?
2. How does the performance
compare to the latest model in
production?
#rapids2018

13. Model Versioning
Two levels of versioning:
1. Which version of the code
produced this model?
2. Which data produced this model?
#rapids2018

14. Infrastructure
● Needs to scale (ideally
automatically)
● Should be easy to maintain,
change or upgrade
● Must be easily reproducible (for
other environments e.g. staging)
● Make it easier for other team
members to develop infrastructure
#rapids2018

15. Data Versioning
Depends largely on how the data input for the model is created.
In our use case, input comes from a database query.
● MongoDB automated backups (snapshots)
● Save relevant snapshots alongside models
#rapids2018

16. Continuous Integration tool
● Trigger a pipeline from code
changes
● Schedule jobs
● Run tests & scripts
#rapids2018

17. Continuous Delivery
serve-skill-recommender-system
Build training container
skill-recommender-system
Private PyPi server
Train new models
Assess models
Build serving API container
Upload models
#rapids2018

18. ● S3 is “primitive” - where is git diff for data?
● Painful transition: prototyping is chaos, production is order sancta sanctorum
● Generalize our process to a proper internal tool
#rapids2018