Se ha denunciado esta presentación.
Se está descargando tu SlideShare. ×

[DSC Europe 22] Reproducibility and Versioning of ML Systems - Spela Poklukar

Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio

Eche un vistazo a continuación

1 de 27 Anuncio

[DSC Europe 22] Reproducibility and Versioning of ML Systems - Spela Poklukar

Descargar para leer sin conexión

Reproducibility of ML system is increasingly important topic in ML community. Reproducibility ensures conclusiveness of the model performance, provides an understanding how ML system works and reduces unnecessary errors when the system is deployed into production. With increasing AI regulation, it will soon become a requirement for many ML applications. In this talk, we will explore different aspects of reproducibility such as reproducibility of the dataset, data processing, ML model, its randomness and hyperparameters, code and SW environment, as well as concepts and practical tools such as data versioning, feature, metadata and artifact store, model registry and containerization that together ensure reproducibility of our experiments.

Reproducibility of ML system is increasingly important topic in ML community. Reproducibility ensures conclusiveness of the model performance, provides an understanding how ML system works and reduces unnecessary errors when the system is deployed into production. With increasing AI regulation, it will soon become a requirement for many ML applications. In this talk, we will explore different aspects of reproducibility such as reproducibility of the dataset, data processing, ML model, its randomness and hyperparameters, code and SW environment, as well as concepts and practical tools such as data versioning, feature, metadata and artifact store, model registry and containerization that together ensure reproducibility of our experiments.

Anuncio
Anuncio

Más Contenido Relacionado

Más de DataScienceConferenc1 (20)

Anuncio

[DSC Europe 22] Reproducibility and Versioning of ML Systems - Spela Poklukar

  1. 1. Reproducibility and Versioning of ML Systems ŠPELA POKLUKAR | MACHINE LEARNING CONSULTANT
  2. 2. DSC 2022 // © COPYRIGHT 2022 ENDAVA 2 "Špela is experienced machine learning consultant with experience mostly in SW engineering services and energy sector. She has successfully lead projects in various domains such as manufacturing, finance, robotics, energy, and IT services. She is currently employed as a data discipline lead in Endava Slovenia and an active member of innovation and gender balance communities. Her background is in mathematics, philosophy, and theology.” Spela.poklukar@endava.com +386 40 545 898 Špela Poklukar MACHINE LEARNING CONSULTANT
  3. 3. DSC 2022 // © COPYRIGHT 2022 ENDAVA 3 Agenda 1. MOTIVATION 2. MODULARITY 3. VERSIONING 4. DOCUMENTATION
  4. 4. DSC 2022 // © COPYRIGHT 2022 ENDAVA 1 Motivation WHY WE NEED REPRODUCIBILITY ANYWAY
  5. 5. DSC 2022 // © COPYRIGHT 2022 ENDAVA 5 Reproducibility: Two Sides of the Same Coin REPRODUCIBILITY OF ML Research Results REPRODUCIBILITY OF ML Systems Reproducibility and Versioning of ML Systems - 1. Motivation Reproducibility of ML research results means being able to recreate a ML workflow of someone else and reach the same or similar conclusions as the original work. Reproducibility of ML system means being able to repeatedly run a ML workflow and reach the same or similar results on each run.
  6. 6. DSC 2022 // © COPYRIGHT 2022 ENDAVA 6 EVIDENCE OF SIGNIFICANCE To ensure the obtained results are accurate and significant. ABLATION To ensure that claimed gain really comes from the intended change and is not random. Why Reproducibility? COST ESTIMATION To inform potential consumers about computational complexity. Reproducibility and Versioning of ML Systems - 1. Motivation
  7. 7. DSC 2022 // © COPYRIGHT 2022 ENDAVA 7 SCALING To be able to scale the machine learning system by replicating its parts. INFERENCE To ensure selected model is the same one used for inference. FAULT TOLERANCE To reduce the risk of errors by consistently obtaining the same results. MODEL ROLLBACK To allow for model rollback in case the new model is not performing as expected. TRUST To create trust and credibility of the machine learning product. REGULATION To adhere to the increasing regulation constraints. Why Reproducibility? Reproducibility and Versioning of ML Systems - 1. Motivation
  8. 8. DSC 2022 // © COPYRIGHT 2022 ENDAVA 2 Modularity ADOPTION OF PIPELINE MENTALITY
  9. 9. DSC 2022 // © COPYRIGHT 2022 ENDAVA 9 Feature Engineering Data Preprocessing Model Training Prediction Service Model Evaluation Feature Engineering Data Preprocessing Model Training Feature Engineering Data Preprocessing Prediction Service Development Pipeline: Training Pipeline: Inference Pipeline: Reproducibility and Versioning of ML Systems - 2. Modularity
  10. 10. DSC 2022 // © COPYRIGHT 2022 ENDAVA 3 Versioning TRACKING THE CHANGES IN ML SYSTEM
  11. 11. DSC 2022 // © COPYRIGHT 2022 ENDAVA Reproducibility can be achieved by tracking and versioning every change in ML system. 11 for Training Datasets Reproducibility and Versioning of ML Systems - 3. Versioning
  12. 12. DSC 2022 // © COPYRIGHT 2022 ENDAVA 12 Environment Source Code Model Parameters Features Preprocessing System Model Dataset Changes to Track Data ‣ Dataset version ‣ Data availability timestamp ‣ Dataset split ‣ Dataset shuffling ‣ Preprocessing parameters ‣ Target variable transformation ‣ Feature computation parameters ‣ Feature selection ‣ Model type ‣ Model hyperparameters ‣ Weights initialization ‣ Evaluation parameters ‣ Dropout ‣ Components source code ‣ Pipeline definition ‣ Dependencies ‣ Environment variables ‣ Infrastructure ‣ Floating point calculation Reproducibility and Versioning of ML Systems - 3. Versioning
  13. 13. DSC 2022 // © COPYRIGHT 2022 ENDAVA 13 Reproducibility and Versioning of ML Systems - 3. Versioning
  14. 14. DSC 2022 // © COPYRIGHT 2022 ENDAVA Experiment Tracking 14 Reproducibility and Versioning of ML Systems - 3. Versioning
  15. 15. DSC 2022 // © COPYRIGHT 2022 ENDAVA Dataset Versioning 15 Reproducibility and Versioning of ML Systems - 3. Versioning
  16. 16. DSC 2022 // © COPYRIGHT 2022 ENDAVA The feature store is a central location where the features are stored and organized for the explicit purpose of being used to either train models or make predictions. Features are computed when the new data become available and stored in the feature store as opposed to being computed on the fly by training and serving services. Feature store should provide: ‣ Updated list of feature consumers ‣ Point-in-time lookup Benefits of using feature store: ‣ Consistent feature engineering for model development, training and serving ‣ Bridging the gap between data scientists and data & ML engineers ‣ Discover and reuse available feature sets, avoid having similar features with different definitions ‣ Point-in-time lookup to prevent data leakage ‣ Accelerate ML innovation ‣ Reproducibility of ML experiments ‣ Empower legal and compliance teams to ensure compliant use of data Feature Versioning – Feature Store 16 for Training Datasets Reproducibility and Versioning of ML Systems - 3. Versioning
  17. 17. DSC 2022 // © COPYRIGHT 2022 ENDAVA Model registry is a service that manages multiple model artifacts, tracks, and governs models at different stages of the ML lifecycle. The model registry provides: ‣ Centralized storage for all types of models, ‣ Collaborative unit for model lifecycle management. ‣ Basis for assessing model risks and model governance. ‣ Fast and seamless model roll-out and roll-back. Model registry should keep track of: ‣ Model name ‣ Model architecture ‣ Model hyperparameters ‣ Trained model/model weights ‣ Model metrics Model Versioning – Model Registry 17 Reproducibility and Versioning of ML Systems - 3. Versioning
  18. 18. DSC 2022 // © COPYRIGHT 2022 ENDAVA Environment Versioning – Container Registry 18 Reproducibility and Versioning of ML Systems - 3. Versioning
  19. 19. DSC 2022 // © COPYRIGHT 2022 ENDAVA Pipeline Versioning – Workflow Orchestration 19 Reproducibility and Versioning of ML Systems - 3. Versioning
  20. 20. DSC 2022 // © COPYRIGHT 2022 ENDAVA Provisioning, configuring and managing infrastructure with machine-readable definition files. Benefits: ‣ Ensures infrastructure consistency and eliminates configuration drift. ‣ Cost reduction. ‣ Increase in speed of deployments. ‣ Scalability and availability. ‣ Fosters collaboration. ‣ Standardizes deployment workflow. ‣ Error risk reduction. Infrastructure Versioning – IaC 20 Reproducibility and Versioning of ML Systems - 3. Versioning
  21. 21. DSC 2022 // © COPYRIGHT 2022 ENDAVA Metadata store is a central place that holds and connects all parameters about ML system. It may hold, for example: ‣ Data version: Reference to the dataset, md5 hash, dataset sample to know which data was used to train the model ‣ Environment configuration: Docker image ID, requirements.txt, conda.yml, Dockerfile, Makefile to know how to recreate the environment where the model was trained ‣ Code version: Git SHA of a commit or an actual snapshot of code to know what code was used to build a model ‣ Model version: Model ID, configuration of the feature preprocessing steps of the pipeline, model training, and inference to reproduce the process if needed ‣ Model performance metrics: Experiment ID, F1, accuracy, ROC on test and validation set to know how your model performs ‣ Hardware metrics: CPU, GPU, TPU, memory to see how much your model consumes during training/inference ‣ Performance visualizations: ROC curve, Confusion matrix, PR curve to understand the errors deeply ‣ Model predictions: to see the actual predictions and understand model performance beyond metrics Version Versioning – Metadata Store 21 Reproducibility and Versioning of ML Systems - 3. Versioning
  22. 22. DSC 2022 // © COPYRIGHT 2022 ENDAVA 22 EXPERIMENT TRACKING SOURCE CODE FEATURE STORE MODEL REGISTRY METADATA STORE EXPERIMENTING AND MODEL DEVELOPMENT ML PIPELINE CI/CD: BUILD, TEST, PACKAGE, DEPLOY DATA ENGINEERING CONTINUOUS MODEL TRAINING MODEL CD PREDICTION SERVICE CONTINUOUS MONITORING DATA ENGINEERING Reproducibility and Versioning of ML Systems - 3. Versioning
  23. 23. DSC 2022 // © COPYRIGHT 2022 ENDAVA 4 Documentation THE ONLY DIFFERENCE BETWEEN SCIENCE AND FOOLING AROUND IS WRITIN G IT DOWN
  24. 24. DSC 2022 // © COPYRIGHT 2022 ENDAVA 24 Reproducibility and Versioning of ML Systems - 4. Documentation
  25. 25. DSC 2022 // © COPYRIGHT 2022 ENDAVA Document as you go. Start from day 1. 25 Reproducibility and Versioning of ML Systems - 4. Documentation
  26. 26. DSC 2022 // © COPYRIGHT 2022 ENDAVA 26 MLOps – New Kid on the Block - Thank You!
  27. 27. Thank You! Q&A

×