Data Science: Good, Bad and Ugly by Irina Kukuyeva
•
3 recomendaciones•586 vistas
Abstract:- As a data scientist, I get to see a broad spectrum of the 'good', 'bad', and 'ugly' implementations of engineering and data practices. I'd be happy to share my tips and experiences with the broader community: the do's and don'ts of working with data in production, for collaboration, and for getting actionable insights.
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Similar a Data Science: Good, Bad and Ugly by Irina Kukuyeva
Similar a Data Science: Good, Bad and Ugly by Irina Kukuyeva (20)
Más de Data Con LA
Más de Data Con LA (20)
Último
Último (20)
Data Science: Good, Bad and Ugly by Irina Kukuyeva
- 1. Data Science: Good, Bad and Ugly Do's and don'ts of working with data in production, for collaboration, and for getting actionable insights Irina Kukuyeva, Ph.D. Big Data Day LA August 5, 2017
- 2. 2 Fashion: Consulting: IoT Healthcare Media & Entertainment Finance CPG Retail Video Game Publisher Online Advertising: Healthcare: Ph.D. My Background
- 3. 3 Spectrum of (Select) Production Environments Online Advertising* Revenue $1M+/year Near-real time Consulting Revenue $3M+/year ASAP Fashion Patentable tech Near-real time Finance Automation Daily Ralph Lauren orange lace top Women’s Top (18mo part-time) (11mo part-time) (9mo part-time) (3wk part-time)
- 4. What ML Model Lifecycle Really Looks Like what customer described what data looked like what DS began to build what budget could buy what pivot looked like what code got reused what got tested [1] what got pushed to prod what got documented what customer wanted 4
- 5. Agenda what customer described what data looked like what DS began to build what budget could buy what pivot looked like what code got reused what got tested [1] what got pushed to prod what got documented what customer wanted Step 1 (Appendix) Step 2: Data QA (Another Talk) Step 3: ML Development (Other Talks) Step 4: Pre-prod Step 5: Prod Step 5: Prod Step 4: Pre-prod Step 1 (Appendix) 5
- 7. 7 Pre-Production: Pay Down Tech Debt Technical Debt — Borrowing against future to tradeoff between quality of code and speed of delivery now [2], [3] • Incur debt: write code, including ML pipelines [4] • Pay down debt: extensively test and refactor pipeline end-to-end [5]
- 8. ￫ Test 1: Joel’s Test: 12 Steps to Better Code [6] • Spec? • Source control? Best practices? [7] • One-step build? Daily builds? • Bug database? Release schedule? QA? • Fix bugs before write new code? 8 Pre-Production: Pay Down Tech Debt [8]
- 9. 9 Consulting Daily stand-up Sprint planning Version control* Fix bugs first Bugs emailed/db Release on bugfix One-step build Atlassian suite Virtual machines Online Advertising* Daily stand-up Sprint planning — — Bug database — — — — Finance — — Version control* Fix bugs first — — One-step build Trello — Fashion Daily stand-up — Version control Fix bugs first — — One-step build Trello Virtual machine Test 1: Joel’s Test: 12 Steps to Better Code … in practice: Pre-Production: Pay Down Tech Debt (18mo part-time) (11mo part-time) (3wk part-time) (9mo part-time)
- 10. ￫ Test 2: ML Test Score [9], [10] • Data and feature quality • Model development • ML infrastructure • Monitoring ML 10 Pre-Production: Pay Down Tech Debt [11]
- 11. 11 ￫ Other tips — ML: • Choose simplest model, most appropriate for task and prod env • Test model against (simulated) “ground truth” or 2nd implementation [12] • Evaluate effects of floating point [12] • Model validation beyond accuracy [13] Pre-Production: Pay Down Tech Debt
- 12. 12 ￫ Other tips — Code: • Learn about production environment ￫ less code to rewrite • Set-up logging • DRY ￫ refactor • Add else to if or try/except + error • Add regression tests (tests to confirm bug fixed) • Comment liberally, especially if have to ask “why” Pre-Production: Pay Down Tech Debt [14]
- 13. 13 Consulting Minimal time to add new feature Unsuitable features excluded Online Advertising* Fashion Finance Minimal time to add new feature Privacy built-in Test 2: ML Test Score … in practice: – Data and Feature Quality – Pre-Production: Pay Down Tech Debt – Model Development – Simulated ground truth Baseline model + 2nd implementation Rolling refresh Performance overall + those most likely to click Proxy + actual metrics Human labeled data Bias correction Performance overall + main clothing categories Baseline model Bias correction Proxy + actual metrics
- 14. 14 Consulting Loosely coupled fcns Central repo for clients Regression testing One-step build, prod Online Advertising* Streaming Fashion Loosely coupled fcns Streaming One-step build, prod Test for decreasing loss Unit test classification Finance Loosely coupled fcns Streaming One-step build, prod* Reproducibility of training Pre-Production: Pay Down Tech Debt – ML Monitoring – Logging Software + package versions check Data availability check Missing titles check Offline = online (virtual) env Missing data check Test 2: ML Test Score … in practice (cont’d): – ML Infrastructure –
- 16. 16 Production: Deploy Code and Monitor Performance ￫ One-button push to prod branch/repo ￫ Model Rollout ￫ Monitoring [15]
- 18. 18 ￫ Documentation + QA gets cut first ￫ Debugging, debugging, debugging ￫ code is never perfect ￫ Bugfix vs. Feature ￫ Post-mortem ￫ Use the product ￫ Support and Training Post-Production: Keep Code Up and Running [16]
- 19. 19 Post-Production: Align Business and Team Goals ￫ Team targets: deadlines and revenue goals ￫ Team competitions [17]
- 20. 20 Key Takeaways ￫ Communication, version control, logging, documentation, debugging ￫ Automatically evaluate all components of ML pipeline ￫ High model accuracy is not always the answer ￫ Scope down, then scale up [18]
- 21. 21 You Did It! Code is in prod! Celebrate!
- 22. 22 You Did It! Code is in prod! Celebrate! … But not too hard. Tomorrow you start on v2.
- 23. Questions? 23 PS: We’re Hiring! https://www.dia.com/careers
- 24. [ 1] http://www.projectcartoon.com/cartoon/1 [ 2] https://research.google.com/pubs/pub43146.html [ 3] https://www.linkedin.com/pulse/when-your-tech-debt-comes-due-kevin-scott [ 4] https://www.sec.gov/news/press-release/2013-222 [ 5] http://dilbert.com/strip/2017-01-03 [ 6] https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-steps-to-better-code/ [ 7] https://solidgeargroup.com/wp-content/uploads/2016/07/tower_cheatsheet_white_EN_0.pdf [ 8] http://geek-and-poke.com/geekandpoke/2014/2/23/dev-cycle-friday-evening-edition [ 9] https://www.eecs.tufts.edu/~dsculley/papers/ml_test_score.pdf [10] http://martin.zinkevich.org/rules_of_ml/rules_of_ml.pdf [11] https://www.slideshare.net/Tech_InMobi/building-machine-learning-pipelines [12] https://www.researchgate.net/publication/262569275_Testing_Scientific_Software_A_Systematic_Literature_Review [13] https://classeval.wordpress.com/introduction/basic-evaluation-measures/ [14] http://geek-and-poke.com/geekandpoke/2015/10/18/why-logging-is-so-important [15] https://www.pinterest.com/explore/programming-humor/ [16] https://www.devrant.io/search?term=debugging [17] https://marketoonist.com/2015/03/hackathons.html [18] https://s-media-cache-ak0.pinimg.com/originals/9c/25/08/9c25082f5c4d3477124356e45673d426.png [19] https://www.pinterest.com/pin/177258935306351629/ 24 References
- 25. 25 Appendix: Establish Business Use Case ￫ Kick-off meeting with stakeholders: • Discuss use case, motivation and scope • Find out about format of deliverable and how it will be used by team • Brainstorm and discuss potential solutions • Iron-out deadlines, checkpoints and on-going support structure • Ask about prod env (if appropriate) • Scope down, then scale up • Close meeting with recap of action items Key Takeaways: communication + clear expectations [19]