The Three Body Problem of Data Science
Denunciar
Dima KaramshukSeguir
6 de Apr de 2019•0 recomendaciones•1,913 vistas
1 recomendaciones
Sé el primero en que te guste
ver más
vistas
Total de vistas
0
En Slideshare
0
De embebidos
0
Número de embebidos
0
Check these out next
The Three Body Problem of Data Science
6 de Apr de 2019•0 recomendaciones•1,913 vistas
1 recomendaciones
Sé el primero en que te guste
ver más
vistas
Total de vistas
0
En Slideshare
0
De embebidos
0
Número de embebidos
0
Descargar para leer sin conexión
Denunciar
Datos y análisis
How Science, Engineering and Product interact to succeed or fail in Data Science.
Dima KaramshukSeguir
Recomendados
Uncertainty Quantification in Complex Physical Systems. (An Inroduction)Ogechi Onuoha
Anatomy of a data science projectAdam Sroka
Scientific Revenue USF 2016 talkScientificRevenue
Causal Inference, Reinforcement Learning, and Continuous OptimizationScientificRevenue
1645 track 2 ard_using our laptopRising Media, Inc.
How to Use Artificial Intelligence by Microsoft Product ManagerProduct School
The Three Body Problem of Data Science
- TheThree Body Problem of DataScience DimaKaramshuk @Skyscanner@SkyscannerEng
- Three Body Problem
- Three Body Problem • One is unable to predict subsequent motions of three bodies acting on each other, from their initial velocities and positions
- Three Body Problem Science Engineering Product Buddies
- Science Engineering Product • What we invest into Actions and Reactions
- Science Engineering Product • What we invest into • How we structure and integrate the team Actions and Reactions
- Science Engineering Product • What we invest into • How we structure and integrate the team • What deliverables we expect Actions and Reactions
- Types of Problems Existing product, existing solution Existing product, new solution New product, existing solution New product, new solution Science Engineering Product
- Optimizing Existing Product Example: Case N1. Replacing heuristics in existing product with ML
- Optimizing Existing Product Good thing: • well defined and controlled environment Bad thing: • integration with existing infrastructure Case N1. Replacing heuristics in existing product with ML
- Offline Evaluation Formulating Machine Learning Problem Product Lifecycle
- Product Lifecycle Data Insights Offline Evaluation Production Prototype Online Evaluation Production Formulating Machine Learning Problem ModelVersioning Data Acquisition only 20% of work
- Product Lifecycle Data Insights Offline Evaluation Production Prototype Online Evaluation Production Formulating Machine Learning Problem ModelVersioning Data Acquisition
- Logging what users click on first page results
- Logging what users click on what users see but do not click on what users do not see first page results
- Product Lifecycle Data Insights Offline Evaluation Production Prototype Online Evaluation Production Formulating Machine Learning Problem ModelVersioning Data Acquisition
- Prototype – model trained in a Jupyter Notebook is a good starting point, but…
- Production Pipeline Data Querying AWS Athena Data Archive AWS S3 Data Collection Current Model Model Training scikit-learn Training Data 7 recent days Validation Data 5% of the last day Model Validation Passed? Skyscanner Traffic Pre-processing Experiments with Challenger Model 5% 5% 90% Training Component (AWS CF + AWS Data Pipeline) Report Failure Update ModelApache Kafka Serving Component ECMLPKDD’2018, https://arxiv.org/pdf/1812.01735.pdf
- Optimizing Existing Product Good thing: • well defined and controlled environment Bad thing: • integration with existing infrastructure Science Engineering Product Case N1. Replacing heuristics in existing product with ML
- Role != Person
- Iterating over Existing Algorithm Case N2. Iterating over existing ML algorithm in existing product
- Iterating over Existing Algorithm Good thing: • return on infrastructure investments Bad thing: • possibly limited impact Science Engineering Product Case N2. Iterating over existing ML algorithm in existing product
- Iterating over Existing Algorithm Science Engineering Product Case N2. Iterating over existing ML algorithm in existing product • How to justify further investment? • Should it become a BAU? • Who should own it?
- Building a New Product Case N3. Building a first version of a new data product
- Building a New Product Good thing: • less dependencies Bad thing: • high level of uncertainty Science Engineering Product Case N3. Building a first version of a new data product
- Managing Uncertainty Levels of uncertainty: – Is the position right? – Is the user flow right? – Is the message right? – Is the design right? – Is the algorithm right? – What is the baseline? – etc. etc.
- Levels of uncertainty: – Is the position right? – Is the user flow right? – Is the message right? – Is the design right? – Is the algorithm right? – What is the baseline? – etc. etc. Managing Uncertainty +30% engagement
- Managing Uncertainty Skyscanner Backpack Levels of uncertainty: – Is the position right? – Is the user flow right? – Is the message right? – Is the design right? – Is the algorithm right? – What is the baseline? – etc. etc.
- Managing Uncertainty Levels of uncertainty: – Is the position right? – Is the user flow right? – Is the message right? – Is the design right? – Is the algorithm right? – What is the baseline? – etc. etc. Start really simple
- Managing Uncertainty Levels of uncertainty: – Is the position right? – Is the user flow right? – Is the message right? – Is the design right? – Is the algorithm right? – What is the baseline? – etc. etc. Study previous experience
- Case N4: NewScience in New Products Existing product, existing solution Existing product, new solution New product, existing solution New product, new solution
- Case N4: NewScience in New Products Existing product, existing solution Existing product, new solution New product, existing solution New product, new solution
- Join our team! @karamshuk @SkyscannerEng