Publicidad
Check these out next
The Three Body Problem of Data Science
6 de Apr de 2019•0 recomendaciones•1,913 vistas
1 recomendaciones
Sé el primero en que te guste
ver más
vistas
Total de vistas
0
En Slideshare
0
De embebidos
0
Número de embebidos
0
Descargar para leer sin conexión
Denunciar
Datos y análisis
How Science, Engineering and Product interact to succeed or fail in Data Science.
Dima KaramshukSeguir
data scientistPublicidad
Publicidad
Publicidad
Recomendados
Uncertainty Quantification in Complex Physical Systems. (An Inroduction)Ogechi Onuoha
Anatomy of a data science projectAdam Sroka
Scientific Revenue USF 2016 talkScientificRevenue
Causal Inference, Reinforcement Learning, and Continuous OptimizationScientificRevenue
1645 track 2 ard_using our laptopRising Media, Inc.
How to Use Artificial Intelligence by Microsoft Product ManagerProduct School
The Three Body Problem of Data Science
- TheThree Body Problem of DataScience DimaKaramshuk @Skyscanner@SkyscannerEng
- Three Body Problem
- Three Body Problem • One is unable to predict subsequent motions of three bodies acting on each other, from their initial velocities and positions
- Three Body Problem Science Engineering Product Buddies
- Science Engineering Product • What we invest into Actions and Reactions
- Science Engineering Product • What we invest into • How we structure and integrate the team Actions and Reactions
- Science Engineering Product • What we invest into • How we structure and integrate the team • What deliverables we expect Actions and Reactions
- Types of Problems Existing product, existing solution Existing product, new solution New product, existing solution New product, new solution Science Engineering Product
- Optimizing Existing Product Example: Case N1. Replacing heuristics in existing product with ML
- Optimizing Existing Product Good thing: • well defined and controlled environment Bad thing: • integration with existing infrastructure Case N1. Replacing heuristics in existing product with ML
- Offline Evaluation Formulating Machine Learning Problem Product Lifecycle
- Product Lifecycle Data Insights Offline Evaluation Production Prototype Online Evaluation Production Formulating Machine Learning Problem ModelVersioning Data Acquisition only 20% of work
- Product Lifecycle Data Insights Offline Evaluation Production Prototype Online Evaluation Production Formulating Machine Learning Problem ModelVersioning Data Acquisition
- Logging what users click on first page results
- Logging what users click on what users see but do not click on what users do not see first page results
- Product Lifecycle Data Insights Offline Evaluation Production Prototype Online Evaluation Production Formulating Machine Learning Problem ModelVersioning Data Acquisition
- Prototype – model trained in a Jupyter Notebook is a good starting point, but…
- Production Pipeline Data Querying AWS Athena Data Archive AWS S3 Data Collection Current Model Model Training scikit-learn Training Data 7 recent days Validation Data 5% of the last day Model Validation Passed? Skyscanner Traffic Pre-processing Experiments with Challenger Model 5% 5% 90% Training Component (AWS CF + AWS Data Pipeline) Report Failure Update ModelApache Kafka Serving Component ECMLPKDD’2018, https://arxiv.org/pdf/1812.01735.pdf
- Optimizing Existing Product Good thing: • well defined and controlled environment Bad thing: • integration with existing infrastructure Science Engineering Product Case N1. Replacing heuristics in existing product with ML
- Role != Person
- Iterating over Existing Algorithm Case N2. Iterating over existing ML algorithm in existing product
- Iterating over Existing Algorithm Good thing: • return on infrastructure investments Bad thing: • possibly limited impact Science Engineering Product Case N2. Iterating over existing ML algorithm in existing product
- Iterating over Existing Algorithm Science Engineering Product Case N2. Iterating over existing ML algorithm in existing product • How to justify further investment? • Should it become a BAU? • Who should own it?
- Building a New Product Case N3. Building a first version of a new data product
- Building a New Product Good thing: • less dependencies Bad thing: • high level of uncertainty Science Engineering Product Case N3. Building a first version of a new data product
- Managing Uncertainty Levels of uncertainty: – Is the position right? – Is the user flow right? – Is the message right? – Is the design right? – Is the algorithm right? – What is the baseline? – etc. etc.
- Levels of uncertainty: – Is the position right? – Is the user flow right? – Is the message right? – Is the design right? – Is the algorithm right? – What is the baseline? – etc. etc. Managing Uncertainty +30% engagement
- Managing Uncertainty Skyscanner Backpack Levels of uncertainty: – Is the position right? – Is the user flow right? – Is the message right? – Is the design right? – Is the algorithm right? – What is the baseline? – etc. etc.
- Managing Uncertainty Levels of uncertainty: – Is the position right? – Is the user flow right? – Is the message right? – Is the design right? – Is the algorithm right? – What is the baseline? – etc. etc. Start really simple
- Managing Uncertainty Levels of uncertainty: – Is the position right? – Is the user flow right? – Is the message right? – Is the design right? – Is the algorithm right? – What is the baseline? – etc. etc. Study previous experience
- Case N4: NewScience in New Products Existing product, existing solution Existing product, new solution New product, existing solution New product, new solution
- Case N4: NewScience in New Products Existing product, existing solution Existing product, new solution New product, existing solution New product, new solution
- Join our team! @karamshuk @SkyscannerEng
Publicidad