Full blog post: https://medium.com/@neal_lathia/machine-learning-for-product-managers-ba9cf8724e57#.k7oexyav6

1. Machine Learning for Product
Managers
Skyscanner PO Workshop

2. What is Machine Learning?
A Machine Learning (ML) system learns a program from
data.

3. Machine Learning
 Given these examples: {data set}
 And this error metric: M
 Learn a function that minimizes M on {data set}

4. Supervised vs. Unsupervised Learning
 Supervised: The training data contains the “right
answer” for each example.
 Unsupervised: The training data does not have the
“right answer” in each example.

5. Unsupervised Learning Example
The data does not tell us whether something is “correct” or not.

6. Unsupervised Learning in Products
The data does not tell us whether something is “correct” or not.

7. Supervised Learning Example
The algorithm can be right or wrong, and the data has examples of each.
1, 3, 4, 2 =
5, 2, 3, 1 =
2, 4, 4, 2 =
7, 1, 3, 5 =
6, 8, 2, 4 =

8. Supervised Learning Example
The algorithm can be right or wrong, and the data has examples of each.

9. Supervised vs. Unsupervised
Two products that look similar but (probably) have different types of ML under the hood.

10. Supervised vs. Unsupervised
Two products that look similar but (probably) have different types of ML under the hood.

11. Most Common Use Cases (Technical Terms)
1. Helping users find the right thing (Ranking)
2. Giving users what they may be interested in (Recommendation)
3. Figuring out what kind of thing something is (Classification)
4. Predicting a numerical value of a thing (Regression)
5. Putting similar things together (Clustering)
6. Finding uncommon things (Anomaly Detection)
Note: many of these could be packaged as “recommendation products”

12. Building an ML Product
There are many different issues: getting started (cold-start),
understanding what is happening (intuition fails in high
dimensions).

13. Building an ML Product
There are domain-specific tasks & product-ML fit tasks. This
session focuses on the latter.

14. Engineering + Data Science
1. Discovering & analysing data to inform what we could do
2. Building data pipelines
3. Feature engineering
4. Selecting algorithms
5. Optimisation & avoiding overfitting
6. Running offline evaluations
7. Putting ML algorithms into production

15. Beyond the ML in the product
1. Does the ML fit the product goal?
2. How does the product behave ”around” the ML?
3. What is the baseline, and how will this product improve?
4. How quickly should this product change?
5. What interactions, actions, & control do users have?
6. How could the product fail catastrophically?

16. ML Objective vs. Product Goal

17. Behaviour
Around the
Machine
Learning

18. Baseline
Is your current approach a “complex heuristic?”

19. Measurable Improvement
Performance is only ever relative
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Random Forest
Product 1
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Random Forest Baseline
Product 1

20. Measurable Improvement
Performance is only ever relative
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Product 1 Product 2 Product 3 Product 4
Relative Performance
Machine Learning Baseline

21. How quickly should the product change?
This will affect how you design the system.

22. Interactions, Actions, Control
Can the user give the ML the “correct answer?” Is the ML helping or
hindering what they want to do?

23. Catastrophic Failure
How is the ML being used/applied in context?

24. Catastrophic Failure
How is the ML being used/applied in context?

25. Summary: Beyond the ML in the product
1. Does the ML fit the product goal?
2. How does the product behave ”around” the ML?
3. What is the baseline, and how will this product improve?
4. How quickly should this product change?
5. What interactions, actions, & control do users have?
6. How could the product fail catastrophically?

26. Next – Discussion
1. How do we decide that a feature would benefit from any ML?
2. Are we logging the right data?
3. What other issues/blockers have you encountered?