Machine Learning is transforming every industry with innovative techniques receiving deserved attention. But turning innovation into value requires integrating into practical technology products, often with the leadership of product managers. We'll talk about how to help your friendly neighborhood Product Owner: identify where ML can make a difference, develop metrics to validate and refine it, identify data to feed it, prioritize work to develop it, and structure teams to deliver it in a satisfying way.
2024: Domino Containers - The Next Step. News from the Domino Container commu...
How to train your product owner
1. How to Train
Your Product Owner
David Murgatroyd ( @dmurga)
MassTLC ML Dev Day
January 24, 2018
(please don’t sue me for copyright infringement, DreamWorks!)
2.
3. Agile & Machine Learning
David Murgatroyd (@dmurga)
(ML Chapter Lead in Quest)
Your Product Owner
4. Agile & Machine Learning
David Murgatroyd (@dmurga)
(ML Chapter Lead in Quest)
Your Product Owner
Your Product
6. @dmurga
What do Product Owners Do?
Lead team in:
‣ establishing vision/hypotheses for
what product should become
‣ prioritizing work to get there
7. @dmurga
What do Product Owners Do?
Lead team in:
‣ establishing vision/hypotheses for
what product should become
‣ prioritizing work to get there
Your Product
10. @dmurga
Outside In … Now to Then
Machine Learning
Machine Learning
Your Product
Your Future Product
11. @dmurga
Outside In … Now to Then
Machine Learning
Problem Metrics Data Models Prioritizing OrganizingDesign
Machine Learning
Your Product
Your Future Product
15. @dmurga
Problem Metrics Data Models
Picking a Problem: Tweak It
Design
‣ What’s the business goal of your
product?
‣ What fuzzy decision does it make
that impacts that goal?
Your Product
16. @dmurga
‣ Perception: a person can do it in
less than a second.
‣ Prediction: done over and over
‣ Personalization: similar need but
met in different ways
Problem Metrics Data Models
Picking a Problem: Think It
Design
Your Product
?
19. @dmurga
1. The role of ‘why’ of ML output?
2. Coping with errors?
3. Designing for varied output?
4. New user behaviors that ML
enables?
Problem Metrics Data Models
Designing with ML
Design
Your Product
20. @dmurga
Design
‣ Trading off why vs. right
‣ Granularity of ‘why’
‣ Creepy vs. personal
‣ First step: designer gives concrete
example on concrete data
Problem Metrics Data Models
Design: the role of ‘why’
Your Product
21. @dmurga
‣ What quality is needed to user
maintain trust or delight them?
‣ Rank multiple outputs?
‣ Provide fallback behavior?
‣ Explicit feedback mechanisms?
Problem Metrics Data Models
Design: coping with errors
Design
Your Product
22. @dmurga
‣ Get realistic output samples from a
variety of users (e.g., synthetic,
persona)
‣ Watch for subtle assumptions
Problem Metrics Data Models
Design: coping with variety
Design
Your Product
23. @dmurga
‣ Users might anthropomorphize ML
Products
‣ Users might express more of
themselves or test the limits of the
systems
‣ Avoid “user bubbles” by encouraging
discovery and crafting metrics
Problem Metrics Data Models
Design: new user behavior
Design
Your Product
24. @dmurga
‣ Alignment to business value
‣ Effort to measure
‣ Useful for team’s decisions
‣ Useful for model’s training
Problem Metrics Data Models
Metrics: Properties
Design
Your Product
Machine Learning
25. @dmurga
Problem Metrics Data Models
Metrics: Matrix of Metrics
Design
Where /
What
Heuristic Modeled
Online Online Heuristic Online
Modeled
Offline Offline Heuristic Offline
Modeled
Your Product
Machine Learning
26. @dmurga
Problem Metrics Data Models
Metrics: Online vs Offline
Design
Online More aligned with business value
Offline Generally less effort to measure
Your Product
Machine Learning
27. @dmurga
Problem Metrics Data Models
Metrics: Heuristic vs Modeled
Design
Heuristic Modeled
More useful for
team’s decisions
interpretable
More useful for
training the model
directly
Your Product
Machine Learning
28. @dmurga
Problem Metrics Data Models
Metrics: Matrix of Metrics
Design
Where /
What
Heuristic Modeled
Online Online Heuristic Online
Modeled
Offline Offline Heuristic Offline
Modeled
Your Product
Machine Learning
29. @dmurga
‣ Just look at some data before/after
‣ Alignment to business value: OK
‣ Effort to measure: OK (one-off)
‣ Useful for team’s decisions: OK
‣ Useful for model’s training: Bad
Problem Metrics Data Models
Metrics: … vs Subjective
Design
Your Product
Machine Learning
30. @dmurga
1. What data do you need?
2. Where can you get it?
3. What biases does it carry?
Problem Metrics Data Models
Data
Design
Your Product
31. @dmurga
‣ Raw input data
‣ Metadata / reference
‣ Has metadata that is a source for
measurement
Problem Metrics Data Models
What data do you need?
Design
Your Product
32. @dmurga
‣ Data is the new Wireframe
‣ Product Owner provides example
inputs / outputs
‣ Use these to also vet metrics
Problem Metrics Data Models
Specifying with Data
Design
Your Product
Machine Learning
33. @dmurga
‣ Logging
‣ Proxies and other 3rd Parties
‣ Annotation
‣ Watch out for drift in the data set
‣ Take advantage of PO’s domain
knowledge!
Problem Metrics Data Models
Where can you get it?
Design
Your Product
34. @dmurga
‣ Is it skewed?
‣ Is it tainted?
‣ Is it likely to stereotype?
Problem Metrics Data Models
What biases does it carry?
Design
Your Product
35. @dmurga
1. Teach tasks rather than techniques
2. Teach trade-offs rather than tools
3. Start simple
Problem Metrics Data Models
Models
Design
Your Product
Machine Learning
36. @dmurga
‣ Classification
‣ Clustering
‣ Regression
‣ (and Embeddings if you must :- )
Problem Metrics Data Models
Models: Tasks
Design
Your Product
Machine Learning
37. @dmurga
‣ Simplicity
‣ Interpretability
‣ Confidence
‣ Accuracy
‣ Adaptability
‣ Speed
‣ Space
‣ Scale
Problem Metrics Data Models
Models: Trade-offs
Design
Your Product
Machine Learning
38. @dmurga
‣ Rules before baseline models
‣ Baselines before adapted models
‣ Adapted models before end-to-end
models
Problem Metrics Data Models
Models: Err toward Simplicity
Design
Your Product
Machine Learning
40. @dmurga
Outside In … Now to Then
Machine Learning
Problem Metrics Data Models Prioritizing OrganizingDesign
Machine Learning
Your Product
Your Future Product
47. @dmurga
Prioritizing: Prod Lifecycle
Prioritizing Organizing
Your Backlog
Stage Characteristics
Exploration by hand examples/rules
Pre-MVP (0.1%
/ early Beta)
measurable & inspectable
MVP (1%, Beta) accurate, not slow, &
documented
v1 (100% / GA) simple & fast
Post-v1 handle new domains
48. @dmurga
Prioritizing: Goals (OKRs)
Prioritizing Organizing
Your Backlog
Stage OKRs
Exploration amount of analysis
Pre-MVP (0.1%
/ early Beta)
having metrics, amount of
experiments
MVP (1%, Beta) moving core metrics,
amount of experiments
v1 (100% / GA) moving all metrics
Post-v1 moving all metrics on new
data
52. @dmurga
Organizing: Team Structure
Organizing
‣ Aligned to Product/Org Maturity
‣ Exploration: Centralized Team
‣ Pre-MVP: Cross-functional
‣ MVP to v1: Cross-functional with
separate work stream
Your Backlog
Prioritizing
53. @dmurga
Organizing: Team Structure
Prioritizing Organizing
‣ Aligned to Product/Org Maturity
‣ Exploration: Centralized Team
‣ Pre-MVP: Cross-functional
‣ MVP to v1: Cross-functional with
separate work stream
‣ Post v1: Dedicated Sibling Team
Your Backlog
54. @dmurga
‣ Applied ML Eng
‣ ML Tool Eng
‣ Core Researcher
Organizing: Roles
Prioritizing Organizing
55. @dmurga
Outside In … Now to Then
Machine Learning
Problem Metrics Data Models Prioritizing OrganizingDesign
Machine Learning
Your Product
Your Future Product
56. Thanks! Questions?
David Murgatroyd (@dmurga)
Suggestions:
What about different kinds of testing?
What are common features ML-based products have?
More on identifying metrics?
Machine Learning vs. Data Science?
Hiring in Boston,
NYC, London,
and Stockholm!