Donal McMahon, Director of Data Science at Indeed, presented how to transition from data-driven to science-driven product development. You’ll make better business decisions. It’s provable!
39. What did we do?
Observation Question Hypothesis Experiment Analysis Conclusion
Nothing
40. Observation Question Hypothesis Experiment Analysis Conclusion
Why was it wrong?
1
Didn’t establish baseline for job seeker experience, or measures
2
When we failed, we had no knowledge backlog for future work
41. Observation Question Hypothesis Experiment Analysis Conclusion
How can you do it better?
Nano
Study real job seeker sessions
Micro
Partner with experts (UX) to gather qualitative data
Macro
Large scale data analysis and observation via experimentation
42. How can you do it better?
Nano: study real job seeker sessions
Query 1
Click on
Job A
Click on
Job C
Query 2
Click on
Job D
Apply
Observation Question Hypothesis Experiment Analysis Conclusion
43. Not only is the universe stranger
than we imagine, it is stranger
than we can imagine.
Sir Arthur Eddington
44. How can you do it better? A shameless plug
Micro: partner with experts (UX) to gather qualitative data
Observation Question Hypothesis Experiment Analysis Conclusion
medium.com/indeed-data-science
45. How can you do it better?
Micro: partner with experts (UX) to gather qualitative data
Observation Question Hypothesis Experiment Analysis Conclusion
1
Real-life observation
2
Interviews
3
Content analysis (surveys)
46. How can you do it better?
Macro: large scale data analysis and observation via experimentation
Common Question
What’s a worthwhile/launchable metric trade-off?
Observation Question Hypothesis Experiment Analysis Conclusion
47. How can you do it better?
Reality
You’re making trade-offs implicitly already
Observation Question Hypothesis Experiment Analysis Conclusion
Macro: large scale data analysis and observation via experimentation
48. How can you do it better?
Learn your implicit local trade-off function
Run multiple simple perturbation experiments, all the time
Observation Question Hypothesis Experiment Analysis Conclusion
Macro: large scale data analysis and observation via experimentation
52. Learn your current implicit trade-offs via experimentation
Applies
JobAlert
Signups
Expt 1: bold Apply with
your Indeed Resume
53. Learn your current implicit trade-offs via experimentation
Applies
JobAlert
Signups
Expt 2: add pixel
whitespace to
JobAlert UI
Expt 1: bold Apply with
your Indeed Resume
54. Learn your current implicit trade-offs via experimentation
Applies
JobAlert
Signups
55. Compare your current state to all pareto efficient alternatives
Applies
JobAlert
Signups
56. For each pareto efficient alternative you have a tradeoff
Applies
JobAlert
Sign-ups
ΔApplies
ΔJobAlerts
57. How can you do it better?
Implicit tradeoff
Each JobAlert sign-up is worth 1.7 Applies
Observation Question Hypothesis Experiment Analysis Conclusion
Macro: large scale data analysis and observation via experimentation
59. What did we do?
Observation Question Hypothesis Experiment Analysis Conclusion
Nothing
60. Why was it wrong?
1
We never prioritized the most important question(s)
2
By bundling questions, we couldn’t answer any, learn and improve
Observation Question Hypothesis Experiment Analysis Conclusion
62. Research Question
Potential
Impact
Complexity
Time To
Learn
What are good measures for job seeker experience? ? ? ?
How can we help job seeker navigate to their desired
job more quickly?
? ? ?
How can we clearly denote sponsored content? ? ? ?
… ... ... ...
How can you do it better?
Observation Question Hypothesis Experiment Analysis Conclusion
66. Why was it wrong?
1
Hypothesis was ill-defined and vague
2
No established metrics
3
No clear success/failure criteria
Observation Question Hypothesis Experiment Analysis Conclusion
67. How can you do it better?
1
Determine one or more hypothesis
“Does extra whitespace between job cards help job seekers to navigate quicker.”
2
Agree on the data, metrics and acceptable trade-offs up front
Suggested metrics: (i) time to click, (ii) click rate, (iii) time to hire
Observation Question Hypothesis Experiment Analysis Conclusion
68. Important Question #1
How many metrics?
Observation Question Hypothesis Experiment Analysis Conclusion
75. How many metrics?
We need a low-dimensional representation
that preserves almost all of the signal
Observation Question Hypothesis Experiment Analysis Conclusion
76. How many metrics?
Singular value decomposition (SVD)
Observation Question Hypothesis Experiment Analysis Conclusion
90. It can be easy to miss bias
Observation Question Hypothesis Experiment Analysis Conclusion
91. Observation Question Hypothesis Experiment Analysis Conclusion
Hidden bias in our example
Estimate “time to hire” for job seekers
92. Job seeker First action Still active Hired
1 01/01/2016 Yes No
2 01/22/2016 No 01/25/2016
3 02/04/2016 No 02/23/2016
4 02/17/2016 No No
... ... ... ...
... ... ... ...
n 04/23/2016 Yes No
Observation Question Hypothesis Experiment Analysis Conclusion
93. Observation Question Hypothesis Experiment Analysis Conclusion
Initial Metric Proposal
Average time to hire for job seekers who were hired
95. Job seeker First action Still active Hired
1 01/01/2016 Yes No
2 01/22/2016 No 01/25/2016
3 02/04/2016 No 02/23/2016
4 02/17/2016 No No
... ... ... ...
... ... ... ...
n 04/23/2016 Yes No
Observation Question Hypothesis Experiment Analysis Conclusion
96. Job seeker First action Still active Hired
1 01/01/2016 Yes No
2 01/22/2016 No 01/25/2016
3 02/04/2016 No 02/23/2016
4 02/17/2016 No No
... ... ... ...
... ... ... ...
n 04/23/2016 Yes No
Observation Question Hypothesis Experiment Analysis Conclusion
97. Job seeker First action Still active Hired
1 01/01/2016 Yes No
2 01/22/2016 No 01/25/2016
3 02/04/2016 No 02/23/2016
4 02/17/2016 No No
... ... ... ...
... ... ... ...
n 04/23/2016 Yes No
Observation Question Hypothesis Experiment Analysis Conclusion
98. Observation Question Hypothesis Experiment Analysis Conclusion
Solution
Estimate typical time to hire using Kaplan-Meier Estimate
104. Variance is fundamental
for valid statistical inference
Observation Question Hypothesis Experiment Analysis Conclusion
105. Science assumes “innocent until proven guilty”
We often term this our null hypothesis (H0)
Observation Question Hypothesis Experiment Analysis Conclusion
106. Proof required beyond reasonable doubt
In order to reject the null hypothesis
Observation Question Hypothesis Experiment Analysis Conclusion
107. Variance is your estimate of uncertainty, i.e. doubt
Observation Question Hypothesis Experiment Analysis Conclusion
108. Observation Question Hypothesis Experiment Analysis Conclusion
Note
We often choose the
Minimum Variance Unbiased Estimator (MVUE)
109. Not Always MVUE
Occasionally you
might trade bias for
variance
e.g. machine learning
Low variance High variance
HighbiasLowbias
Observation Question Hypothesis Experiment Analysis Conclusion
117. There is no catch-all mathematical formula
to measure and account for system complexity
Observation Question Hypothesis Experiment Analysis Conclusion
118. But that doesn’t mean you shouldn’t try to estimate
it and factor it into decisions
Observation Question Hypothesis Experiment Analysis Conclusion
123. You predict a winner for each game and awarded points if correct
16
9
5
4
✅
✅
✅
1
9
5
4
̶ my prediction ̶ actual result
124. If you predict an upset early, success/failure compounds
16
9
5
4
✅
✅
✅
1
9
5
4
̶ my prediction ̶ actual result
9 1
4 ✅ 4
4 1
125. ● Downstream compounded loss
● Number of bracket participants
● Points awarded at each stage
Observation Question Hypothesis Experiment Analysis Conclusion
System
complexity
factors
126. How to win your NCAA pool
Simulate the downstream effect of all potential decisions
Check whether it increases/decreases your win probability
Observation Question Hypothesis Experiment Analysis Conclusion
127. Reminder - How can you do it better?
1
Determine one or more hypothesis
“Does extra whitespace between job cards help job seekers to navigate quicker.”
2
Agree on the data, metrics and acceptable trade-offs up front
Metrics: (i) time to click, (ii) click rate, (iii) time to hire
Observation Question Hypothesis Experiment Analysis Conclusion
129. What did we do?
Ran a single treatment experiment where we
simultaneously changed four components
Observation Question Hypothesis Experiment Analysis Conclusion
137. What did we do?
1
Cobbled data together from different sources
2
Defined different metrics
3
Invested a lot of time analysing tests
Observation Question Hypothesis Experiment Analysis Conclusion
138. To consult the statistician after an
experiment is finished is often merely
to ask her to conduct a post mortem
examination. She can perhaps say
what the experiment died of.
R.A. Fisher
139. Why was it wrong?
Observation Question Hypothesis Experiment Analysis Conclusion
Opinion-driven, time sink, unsatisfying for all involved
140. How can you do it better?
Observation Question Hypothesis Experiment Analysis Conclusion
With correct setup, this should be trivial
141. Existing metric New metric
Existing product
New product
Observation Question Hypothesis Experiment Analysis Conclusion
142. Existing metric New metric
Existing product Uninteresting
New product
Observation Question Hypothesis Experiment Analysis Conclusion
143. Existing metric New metric
Existing product Uninteresting Metric Innovation
New product
Observation Question Hypothesis Experiment Analysis Conclusion
144. Existing metric New metric
Existing product Uninteresting Metric Innovation
New product Product Innovation
Observation Question Hypothesis Experiment Analysis Conclusion
148. What did we do?
Observation Question Hypothesis Experiment Analysis Conclusion
Drew two different conclusions
149. Why was it wrong?
1
Didn’t learn anything
2
Lost team trust
Observation Question Hypothesis Experiment Analysis Conclusion
150. How can you do it better?
Observation Question Hypothesis Experiment Analysis Conclusion
Should follow directly from analysis
151. The Goldilocks syndrome
Observation Question Hypothesis Experiment Analysis Conclusion
A/B test
(-1%, 1%] (1%, 5%] (5%, ∞](-5%, -1%][-∞, -5%]Outcome
Conclusion too cold too cold too cold
Just right,
declare victory
too hot
153. The Complete Scientific Method
Observation Question Hypothesis Experiment Analysis Conclusion
nano,
micro,
macro
prioritize,
implicit
trade-offs
bias &
variance,
3 metrics
full
factorial
design
trivial,
no data
innovation
Goldilocks
syndrome,
repeatability
154. Observation Question Hypothesis Experiment Analysis Conclusion
nano,
micro,
macro
prioritize,
implicit
trade-offs
bias &
variance,
3 metrics
full
factorial
design
trivial,
no data
innovation
Goldilocks
syndrome,
repeatability
155. Data-driven can be disorientating in a world of abundant data
Be science-driven, i.e. use the scientific method to add necessary structure
Invest in the observation, question and hypothesis stages
Parting Thoughts