5. Let’s talk about how to build a successful
experimentation program.
6. "To invent you have to
experiment, and if you
know in advance that it's
going to work, it's not an
experiment.”
- Jeff Bezos
Image source: huffpost.com
7. - Limited resources
- Few experiments per quarter
- No clear process
- Very little revenue contribution
coming from experiments
Before
8. - Limited resources
- Few experiments per quarter
- No clear process
- Very little revenue contribution
coming from experiments
Before
- 4 agile teams
- 40 - 50 experiments per quarter
- Established process
- Significant revenue contribution
Today
22. Pathway Analysis
Traffic Channel Landing Page Discovery Conversion
Note: Does not represent real Udemy numbers
30% Direct
40% SEO
20% Paid Acq
23. Pathway Analysis
Traffic Channel Landing Page Discovery Conversion
Note: Does not represent real Udemy numbers
30% Direct
40% SEO
20% Paid Acq
70% Product Page
30% Home Page
24. Pathway Analysis
Traffic Channel Landing Page Discovery Conversion
Note: Does not represent real Udemy numbers
30% Direct
40% SEO
20% Paid Acq
70% Product Page
30% Home Page
20% Search
10% Browse
2% Success Page
25. Pathway Analysis
Traffic Channel Landing Page Discovery Conversion
30% Direct
40% SEO
20% Paid Acq
70% Product Page
30% Home Page
1% Success Page20% Search
10% Browse
2% Success Page
Note: Does not represent real Udemy numbers
26. How Do People Use Your Product: Funnel Analysis
Understand the leakiest parts of your funnel
Product Landing Page
1,000,000
Watched the Preview Video
600,000
Created an
Account
60,000
Purchased
1,200
60%
10%
2%
Note: Does not represent real Udemy numbers
29. 3 Questions to Ask to Figure Out the Right Segment
1) Which type of users might be using your product differently?
30. 3 Questions to Ask to Figure Out the Right Segment
1) Which type of users might be using your product differently?
2) Which one of these users actually behave differently in your product?
31. 3 Questions to Ask to Figure Out the Right Segment
1) Which type of users might be using your product differently?
2) Which one of these users actually behave differently in your product?
3) Is the segment that’s behaving differently big enough for you to care?
32. 3 Questions to Ask to Figure Out the Right Segment
1) Which type of users might be using your product differently?
2) Which one of these users actually behave differently in your product?
3) Is the segment that’s behaving differently big enough for you to care?
For Udemy
We ran the pathway analysis and funnel analysis for these user segments:
- First time Udemy visitors vs returning visitors
- Mobile users vs desktop users
- Users coming from different traffic channels
- Users coming to buy different category of courses
33. We decided to focus on
First time Udemy visitors that landed on one of our product pages and didn’t take
any action on the page.
34. Process: The Experimentation Loop
Prioritize User
Problems
Form
Hypotheses
Gather Data
Prioritize
Hypotheses
Design
Build
Learn &
Improve
Test
36. Quantitative data is helpful
to understand how users behave
but it’s not very helpful to understand
“why”
37. When users showed exit intent,
we asked a simple question:
What is preventing you from
purchasing this course right
now?
Qualitative User Research
Image source: blog.informizely.com
38. Prioritization of User Problems
1) Start with an open ended survey question to your users.
E.g - “What is preventing you from purchasing this course right now?”
39. Prioritization of User Problems
1) Start with an open ended survey question to your users.
E.g - “What is preventing you from purchasing this course right now?”
2) Bucket the answers in meaningful categories and this time ask as a
multi-choice question.
E.g “I will not purchase because the price is not right.” or “I don’t trust
Udemy yet”.
40. Top 3 Reasons For Udemy Were
- I don’t know if Udemy is the right place to learn this skill.
- I don’t understand how this course compares to the other courses.
- I don’t know if this course fits my skill level.
41. Now, we are so much more focused
The new focus:
“We would like to increase
Revenue Per Visitor for first
time visitors who land on
one of our product pages.
And we will do that by
solving these 3 problems for
users”
The old focus:
“We would like to increase
the Revenue Per Visitor.”
42. Process: The Experimentation Loop
Prioritize User
Problems
Form
Hypotheses
Gather Data
Prioritize
Hypotheses
Design
Build
Learn &
Improve
Test
43. Form a clear hypothesis that ties your idea to the problem you are trying to solve.
Your hypothesis statement should answer these questions:
Form Hypotheses
44. 1) What is the problem you are solving?
Form Hypotheses
Form a clear hypothesis that ties your idea to the problem you are trying to solve.
Your hypothesis statement should answer these questions:
45. 1) What is the problem you are solving?
2) Which user segments have this problem?
Form a clear hypothesis that ties your idea to the problem you are trying to solve.
Your hypothesis statement should answer these questions:
Form Hypotheses
46. 1) What is the problem you are solving?
2) Which user segments have this problem?
3) Is this a real problem? What evidence do we have to support that?
Form a clear hypothesis that ties your idea to the problem you are trying to solve.
Your hypothesis statement should answer these questions:
Form Hypotheses
47. 1) What is the problem you are solving?
2) Which user segments have this problem?
3) Is this a real problem? What evidence do we have to support that?
4) How big is the problem?
Form a clear hypothesis that ties your idea to the problem you are trying to solve.
Your hypothesis statement should answer these questions:
Form Hypotheses
48. Form a clear hypothesis that ties your idea to the problem you are trying to solve.
Your hypothesis statement should answer these questions:
Form Hypotheses
1) What is the problem you are solving?
2) Which user segments have this problem?
3) Is this a real problem? What evidence do we have to support that?
4) How big is the problem?
5) How are we going to solve the problem
49. Form a clear hypothesis that ties your idea to the problem you are trying to solve.
Your hypothesis statement should answer these questions:
Form Hypotheses
1) What is the problem you are solving?
2) Which user segments have this problem?
3) Is this a real problem? What evidence do we have to support that?
4) How big is the problem?
5) How are we going to solve the problem
6) How will we know if this experiment succeeds?
50. Hypotheses Template
Based on user research, we know that [X] is an important user problem. If we do
[Y], we expect to see [this] change. This change should cause an increase in
[KPIs].
51. Hypotheses Template
Based on user research, we know that [X] is an important user problem. If we do
[Y], we expect to see [this] change. This change should cause an increase in
[KPIs].
Example: “Based on user research, we know that 40% of first-time visitors
are not sure if Udemy is the right place to learn a new skill. If we feature
successful student stories on our product pages, we expect to increase the
trust of our first-time visitors. This should cause lower bounce rates and
increased conversion rates."
52. Process: The Experimentation Loop
Prioritize User
Problems
Form
Hypotheses
Gather Data
Prioritize
Hypotheses
Design
Build
Learn &
Improve
Test
55. Customized Framework
Above the Fold
(Y/N)
Addressing an
issue discovered
via user
research?
(Y/N)
Did we have
success in a
similar experiment
before?
(Y/N)
Confidence
(1-5)
Ease of Building
(1-5)
Total Score
1 1 1 5 5 20
0 1 1 4 3 11
1 0 0 3 5 10
56. Process: The Experimentation Loop
Prioritize User
Problems
Form
Hypotheses
Gather Data
Prioritize
Hypotheses
Design
Build
Learn &
Improve
Test
57. Process: The Experimentation Loop
Prioritize User
Problems
Form
Hypotheses
Gather Data
Prioritize
Hypotheses
Design
Build
Learn &
Improve
Test
58. Analyzing the Results
Two key questions to answer:
1) Did the experiment solve the user problem?
2) Did solving the user problem move the needle?
61. 1) How strong your hypotheses are
2) How strong your prioritization framework is
Your Success Depends On:
62. Your Success Depends On:
1) How strong your hypotheses are
2) How strong your prioritization framework is
3) How fast you can move through the cycle
63. Your Success Depends On:
1) How strong your hypotheses are
2) How strong your prioritization framework is
3) How fast you can move through the cycle
4) How much you are learning in each cycle
64. Step 2: Building the Right Team
Image source: www.scrumalliance.org
65. The Roles on the Team
- The Experiment Owner (Product Manager, Marketer, CRO Expert)
- Engineers
- User Experience Designer
- Data Analyst
- Data Scientist
- User Researcher