Slides of my ROAR keynote at #elitecamp2015 from Peep Laja / ConversionXL in Estonia. I spoke about Risk, +Optimization, +Automation and Re-invent and explained how http://testing.agency can help you getting things done.
Aspirational Block Program Block Syaldey District - Almora
Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments
1. D i g i t a l E l i t e C a m p E s t o n i a , J u n e 1 3 t h 2 0 1 5
To n W e s s e l i n g – C E O – Te s t i n g . A g e n c y
# E l i t e C a m p - @ To n W
When to run more experiments?
2. Ton Wesseling | @ TonW
Run more experiments #TonW
http://testing.agency
3. Ton Wesseling | @ TonW
Run more experiments #TonW
Today
the
ROAR
model
4. Ton Wesseling | @ TonW
Run more experiments #TonW
Methods used to improve conversion
5. Ton Wesseling | @ TonW
Run more experiments #TonW
Methods companies plan to use
6. Ton Wesseling | @ TonW
Run more experiments #TonW
A/B-testing is
HOT
7. Ton Wesseling | @ TonW
Run more experiments #TonW
Everyone is pushing to test more
8. Ton Wesseling | @ TonW
Run more experiments #TonW
Never satisfied with conversion rates
9. Ton Wesseling | @ TonW
Run more experiments #TonW
Start testing or test more
10. Ton Wesseling | @ TonW
Run more experiments #TonW
NO
Growth problems will dissapear…
11. Ton Wesseling | @ TonW
Run more experiments #TonW
A/B-testing is not a solution
It’s a methodology
12. Ton Wesseling | @ TonW
Run more experiments #TonW
It’s part of a:
Continuous
Optimization
Program
13. Ton Wesseling | @ TonW
Run more experiments #TonW
And you need to know:
When
How
What
14. Ton Wesseling | @ TonW
Run more experiments #TonW
Growth curve – how it was teached
15. Ton Wesseling | @ TonW
Run more experiments #TonW
the
ROAR
model
16. Ton Wesseling | @ TonW
Run more experiments #TonW
Optimization phases - ROAR
Time span!
Conversionspermonth!
17. Ton Wesseling | @ TonW
Run more experiments #TonW
Optimization phases - ROAR
Time span!
Conversionspermonth!
18. Ton Wesseling | @ TonW
Run more experiments #TonW
Optimization phases - ROAR
Time span!
Conversionspermonth!
19. Ton Wesseling | @ TonW
Run more experiments #TonW
Optimization phases - ROAR
Time span!
Conversionspermonth!
Risk!
20. Ton Wesseling | @ TonW
Run more experiments #TonW
Optimization phases - ROAR
Time span!
Conversionspermonth!
Risk! + Optimization!
21. Ton Wesseling | @ TonW
Run more experiments #TonW
Optimization phases - ROAR
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation!
22. Ton Wesseling | @ TonW
Run more experiments #TonW
Optimization phases - ROAR
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
23. Ton Wesseling | @ TonW
Run more experiments #TonW
Optimization phases - ROAR
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!
per month!
24. Ton Wesseling | @ TonW
Run more experiments #TonW
Optimization phases - ROAR
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
10.000 conversions
per month!
1.000 conversions!
per month!
25. Ton Wesseling | @ TonW
Run more experiments #TonW
Statistics
26. Ton Wesseling | @ TonW
Run more experiments #TonW
Optimization phases - ROAR
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
10.000 conversions
per month!
1.000 conversions!
per month!
15% impact
needed!
5% impact!
needed!
Impact needed: all conversions with an average conversion rate of 2%, a test length of 3 weeks and a power of 80%!
27. Ton Wesseling | @ TonW
Run more experiments #TonW
Low number testing?
Why can it hurt you?
28. Ton Wesseling | @ TonW
Run more experiments #TonW
Warning: low significance levels
ü Big chance the winner is not a real winner
29. Ton Wesseling | @ TonW
Run more experiments #TonW
Warning: low power levels
ü Big chance a real winner is not recognized
30. Ton Wesseling | @ TonW
Run more experiments #TonW
Example test set-up
ü 90% significance level
ü 100 conversions per variation
ü Average conversion rate: 2%
ü 20 out of 100 ideas is a winner…
ü …with an average uplift of 10%
31. Ton Wesseling | @ TonW
Run more experiments #TonW
Average: 75% significance & 40% power
32. Ton Wesseling | @ TonW
Run more experiments #TonW
So:
ü 100 tests will give 10 false positives (90% significance)
ü 40% of the 20 winners are recognized(40% power)
ü 8 real winners & 10 false positives
ü NOT 18 improvements of 10%+
33. Ton Wesseling | @ TonW
Run more experiments #TonW
Yes, I’ve got those 1000+ conversions
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!
per month!
10.000 conversions
per month!
34. Ton Wesseling | @ TonW
Run more experiments #TonW
With 1000+ conversions a month
Your test capacity is max.
20+ tests a year
with a statistical power of 80%
on predicted uplifts of 15%
35. Ton Wesseling | @ TonW
Run more experiments #TonW
Tests with measureable impact
On average
1 out of 3
Simply truth: you’re just not always able to create a winner with enough impact
36. Ton Wesseling | @ TonW
Run more experiments #TonW
So in this case, you’ve got impact
Every 7 or 8 weeks
37. Ton Wesseling | @ TonW
Run more experiments #TonW
Slower? Energy will run out!
And your continous optimization program will die
38. Ton Wesseling | @ TonW
Run more experiments #TonW
What to test? Business cases!
Delivery –test with
in stock. 4 weeks
or 2 days?!
39. Ton Wesseling | @ TonW
Run more experiments #TonW
What to test? Business cases!
Service charge, do
it or not? Include in
price or not?!
40. Ton Wesseling | @ TonW
Run more experiments #TonW
What to test? Big design changes
Yes this can be done with an A/B-test tool, you only need A/B-test coders!
41. Ton Wesseling | @ TonW
Run more experiments #TonW
ROAR – Optimization – moving up
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!
per month!
10.000 conversions
per month!
42. Ton Wesseling | @ TonW
Run more experiments #TonW
Finally you can test details…
Is Multi Variate Testing (which is also A/B-testing but with more varations) a good idea? Almost never!!
43. Ton Wesseling | @ TonW
Run more experiments #TonW
You should grow to
Customer Buying Reasons
44. Ton Wesseling | @ TonW
Run more experiments #TonW
Reasons behind the customer journey
45. Ton Wesseling | @ TonW
Run more experiments #TonW
A continuous optimization team
Analyst
Psychologyst
Designer
Developer
Team Lead
46. Ton Wesseling | @ TonW
Run more experiments #TonW
They keep on asking:
Why are these users here?
47. Ton Wesseling | @ TonW
Run more experiments #TonW
And they want to know
How they can
help these users’ brains
to fullfil their needs?
48. Ton Wesseling | @ TonW
Run more experiments #TonW
Data & Psychology
Digital Data
Persuasion Psychology
Analyze
&
Experiment
49. Ton Wesseling | @ TonW
Run more experiments #TonW
A lack of resources
50. Ton Wesseling | @ TonW
Run more experiments #TonW
Testing takes resources
Use it to LEARN
51. Ton Wesseling | @ TonW
Run more experiments #TonW
FA C T & A C T
&
Tell
Conclude
Analyze
Test
Create
Analyze
Find
52. Ton Wesseling | @ TonW
Run more experiments #TonW
Know the company
Mission, Vision,
Stategy & Goals
53. Ton Wesseling | @ TonW
Run more experiments #TonW
Know your user (journeys)
54. Ton Wesseling | @ TonW
Run more experiments #TonW
Based on: asking – why are you here?
55. Ton Wesseling | @ TonW
Run more experiments #TonW
FA C T & A C T
&
Tell
Conclude
Analyze
Test
Create
Analyze
Find
Create
And yes, your team needs an economic / consumer Psychologyst and a UX designer!
56. Ton Wesseling | @ TonW
Run more experiments #TonW
Tests that can be done: A variation
57. Ton Wesseling | @ TonW
Run more experiments #TonW
Tests that can be done: B variation
Need help with coding A/B-test variations like this one? – http://testing.agency!
58. Ton Wesseling | @ TonW
Run more experiments #TonW
Tests that can be done: B variation
59. Ton Wesseling | @ TonW
Run more experiments #TonW
Tests that can be done: A variation
60. Ton Wesseling | @ TonW
Run more experiments #TonW
Tests that can be done: B variation
Need help with coding A/B-test variations like this one? – http://testing.agency!
61. Ton Wesseling | @ TonW
Run more experiments #TonW
Tests that can be done: A & B variation
62. Ton Wesseling | @ TonW
Run more experiments #TonW
Tests that can be done: A variation
63. Ton Wesseling | @ TonW
Run more experiments #TonW
Tests that can be done: B variation
64. Ton Wesseling | @ TonW
Run more experiments #TonW
Tests that can be done: A variation
65. Ton Wesseling | @ TonW
Run more experiments #TonW
Tests that can be done: B variation
66. Ton Wesseling | @ TonW
Run more experiments #TonW
FA C T & A C T
&
Tell
Conclude
Analyze
Test
Create
Analyze
Find
Analyze
Your analyst will analyze the results!
67. Ton Wesseling | @ TonW
Run more experiments #TonW
We measure in Analytics software
68. Ton Wesseling | @ TonW
Run more experiments #TonW
Calculate results: ABTestGuide.com/calc
69. Ton Wesseling | @ TonW
Run more experiments #TonW
Not significant
ü It’s not a winner, stop
ü But, it does not mean it’s a losing one!
ü Average: 33% of your tests will have winners
If you are optimizing dialogues!
70. Ton Wesseling | @ TonW
Run more experiments #TonW
Choose what to analyze up front
ü You just go and dig until you find something
(you will always find something that seems true BUT is not!)
71. Ton Wesseling | @ TonW
Run more experiments #TonW
Don’t stop the test once it’s significant!
72. Ton Wesseling | @ TonW
Run more experiments #TonW
You just let it run until it’s significant
73. Ton Wesseling | @ TonW
Run more experiments #TonW
You just let it run until it’s significant
Your test has a fixed lenght!
It’s even a fixed amount of test population, but we love to test business cycles and/or weeks!
74. Ton Wesseling | @ TonW
Run more experiments #TonW
Be aware: Your test takes too long
ü Cookie deletion
(you will lose 10% in two weeks)
They will re-enter the and pollute your samples!
75. Ton Wesseling | @ TonW
Run more experiments #TonW
Test length: ABTestGuide.com/calc
76. Ton Wesseling | @ TonW
Run more experiments #TonW
Tip: Don’t test broken stuff
ü Always check browser compatibility!
ü Don’t break dynamic stuff!
77. Ton Wesseling | @ TonW
Run more experiments #TonW
Tip: Slow down the control down
ü Add the same code to the control
78. Ton Wesseling | @ TonW
Run more experiments #TonW
Tip: leave out s o m e visitors
ü Only test on those who can be changed
(if you test a sales promotion – leave out your current clients who just logon)
79. Ton Wesseling | @ TonW
Run more experiments #TonW
Tip: Only test on fresh people
ü Start with control for a full purchase cycle
(and then start sending traffic to your variations – setting the control to 0%)
80. Ton Wesseling | @ TonW
Run more experiments #TonW
Tip: give them time to buy!
ü Stop getting fresh visitors after your test time
(send new traffic to the control, but give you tested people time to finish converting)
81. Ton Wesseling | @ TonW
Run more experiments #TonW
+ Optimization! + Automation!
Back to the ROAR model
Time span!
Conversionspermonth!
Risk! Re-think!
1.000 conversions!
per month!
10.000 conversions
per month!
82. Ton Wesseling | @ TonW
Run more experiments #TonW
ROAR – Optimization – moving up
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!
per month!
10.000 conversions
per month!
83. Ton Wesseling | @ TonW
Run more experiments #TonW
Test capacity
maximum number of
experiments per year
that your site or app
can handle
84. Ton Wesseling | @ TonW
Run more experiments #TonW
Regret
amount of money
you are losing per year by
not using your full test capacity
85. Ton Wesseling | @ TonW
Run more experiments #TonW
Calculate your test capicity
http://ondi.me/size!
86. Ton Wesseling | @ TonW
Run more experiments #TonW
ROAR – Automation
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!
per month!
10.000 conversions
per month!
87. Ton Wesseling | @ TonW
Run more experiments #TonW
Automation
88. Ton Wesseling | @ TonW
Run more experiments #TonW
You tested which locations are critical
89. Ton Wesseling | @ TonW
Run more experiments #TonW
You know which dialogues are critical
90. Ton Wesseling | @ TonW
Run more experiments #TonW
Automation
Contextual bandits
If you are ready – give me a call!!
91. Ton Wesseling | @ TonW
Run more experiments #TonW
Now you’ve moved to
Algorithm testing
Which internal and external influencers do have an impact on which segment of users?!
92. Ton Wesseling | @ TonW
Run more experiments #TonW
+ Optimization! + Automation!
ROAR – Automation
Time span!
Conversionspermonth!
Risk! Re-think!
1.000 conversions!
per month!
10.000 conversions
per month!
93. Ton Wesseling | @ TonW
Run more experiments #TonW
ROAR – You don’t stop optimizing
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!
per month!
10.000 conversions
per month!
94. Ton Wesseling | @ TonW
Run more experiments #TonW
ROAR – Maybe you’ll never automate
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!
per month!
10.000 conversions
per month!
95. Ton Wesseling | @ TonW
Run more experiments #TonW
ROAR – Re-think
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!
per month!
10.000 conversions
per month!
96. Ton Wesseling | @ TonW
Run more experiments #TonW
The ultimate Re-think resource
97. Ton Wesseling | @ TonW
Run more experiments #TonW
CRO brings user knowledge
98. Ton Wesseling | @ TonW
Run more experiments #TonW
You will make big impact on Discover
99. Ton Wesseling | @ TonW
Run more experiments #TonW
Customer knowledge center
Making money from conversion improvements
Creating more and more customer insights
Which can be monetized
on even a larger scale
100. Ton Wesseling | @ TonW
Run more experiments #TonW
Now, that’s
HOT
101. Ton Wesseling | @ TonW
Run more experiments #TonW
Done testing? No!
TESTING NEVER STOPS
Okay, end of life, cash cow, no need for any further growth accelaration or user knowledge!
102. Ton Wesseling | @ TonW
Run more experiments #TonW
A/B-testing
Don’t start to soon
103. Ton Wesseling | @ TonW
Run more experiments #TonW
A/B-testing
Test the right way
104. Ton Wesseling | @ TonW
Run more experiments #TonW
A/B-testing
Minimize regret
Run more experiments, maximize your capacity!
105. Ton Wesseling | @ TonW
Run more experiments #TonW
http://testing.agency
106. D i g i t a l E l i t e C a m p E s t o n i a , J u n e 1 3 t h 2 0 1 5
To n W e s s e l i n g – C E O – Te s t i n g . A g e n c y
# E l i t e C a m p - @ To n W
When to run more experiments?