The document discusses common mistakes in A/B testing and provides guidance on properly designing, interpreting results, and making decisions from A/B tests. It notes the most common serious errors include not considering multiple testing, choosing the wrong metrics, and improper stopping rules. It emphasizes the importance of considering significance and risk during test design, focusing on actual key performance indicators like profit, and recognizing the different risks between medical testing and business experimentation.

1. @THCapper
YOUR RESULTS
ARE INVALID
STATISTICS FOR CRO

2. A Good A/B Test Result:
“10% Uplift, With 95%
Significance”

3. ● What does this mean?
● Is this correct?
“10% Uplift, With 95%
Significance”

4. Easy Questions?

5. Most Tools Encourage
Mistakes

7. Risk

9. Marketer: “Roll it out!”
Statistician (me): *sobs*

10. “That’s Risky”

12. “Advanced”

13. You will learn today:
● The most common serious errors in
A/B testing
● How to avoid them
● How to interpret your result
● Whether to roll it out

14. How to Run an A/B Test

15. 1. Test design
2. Results interpretation
3. Decision

16. Jargon: Null Hypothesis

17. Jargon: Null Hypothesis
● The hypothesis that your variant and
original are functionally equivalent

18. e.g. an A/A Test
vs.
A A

19. Jargon: P-Value

20. Jargon: P-Value
● The chance of a result this extreme if
the null hypothesis is true
● E.g. 0.05 for 95% significance

21. Jargon: Critical Value

22. Jargon: Critical Value
● What you compare your p-value with
when deciding whether to reject the
null hypothesis

23. 1. Test Design

24. How Many Tests?

25. A B
C D E F
Multivariate Testing
Landing Page:
Product Pages:

26. Multivariate Testing
A B
C D
Landing Page:
Product Page:

27. Multivariate Testing
A
C D
B

28. Multivariate Testing
A
C D
BA
C D
B

29. Multivariate Testing
A
C D
BA
C D
BA
C D
B

30. Multivariate Testing
A
C D
B A
C D
BA
C D
BA
C D
B

31. Multivariate Testing
A BLanding Page:
A: 5%
B: 7.5%

32. Multivariate Testing
C D
C: 5%
D: 7.5%
Product Page:

33. Multivariate Testing
A
C D
B A
C D
BA
C D
BA
C D
B

34. A
C
Multivariate Testing
D
B A
C D
BA
C D
BA
C D
B
AC:
0%
BD:
5%
BC:
10%
AD:
10%

35. A B

37. “Constantly Iterate”

38. Multiple Testing
A B C D E F

39. False Positives
Test: Healthy
Test: Ill

40. False Positives
Test: Healthy
Test: Ill
False Negatives

41. False Positives
Test: Healthy
Test: Ill False Positives
False Negatives

42. Multiple Testing
1 A/A test:
5% chance of achieving 95% significance.

43. Multiple Testing
1 A/A Test: 5% chance

44. Multiple Testing
1 A/A Test:
2 A/A Tests:
5% chance
9.75% chance

45. Multiple Testing
1 A/A Test:
2 A/A Tests:
3 A/A Tests:
5% chance
9.75% chance
14.26% chance

46. Multiple Testing
1 A/A Test:
2 A/A Tests:
3 A/A Tests:
4 A/A Tests:
5% chance
9.75% chance
14.26% chance
18.55% chance

47. Multiple Testing
1 A/A Test:
2 A/A Tests:
3 A/A Tests:
4 A/A Tests:
n A/A Tests:
5% chance
9.75% chance
14.26% chance
18.55% chance
1-0.95^n

48. Multiple Testing
Solutions:
1. Accept risk of false positives

49. Multiple Testing
Solutions:
1. Accept risk of false positives
2. Bonferroni correction

50. Bonferroni Approximation
Standard: P-value vs………..…. 0.05

51. Bonferroni Approximation
Standard: P-value vs………..….
Approximation: P-value vs…...
0.05
0.05/N

52. Bonferroni Correction
Standard: P-value vs………..….
Bonferroni: P-value vs……….
0.05
1-(1-0.05)^(1/N)

53. Multiple Testing
Solutions:
1. Accept risk of false positives
2. Bonferroni correction
3. Holm-Bonferroni correction

54. Choosing the Right Metric

55. Choosing the Right Metric
Conversion Rate
vs.
Average Session Value

56. Choosing the Right Metric
Conversion Rate
vs.
Average Session Value Profit?

57. Stopping Rules

58. Stopping Rules
Common: When my test reaches
significance.

59. “Significance so far” varies over time.

60. Stopping Rules
Y Y Y Y Y N N N N N

61. Stopping Rules
Y Y Y Y Y Y YN N N

62. Stopping Rules
20000

63. 20000

64. Exceptions
https://en.wikipedia.org/wiki/Sequential_probability_ratio_test

65. Stopping Rules
Solutions:
1. Sequential testing - e.g. Optimizely
2. Bayesian testing - e.g. VWO
3. Predetermined sample size

66. evanmiller.org/ab-testing/sample-size.html

67. Sample Size for Average
Session Value Testing

68. =stdev(B:B)
=stdev.s(B:B)
Standard Deviation

69. powerandsamplesize.com/Calculators/

70. Cutting Your Losses

71. Test Design Recap
Contamination
Multiple
Testing
Metric
Choice
Stopping
Rules

73. 1. Test design
2. Results interpretation
3. Decision

74. 2. Results Interpretation

75. Interpreting the P-Value

76. Interpreting the P-value
1 test reaches 95% significance:
5% chance of data this extreme if
variants functionally equivalent.

77. 0

78. Analogy

79. Analogy
Question: How likely is it that my
analytics or site are broken?

80. Analogy
Question: How likely is it that my
analytics or site are broken?
Non-Answer: We only go a whole day
with no conversions once every 2
months.

81. Analytics is broken with
probability 1 or 0.

82. Interpreting the P-value
Question: How likely is it that this
variation actually does nothing?
Non-Answer: We’d only see a
difference this big 5% of the time.

83. Meanwhile in Industry Tools:
● “Chance to beat baseline”
● “We are 95% certain that the changes
in test “B” will improve your
conversion rate”

84. Unanswered Questions

85. Unanswered Questions
Question: How likely is it that the
increase will be less than predicted?

86. Unanswered Questions
Question: How likely is it that the
increase will be negative?

87. One Mistake
Probability of Outcome given Data
vs.
Probability of Data given Null

88. Unanswered Questions
Question: How likely is it that these
results are a fluke?

89. Confidence Intervals

90. Confidence Interval of
Conversion Rate

91. Overlapping Confidence
Intervals

92. Everything Else Still Applies

93. Choosing the Right Metric

94. evanmiller.org/ab-testing/t-test.html

95. Results Interpretation Recap
Check
Revenue
P-Value
Confidence
Intervals

96. 1. Test design
2. Results interpretation
3. Decision

97. 3. Decision

98. A Good A/B Test Result:
“10% Uplift, With 95%
Significance”

99. But what about this?
“10% Uplift, With 60%
Significance”

100. Jargon: P-Value
● The chance of a result this extreme if
the null hypothesis is true
● E.g. 0.05 for 95% significance

101. “10% Uplift, With 60% Significance”
● 40% chance of data at least this
extreme if variation functionally
identical

102. “10% Uplift, With 60% Significance”
● 40% chance of data at least this
extreme if variation functionally
identical
● The variation is probably better than
the baseline

103. Drug Trials
vs.
Investment Banking

104. Are You OK With False
Positives?

105. Please Don’t Tweet This:

106. Tweet This Instead:

107. Data is Expensive

108. Data is Expensive:
● Opportunity Cost
● Exploration vs. Exploitation

109. Historical Comparisons are
Invalid

110. Hang on…
Why Should I Care About
Significance?

111. 1. Ignoring Significance
Doesn’t Allow You to Ignore
Statistics

112. 2. Risk Aversion

113. Risk Factors:
● Agility
● Business attitudes
● What’s the worst that
could happen?

114. Decision Recap
Significant
vs. Winning
Risk
Exploration
vs. Exploitation

115. Conclusion:
3 Takeaways

116. 1. Think about significance and risk
during test design

117. 2. Remember your real KPI: Profit

118. 3. You’re not testing medicines

119. @THCapper
Takeaways:
1. Think about significance and risk
during test design
2. Remember your real KPI: Profit
3. You’re not testing medicines