1. The math behind A/B testing
How to perform a non-­‐biased test

2. A/B testing
Not a replacement for common sense
It only gives you a level of confidence
Helps you achieve only local maxima

3. AB experiment: Toss a coin
Heads = successful conversion. Tails = no conversion.
Hypothesis: Wearing a Red color t-­‐shirt will increase conversion

4. 46 heads out
of 100
54 heads out
of 100

5. Conversion increased by 17%
Changing your t-­‐shirt to red increases conversion

6. Whats wrong
Conversion is never a single number. Its a range.
Probability
µ
Variance/Noise

7. Whats wrong
Sample = 100 tosses.
µ
=
0.5
µ
=
0.46
∞
coin
tosses

8. Whats wrong
Sample = 100 tosses.
µ
=
0.5
µ
==
0.46
µ
0.46
∞
coin
tosses 100
coin
tosses
Sample mean ≠ population mean

9. The role of chance
Red Blue
Comparison between two noisy samples

10. Statistical significance
Red Blue
Standard Error (SE) = Square root of (p * (1-p) / n)
p = conversion rate, n = sample size
How much deviation from average conversion rate (p)
can be expected if this experiment is repeated multiple
times.

11. Statistical significance
95% confidence:
True conversion rate lies within this range: p ± 2 * SE

12. Statistical significance
95% confidence:
True conversion rate lies within this range: p ± 2 * SE

13. Statistical significance
95% confidence:
True conversion rate lies within this range: p ± 2 * SE
h3p://visualwebsiteop=mizer.com/ab-­‐split-­‐significance-­‐calculator/

14. Sample size
Standard Error (SE) = Square root of (p * (1-p) / n)

15. Sample size
Standard Error (SE) = Square root of (p * (1-p) / n)
Min. sample size to calculate the statistical signiLicance
Statistical conLidence
Existing conversion rate of website
Difference in conversion rate you want to detect
Number of variations you want to test
h3p://www.testsignificance.com/

16. Ideal test
Determine the sample size
Check the results only once you have reached the sample size
Determine the statistical signiLicance
Pick based on long term plan if no clear winner

17. Thanks