This document discusses the lessons learned from AB testing on a political campaign website. It shows the results of testing different button text and media on key metrics like signups, donations, and engagement. Testing variations of the call to action button increased subscriptions by 40.6% and funds raised by over $57 million. Overall it advocates defining metrics, focusing tests, failing fast from poor performers, and continually testing to find the options with the highest conversion rates. The goal is to use AB testing to continuously optimize the site and improve outcomes.

1. AB
Testing

2. AB Testing?
+17.1%
$ per visitor

3. 2008

4. Lesson #1
Define Quantifiable Success
Metrics

5. visitors
per day
Visit
sign up
rate
$ per recipient
Email
$

6. Optimizely
visitors
per day
Visit
sign up
rate
$ per recipient
Email
$

7. Media
Button

8. Button
Media
1. Sign up
1. Get Involved Image
2. Learn More
2. Family Image
3. Join Us Now
3. Change Image
4. Sign Up Now
4. Barack’s Video
5. Springfield Video
6. Sam’s Video

9. Button: “Sign Up”

10. Button: “Learn More”

11. Button: “Join Us Now”

12. Button: “Sign Up Now”

13. Button
Media
1. Sign up
1. Get Involved Image
2. Learn More
2. Family Image
3. Join Us Now
3. Change Image
4. Sign Up Now
4. Barack’s Video
5. Springfield Video
6. Sam’s Video

14. Media: “Get Involved”

15. Media: “Fam ily”

16. Media: “Change”

17. Media: “Change”
Media: “Barack’s Video”

18. Media: “Change”
Media: “Springfield Video”

19. Media: “Change”
Media: “Sam’s Video”

20. And the Winner Was…
Button
Media
1. Sign up
1. Get Involved Image
2. Learn More
2. Family Image
3. Join Us Now
4. Sign Up Now
??????
?????
3. Change Image
4. Barack’s Video
5. Springfield Video
6. Sam’s Video
??????
?????

21. And the Winner Was…
Button
Media
1. Sign up
1. Get Involved Image
2. Learn More
2. Family Image
3. Join Us Now
3. Change Image
4. Sign Up Now
4. Barack’s Video
5. Springfield Video
6. Sam’s Video

22. And the Winner Was…

25. What Did That Mean?
Original
Email
Subscription
s
7,120,000
+40.6%
+2,880,000
288,000
+$57,000,000
New
10,000,000
10,000
$200,000,000
Volunteers
Amount
Raised
712,000
$143,000,000

26. Lesson #1
Define Quantifiable Success
Metrics

27. Lesson #2
Less Is More

28. Vs
+8% Engagement

29. +15% Conversion Rate
+10% $ per visitor

30. +16.8%
$ per visitor

31. Lesson #2
Less Is More

32. Lesson #3
Words Matter
Focus on Your Call to Action

33. Vs
+15.75%
$ per visitor

34. Never Signed
Signed Up,
Up
Never Donated
0.0%
0.0%
Previously
Donated
0.0%
+2.3%
+27.8%
+16.3%
-27.8%
N/A
N/A
+15.2%
-24.6%
+11.9%
+8.5%
+2.9%
+18.4%

35. Vs
+14.6%
Clicks

36. Lesson #3
Words Matter
Focus on Your Call to Action

37. Lesson #4
Fail Fast

38. Vs
-92.3%
Video
Clicks

39. Vs
-10% Conversion Value

40. Lesson #4
Fail Fast

41. Lesson #5
Seek the Global Maximum

43. +600%
Engagement

44. “This new interface is just awful”
+0.5% Retention
+1.4% Engagement

45. Vs
+49% Conversion Rate
from 240 A/B Tests

46. Lesson #5
Seek the Global Maximum

47. Lesson #6
Start Today

48. Optimizely for Magento

49. Lessons Learned
1.
2.
3.
4.
5.
6.
Define Quantifiable Success Metrics
Less is More
Words Matter
Fail Fast
Seek The Global Maximum
Start Today

50. Happy
Testing!