An updated deck of a short talk (30m) given at the first Brighton CRO meetup. Contains useful AB testing tools as well as full speaker notes for most of the slides.

1. Oh Boy!
These A/B tests appear to
be bullshit!

2. @OptimiseOrDie
• UX, Analytics, Testing and Innovation
• Started doing testing & CRO 2004
• Split tested over 40M visitors in 19
languages
• 60+ mistakes with AB testing
• I’ve made every one of them
• Like riding a bike…
• Testing, Workshops, Mentoring,
CRO Methodology, Growth

4. Oh Boy!
These A/B tests appear to
be bullshit!

5. Top F***ups for 2014
1. Testing in the wrong place
2. Your hypothesis inputs are crap
3. No analytics integration
4. Your test will finish after you die
5. You don’t test for long enough
6. You peek before it’s ready
7. No QA for your split test
8. Opportunities are not prioritised
9. Testing cycles are too slow
10. You don’t know when tests are ready
11. Your test fails
12. The test is ‘about the same’
13. Test flips behaviour
14. Test keeps moving around
15. You run an A/A test and waste time
16. Nobody ‘feels’ the test
17. You forgot you were responsive
18. You forgot you had no traffic
19. You ran the wrong test type
20. You didn’t try all the flavours of testing
slidesha.re/1wBbZ9c
@OptimiseOrDie

6. #fail
@OptimiseOrDie

7. @OptimiseOrDie

8. @OptimiseOrDie

9. Oppan Gangnam Style!
@OptimiseOrDie

10. @OptimiseOrDie

11. @OptimiseOrDie

12. @OptimiseOrDie

13. The 95% Stopping Problem
• Many people use 95, 99% ‘confidence’ to stop
• This value is unreliable
• Read this Nature article : bit.ly/1dwk0if
• You can hit 95% early in a test
• If you stop, it could be a false positive
• Tools need to be smarter about inference
• This 95% thingy – it’s last on your list for reasons to
stop testing
• Let me explain
@OptimiseOrDie

14. @OptimiseOrDie
The 95% Stopping Problem

15. The 95% Stopping Problem
@OptimiseOrDie
abtestguide.com/calc/

16. The 95% Stopping Problem
@OptimiseOrDie

17. The 95% Stopping Problem
“You should know that stopping a test once it’s significant is
deadly sin number 1 in A/B testing land. 77% of A/A tests (testing
the same thing as A and B) will reach significance at a certain
point.”
Ton Wesseling, Online Dialogue
“I always tell people that you need a representative sample if
your data needs to be valid. What does ‘representative’ mean?
First of all you need to include all the weekdays and weekends.
You need different weather, because it impacts buyer behavior.
But most important: Your traffic needs to have all traffic sources,
especially newsletter, special campaigns, TV,… everything! The
longer the test runs, the more insights you get.
Andre Morys, Web Arts

18. Three Articles you MUST read
“Statistical Significance does not equal Validity”
http://bit.ly/1wMfmY2
“Why every Internet Marketer should be a Statistician”
http://bit.ly/1wMfs1G
“Understanding the Cycles in your site”
http://mklnd.com/1pGSOUP

19. Business & Purchase Cycles
@OptimiseOrDie
Start Test Finish Avg Cycle
• Customers change
• Your traffic mix changes
• Markets, competitors
• Be aware of all the waves
• Always test whole cycles
• Minimum 2 cycles (wk/mo)
• Don’t exclude slower buyers

20. 20

21. When to stop?
• MINIMUM two business cycles (week/mo)
• MINIMUM of 1 purchase cycle
• MINIMUM 250 outcomes/conversions per creative
• MORE if relative difference is low
• ALWAYS test full weeks
• KNOW what marketing and cycles are doing
• RUN a test length calculator - bit.ly/XqCxuu
• SET your test run time
• Run it
• Stop it
• Analyse the data
• Sometimes I run longer but beware!
@OptimiseOrDie

22. No QA testing for
the AB test?

23. QA Test or Die!
• Over 40% of tests have had QA issues.
• It’s very easy to break or bias the testing
Browser testing www.crossbrowsertesting.com
www.browserstack.com
www.spoon.net
www.saucelabs.com
www.multibrowserviewer.com
Mobile devices www.appthwack.com
www.deviceanywhere.com
www.opendevicelab.com
Article bit.ly/1wBccsJ
@OptimiseOrDie

24. Gamble the Company AWAY!
• I get 60-70% right
• UX and Copywriters good at picking!
• C level execs are easy marks
• Ironically, many decide ‘designs’
• You need collaborative test design
• It’s a team game, with customers
• Flip a coin, anyone?

25. WE’RE ALL WINGING
IT

26. 2004 Headspace
What I thought
I knew in 2004
Reality

27. 2014 Headspace
What I
KNOW
I know
Me, on a
good day

28. Guessaholics Anonymous

29. Rumsfeldian Space
@OptimiseOrDie

30. Rumsfeldian Space
@OptimiseOrDie

31. 23 : Business Future Testing?
“Congratulations!
Today you’re the
lucky winner of our
random awards
programme. You
get all these extra
features for free,
on us. Enjoy.”

32. #1 Smart Talented Polymath People
The 5 Legged Optimisation Barstool
@OptimiseOrDie
Flexible and Agile teams

33. Fittest? Agile!
@OptimiseOrDie

34. #2 : Analytics Investment (tools, people, dev time)
@OptimiseOrDie

35. @OptimiseOrDie
#3 : User research and insight

36. #3 : THE BEST IDEAS COME FROM?
@OptimiseOrDie

37. #4 : GREAT COPYWRITING
“On the average, five times as many people
read the headline as read the body copy. When
you have written your headline, you have spent
eighty cents out of your dollar.”
David Ogilvy
“In 9 years and 40M split tests with visitors, the
majority of my testing success came from
playing with the words.”
@OptimiseOrDie

38. • Google Content Experiments
bit.ly/Ljg7Ds
• Optimizely
www.optimizely.com
• Visual Website Optimizer
www.visualwebsiteoptimizer.com
• Multi Armed Bandit Explanation
bit.ly/Xa80O8
• New Machine Learning Tools
www.conductrics.com
www.rekko.com
@OptimiseOrDie
#5 : Split Testing Tools

39. The 5 Legged Optimisation Barstool
@OptimiseOrDie
#1 Culture & Team
#2 Toolkit & Analytics investment
#3 UX, CX, Service Design, Insight
#4 Persuasive Copywriting
#5 Experimentation (testing) tools

40. READ STUFF

41. READ STUFF

42. READ STUFF

43. #5 : FIND STUFF
@OptimiseOrDie
@danbarker Analytics
@fastbloke Analytics
@timlb Analytics
@jamesgurd Analytics
@therustybear Analytics
@carmenmardiros Analytics
@davechaffey Analytics
@priteshpatel9 Analytics
@cutroni Analytics
@avinash Analytics
@Aschottmuller Analytics, CRO
@cartmetrix Analytics, CRO
@Kissmetrics CRO / UX
@Unbounce CRO / UX
@Morys CRO / Neuro
@UXFeeds UX / Neuro
@Psyblog Neuro
@Gfiorelli1 SEO / Analytics
@PeepLaja CRO
@TheGrok CRO
@UIE UX
@LukeW UX / Forms
@cjforms UX / Forms
@axbom UX
@iatv UX
@Chudders Photo UX
@JeffreyGroks Innovation
@StephanieRieger Innovation
@BrianSolis Innovation
@DrEscotet Neuro
@TheBrainLady Neuro
@RogerDooley Neuro
@Cugelman Neuro
@Smashingmag Dev / UX
@uxmag UX
@Webtrends UX / CRO

44. #5 : LEARN STUFF
@OptimiseOrDie
Baymard.com
Lukew.com
Smashingmagazine.com
ConversionXL.com
Medium.com
Whichtestwon.com
Unbounce.com
Measuringusability.com
RogerDooley.com
Kissmetrics.com
Uxmatters.com
Smartinsights.com
Econsultancy.com
Cutroni.com
www.GetMentalNotes.com

45. #12 : The Best Companies…
• Invest continually in analytics instrumentation, tools, people
• Use an Agile, iterative, cross-silo, one team project culture
• Prefer collaborative tools to having lots of meetings
• Prioritise development based on numbers and insight
• Practice real continuous product improvement, not SLEDD*
• Are fixing bugs, cruft, bad stuff as well as optimising
• Source photos and content that support persuasion and utility
• Have cross channel, cross device design, testing and QA
• Segment their data for valuable insights, every test or change
• Continually reduce cycle (iteration) time in their process
• Blend ‘long’ design, continuous improvement AND split tests
• Make optimisation the engine of change, not the slave of ego
* Single Large Expensive Doomed Developments

46. THE FUTURE OF TESTING

47. Thank You!
Mail : sullivac@gmail.com
Deck : slideshare.com/sullivac
Linkedin : linkd.in/pvrg14
@OptimiseOrDie