With a variety of easy-to-use tools available, A/B testing is something that every business can and should be doing. However, there’s more to A/B testing than just setting up the test. So many companies waste time and money by making simple and easy to avoid mistakes. In this presentation, conversion rate expert and guest presenter, Peep Laja, discusses 11 of the biggest testing mistakes he frequently sees companies make. Leave with actionable tips and strategies on how to avoid them. The webinar was moderated by Hanapin’s CRO Manager, Roopa Carpenter. You’ll get expert-level CRO tips like: *Why testing random ideas comes at a huge expense *How to avoid calling a test too early *Utilizing small wins to make a big impact

1. #thinkppc
HOSTED BY:
The 11 13 Biggest A/B Testing
Mistakes and How To Avoid
Them
Featuring Peep Laja

2. #thinkppc
Presenters
• Peep Laja
– Guest Presenter
– Founder of Conversion XL
– @peeplaja
• Roopa Carpenter
– Moderator
– CRO Manager at Hanapin Marketing
– @roopacarpenter

3. #thinkppc
Join the conversation
• Include the hashtag #thinkppc in your
Twitter tweets.
Or use the webinar question box to send
us questions.

4. #thinkppc
Live Poll Question #1
How long have you been doing CRO?
#thinkppc
A. Less than 1 year
B. 1-3 years
C. 3-5 years
D. 5+ years

5. #thinkppc
Live Poll Question #2
How do you manage your account(s)?
#thinkppc
a) I’m part of an in-house PPC team.
b) I do all the marketing myself.
c) I work with an agency.
d) I’m a consultant.

6. #thinkppc
Peep Laja, ConversionXL @peeplaja
Typical scenario:
A company runs 100 tests
/ year. Lots of wins. But no
difference on the bank
account.

7. #thinkppc
Precious time wasted
on stupid tests
Testing mistake #1:

8. #thinkppc
You don’t have a data-
driven, learning-
oriented hypothesis
Testing mistake #2:

9. #thinkppc
Learn more: http://conversionxl.com/how-to-come-up-with-more-
winning-tests-using-data/

10. #thinkppc
You think you know
what will work
Testing mistake #3:

11. #thinkppc
Your sample size is
way too small
Testing mistake #4:

12. #thinkppc

13. #thinkppc
Calculate needed sample size in advance:

14. #thinkppc
I won’t even look at results until
there’s at least 250-350
conversions PER variation.
Way more if you want to segment the data

15. #thinkppc
Your tests don’t run
long enough
Testing mistake #5:

16. #thinkppc

17. #thinkppc
You don’t test full
weeks at a time
Testing mistake #6:

18. #thinkppc

19. #thinkppc
You run tests when
you have no traffic
Testing mistake #7:

20. #thinkppc
•Learn more: http://conversionxl.com/how-to-do-conversion-
optimization-with-very-little-traffic/

21. #thinkppc
Test data is not sent
to third-party
analytics
Testing mistake #8:

23. #thinkppc

25. #thinkppc
(function(){
var experimentID = 2809164105;
window.optimizely = window.optimizely || [];
if (typeof window.optimizely.data.experiments[experimentID] !== 'undefined') {
var experimentName = window.optimizely.data.experiments[experimentID].name;
var variationName =
window.optimizely.data.state.variationNamesMap[experimentID];
window.ga = window.ga || function() {(window.ga.q = window.ga.q ||
[]).push(arguments);}; window.ga.l =+ new Date();
window.ga('send', 'event', 'Optimizely', experimentID + ' ' + experimentName,
variationName, {'nonInteraction': 1});
}
})();

26. #thinkppc
Peep Laja, ConversionXL @peeplaja

27. #thinkppc
You give up after your
first test for a
hypothesis fails.
Testing mistake #9:

28. #thinkppc

29. #thinkppc
Full case study:
http://conversionxl.com/case-study-how-we-improved-landing-page-conversion/
79.3% uplift in conversions

30. #thinkppc

31. #thinkppc

32. #thinkppc

33. #thinkppc

34. #thinkppc

35. #thinkppc

38. #thinkppc
You’re not aware of
validity threats
Testing mistake #10:

39. #thinkppc
History effect
Instrumentation effect
Selection effect

40. #thinkppc
You’re ignoring small
gains
Testing mistake #11:

41. #thinkppc
5% monthly increase in
your conversion rate will
result in 80% uplift for
the year. That’s how
math works.

42. #thinkppc
You copy competitors
/ leaders / other
people’s tests
Testing mistake #12:

43. #thinkppc
Which Test Won?

44. #thinkppc
WhichTestWon?
WhichTestWasNot
RunProperly?

45. #thinkppc
Somebody else’s test
results are useless

46. #thinkppc
“Site shows stats from
various A/B tests - Finally
I've got evidence to show
clients on a load of
design decisions!”

47. #thinkppc

48. #thinkppc
The process, the
method, the analysis

49. #thinkppc
You’re not running tests
at all times
Testing mistake #13:

50. #thinkppc
PPC Retainers
Need some assistance for your PPC?
We offer on-going and one-time retainers.
Learn More:
http://www.hanapinmarketing.com/ppc-retainers/

51. #thinkppc
Live Q&A Time!

52. #thinkppc
Have more questions?
Thank you for attending our webinar! #thinkppc
• Contact us Directly:
» Hanapin Feedback: marketing@hanapinmarketing.com