Many marketers are eager to start A/B testing but aren't aware of dangerous misconceptions and prioritize tactics over strategy. This presentation explores the evolution of A/B testing, discusses what it can and cannot do, and covers the basic principles of conversion rate optimization so you can avoid risk, use data effectively and find solutions to YOUR problems. Originally presented at National Multifamily Housing Council NMHC Optech Nov. 2016 on behalf of G5. NMHC OPTECH is the premier destination for what’s next in multifamily. Open to NMHC members and non-members alike, OPTECH provides a forum for senior practice leaders from a wide variety of disciplines to collaborate on strategic action, incubate new ideas and elevate the industry across all areas of business operations and technology.

1. THE
EVOLUTION
OF A/B
TESTING
@Nicole_Mintiens
Data Analysis & Insights Manager
@GetG5

2. 41 Shades
of Blue
The Evolution of A/B Testing
@Nicole_Mintiens

3. $200 MILLION
The Evolution of A/B Testing
@Nicole_Mintiens

4. The Evolution of A/B Testing
@Nicole_Mintiens

5. Advertising and Marketing
• Improved ROI = Good
The Evolution of A/B Testing
@Nicole_Mintiens

6. Advertising and Marketing
• Improved ROI = Good
• Wasted Resources = Bad
The Evolution of A/B Testing
@Nicole_Mintiens

7. Advertising and Marketing
• Improved ROI = Good
• Wasted Resources = Bad
• Marketing = Complex
The Evolution of A/B Testing
@Nicole_Mintiens

8. Advertising and Marketing
• Improved ROI = Good
• Wasted Resources = Bad
• Marketing = Complex
• Facts > Opinion
The Evolution of A/B Testing
@Nicole_Mintiens

9. WHY SPEND TIME TESTING?
• $92 traffic /$1 optimization
Source: Econsultancy & Redeye Conversion Rate Optimization Report 2014
The Evolution of A/B Testing
@Nicole_Mintiens

10. WHY SPEND TIME TESTING?
• $92 traffic /$1 optimization
• 87% run 1-5 tests/month
Source: Econsultancy & Redeye Conversion Rate Optimization Report 2014
The Evolution of A/B Testing
@Nicole_Mintiens

11. WHY SPEND TIME TESTING?
• $92 traffic /$1 optimization
• 87% run 1-5 tests/month
– Best brands run
hundreds per month
Source: Econsultancy & Redeye Conversion Rate Optimization Report 2014
The Evolution of A/B Testing
@Nicole_Mintiens

12. A/B TESTING FACTS
• Improves conversion rate
The Evolution of A/B Testing
@Nicole_Mintiens

13. A/B TESTING FACTS
• Improves conversion rate
• Enhances effectiveness
The Evolution of A/B Testing
@Nicole_Mintiens

14. A/B TESTING FACTS
• Improves conversion rate
• Enhances effectiveness
• Optimizes ROI
The Evolution of A/B Testing
@Nicole_Mintiens

15. A/B TESTING FACTS
• Improves conversion rate
• Enhances effectiveness
• Optimizes ROI
So… YOU should test!
The Evolution of A/B Testing
@Nicole_Mintiens

16. The Evolution of A/B Testing
@Nicole_Mintiens

17. MISCONCEPTIONS
• Hurt conversion rate
The Evolution of A/B Testing
@Nicole_Mintiens

18. MISCONCEPTIONS
• Hurt conversion rate
• Reduce effectiveness
The Evolution of A/B Testing
@Nicole_Mintiens

19. MISCONCEPTIONS
• Hurt conversion rate
• Reduce effectiveness
• Waste resources
The Evolution of A/B Testing
@Nicole_Mintiens

20. MISCONCEPTIONS
• Hurt conversion rate
• Reduce effectiveness
• Waste resources
• Damage ROI
The Evolution of A/B Testing
@Nicole_Mintiens

21. TESTING
IS NOT
OPTIMIZATION
The Evolution of A/B Testing
@Nicole_Mintiens

22. TESTING
IS NOT
OPTIMIZATION
…but it plays a role.
The Evolution of A/B Testing
@Nicole_Mintiens

23. DON’T JUST TEST, OPTIMIZE.
• Avoid mistakes
The Evolution of A/B Testing
@Nicole_Mintiens

24. DON’T JUST TEST, OPTIMIZE.
• Avoid mistakes
• Increase effectiveness
The Evolution of A/B Testing
@Nicole_Mintiens

25. DON’T JUST TEST, OPTIMIZE.
• Avoid mistakes
• Increase effectiveness
• Improve ROI
The Evolution of A/B Testing
@Nicole_Mintiens

26. I’LL BE YOUR GUIDE
• 7+ years optimizing multifamily
– PPC, SMM, SEO & CRO
The Evolution of A/B Testing
@Nicole_Mintiens

27. A/B TESTING BASICS
• Statistical hypothesis-based
experimentation between
variations of a control.
The Evolution of A/B Testing
@Nicole_Mintiens

28. A/B TESTING BASICS
• Statistical hypothesis-based
experimentation between
variations of a control.
• Goal:
– Isolate differences that
maximize an outcome
• i.e. conversion
The Evolution of A/B Testing
@Nicole_Mintiens

29. CONVERSION DEFINITION
Action YOU want the user to take.
The Evolution of A/B Testing
@Nicole_Mintiens

30. A/B/N TESTING
The Evolution of A/B Testing
@Nicole_Mintiens

31. A/B/N TESTING
The Evolution of A/B Testing
@Nicole_Mintiens

32. A/B/N TESTING
The Evolution of A/B Testing
@Nicole_Mintiens

33. • 2000
– Google runs first test to find optimum number of SERP
results.
A/B TESTING HISTORY
The Evolution of A/B Testing
@Nicole_Mintiens

34. • 2000
– Google runs first test to find optimum number of SERP
results.
• 2008
– A/B tests raise additional $60 million for Obama campaign.
A/B TESTING HISTORY
The Evolution of A/B Testing
@Nicole_Mintiens

35. • 2000
– Google runs first test to find optimum number of SERP
results.
• 2008
– A/B tests raise additional $60 million for Obama campaign.
• 2010
– Obama campaign’s Director of Analytics founds Optimizely.
A/B TESTING HISTORY
The Evolution of A/B Testing
@Nicole_Mintiens

36. • 2000
– Google runs first test to find optimum number of SERP
results.
• 2008
– A/B tests raise additional $60 million for Obama campaign.
• 2010
– Obama campaign’s Director of Analytics founds Optimizely.
• 2011
– Google has run over 7,000 A/B tests
A/B TESTING HISTORY
The Evolution of A/B Testing
@Nicole_Mintiens

37. MULTIFAMILY AVG CONVERSION RATE
Source: 2016 G5 Multifamily Google Analytics Data
The Evolution of A/B Testing
@Nicole_Mintiens

38. MULTIFAMILY AVG CONVERSION RATE
Source: 2016 G5 Multifamily Google Analytics Data
~2%
The Evolution of A/B Testing
@Nicole_Mintiens

39. ECOMMERCE AVG
CONVERSION RATE
~3%
The Evolution of A/B Testing
@Nicole_Mintiens

40. AMAZON PRIME’S CONVERSION RATE
Source: https://www.internetretailer.com/2015/06/25/amazon-prime-members-convert-74-time
The Evolution of A/B Testing
@Nicole_Mintiens

41. 2366%
Better than
average!
AMAZON PRIME’S CONVERSION RATE
Source: https://www.internetretailer.com/2015/06/25/amazon-prime-members-convert-74-time
The Evolution of A/B Testing
@Nicole_Mintiens

42. AMAZON’S TECHNOLOGY
Source: Builtwith.com
The Evolution of A/B Testing
@Nicole_Mintiens

43. AMAZON’S TECHNOLOGY
Source: Builtwith.com
The Evolution of A/B Testing
@Nicole_Mintiens

44. SOFTWARE
ADVANCEMENTS
• Marketing automation
The Evolution of A/B Testing
@Nicole_Mintiens

45. SOFTWARE
ADVANCEMENTS
• Marketing automation
• Easy to install
The Evolution of A/B Testing
@Nicole_Mintiens

46. SOFTWARE
ADVANCEMENTS
• Marketing automation
• Easy to install
• WYSIWYG editors
The Evolution of A/B Testing
@Nicole_Mintiens

47. WHAT A/B TESTING
CAN DO
• Validate hypotheses
The Evolution of A/B Testing
@Nicole_Mintiens

48. WHAT A/B TESTING
CAN DO
• Validate hypotheses
• Inform choices
The Evolution of A/B Testing
@Nicole_Mintiens

49. WHAT A/B TESTING
CAN DO
• Validate hypotheses
• Inform choices
• Insight for iteration
The Evolution of A/B Testing
@Nicole_Mintiens

50. WHAT A/B TESTING CAN’T DO
• Determine YOUR
–Problem
Thanks, Boss!
The Evolution of A/B Testing
@Nicole_Mintiens

51. WHAT A/B TESTING CAN’T DO
• Determine YOUR
–Problem
–User’s
needs/wants
Thanks, Boss!
The Evolution of A/B Testing
@Nicole_Mintiens

52. WHAT A/B TESTING CAN’T DO
• Determine YOUR
–Problem
–User’s
needs/wants
• Improve traffic
–Quality
Thanks, Boss!
The Evolution of A/B Testing
@Nicole_Mintiens

53. WHAT A/B TESTING CAN’T DO
• Determine YOUR
–Problem
–User’s
needs/wants
• Improve traffic
–Quality
–Volume
Thanks, Boss!
The Evolution of A/B Testing
@Nicole_Mintiens

54. CAN DOES NOT EQUAL SHOULD
Testing will waste
your time without:
• Enough traffic
The Evolution of A/B Testing
@Nicole_Mintiens

55. CAN DOES NOT EQUAL SHOULD
Testing will waste
your time without:
• Enough traffic
• Enough research
The Evolution of A/B Testing
@Nicole_Mintiens

56. CAN DOES NOT EQUAL SHOULD
Testing will waste
your time without:
• Enough traffic
• Enough research
• Technical resources
The Evolution of A/B Testing
@Nicole_Mintiens

57. THEN HOW DO YOU IMPROVE?!
The Evolution of A/B Testing
@Nicole_Mintiens

58. A PROCESS
to improve performance
informed by INSIGHT
and defined by GOALS.
CONVERSION RATE OPTIMIZATION
(CRO)
The Evolution of A/B Testing
@Nicole_Mintiens

59. A PROCESS
to improve performance
informed by INSIGHT
and defined by GOALS.
GOALS
CONVERSION RATE OPTIMIZATION
(CRO)
The Evolution of A/B Testing
@Nicole_Mintiens

60. ALWAYS ROOM FOR IMPROVEMENT
• Customer acquisition cost
The Evolution of A/B Testing
@Nicole_Mintiens

61. ALWAYS ROOM FOR IMPROVEMENT
• Customer acquisition cost
• Return on investment
The Evolution of A/B Testing
@Nicole_Mintiens

62. ALWAYS ROOM FOR IMPROVEMENT
• Customer acquisition cost
• Return on investment
• Customer understanding
The Evolution of A/B Testing
@Nicole_Mintiens

63. ALWAYS ROOM FOR IMPROVEMENT
• Customer acquisition cost
• Return on investment
• Customer understanding
• Google’s perception
The Evolution of A/B Testing
@Nicole_Mintiens

64. GOALS & TRACKING
Source: http://borsen.dk/opinion/blogs/view/17/4159/den_svaereste_marketingdisciplin.html
The Evolution of A/B Testing
@Nicole_Mintiens

65. YOUR IDEAL RESIDENTS
Prospects
Value
I only rent the
most affordable
apartments.
The Evolution of A/B Testing
@Nicole_Mintiens

66. YOUR IDEAL RESIDENTS
Prospects
Value
I only rent the
most affordable
apartments.
I’ll rent near
my tech
coworkers
regardless of
price.
The Evolution of A/B Testing
@Nicole_Mintiens

67. CUSTOMER RESEARCH
Source: https://blog.kissmetrics.com/wp-content/uploads/2015/02/NYU.lowres.ifiwere.jpg
The Evolution of A/B Testing
@Nicole_Mintiens

68. “The EFFECTIVENESS of
any lead generation campaign
comes down to your HOOK.
”Mike King
@IPullRank
Founder iPullRank
Consultant to Fortune/INC500 brands.
The Evolution of A/B Testing
@Nicole_Mintiens

69. VOICE OF CUSTOMER RESEARCH
• What makes your
property unique?
The Evolution of A/B Testing
@Nicole_Mintiens

70. VOICE OF CUSTOMER RESEARCH
• What makes your
property unique?
• Why you and not
the competition?
The Evolution of A/B Testing
@Nicole_Mintiens

71. VOICE OF CUSTOMER RESEARCH
• What makes your
property unique?
• Why you and not
the competition?
The Evolution of A/B Testing
@Nicole_Mintiens

72. UNIQUE VALUE PROPOSITION (UVP):
Above-the-fold EXPERIENCE
that communicates WHAT
you do and WHY
it’s UNIQUE.
The Evolution of A/B Testing
@Nicole_Mintiens

73. Original (A)
UVP: FIVE SECOND TEST
The Evolution of A/B Testing
@Nicole_Mintiens

74. Original (A) Variation (B)
UVP: FIVE SECOND TEST
The Evolution of A/B Testing
@Nicole_Mintiens

75. Original (A) Variation (B)
UVP: FIVE SECOND TEST
The Evolution of A/B Testing
@Nicole_Mintiens

76. CUSTOMER SEGMENTATION
The Evolution of A/B Testing
@Nicole_Mintiens

77. CUSTOMER SEGMENTATION
User Persona Creator: https://xtensio.com/user-persona/
The Evolution of A/B Testing
@Nicole_Mintiens

78. Slide credit to Angie Schottmuller - Check out the full presentation:
http://www.slideshare.net/aschottmuller/content-marketing-onpage-optimization-holy-grail
The Evolution of A/B Testing
@Nicole_Mintiens

79. LARGEST LEAKS
Image source: https://s-media-cache-ak0.pinimg.com/564x/7e/c9/f9/7ec9f92ab4e7819c76011e432e760ece.jpg
The Evolution of A/B Testing
@Nicole_Mintiens

80. LARGEST LEAKS
Image source: https://s-media-cache-ak0.pinimg.com/564x/7e/c9/f9/7ec9f92ab4e7819c76011e432e760ece.jpg
The Evolution of A/B Testing
@Nicole_Mintiens

81. ANALYZE ENGAGEMENT PATTERNS
Heatmaps
The Evolution of A/B Testing
@Nicole_Mintiens

82. ANALYZE ENGAGEMENT PATTERNS
Heatmaps
The Evolution of A/B Testing
@Nicole_Mintiens

83. ANALYZE PERCEPTIONS
The Evolution of A/B Testing
@Nicole_Mintiens

84. ANALYSIS FRAMEWORK
G5 D-LITE
Deliver Relevance - Match then exceed user expectations.D
L
I
T
E
The Evolution of A/B Testing
@Nicole_Mintiens

85. ANALYSIS FRAMEWORK
G5 D-LITE
Lower Friction - Reduce user anxiety. Simplify their process.
Deliver Relevance - Match then exceed user expectations.D
L
I
T
E
The Evolution of A/B Testing
@Nicole_Mintiens

86. ANALYSIS FRAMEWORK
G5 D-LITE
Improve Clarity - Ensure messaging is clear, concise & valued.
Lower Friction - Reduce user anxiety. Simplify their process.
Deliver Relevance - Match then exceed user expectations.D
L
I
T
E
The Evolution of A/B Testing
@Nicole_Mintiens

87. ANALYSIS FRAMEWORK
G5 D-LITE
Improve Clarity - Ensure messaging is clear, concise & valued.
Lower Friction - Reduce user anxiety. Simplify their process.
Deliver Relevance - Match then exceed user expectations.
Trigger Action - Motivate conversion. Remove distraction.
D
L
I
T
E
The Evolution of A/B Testing
@Nicole_Mintiens

88. ANALYSIS FRAMEWORK
G5 D-LITE
Improve Clarity - Ensure messaging is clear, concise & valued.
Lower Friction - Reduce user anxiety. Simplify their process.
Deliver Relevance - Match then exceed user expectations.
Experiment - Learn, test and optimize for your users.
Trigger Action - Motivate conversion. Remove distraction.
D
L
I
T
E
The Evolution of A/B Testing
@Nicole_Mintiens

89. “A solid test hypothesis is an
INFORMED SOLUTION
to a real problem - not an
arbitrary guess.
”Michael Aagaard
@ContentVerve
International Speaker and
Senior CRO @Unbounce
The Evolution of A/B Testing
@Nicole_Mintiens

90. HYPOTHESIS TEMPLATE
If [cause], then [effect], because [rationale].
Source: https://help.optimizely.com/Ideate_and_Hypothesize/Design_a_hypothesis_that_drives_your_business_goals
The Evolution of A/B Testing
@Nicole_Mintiens

91. TESTING
The Evolution of A/B Testing
@Nicole_Mintiens

92. TESTING
Wait… do you even have enough traffic?
The Evolution of A/B Testing
@Nicole_Mintiens

93. Gives you CONFIDENCE that
the changes you make to
your website ACTUALLY
have a positive IMPACT.
https://www.optimizely.com/optimization-glossary/statistical-significance/
STATISTICAL SIGNIFICANCE
“
”
The Evolution of A/B Testing
@Nicole_Mintiens

94. MULTIFAMILY AVG SESSIONS/DAY
Source: 2016 G5 Multifamily Google Analytics Data
The Evolution of A/B Testing
@Nicole_Mintiens

95. MULTIFAMILY AVG SESSIONS/DAY
Source: 2016 G5 Multifamily Google Analytics Data
66
The Evolution of A/B Testing
@Nicole_Mintiens

96. TEST DURATION CALCULATOR
Source: https://www.convert.com/tools/ab-test-duration-calculator/
Image source: http://www.vintag.es/2016/09/the-good-old-days-12-crazy-vintage-ads.html
The Evolution of A/B Testing
@Nicole_Mintiens

97. TEST DURATION CALCULATOR
Source: https://www.convert.com/tools/ab-test-duration-calculator/
Image source: http://www.vintag.es/2016/09/the-good-old-days-12-crazy-vintage-ads.html
The Evolution of A/B Testing
@Nicole_Mintiens

98. LOW TRAFFIC OPTIMIZATION
Source: http://unbounce.com/landing-pages/i-dont-have-enough-traffic-to-ab-test-my-landing-pages/
The Evolution of A/B Testing
@Nicole_Mintiens

99. A/B TESTING TOOLS
The Evolution of A/B Testing
@Nicole_Mintiens

100. A/B TESTING TOOLS
The Evolution of A/B Testing
@Nicole_Mintiens

101. BUILD VARIATIONS
Use technical resources to ensure proper setup and tracking
The Evolution of A/B Testing
@Nicole_Mintiens

102. BUILD VARIATIONS
Use technical resources to ensure proper setup and tracking
The Evolution of A/B Testing
@Nicole_Mintiens

103. LAUNCH TEST
Original
(A)
Variation
(B)
The Evolution of A/B Testing
@Nicole_Mintiens

104. LAUNCH TEST
Original
(A)
Variation
(B)
The Evolution of A/B Testing
@Nicole_Mintiens

105. MEASURE RESULTS
Image source: https://www.optimizely.com/ab-testing/
The Evolution of A/B Testing
@Nicole_Mintiens

106. MEASURE RESULTS
Image source: https://www.optimizely.com/ab-testing/
The Evolution of A/B Testing
@Nicole_Mintiens

107. DRAW INSIGHTS AND ITERATE
The Evolution of A/B Testing
@Nicole_Mintiens

108. Your properties aren't 1-size fits all and
your strategy shouldn't be either.
DRAW INSIGHTS AND ITERATE
The Evolution of A/B Testing
@Nicole_Mintiens

109. MOST IMPORTANTLY
Research and
user feedback is
VITAL.
The Evolution of A/B Testing
@Nicole_Mintiens

110. REPEAT
GOALS
The Evolution of A/B Testing
@Nicole_Mintiens

111. THERE IS
NO
BEST COLOR
41 Shades
of Blue
The Evolution of A/B Testing
@Nicole_Mintiens

112. INDUSTRY
BENCHMARKS
• Contextless data
The Evolution of A/B Testing
@Nicole_Mintiens

113. INDUSTRY
BENCHMARKS
• Contextless data
• Weak conclusions
The Evolution of A/B Testing
@Nicole_Mintiens

114. INDUSTRY
BENCHMARKS
• Contextless data
• Weak conclusions
• Blind duplication
The Evolution of A/B Testing
@Nicole_Mintiens

115. INDUSTRY
BENCHMARKS
• Contextless data
• Weak conclusions
• Blind duplication
• Tactics over strategy
The Evolution of A/B Testing
@Nicole_Mintiens

116. USE STRATEGY,
NOT JUST TACTICS
• Avoid risks
The Evolution of A/B Testing
@Nicole_Mintiens

117. USE STRATEGY,
NOT JUST TACTICS
• Avoid risks
• Use data effectively
The Evolution of A/B Testing
@Nicole_Mintiens

118. USE STRATEGY,
NOT JUST TACTICS
• Avoid risks
• Use data effectively
• Discover solutions to
YOUR problems
The Evolution of A/B Testing
@Nicole_Mintiens

119. FREEMIUM (OR CLOSE) RESOURCES
• Research:
– Personas
– Tracking
– Heatmapping
– Test Duration Calculator
The Evolution of A/B Testing
@Nicole_Mintiens

120. FREEMIUM (OR CLOSE) RESOURCES
• Research:
– Personas
– Tracking
– Heatmapping
– Test Duration Calculator
• Analysis Tools:
G5 D-LITE CRO Framework
The Evolution of A/B Testing
@Nicole_Mintiens

121. FREEMIUM (OR CLOSE) RESOURCES
• Research:
– Personas
– Tracking
– Heatmapping
– Test Duration Calculator
• Analysis Tools:
G5 D-LITE CRO Framework
• Hypotheses:
If [cause], then [effect], because [rationale]
The Evolution of A/B Testing
@Nicole_Mintiens

122. FREEMIUM (OR CLOSE) RESOURCES
• Research:
– Personas
– Tracking
– Heatmapping
– Test Duration Calculator
• Analysis Tools:
G5 D-LITE CRO Framework
• Hypotheses:
If [cause], then [effect], because [rationale]
• Testing:
The Evolution of A/B Testing
@Nicole_Mintiens

123. “CRO IS LEARNING WHY
YOUR USERS aren’t
converting and FIXING IT.”
The Evolution of A/B Testing
@Nicole_Mintiens

124. GET THESE SLIDES:
goo.gl/TPUOgA
THANKS!
DON’T JUST TEST, OPTIMIZE.
The Evolution of A/B Testing
@Nicole_Mintiens