ClickZ Live -- Chicago 2014: 3 Common Testing Mistakes (and how to avoid them)

November 3–6, 2014
#CZLCHI | @ClickZLive
The Global Conference Series Designed by Digital Marketers, for Digital Marketers
3 Common Mistakes in Online Testing
(And How to Avoid Them)
Taylor Kennedy
MECLABS
Senior Manager, Optimization & Strategy

@OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
30%
16%
Do you understand your results?
60%
12%
43%
8%
62%
6%

@OptimizeDude
30%
16%
60%
12%
43%
8%
62%
6%

What does it mean?

@OptimizeDude
You’re (probably) testing wrong if you:
•
Are stuck optimizing a part of your funnel that lacks significant impact on the bottom line
•
Have been blindly following “best practices”
•
Are not further establishing customer theory with each test

@OptimizeDude
Our testing process

Case Study
Optimizing the wrong part of the funnel

@OptimizeDude
Cart Test 1: Control

@OptimizeDude
Cart Test 1: Treatment

@OptimizeDude
Cart Test 1: Results
Results were flat.
Shopping Cart
Conversion Rate
Rel. Difference
Level of Confidence
Control
14.3%
NA
---
Treatment
13.5%
-4.99%
44%
?

@OptimizeDude
Cart Test 2: Control

@OptimizeDude
Cart Test 2: Treatment

@OptimizeDude
Cart Test 2: Results
Results were flat, even trending downward.
Shopping Cart
Conversion Rate
Rel. Difference
Level of Confidence
Control
13.7%
NA
---
Treatment
11.1%
-19.26%
88%
?

@OptimizeDude
Why aren’t tests getting results?
5.7%
41%
All visitors
Add to cart
Payment info
Convert
40%

@OptimizeDude
Through proper planning you:
Identifythe highest impact area of your site to test.
Findpotential validity threats linked to normal traffic that may impact results.
Discover visitor behavior that helps you create long-term test plans.

@OptimizeDude
Product Page 1: Control

@OptimizeDude
Product Page 1: Treatment

@OptimizeDude
Product Page Test 1: Results
•
“Add to Cart” increased by over 15%
•
Conversion increased by 40%
Shopping Cart
Conversion Rate
Rel. Difference
Level of Confidence
Control
1.07%
NA
---
Treatment
1.50%
40.05%
98%
!

@OptimizeDude
What motivated the cart test?
A.
Needed improvements were obvious
B.
Our client or boss told us to
C.
Testing close to the conversion event has the biggest ability to increase revenue
D.
The data told us to

You’ve been following “best practices”
Case study examples

@OptimizeDude
How are successful and unsuccessful ecommerce companies approaching testing?
MarketingSherpa asked marketers:

@OptimizeDude

@OptimizeDude
Your peers are following best practices too.

@OptimizeDude
How often do you hear the following?
“Keep your call-to-action above the fold”
“Video converts”
“Use a compelling image”
“Mobile-friendly is critical”
“Include privacy and security statements to establish trust”

@OptimizeDude
Security Seal Test: Control

@OptimizeDude
Security Seal Test: Treatment 1

@OptimizeDude

@OptimizeDude
Security Seal Test: Results
Shopping Cart
ClickthroughRate
Rel. Difference
Level of Confidence
Control
66.14%
NA
---
Treatment 1
66.95%
1.2%
50%
Treatment 2
65.00%
-1.7%
57%
Treatment 3
64.60%
-2.3%
54%
Results were flat.
?

@OptimizeDude
Mobile Test: Control

@OptimizeDude
Mobile Test: Treatment

@OptimizeDude
Mobile Redesign Test: Results
Results were significantly negative.
Shopping Cart
Conversion Rate
Rel. Difference
Level of Confidence
Control
0.5%
NA
---
Treatment
0.3%
-40.75%
96%
!

Redefine best practice:
•
There are no shortcuts to customer wisdom.
•
Just because a test worked for someone else, doesn’t mean it will work for you.
•
Rigorous, methodology-driven, hypothesis testing ensures you are generating real customer insights.

@OptimizeDude
Framework
©
C = 4m + 3v + 2(i-f) -2a

@OptimizeDude
Framework
©
C= 4m + 3v + 2(i-f) -2a
Probability of conversion

@OptimizeDude
Framework
©
C = 4m+ 3v + 2(i-f) -2a
Motivation of user

@OptimizeDude
Framework
©
C = 4m + 3v+ 2(i-f) -2a
Clarity of value proposition

@OptimizeDude
Framework
©
C = 4m + 3v + 2(i-f)-2a
Incentive -Friction

@OptimizeDude
A bad hypothesis:
“By reducing friction on the page associated with product selection, we will see an increase in clickthrough.”
•
Unclear link to a particular problem
•
Not related to specific elements on a particular page or process
•
Not linked to information or data about the page

@OptimizeDude
A bad hypothesis:
“By reducing friction on the page associated with product selection, we will see an increasein clickthrough.”
•
Unclear link to a particular problem
•
Not related to specific elements on a particular page or process
•
Not linked to information or data about the page

@OptimizeDude
A good hypothesis:
CHANGING __________________ INTO__________________
WILL_______________________
What your analysis indicates is the problem
What change you think will solve the problem
The effect it will have on your KPI

@OptimizeDude
A good hypothesis:
“Changing the presentation of products from a single column format into a dual column format will increase the number of product options visible and boost clickthrough to the product details page”

@OptimizeDude
A good hypothesis:
“Changing the presentation of products from a single column format into a dual column format willincrease the number of product options visible and boost clickthrough to the product details page”

@OptimizeDude
A good hypothesis:
“Changing the presentation of products from a single column format into a dual column format will increase the number of product options visible and boost clickthrough to the product details page.”

@OptimizeDude
A good hypothesis is:
•
A proposed statement
•
Made on the basis of limited evidence related to a specific problem
•
Can be proved or disproved
•
Used as a starting point for further investigation

@OptimizeDude
Motivation Test: Control

@OptimizeDude
Motivation Test: Control
Hypothesis:
Changing the right-column form intoa call-to- action requiring no personal information will reduce friction and boost clickthrough to the subsequent page.

@OptimizeDude
Removed form completely
Treatment 1:

@OptimizeDude
Removed form, added starting price
Treatment 2:

@OptimizeDude
Kept form, added starting price
Treatment 3:

@OptimizeDude
Homepage Motivation Test: Treatments
Shopping Cart
Rel. CTR
Level of Confidence
Control
--
---
Treatment 1
202%
99%
Treatment2
149%
99%
Treatment 3
-20%
99%

@OptimizeDude
Remember:
•
Proper planning allows you to run tests with clear outcomes predetermined in advance
•
Formulate test sequences that generate big-picture customer analysis
!

Key takeaways
Follow a “testing process.” Don’t just “test.”
1.
Always perform at least a basic site data analysis (funnel analysis) before beginning testing.
2.
Analyze each page through the mind of your prospect to identify areas of opportunity for testing.
3.
Formulate hypotheses that enable you to learn from each and every test. That will lead you to a more complete picture of your ideal prospect.

http://bit.ly/OptimizeTK
Presentation Link

ClickZ Live -- Chicago 2014: 3 Common Testing Mistakes (and how to avoid them)

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a ClickZ Live -- Chicago 2014: 3 Common Testing Mistakes (and how to avoid them)

Similar a ClickZ Live -- Chicago 2014: 3 Common Testing Mistakes (and how to avoid them) (20)

Más de MarketingExperiments

Más de MarketingExperiments (20)

Último

Último (20)

ClickZ Live -- Chicago 2014: 3 Common Testing Mistakes (and how to avoid them)

Notas del editor