Benchmark Your Optimization Program
❯
Featuring speakers from Optimizely
Shana Rusonis, Content Marketing Manager, Optimizely
What does 'normal' look like for an optimization program? What about best-in-class? What are the opportunities and challenges facing teams that are optimizing, and how do they allocate their manpower and budget towards them? We sought out the answers to these tough questions for our first industry benchmark report. Join Optimizely for an in-depth discussion of how to quantify and benchmark optimization performance.
7. #opticon2015@srusonis
Benchmark Methodology
n = 417
1/4 Large Enterprise
1/2 Mid-Size to
Small Enterprise
1/4 Small Businesses
Company Size
1/4 Executives
1/2 Managers
1/4 Associate,
Consultants
Level
Marketers
Product Managers
Engineers
Titles
8. #opticon2015@srusonis
94%
Improves my understanding of my customer’s needs and desires
Helps me to deliver relevant content to my customers
6%
Do you agree or disagree with the following statements
about optimization?
93%
Helps me to deliver relevant content to my customers
Increases my conversion and engagement rates
7%
Do you agree or disagree with the following statements
about optimization?
87%
Allows me to take action on data quickly
Has created a culture of learning and experimentation at my company
13%
Do you agree or disagree with the following statements
about optimization?
87%
Has created a culture of learning and experimentation at my company
Improves the creative output of my team
13%
Do you agree or disagree with the following statements
about optimization?
91%
Increases my conversion and engagement rates
9%
Do you agree or disagree with the following statements
about optimization?
88%
Improves the ROI of my marketing campaigns
Allows me to take action on data quickly
12%
Do you agree or disagree with the following statements
about optimization?
80%
Improves the creative output of my team
Improves my team’s speed to execute on ideas
20%
Do you agree or disagree with the following statements
about optimization?
98%
94%
93%
91%
88%
87%
87%
80%
76%
Is valuable to the success of my marketing campaigns and/or product strategy
Improves my understanding of my customer’s needs and desires
Helps me to deliver relevant content to my customers
Increases my conversion and engagement rates
Improves the ROI of my marketing campaigns
Allows me to take action on data quickly
Has created a culture of learning and experimentation at my company
Improves the creative output of my team
Improves my team’s speed to execute on ideas
6%
7%
9%
12%
13%
13%
20%
24%
Agree
Disagree
Do you agree or disagree with the following statements
about optimization?
98% of optimizers agree that it’s valuable to their success
98%
Is valuable to the success of my marketing campaigns and/or product strategy
Do you agree or disagree with the following statements
about optimization?
9. #opticon2015@srusonis
… Along with many other qualitative and quantitative benefits
94%
Improves my understanding of my customer’s needs and desires
Helps me to deliver relevant content to my customers
6%
Do you agree or disagree with the following statements
about optimization?
87%
Has created a culture of learning and experimentation at my company
Improves the creative output of my team
13%
Do you agree or disagree with the following statements
about optimization?
91%
Increases my conversion and engagement rates
9%
Do you agree or disagree with the following statements
about optimization?
88%
Improves the ROI of my marketing campaigns
Allows me to take action on data quickly
12%
Do you agree or disagree with the following statements
about optimization?
80%
Improves the creative output of my team
Improves my team’s speed to execute on ideas
20%
Do you agree or disagree with the following statements
about optimization?
Quantitative Qualitative
10. #opticon2015@srusonis
Increase
57%
61%
43%
How do you plan to change the amount of budget allocated to
optimization in the next 12 months?
Optimization budgets are set to increase
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
55%
53%
4%
43%
Increase
Decrease
No change
How do you plan to change the amount of budget allocated to
optimization in the next 12 months?
11. #opticon2015@srusonis
Increase
57%
61%
43%
How do you plan to change the amount of budget allocated to
optimization in the next 12 months?Executives & Managers are more certain of budget increases
• 57% of Executives reported a
planned increase
• 61% of Managers reported a
planned increase
Executive
Manager
Individual
et allocated to
12. “Optimization has become even
more crucial in today's market,
so more budget is required to
make a position in the market.”
– Anonymous Survey Respondent
17. “I think the single best thing to do is to
allocate headcount towards optimization.
If you can swing it, hire a dedicated person.
If you can’t, get everyone on the team
(designers, engineers, product managers)
evangelized. Let everyone run an experiment
so they can see how mission critical it is.”
– Kyle Rush, Director of Frontend
Engineering and Optimization,
Hillary for America
18. #opticon2015@srusonis
Optimization draws from a wide range of teams and skills
Marketing
62%
Design
53%
UI or Front-End Development
40%
Product
36%
Engineering
35%
E-commerce
31%
Sales
30%
Customer Service or Support
23%
Executive Staff
20%
Which teams are involved with optimization at your company?
Marketing
32%
E-commerce
14%
UI or Front-End Development
12%
Product
12%
Engineering
9%
Design
8%
Executive Staff
5%
Other
3%
Customer Service or Support
3%
Sales
Which team leads optimization at your company?
23. “If you can’t state your reason
for running a test simply, then
you probably need to examine
why and what you are testing.”
.”
– Brian Schmitt, CROmetrics
24. #opticon2015@srusonis
Analytics are the top input for optimization
Analytics
72%
Results from Previous Experiments
57%
Customer Surveys
51%
User Research Studies
50%
A/B Testing Case Studies
44%
Support Tickets or Live Chat Feedback
38%
Heat Maps
28%
Which of the following inform your optimization strategy?
Analytics
72%
Which of the following inform your optimization strategy?
Results from Previous Experiments
57%
Which of the following inform your optimization strategy?
Customer Surveys
51%
Which of the following inform your optimization strategy?
User Research Studies
50%
Which of the following inform your optimization strategy?
A/B Testing Case Studies
44%
Which of the following inform your optimization strategy?
A/B Testing Case Studies
44%
Which of the following inform your optimization strategy?
Support Tickets or Live Chat Feedback
38%
Which of the following inform your optimization strategy?
Heat Maps
28%
Which of the following inform your optimization strategy?
25. #opticon2015@srusonis
… Along with other qualitative inputs and sources of inspiration
Customer Surveys
51%
Which of the following inform your optimization strategy?
User Research Studies
50%
Which of the following inform your optimization strategy?
Support Tickets or Live Chat Feedback
38%
Which of the following inform your optimization strategy?
Heat Maps
28%
Which of the following inform your optimization strategy?
Results from Previous Experiments
57%
Which of the following inform your optimization strategy?
A/B Testing Case Studies
44%
Which of the following inform your optimization strategy?
Qualitative Inputs
Institutional &
Industry Knowledge
27. #opticon2015@srusonis
Optimizers prioritize for revenue, customer satisfaction
Potential to increase the value of each customer interaction, or increase overall revenue
65%
The impact on ease of use and satisfaction for my customer
58%
Improvement to website or app stability or performance
48%
The inefficiencies that currently exist in my marketing and product funnels
33%
The cost to acquire customers on various platforms
31%
We don’t do this kind of prioritization
2%
How do you prioritize which digital interactions with your audience
require optimization?
28. #opticon2015@srusonis
Optimizers care deeply about the customer experience,
in addition to positive ROI.
The values we use to
prioritize our tests are a
snapshot of why we choose
to optimize in the first place.
29. “The way your customers use
your product and the way they
experience it … is everything.”
”
.”
—Lev Tatarov, Marktplaats, eBay
30. #opticon2015@srusonis
Research to create
solid experiment
hypotheses
Processes help scale optimization efficiency
Prioritize
backlog of
experiment ideas
Build, QA,
and launch
experiments
OPTIMIZATION
LEAD
31. #opticon2015@srusonis
Optimizers are testing a variety of elements,
each with different degrees of complexity
User Experience Components (Onboarding Patterns, Search Functionality, Modals, etc.)
63%
Landing Pages
61%
Messaging and Copy
57%
Forms
54%
Offers and Promotions
42%
Recommended Content Functionality
38%
Social Sharing Functionality
37%
Payment Functionality
37%
Other
What do you test when you are optimizing?
User Experience Components (Onboarding Patterns, Search Functionality, Modals, etc.)
63%
Landing Pages
61%
Messaging and Copy
57%
Forms
What do you test when you are optimizing?
32. #opticon2015@srusonis
What you’re likely to test depends on which industry you’re in
• B2B more likely to test
landing pages
E-Commerce B2B Media
65%
74%
67%
65%
Landing Pages
Messaging and Copy
What do you test when you are optimizing?
51%
46%
53%
44%
Offers and Promotions
What do you test when you are optimizing?
What do you test when you are optimizing?
32%
36%
42%
50%
Social Sharing Functionality
Payment Functionality
What do you test when you are optimizing?
• E-commerce more likely to
test offers, payments
• Media more likely to test
social sharing
33. #opticon2015@srusonis
Multiple Times per Week
Weekly
Multiple Times per Month
Monthly
Multiple Times per Year
37%
21%
15%
12%
9%
Approximately how often do you run A/B or multivariate tests to opti-
mize your website, mobile app, or other digital channels?
Multiple Times
per Week
Multiple Times
per Month
58% of optimizers are testing at least weekly
35. #opticon2015@srusonis
Research to create
solid experiment
hypotheses
Processes help scale optimization efficiency
Prioritize
backlog of
experiment ideas
Build, QA,
and launch
experiments
Conclude and
analyze experiments,
share results OPTIMIZATION
LEAD
36. #opticon2015@srusonis
Measuring optimization success is multi-faceted
Conversion Improvements from Experiments
58%
Revenue Generated from Experiments
52%
ROI (Combination of Conversion Improvement and Revenue)
49%
Output or Velocity of Experiments
37%
Number of Team Members or Departments Involved
25%
Other
1%
0 5 10 15 20 25 30 35 40 45 50 55
How do you measure the success of your optimization program?
41. #opticon2015@srusonis
• Evaluate the makeup of your team; focus
on communication and removing
bottlenecks
• Diversify your inputs for optimization
beyond analytics
• Maintain a relentless focus on process
to improve experiment quality and
quantity
Conversation starters and next steps
50. #opticon2015@srusonis
We all have challenges …
• Maximizing ROI
• Delivering custom content
• Making quick decisions
• Understanding, taking action
on data
0 10 20 30 40 50 60 70 80 90 100
20%
24%
24%
27%
30%
31%
31%
Maximize the ROI of my resources
Deliver customized experiences to customer segments
Make quick decisions
Take action on company data
Embrace the creative talents of my team
Make data-driven decisions
Understand company data
37%
26%
35%
34%
35%
39%
35%
31%
26%
23%
31%
24%
21%
27%
12%
15%
15%
6%
10%
8%
6%
9%
Very Good Good OK Not Good Very Bad