This document provides an overview of an optimization process for testing and improving marketing efforts. It discusses 13 steps in the process: 1) defining business goals, 2) understanding the target audience, 3) analyzing web data, 4) performing heuristic analysis, 5) capturing challenges, 6) forming hypotheses from challenges, 7) designing experiments, 8) creating variations, 9) building variations, 10) implementing experiments, 11) analyzing results, 12) reporting findings, and 13) determining next steps. Following a scientific process helps ensure statistically valid results that can add to the bottom line through reasonable testing efforts.
8. Process lets you…
1. Achieve valid, effective tests
2. Take full advantage of business
opportunities
9. Without Process With Process
Unclear, misleading results Statistically valid results
Results that don’t add $ Results that add to the bottom line
High effort, low return Reasonable effort, huge return
Opinion-based testing Data-driven testing
“Inconclusive” results (a costly tie) Clear winners and losers
10. Take advantage of opportunities
1. Time to market
– How fast can you get
improvements in place?
2. Test the right stuff, in the right
order
– Low Investment, high Return
12. Today, You Will…
1. Learn the Scientific Method…for
Marketers
2. Learn a proven Optimization
process
3. Discover 4 hidden BestPractices
nuggets
13. The Scientific Method…for Marketers
1. Define the problem
2. Research/observe the problem
3. Form a hypothesis
4. Conduct experiment
5. Analyze experiment results Look familiar?
6. Form a conclusion
7. Socialize results
16. 1) Business Goals
• In science-speak, this is “define
the problem”
• Resist temptation to:
– skip it
– assume
• Multiple goals OK
– prioritize
17. 1) Business Goals
• Example:
– Conversion Rate = primary business goal
– Forward to a friend engagement =
secondary
– Test for Conversion Rate
– Analyze/monitor F2F engagement
18. Hidden BestPractice Nugget #1:
Test as “close to” revenue as you can.
Testing further upstream can be
misleading.
20. 2) Understand Target Audience
• In science-speak, this is “research the
problem”
• Generally qualitative data (“market
research”)
– personas
– feedback forms
– focus groups
– social media listening
– chat transcripts
– call center monitoring
– demographics/psychographics
22. 3) Data Analysis (Quantitative)
• In science-speak, this is
“research the problem”
• Web Analytics
– High traffic points
– Decision/conversion points
– Persuasion points
– High abandonment points
• Survey Data
24. 4) Heuristic Analysis
• In science-speak, this is
“research the problem”
• Are “best practices” (rules of thumb)
applied?
• Are you at parity with competitors
and industry leaders?
25. 5) Capture Challenges (roadblocks)
No, not
this kind of
Roadblock.
Image credit: YoJoe.com
29. 6) Challenges » Hypotheses
• Every challenge/roadblock has
potential solution(s)
• Best guess, best solution =
hypothesis
• Ex: “Adding ‘social proof’
content will increase conversion
rate.”
30. Hidden BestPractice Nugget #2:
• Hypothesis: “Adding ‘social proof’
content will increase conversion rate.”
• Research Question: “Which version of
social proof content will perform
best?”
33. Hidden BestPractice Nugget #3:
“Confidence level” - the statistical probability
that your results are not due to chance.
Own your confidence level. Most tools are 95%.
Get buy-in on tool’s default or other confidence
level.
42. 11) Analyze Results
• Google Analytics integrations
– Optimizely, Visual Website Optimizer, etc.
43. 11) Analyze Results
• What does the result mean
about the target audience?
• Are the learnings applicable
elsewhere?
– Other sites?
– Other campaigns?
– Other channels?
46. 12) Report Findings
• If +, tie it to revenue
– “estimated”
• If -, what did you learn?
47. 13) Next Steps
• Install winning version
– You don’t make $ until you install
the winner!
– Production-ready code, SEO, etc.
• Follow-up test ideas
• Add to Marketing knowledge
base
49. Key Takeaways
1. Optimization Process is your friend
2. Optimization Process maps to science…
– …but bows to Marketing
3. Optimize your own Optimization efforts
– “…the best is the enemy of the good.”
Voltaire (1694–1778)