A testing playbook combines the best practices of testing and optimization, along with communication strategies, education, and gaining buy-in from your client. Andrew Richardson, Senior Director of Analytics at Delphic Digital, provides a peek behind the curtain to reveal how Delphic prioritizes tests, recruits/trains/staffs-up for a testing practice, and moved from A/B to multivariate testing. Come with and open mind, walk away with a Testing Playbook Template you can put to use at once.
3. This is your
BRAIN
An attention span of
8 seconds
*https://advertise.bingads.microsoft.com/en-us/insights @analyticandrew #eMetrics
4.
5. IS MY CONVERSION
RATE GOOD OR BAD?
WHY DO I NEED TO
TEST?
BUT NO ONE WANTS TO
DO TESTING AT MY
ORGANIZATION.
PLATFORM, PEOPLE,
PROCESS
WHAT DO I TEST AND
HOW DO I TRACK IT?
TIPS AND
SUGGESTIONS FOR
GETTING STARTED
TAKEAWAYS
@analyticandrew #eMetrics
6. IS MY CONVERSION
RATE GOOD OR BAD?
WHY DO I NEED TO
TEST?
BUT NO ONE WANTS TO
DO TESTING AT MY
ORGANIZATION.
PLATFORM, PEOPLE,
PROCESS
WHAT DO I TEST AND
HOW DO I TRACK IT?
TIPS AND
SUGGESTIONS FOR
GETTING STARTED
TAKEAWAYS
@analyticandrew #eMetrics
10. IS MY CONVERSION
RATE GOOD OR BAD?
WHY DO I NEED TO
TEST?
BUT NO ONE WANTS TO
DO TESTING AT MY
ORGANIZATION.
PLATFORM, PEOPLE,
PROCESS
WHAT DO I TEST AND
HOW DO I TRACK IT?
TIPS AND
SUGGESTIONS FOR
GETTING STARTED
TAKEAWAYS
@analyticandrew #eMetrics
14. IS MY CONVERSION
RATE GOOD OR BAD?
WHY DO I NEED TO
TEST?
BUT NO ONE WANTS TO
DO TESTING AT MY
ORGANIZATION.
PLATFORM, PEOPLE,
PROCESS
WHAT DO I TEST AND
HOW DO I TRACK IT?
TIPS AND
SUGGESTIONS FOR
GETTING STARTED
TAKEAWAYS
@analyticandrew #eMetrics
21. IS MY CONVERSION
RATE GOOD OR BAD?
WHY DO I NEED TO
TEST?
BUT NO ONE WANTS TO
DO TESTING AT MY
ORGANIZATION.
PLATFORM, PEOPLE,
PROCESS
WHAT DO I TEST AND
HOW DO I TRACK IT?
TIPS AND
SUGGESTIONS FOR
GETTING STARTED
TAKEAWAYS
@analyticandrew #eMetrics
26. IS MY CONVERSION
RATE GOOD OR BAD?
WHY DO I NEED TO
TEST?
BUT NO ONE WANTS TO
DO TESTING AT MY
ORGANIZATION.
PLATFORM, PEOPLE,
PROCESS
WHAT DO I TEST AND
HOW DO I TRACK IT?
TIPS AND
SUGGESTIONS FOR
GETTING STARTED
TAKEAWAYS
@analyticandrew #eMetrics
35. THE PEOPLE
The most important thing
one: focus efforts where they matter most
two: avoid conflict-put opinions to the test
three: empower people to optimize & be data-drive
four: align teams around shared goals
five: collaborate around hypothesis generation,
test
ideas & execution
38. RESEARCH
1.
“If A/B tests are used in lieu of research, the variations
are essentially guesses. We can improve outcomes of
A/B tests by incorporating UX research to:
•improve cause identification
•develop more realistic hypotheses
•identify more opportunities for experimentation”
Nielsen Norman Group: www.nngroup.com
44. A theory can be
proved by
experiment; but no
path leads from
experiment to the
birth of a theory.
45. RESEARCH
1.
Issue
• What is the
conversion
problem?
Cause
Theory
• What do you
think is causing
the problem?
Variation
Hypotheses
• What do you
propose
changing?
Goals
• What outcomes
are expected?
46. RESEARCH
1.
Issue
• What is the
conversion
problem?
Cause
Theory
• What do you
think is causing
the problem?
Variation
Hypotheses
• What do you
propose
changing?
Goals
• What outcomes
are expected?
48. RESEARCH
1.
User-Experience to Improve Optimization Testing
• Defining user intent and objections
• Why did you visit? Were you successful? If not, why?
• Exposing interface flaws
• Changing colors vs. changing form fields
• Measuring findability
• Tree testing, site search, heatmaps
• Cleaning up design variations
• Reduce friction, maximize testing
50. PRIORITIZE
2.
• TEST
• Where you put stuff for testing
• INSTRUMENT
• Fixing, adding, improving tagging, events, etc.
• HYPOTHESIZE
• Found something, but no clear solution
• JUST DO IT (JDI)
• No brainers
• INVESTIGATE
• Need to dig further, ask more questions
Craig Sullivan: @optimiseordie
52. PRIORITIZE
2.
Craig Sullivan Hypothesis Kit
one: Because we saw (data/feedback)
two: We expect that (change) will cause (impact)
three: We’ll measure this using (data metric)
https://medium.com/@optimiseordie/hypothesis-kit-2-eff0446e09fc
53. PRIORITIZE
2.
Craig Sullivan Advanced Kit
one: Because we saw (qual and quant data)
two: We expect that (change) for (population)
will cause (impact(s))
three: We except to see (data metric(s)
change) over a period of (x business
cycles)
https://medium.com/@optimiseordie/hypothesis-kit-2-eff0446e09fc
59. IS MY CONVERSION
RATE GOOD OR BAD?
WHY DO I NEED TO
TEST?
BUT NO ONE WANTS TO
DO TESTING AT MY
ORGANIZATION.
PLATFORM, PEOPLE,
PROCESS
WHAT DO I TEST AND
HOW DO I TRACK IT?
TIPS AND
SUGGESTIONS FOR
GETTING STARTED
TAKEAWAYS
@analyticandrew #eMetrics
60. • Headlines
• Sub headlines
• Paragraph Text
• Testimonials
• Call to Action text
• Call to Action Button
• Links
• Images
• Content near the
fold
• Social proof
• Media mentions
• Awards and badges
63. IS MY CONVERSION
RATE GOOD OR BAD?
WHY DO I NEED TO
TEST?
BUT NO ONE WANTS TO
DO TESTING AT MY
ORGANIZATION.
PLATFORM, PEOPLE,
PROCESS
WHAT DO I TEST AND
HOW DO I TRACK IT?
TIPS AND
SUGGESTIONS FOR
GETTING STARTED
TAKEAWAYS
@analyticandrew #eMetrics