This document outlines Erin Zefkeles' process for A/B testing marketing campaigns at her company. She leads a core testing team of 3 people who review competitor sites, gather ideas from partners, and prioritize tests based on potential impact. Smaller markets are tested first for quick wins. Assumptions are verified with reality checks, and tests aim to solve business problems or improve customer experience. The process is nimble, with quick updates made based on customer feedback. Tests have resulted in up to 40% reduced expenses and more customer engagement on landing pages. Erin advocates for using Optimizely for small changes, coming up with a plan, being willing to make mistakes, continually testing and refining, and understanding how
3. A/B Testing on Our
Campaigns
Fixed design & branding
webpage template
Resource constrained
Optimizely & competitor
product available for us
Core Testing Team of 3
@ErinZef
4. Review competitor sites & our personal favorites sites
Monthly call for ideas from partner teams
Compiled list on SharePoint for people to add to
Review how much time each test will take
Prioritize based on country size & business impact
Ideation Process
@ErinZef
5. Smaller markets = quick wins, fast to implement
Review & Selection
@ErinZef
6. we got a 10% lift
or more in visitors
moving on in the
process by testing
CTA messages
Submit
vs
Real world example:
Test & Refine….
10. Assumption:
visitors prefer a
visually appealing
page with imagery
Reality Check:
It depends, results
differed by browser
type, audience and
market
Image vs. No Image
Original
Main Image
w/Simple Text
No Image
@ErinZef
11. Small quick wins, fast to implement
Verify our assumptions
Look to solve current business problems
Review & Selection
@ErinZef
15. Small quick wins, fast to implement
Verify our assumptions
Look to solve current business problems
Innovate
Review & Selection
Launch - Test - Analyze - Refine - Test - Grow
@ErinZef
16. Nimble
Taking quick action on customer feedback
Customer posted concern on Facebook
Determined landing page could be updated to avoid similar dissatisfaction
Went into Optimizely and immediately updated the page
Took minutes to improve the customer experience & help avoid negative PR
online
@ErinZef
18. Take Note
◊ Use Optimizely for the small changes pending in
your web team’s queue
◊ Come up with plan
◊ Be willing to make mistakes
◊ Test, refine & test again!
◊ Understand your funnel – what impact your test
has during each phase
◊ It’s so easy! If you don’t try how will you
determine what works?
@ErinZef
We started using Optimizely early last year and saw great results by testing the small stuff.
Brainstorm:
What pages do we love? How can we incorporate they’re features & test the impact?
For example….
Highlight
Continuous testing – compounding interest – continual increase (graph – slowing lift of one vs. another)
Quick wins as well as larger strategies….. However when I wanted to push the team for larger more creative ideas I asked them what Sheldon from the Bing Band Theory would ask……
Sheldon
ADD: quotation marks on the top left
Don’t assume your customers share your assumptions.
Recently forrester came out with a stat that said 50% of sales don’t happen because customers can’t find what they’re looking for. That leads me to believe that what we’re providing to customers isn’t what they’re seeking.
If you’re just starting with testing, step back and think to yourself what is one of the most simplistic assumption we’ve made, then try a quick A/B test to validate it’s correct.
Reality Check: It depends, results differed by browser type, audience and market
In UK – it varied by Browser – Chrome users preferred no image, while the other browsers where inconclusive
In FR – all browsers, preferred
Speaking to customers in a way they understand has helped us make drastic improvement (“Market Share” = SMB confusion) --- Keep it short & sweet
Move coupon form above the fold by reducing text
Quick wins,
What’s our biggest assumption?
This could be a simple as a CTA button or as complicated as a web scheme and navigation
What is a current business problem?
CTA button – large fraud – identified that traffic from India and Turkey drove the highest fraudulent requests
“Get started today” outperforms “Request a coupon”
Possibility to differentiate UK Geolocation from the potential fraudulent ones
Geo-location based on country --
After launching our tests, completing a post-analysis, then it’s all about analyzing, testing, and refining again
Helps our KPIs but also allows us to be Nimble – not wait weeks for a change.
Low priority small changes in web teams queue could be Web teams queue business days to make adjustments to a web page, preventing quick action to be taken based on customer feedback.
on Facebook he didn’t receive a coupon
Investigated & learned he was an existing customer, the coupon was offered only to new customers
after investigating determined landing page could be updated to better set expectations
Webpage needed to be updated to highlight it offer was only for new customers
Went to Optimizely, within minutes updated the webpage to include “for new customers only”
Took minutes instead of a day or two to improve the customer experience & help avoid negative PR online.
if you have a three step funnel, analysis what impact the landing page has to each step
Have a landing page that is rock solid you’ve already tested it and it works like a charm – still put the Optimizely script on there – if your web team can’t get around to changing something same day then you can quickly go into Optimizely and make the small content change while it’s in your web teams queue
Launch - Test - Analyze - Refine - Test - Grow
You can launch a quick test in minutes, it’s better than no test at all.