Everything You Need to Know About A/B Testing for Lean Startups
1. A/B Testing for Lean Startups
Everything you ever Secrets from the
wanted to know experts revealed
about split testing.
Watch Presentation Learn More
Pete Mauro @mauronic
Mucker Labs, February 27, 2012
13. Let’s pretend we work at Facebook
Ask Question How can we stop more people from deactivating their
Facebook account?
Do research The reason why people use Facebook is because their friends
are on Facebook.
Hypothesis If you remind people about their friends who are being left
behind, we will lower the deactivation rate.
19. 10
Quiz: Guess the Winner Points
A
B
Source: http://www.abtests.com/
✔ +60% signups
20. Designing a Test
• Anyone at the company should contribute
• Ideas can come from user testing
• Use best practices
• Be creative, try crazy ideas
• Use common sense: don’t piss off your users
for the sake of metrics
25. Further Reading
• Metrics Driven Design (See Joshua Porter on
Slideshare)
• The New Science of Product Development
(See James Birchler on Slideshare)
• Quora: “What are the most unexpected things
people have learned from A/B tests?”
• Abtests.com & whichtestwon.com
The marketing people felt that they needed to modernize their imageTropicana hired Arnell Group, a subsidiary of Omnicom Group, to reimagine the packaging
His team spent 6 months and tens of thousands of dollars to develop this new package
Customers hated itAd agency CEO defends design in a press conference saying “emotionally it is very difficult for people to grasp this bold new change”Two moths after introduction its pulled from the shelves
Eric Ries’ big ideaProgress is not measured by what you make, but what you learn.Goal: spin through this loop as fast as possible
Steve blank says get out of the buildingYour opinion doesn’t matter, your customer’s does
Just because we use this method does not be that we lack vision or creativity
Lets look at a case study
Your hypothesis must be measurable
Millions of users retainedFacebook likes to say “the code wins”. What this really means is let’s stop arguing about what we think is right and test it with our users.
Kiss metrics video anecdote
Can you even tell what the difference is?
Just about every conversion goal that tracked showed a dramatic improvement. Signups for free trial: +60%Engagement rate (inverse of bounce rate): +20%Visit to signup page: +30%Visit to homepage: +13%Visit to blog main page: +78%
Tests are powerful because opinions don’t matter anymore Don’t piss off users: Appsumo pop-up, signup on blog post, ipad app download interstitial on
This is testing gone astray~ 2.5% of users each got a shade for 2 weeksDoug Bowman cited this test as epitomizing the reason why he left as head designer at google
Don’t get caught in a micro-optimization loopBig gains come from bold changes
Nowadays there is no excuse not to A/B testEach of these sites have great resources on things like landing page best practices
Web based testing software is ridiculously easy to use now