A/B Testing - What your mother didn't tell you
•
3 recomendaciones•7,276 vistas
Many people think A/B testing is merely showing customers two versions of something and choosing the one that customers respond to better. It's much, much more complex than that.
Recomendados
Recomendados
Más contenido relacionado
Similar a A/B Testing - What your mother didn't tell you
Similar a A/B Testing - What your mother didn't tell you (20)
Más de Curtis Poe
Más de Curtis Poe (19)
Último
Último (20)
A/B Testing - What your mother didn't tell you
- 1. January 14, 2014 P.O.P. Copyright 2013, All Around The World
- 2. A/B TesAng What your mother didn’t tell you hDp://www.allaroundtheworld.fr/ January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
- 3. What is A/B TesAng? • TesAng customers, not code • Show different versions • Pick the best one January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
- 4. Who Relies On A/B TesAng? January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
- 5. Transform Your Company • Test your customers, not your code • Stop guessing what works January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
- 6. Should your paycheck depend on my hunch? January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
- 7. Conversion Funnel January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
- 8. Conversion Funnel → January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
- 9. A Basic Test • • • • Your current behavior is the base A variant is a new version ( just one change) You can have one or more variants But we assume only one for this talk January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
- 10. January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
- 11. Which Generates More Clicks? January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
- 12. Which Generates More Clicks? 10,017 Visits 125 Conversions January 14, 2014 9,983 Visits 132 Conversions © 2014 hDp://www.allaroundtheworld.fr/
- 13. Which Generates More Clicks? 9,950 Visits 125 Conversions January 14, 2014 10,550 Visits 155 Conversions © 2014 hDp://www.allaroundtheworld.fr/
- 14. The Numbers (95% Confidence) Version Visits Conversion Rate Not Random Red 10,017 125 1.25% — Blue 9,983 132 1.32% 51% hDp://getdelve.com/2011/12/math-‐behind-‐ab-‐tesAng-‐for-‐curious-‐execuAves-‐part-‐1/ January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
- 15. The Numbers (95% Confidence) Version Visits Conversion Rate Not Random Red 9,950 125 1.26% — Blue 10,550 155 1.47% 94% hDp://getdelve.com/2011/12/math-‐behind-‐ab-‐tesAng-‐for-‐curious-‐execuAves-‐part-‐1/ January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
- 16. Which Sells More Product? January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
- 17. Which Creates More Profit? January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
- 18. What Confidence Level? Is 99% beDer than 90%? January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
- 19. The Cycle January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
- 20. Create Experiment Only one difference per version! January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
- 21. Roll the Dice January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
- 22. Track Performance Sta:s:cs January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
- 23. Choose the Best Pa:ence January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
- 24. Create Experiment Wash, rinse, repeat January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
- 25. Choose the Best • • • • Ignore the first day Full-‐business cycle What if you’re wrong? False posiAves hDp://www.einarsen.no/is-‐your-‐ab-‐tesAng-‐effort-‐just-‐chasing-‐staAsAcal-‐ghosts/ January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
- 26. LimitaAons of A/B TesAng • Doesn’t tell you why • Side effects • Doesn’t tell you loyalty January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
- 27. Benefits • • • • • • • • Hyundai: 62% increase in test drive requests Artsy Editor: 47% increase in click-‐throughs Visual Website OpAmizer: 50% signup increase AssessmentDay: 62% increase in sales SmileyCookie: 41% increase in sales BaseKit: 25% increase in conversions Mobal: 27% increase in sales You Need A Budget: 85% increase in downloads hDp://visualwebsiteopAmizer.com/case-‐studies.php
- 28. What Are They TesAng? • • • • • • • • Add-‐to-‐cart buDon placement Call to acAon buDons Using popup forms Removing content Link color Adding tesAmonials NavigaAon menus Removing signup forms January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
- 29. Gemng Started • • • • Vendor or in-‐house? Go for quick wins Run an AA test Be pa:ent! January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
- 30. Problems Ego January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
- 31. Problems • • • • • • • Forgemng caching Non-‐random selecAon Forgemng selecAon TesAng only the view Performance Dead code Profit versus conversion January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
- 32. Tips • • • • • Most experiments fail Don’t roll the dice prematurely A/B test bugs Test “stupid” ideas Don’t require rollouts! January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
- 33. SegmentaAon • • • • • • New versus returning customers Different browsers Accept-‐Language Time of day IP country Gender? Age? Something else? January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
- 34. See Also • The MulA-‐Armed Bandit hDp://stevehanov.ca/blog/index.php?id=132 January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
- 35. Conclusion • • • • • Lots of money lying on the table Egos are bad Data is good Customer behavior trumps expert opinion Customer behavior trumps code behavior January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
- 36. QuesAons? hDp://www.allaroundtheworld.fr/ January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/