This presentation by Anna Marie Clifton, Product Manager at Yammer, covers the important topics of when to use A/B testing, how to implement it and most importantly, how to measure the results. The content is directed for software engineers who want to transition to product management, MBA's with finance/consulting background who wish to work high-tech companies as product managers and Project Managers, Marketers, and Designers who are seeking opportunities in product management.

1. A/B Testing with Yammer’s Product
Manager
www.productschool.com

2. FREE INVITE
Join 10,000+ Product Managers on
productschool.com/slack-community

3. Product
Management
Courses

4. Coding
for Managers
Courses

5. Data Analytics
for Managers
Courses

6. Include @productschool and #prodmgmt at the
end of your tweet
Tweet to get a free ticket
for our next Event!

7. Anna Marie Clifton
Tonight’s Speaker

8. Anna Marie Clifton
Product Manager at Yammer
Co-host of Clearly Product Podcast
All the ways that
A/B testing sucks

9. The Usual Story

10. Anna Marie Clifton
Product Manager, Yammer

11. ● @TweetAnnaMarie
○ Medium & Twitter
● Clearly Product
○ iTunes & Stitcher
Anna Marie Clifton
Product Manager, Yammer

12. 1. The rosy side of A/B testing
Our roadmap

13. 1. The rosy side of A/B testing
2. The grey side of A/B testing
Our roadmap

14. Our roadmap
1. The rosy side of A/B testing
2. The grey side of A/B testing
3. Somewhere in the middle

15. Our roadmap
1. The rosy side of A/B testing
2. The grey side of A/B testing
3. Somewhere in the middle

16. What is it?
1. The rosy side of A/B testing

17. New Signup — sequence test
1. The rosy side of A/B testing

18. ● 80% of users complete the sign up
New Signup — sequence test
1. The rosy side of A/B testing

19. ● 80% of users complete the sign up
● Release a new flow to all your users
New Signup — sequence test
1. The rosy side of A/B testing

20. ● 80% of users complete the sign up
● Release a new flow to all your users
● Measure again
New Signup — sequence test
1. The rosy side of A/B testing

21. ● 80% of users complete the sign up
● Release a new flow to all your users
● Measure again
● Now only 70% of users complete.
New Signup — sequence test
1. The rosy side of A/B testing

22. ● 80% of users complete the sign up
● Release a new flow to all your users
● Measure again
● Now only 70% of users complete.
New Signup — sequence test
1. The rosy side of A/B testing

23. New Signup — A/B test
1. The rosy side of A/B testing

24. ● 80% of users complete the sign up
New Signup — A/B test
1. The rosy side of A/B testing

25. ● 80% of users complete the sign up
● Release a new flow to 50% of your users — “Treatment”
New Signup — A/B test
1. The rosy side of A/B testing

26. ● 80% of users complete the sign up
● Release a new flow to 50% of your users — “Treatment”
● 70% of users in the treatment complete
New Signup — A/B test
1. The rosy side of A/B testing

27. ● 80% of users complete the sign up
● Release a new flow to 50% of your users — “Treatment”
● 70% of users in the treatment complete
● Meanwhile, only 60% of control users completed
New Signup — A/B test
1. The rosy side of A/B testing

28. ● 80% of users complete the sign up
● Release a new flow to 50% of your users — “Treatment”
● 70% of users in the treatment complete
● Meanwhile, only 60% of control users completed
New Signup — A/B test
1. The rosy side of A/B testing

29. Our roadmap
1. The rosy side of A/B testing
2. The grey side of A/B testing
3. Somewhere in the middle

30. Engineering
2. The grey side of A/B testing

31. Engineering issues:
2. The grey side of A/B testing

32. 1. Takes longer to build
Engineering issues:
2. The grey side of A/B testing

33. 1. Takes longer to build
2. Can’t make changes in flight
Engineering issues:
2. The grey side of A/B testing

34. Engineering issues:
2. The grey side of A/B testing
1. Takes longer to build
2. Can’t make changes in flight
3. Confusion about expected experience

35. Engineering issues:
2. The grey side of A/B testing
1. Takes longer to build
2. Can’t make changes in flight
3. Confusion about expected experience
4. Code cleanup isn’t always perfect

36. Product
2. The grey side of A/B testing

37. Product issues:
2. The grey side of A/B testing

38. 1. It can take Forever™
Product issues:
2. The grey side of A/B testing

39. 1. It can take Forever™
2. Need tons of users
Product issues:
2. The grey side of A/B testing

40. 1. It can take Forever™
2. Need tons of users
3. Lots of up front work
Product issues:
2. The grey side of A/B testing

41. 1. It can take Forever™
2. Need tons of users
3. Lots of up front work
4. Lots of after-the-fact work
Product issues:
2. The grey side of A/B testing

42. 1. It can take Forever™
2. Need tons of users
3. Lots of up front work
4. Lots of after-the-fact work
5. Geared toward small optimizations
Product issues:
2. The grey side of A/B testing

43. 1. It can take Forever™
2. Need tons of users
3. Lots of up front work
4. Lots of after-the-fact work
5. Geared toward small optimizations
6. Only useful in single-player mode
Product issues:
2. The grey side of A/B testing

44. 1. It can take Forever™
2. Need tons of users
3. Lots of up front work
4. Lots of after-the-fact work
5. Geared toward small optimizations
6. Only useful in single-player mode
7. Can’t be scientific
Product issues:
2. The grey side of A/B testing

45. 1. It can take Forever™
2. Need tons of users
3. Lots of up front work
4. Lots of after-the-fact work
5. Geared toward small optimizations
6. Only useful in single-player mode
7. Can’t be scientific
Product issues:
2. The grey side of A/B testing

46. Our roadmap
1. The rosy side of A/B testing
2. The grey side of A/B testing
3. Somewhere in the middle

47. Totally sucks
3. Somewhere in the middle

48. Still worth it
3. Somewhere in the middle

49. Questions?
@TweetAnnaMarie

50. Part-time Product Management Courses
in San Francisco
www.productschool.com