This document outlines an approach for continuous experimentation and improvement. It recommends starting with data to identify high traffic, low conversion areas to improve. Form hypotheses about potential solutions and test them through small experiments. Document learnings to establish best practices and avoid regressions. The goal is to continuously run small tests to drive learning and improvements.

1. Always Be Testing
Learn from every thing you do!
Hiten Shah
hnshah@gmail.com

2. @hnshah #FILIVE

3. Product, Design, Engineering, Marketing, Growth, Sales
What do you do?
 

5. How do you make
all the things
be er?!
Figuring this out is your job.

6. A/b testing

7. Only
1
out
of
5
tests
win.
4
out
of
5
tests
cost
you
money.

8. So, why experiment if so many tests fail?

9. Go Where Customers Are!
Long, slow, SaaS ramp of death from 2006…

10. Go Where Customers Are!
Now…

11. Go Where Customers Are!
Growth can be sped up!

12. Increase Hello Bar’s installation rate

13. Improve the KISSmetrics features page

14. Increase signups from Crazy Egg’s homepage

15. Grow Crazy Egg’s revenue

16. Results from 3 months of focused growth effort

17. Results: KISSmetrics signups over time

18. Simple 4-step experimentation process
Data
Figure out what to improve
Hypothesis
Make an educated guess
Experiment
Test your guess
Act
Decide what’s next

19. Data
Figure out what to improve

20. Data doesn’t make decisions for you.

21. Data informs your decision making.

22. Data helps you identify the most
painful problems people have.
(not the ones that bother you the most)

23. Find problems worth solving by
starting with quantitative data.

24. When in doubt, look at your top pages. Usually the homepage.

25. A simple framework for choosing what to improve

26. Find pages with high traffic and high bounce rate
HTTP://GOOGLE.COM/ANALYTICS

27. Find pages with high traffic and low conversion rate
HTTP://UNBOUNCE.COM

28. Find sources with high traffic and low conversion rate
HTTP://HUBSPOT.COM

29. Find opportunities based on user paths
HTTP://KISS.LY/KMPATH

30. Increase Conversions
By A/B Testing
Do a Happy Dance
But Watch it Closely
Get More Traffic
So You Can Start A/B Testing
Get More Traffic
So You Can Do a Happy Dance
High Traffic
Low Traffic
High ConversionsLow Conversions
How To Figure Out What To Improve

31. What percentage of your
customers or users have
adopted each feature?
Intercom on Product Management
When you’ve already got usage, analyze it.

32. Where do you have the most
visitors, users, customers,
emails, or notifications?
More volume helps you test faster.

33. ๏ Website Traffic / App Installs
๏ Welcome Emails
๏ Web / Mobile Onboarding
๏ Email Digests
๏ Triggered Notifications
๏ Cart / Funnel Abandonment
๏ Dormant / Churned users
Common areas with large volume and high impact potential

34. Find out why by ge ing
qualitative data.

35. A few qualitative tools
HTTP://USERTESTING.COM HTTP://SURVEYMONKEY.COM

36. A few more qualitative tools
HTTP://QUALAROO.COM HTTP://OLARK.COM

37. Use exit surveys to learn why people bounce
HTTP://QUALAROO.COM

38. Use post-conversion surveys to understand why
HTTP://QUALAROO.COM

39. Discover if your web pages are effective at their goal
HTTP://QUALAROO.COM

40. What frustrates people?
EMAIL

41. Find people who don’t do things and email them
EMAIL + SURVEYMONKEY

42. User Research gives you a headstart

43. Hypothesis
Make an educated guess

44. Ideas
Designs
Solutions
Assumptions
(A guess about the best way to solve a problem)}

45. Everything can start with a
testable hypothesis.

46. Before you create anything, make an educated guess.

47. Use simple equations to assess potential impact of each idea
DAN MCKINLEY HTTP://KISS.LY/DANETSY
visits
x
conversion
rate
x
order
value
x
li4
=
?
users
x
order
value
x
resurrec7on
rate
=
?
ESTIMATED
ESTIMATED
MODELED
MODELED

48. Assess the potential impact,
difficulty and priority of
each test idea you have.
Prioritize your tests

49. Experiment
Test your guess

50. Test your hypothesis by starting
an experiment.

51. Not bu on colors…

52. Don’t copy “best practices” blindly

53. Test specific bu on copy

54. Sensitivity tests help you
learn what your visitors care
about and what they don’t.
Run sensitivity tests on your web pages

55. Remove graphical elements and see what happens

56. Does social proof and logos ma er for your visitors?

57. Remove social proof and learn what happens

58. Act
Decide what’s next

59. You’ve probably seen an A/B testing report before

60. 1. Your starting hypothesis.
2. Dates for when you ran the test.
3. Screenshots of control and all variations used during the test.
4. Expected change in conversion.
5. Probability of change.
6. Screenshot of the raw data used in your analysis.
7. A screenshot to the A/B Test Report in KISSmetrics.
8. The decision you've made from the test.
9. What you learned from the test.
Document every A/B test and create a playbook

61. What learnings do we want to
carry over from the past?
Helps you avoid unknowingly
decreasing your conversions.

62. ✓ Keep the footer links
✓ Autofocus on the first form field
✓ Do not include social proof
✓ Link to the features page
✓ Use “Log in” copy on the bu on
Create experiment checklists based on learnings

63. If you’re not testing,
you’re not learning.
☀

64. Always have tests running,
no ma er how small.
⌾

65. If you don't have a process,
you won't continuously improve.
∞

66. Work on things that ma er.
Make those things be er.
Be happier ☺

67. Thank Y0u =)
Got questions? Need help?
Contact me any time!
Hiten Shah
hnshah@gmail.com