Presentation on Lean Analytics at MicroConf 2013. Understanding what metrics are the most value, when, for your type of business.
* What makes a good metric?
* Types of metrics (qualitative vs. quantitative, vanity vs. actionable, etc.)
* Lean Analytics framework
Shared a number of case studies: Airbnb, Buffer, ClearFit, OffceDrop and others.
9. The basics of Lean Startup
Everyone’s idea is
the best right?
People love
this part!
(but that’s not always a
good thing)
This is where things
fall apart.
No data, no
learning.
10. What I Hope You Get From It
h"p://www.flickr.com/photos/romtomtom/4382603005/sizes/o/in/photostream/
11. Follow the Lean model, and it
becomes increasingly hard to lie,
especially to yourself.
The importance of intellectual honesty
12. Using your gut properly
Instincts are experiments.
Data is proof.
13. Better decision making abilities
Everyone has data, the key is
figuring out what pieces will
improve your learning and
decision making.
15. Measure what matters
1 What makes a good metric?
2 Types of metrics
3 Analytical superpowers
4 Lean Analytics framework
5 The One Metric That Matters
6 Lean Analytics Cycle
16. What Makes a Good Metric?
h"p://www.flickr.com/photos/artnoose/2263480871/sizes/l/in/photostream/
17. Analytics is the measurement of
movement towards your
business goals.
What is analytics?
22. If it won’t change
how you behave,
it’s a
bad
metric.
If a metric won’t change how
you behave, it’s a
h"p://www.flickr.com/photos/circasassy/7858155676/
24. Warm and fuzzy. Cold and hard.
Unstructured,
anecdotal,
revealing, hard to
aggregate.
Numbers and stats;
hard facts but less
insight.
Qualitative
vs.
Quantitative
27. Professional photography helps Airbnb’s business
Gut instinct
Concierge MVP
20 photographers in the field
Test results
Two to three times more bookings!
Back to the beginning
Use additional data to keep experimenting
29. Makes you feel
good but doesn’t
change how you’ll
act.
Helps you pick a
direction and
change your
behavior.
“Up and to the right.” These are good.
Vanity Actionable
vs.
30. Hits
A metric from the early, foolish days of the Web.
Count people instead.
Page views
Marginally better than hits. Unless you’re displaying
ad inventory, count people.
Visits
Is this one person visiting a hundred times, or are a
hundred people visiting once? Fail.
Users
This tells you nothing about what they did, why they
stuck around, or if they left.
Followers/
friends/likes
Count actions instead. Find out how many followers
will do your bidding.
Logins
But what are they actually doing when they login?
Logins don’t tell you about actions and value.
Vanity metrics are bad!
31. Speculative, tries to
find unexpected or
interesting insights.
Predictable, keeps
you abreast of
normal, day-to-day
operations.
The cool stuff. The necessary stuff.
Exploratory Reporting
vs.
32. Pivoting from friends to moms
•Started as Circle of Friends
•Leveraged Facebook early
•Grew to 10M users
But engagement sucked!
33. Moms are crazy!
(in a good way)
Engagement solved!
• Messages to one another were on average 50% longer.
• 115% more likely to attach a picture to a post they wrote.
• 110% more likely to engage in a threaded (i.e. deep) conversation.
• Friends, once invited, were 50% more likely to become engaged users.
• 180% more likely to click on Facebook news feed items.
• 60% more likely to accept invitations to the app.
34. Historical metric
that shows you
how you’re doing:
reports the news.
Number today that
shows a metric
tomorrow: makes
the news.
Try and get here.Start here.
Lagging Leading
vs.
38. Correlation lets you
predict the future
Causality lets you
change the future
“I will have 420 engaged users
and 75 paying customers next
month.”
“If I can make more first-time
visitors stay on for 17 minutes I
will increase sales in 90 days.”
Find correlation Test causality
Optimize the
causal factor
Causality is a superpower, because it lets
you change the future.
40. Your Business + Stage
What business
are you in?
What stage
are you at?
•E-Commerce
•SaaS
•Free Mobile App
•2-Sided Marketplace
•Media
•User-Generated Content
•Empathy
•Stickiness
•Virality
•Revenue
•Scale
42. The SaaS Customer
Lifecycle
Customer Acquisition Cost
paid direct search wom
inherent
virality
VISITOR
Freemium/trial offer
Enrollment
User
Disengaged User
Cancel
Freemium
churn
Engaged User
Free user
disengagement
Reactivate
Cancel
Trial abandonment rate
Invite Others
Paying Customer
Reactivation rate
Paid
conversion
FORMER USERS
User Lifetime Value
Reactivate
FORMER CUSTOMERS
Customer Lifetime Value
Viral coefficient
Viral rate
Resolution
Support data
Account Cancelled Billing Info Exp.
Paid Churn Rate
Tiering
Capacity Limit
Upselling
rate Upselling
Disengaged DissatisfiedTrial Over
43. •Stage: Revenue / Scale
•Model: SaaS (Paid)
•Recruitment marketing and
assessment software
•Switched business models from
monthly subscription to pay per
job posting
Does recurring revenue
work for everyone?
44. 10x
revenue increase
off of 3x in sales
volume
“People don’t do subscriptions for haircuts, hamburgers,
and hiring. You have to understand your customer, who
they are, how and why they buy, and how they value
your product or service.” - Ben Baldwin, co-founder
Lots of money!
47. EMPATHY
STICKINESS
GROWTHRATE
VIRALITY
REVENUE
SCALE
Lean Analytics
Stages
I’ve found a real, poorly-met need that
a reachable market faces.
I’ve figured out how to solve the problem in
a way they will adopt and pay for.
I’ve built the right product/features/
functionality that keeps users around.
The users and features fuel growth
organically and artificially.
I’ve found a sustainable, scalable business
with the right margins in a healthy
ecosystem.
“Gates” needed to
move forward
48. •Stage: Scale
•Model: SaaS
•Popular social sharing application
•Focused primarily on customer
acquisition
•Charged from day one
From Stickiness to Scale
(through Revenue)
49. 20%
60%
20%
2%
of visitors created an account
(acquisition / Empathy)
of sign-ups returned in the 1st month
(engagement / Stickiness)
of sign-ups were active after 6 months
(engagement / Stickiness)
convert from free to paid
(Virality & Revenue)
Buffer charges early to prove
people want the problem solved
51. One Metric
That Matters.
How It All Comes Together
The business you’re in
E-Com SaaS Mobile 2-Sided Media UCG
Empathy
Stickiness
Virality
Revenue
Scale
Thestageyou’reat
52. Choose only one metric and
draw a line in the sand.
Putting the pieces together...
53. •Stage: Revenue
•Model: SaaS (Freemium)
•Paper and digital collaboration
•180,000 users
•Paid churn is their One Metric
That Matters (OMTM)
Building a revenue engine
54. • Target < 4% paid churn (hitting 2% lately
on a monthly basis)
•Anything over 5% means they don’t have
a business that will generate positive
margin returns: the bucket is too leaky
The OMTM: Paid Churn
55. • Can we acquire more valuable customers?
•What product features can increase engagement?
• Can we improve customer support?
•Was a marketing campaign successful?
•Were customer complaints lowered?
•Was a product upgrade valuable?
If Paid Churn: Why & Next Steps:
Paid Churn = “business health” indicator
•Are the new customers not the right segment?
• Did a marketing campaign fail?
• Did a product upgrade fail somehow?
• Is customer support falling apart?
56. Some interesting benchmarks
Growth
5% / week (revenue or active
users)
Churn
2% / month
Engaged visitors
30% monthly users
10% daily users
Time on site
17 minutes
Page load time
< 5 seconds
CLV:CAC
3:1
Mobile file size
< 50MB
Free to paid
2% of free users
58. Identify a key business problem,
pick the OMTM, draw a line in the
sand, and get started.
Time to experiment
59. Draw a new line
Pivot or
give up
Try again
Success!
Did we move the
needle?
Measure the
results
Make changes in
production
Design a test
Hypothesis
With data:
find a
commonality
Without data:
make a good
guess
Find a potential
improvement
Draw a linePick a KPI
The Lean Analytics Cycle