http://leananalyticsbook.com
An Introduction
to Lean Analytics
Ben Yoskovitz
@byosko

A bit about me...
http://instigatorblog.com
1996 - 2006
2007 - 2010
2010 - 2011
2011 - ....

Why did I write it?
h"p://loreindustries.com/wp-­‐content/uploads/2012/05/Monkeys-­‐typing.jpg

We’re all liars.

The basics of Lean Startup
Everyone’s idea is
the best right?
People love
this part!
(but that’s not always a
good thing)
This is where things
fall apart.
No data, no
learning.

Analytics is the measurement of
movement towards your
business goals.
What is analytics?

In a startup, the purpose of analytics
is to iterate to product/market fit
before the money runs out.
And for startups?

Good Metrics are:
1 Understandable
4 Behavior Changing
2 Comparative
Active Users vs.
Active Users/month
3 Ratio / Rate
% Monthly
Active Users

If it won’t change
how you behave,
it’s a
bad
metric.
If a metric won’t change how
you behave, it’s a
h"p://www.flickr.com/photos/circasassy/7858155676/

h"p://www.flickr.com/photos/maCmaCla/3822631755/
Types of Metrics

Warm and fuzzy. Cold and hard.
Unstructured,
anecdotal,
revealing, hard to
aggregate.
Numbers and stats;
hard facts but less
insight.
Qualitative
vs.
Quantitative

Discover qualitatively.
Prove quantitatively.
Quantitative vs. qualitative data

Speculative, tries to
find unexpected or
interesting insights.
Predictable, keeps
you abreast of
normal, day-to-day
operations.
The cool stuff. The necessary stuff.
Exploratory Reporting
vs.

Finding the answer in the data
•Started as Circle of Friends
•Leveraged Facebook early
•Grew to 10M users
But engagement sucked!

Moms are crazy!
(in a good way)
Engagement solved!
• Messages to one another were on average 50% longer.
• 115% more likely to attach a picture to a post they wrote.
• 110% more likely to engage in a threaded (i.e. deep) conversation.
• Friends, once invited, were 50% more likely to become engaged users.
• 180% more likely to click on Facebook news feed items.
• 60% more likely to accept invitations to the app.

Historical metric
that shows you
how you’re doing:
reports the news.
Number today that
shows a metric
tomorrow: makes
the news.
Try and get here.Start here.
Lagging Leading
vs.

h"p://www.flickr.com/photos/bloke_with_camera/401812833/sizes/o/in/photostream/
Analytical Superpowers
(or what the heck is growth hacking?)

1
10
100
1000
10000
Ice cream consumption Drownings
Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec

Summer
Ice cream
consumption
Drowning
Correlated
Causal
Causal
Two variables that
change in similar ways,
perhaps because
they’re linked to
something else.
Correlated
An independent factor
that directly impacts a
dependent one.
Causal
vs.

Correlation lets you
predict the future
Causality lets you
change the future
“I will have 420 engaged users
and 75 paying customers next
month.”
“If I can make more first-time
visitors stay on for 17 minutes I
will increase sales in 90 days.”
Find correlation Test causality
Optimize the
causal factor
Causality is a superpower, because it lets
you change the future.

Lean Analytics Framework
h"p://www.flickr.com/photos/ikhlasulamal/2331176652/

Your Business + Stage
What business
are you in?
What stage
are you at?
•E-Commerce
•SaaS
•Free Mobile App
•2-Sided Marketplace
•Media
•User-Generated Content
•Empathy
•Stickiness
•Virality
•Revenue
•Scale

business models

The 2-Sided Market Customer Lifecycle
Visits
SELLERS
Seller
visits
Account created
Listings created
Listing count
New listings/day
Fraud rate
Fraudulent / Flagged
Results
% blank searches
Additional Services
Other revenue
Conversion rates
Delivery
Transaction
Click-throughs
Good results
Conversions
Shortlist
Buyer rates Seller
Ratings, rate
Repeat Seller
Account Cancelled Listing Expired Disengagement
Churn RateSeller churn
Seller lifetime value
Visits
BUYERS
Signup rate
Account created
Notifications
Disengagement /
opt-out
Repeat Buyer
Buyer lifetime value
Email bounce
Open
Clicks
Buyer
churn
Click-through rate
Searches
Searches / visit
Seller rates
Buyer
Ratings, rate
One-time
purchase
Gross revenue
Net revenue

2-Sided Marketplaces
1 Do things that don’t scale
2 Focus on the money
5 The scale has to be enormous
3 Seller quality matters
4 Find extra revenue sources

lean analytics stages

EMPATHY
STICKINESS
GROWTHRATE
VIRALITY
REVENUE
SCALE
Lean Analytics
Stages
I’ve found a real, poorly-met need that
a reachable market faces.
I’ve figured out how to solve the
problem in a way they will adopt and
pay for.
I’ve built the right product/features/
functionality that keeps users around.
The users and features fuel growth
organically and artificially.
I’ve found a sustainable, scalable
business with the right margins in a
healthy ecosystem.
“Gates” needed to
move forward

•Stage: Empathy
•Model: UCG/Mobile
•Real-time Q&A platform tied to locations
•Needed to find out if a core behavior--
answering questions about a place--happened
enough to make the business real
Localmind hacks Twitter

Localmind hacks Twitter
•Before writing a line of code, they wanted to answer this question
•This was their biggest risk: if questions went unanswered, the experience would suck
Would people answer questions?
Test results
•The response rate to tweeted questions was very high
•Good enough proxy to de-risk the solution and convince the team to continue
Ran an experiment on Twitter
•Tracked geolocated tweets in Time Square
•Sent @ messages to people who had just tweeted, asking a question about the area

•Stage: Empathy / Stickiness
•Model: E-Commerce
•Eco-friendly prints of your photos
•Originally tied exclusively to Instagram and
built a custom InstaOrder feature
Static Pixels simplifies to focus on
immediate success

★ 2x transactions
★ lower bounce rate
★ sign-in goals increased
with InstaOrder
Click checkout
Sign into PayPal
Confirmation page
Confirm order
Back to PayPal
Authorize pre-
approved payments
Success page
without InstaOrder
Click checkout
Sign into PayPal
Confirmation page
Confirm order
Success page
Optimize for 1st time purchases or
repeat orders?

skip steps at your
own risk

One Metric
That Matters.
How It All Comes Together
The business you’re in
E-Com SaaS Mobile 2-Sided Media UCG
Empathy
Stickiness
Virality
Revenue
Scale
Thestageyou’reat

•Stage: Scale
•Model: SaaS
•SEO toolkit (product suite)
•Reduced KPIs to focus on Net Adds
SEOMoz reduces the KPIs it
tracks

If Net Adds: Why & Next Steps:
Net Adds = “business health”
indicator
•Was a marketing campaign successful?
•Were customer complaints lowered?
•Was a product upgrade valuable?
• Can we acquire more valuable customers?
•What product features can increase engagement?
• Can we improve customer support?
•Are the new customers not the right segment?
• Did a marketing campaign fail?
• Did a product upgrade fail somehow?
• Is customer support falling apart?

Choose only one metric and
draw a line in the sand.
Choosing your OMTM isn’t enough...

•Stage: Virality
•Model: Mobile app
•Social network around the past
•Focused on virality (but not the
viral coefficient)
Timehop aims for virality through
content sharing

• Focused on % of daily active users that
share content
• Aiming for 20-30% of daily active users
to share content
“All that matters now is virality. Everything else—be it
press, publicity stunts or something else—is like
pushing a rock up a mountain: it will never scale. But
being viral will.” - Jonathan Wegener, co-founder
The One Metric That Matters:
Content sharing

Some interesting benchmarks
Growth
5% / week (revenue or active
users)
Churn
2% / month
Engaged visitors
30% monthly users
10% daily users
Time on site
17 minutes
Page load time
< 5 seconds
CLV:CAC
3:1
Mobile file size
< 50MB
Free to paid
2% of free users

Lean Analytics cycle
h"p://www.flickr.com/photos/jrodmanjr/4728457415/

Identify a key business problem,
pick the OMTM, draw a line in the
sand, and get started.
Time to experiment

Draw a new line
Pivot or
give up
Try again
Success!
Did we move the
needle?
Measure the
results
Make changes in
production
Design a test
Hypothesis
With data:
find a
commonality
Without data:
make a good
guess
Find a potential
improvement
Draw a linePick a KPI
The Lean Analytics Cycle

Thank you.
byosko@gmail.com
@byosko
ORDER!
follow me.
instigatorblog.com
leananalyticsbook.com
subscribe.
email me.