The document discusses metrics and lessons learned for product validation. It recommends focusing on retention and user happiness before product-market fit by measuring activation, retention, referral, and revenue (AARRR). Narrow metrics to understand product health rather than traffic. Marketing stunts can distort numbers, so isolate registered users. Keep the number of metrics small and focus on those relevant to current product iterations. Various tools like Google Analytics, Kissmetrics, and custom databases each have strengths and weaknesses for different analysis needs.

1. METRICS
LESSONS
LEARNED
LEANCAMP | @serenestudios | @andreasklinger

2. Dan Hill
CTO of Crashpadder
@serenestudios
Andreas Klinger
CWTFO of LOOKK
@andreasklinger

3. BEFORE
PRODuCT MARKET FIT
FOCuS ON
RETENTION
LEANCAMP | @serenestudios | @andreasklinger

4. BEFORE PMF
FOCuS ON RETENTION
http://www.ashmaurya.com/2010/07/3-rules-to-
actionable-metrics/
Summary:
In Discovery/Validation
Focus on Retention & Activation
In Validation/Growth
Focus on Revenue and Referral

5. ThE REASON:
RETENTION
=
f(uSERhAPPINESS)
LEANCAMP | @serenestudios | @andreasklinger

6. RETENTION =
f(uSERhAPPINESS)
Actually you don’t want to measure retention.
You want to measure user happiness.
Especially before PMF.
Rentention is the best signal for
userhappiness.
But if you can - narrow down your KPIs
to show user happiness and
the health of your product.
IF YOu WILL DO ONE ThING ONLY:
FOCuS ON uSER hAPPINESS

7. KPI’S
NEED TO BE
YOuR hEALTh
MONITOR
NOT YOuR
PENISPuMP
LEANCAMP | @serenestudios | @andreasklinger

8. KPI’S
NEED TO BE
YOuR hEALTh
MONITOR
What numbers do actually show you the
(real) health of your product?
What numbers could show user happiness?
It’s not visits or one-time engagement.
In Discovery / Validation “Visits” or even
“Engagementclicks” don’t mean much.

9. IN DISCOVERY
MARKETING
CAN huRT
YOuR
NuMBERS
LEANCAMP | @serenestudios | @andreasklinger

10. (WRONG) MARKETING
STuNTS (CAN) huRT YOuR
(ACTIONABLE) NuMBERS
E.g. PR BuRSTS OR COMPETITIONS
- Created visits in Google Analytics
- Created userentries in Database
- Disappeared.
- Lower your retention and activation (%) rates
- Defocus you from understanding our product.
Example: LOOKK
We asked users to vote designers in a competition.
Effect: People voted their friends - didn’t gave a rat’s
ass about LOOKK.
Huge traffic spikes. Huge “feelgood”. Little learnings
and longterm effect.
They just created “Dataschmutz”

11. DRILL DOWN
TO REMOVE
DATASChMuTZ
LEANCAMP | @serenestudios | @andreasklinger

12. DRILL DOWN TO REMOVE
DATASChMuTZ*
What numbers really show the health of
your business and are affected by your
product changes
and how can you make them more
“stable” against outside effects.
Example:
Base your activation and retention KPIs
on registered users not visitors to remove
traffic spikes.
* Schmutz (noun, german/jiddish) means “dirt”

13. ThE METRICS FAIRY SAYS…
METRICS
NEED
TO
huRT
LEANCAMP | @serenestudios | @andreasklinger

14. METRICS
NEED TO huRT
If the metrics you focus on do not hurt you
everytime you look at them and you are too
embarassed to show them around.
They are either
a) not narrowed down enough
or
b) nothing you should focus on.

15. … AND
“AARRR”
ShALL BE YOuR
SOuND OF PAIN
EXAMPLE - LOOKK Nice to look at. useless to work with.
ACQuISITION - user registered Eg.
ACTIVATION - user Voted (eg 90%) We changed activiation from
“Registered users that voted”
RETENTION - user Visited Again
to
REFERRAL - … “Registered users that voted
REVENuE - … for more than two designers”
Everybody else is just voting for a friend.
Now:
AARRRR = ThE PAIN
ThE PAIN = ACTIONABLE

16. “AARRR” IS ThE
SOuND OF PAIN
ACQuISITION
You need a way to reach the user.
(Dave is wrong. A visit is just fog.s)
(eg. registered + confirmed email)
ACTIVATION
The user has seen the real core of your
product and used it.
(segment in user groups if multifaced)
RETENTION
The user did an action that leads to value. (if
possible close to money or coreaction)
REFERRAL
Not only shared but invited users came.
REVENuE
Money.

17. TOO MANY
METRICS
ARE
BAD FOR
YOuR EYES
FOCuS
LEANCAMP | @serenestudios | @andreasklinger

18. TOO MANY
METRICS
ARE
BAD FOR
YOuR EYES
On your dashboards only watch those
metrics you are currently iterating with
your product on.
There are always numbers going up and
down until you have a stable userflow.
Don’t go crazy because of that.
Focus your product discovery around
assumptions. Align your metrics to that.
Example: At LOOKK we removed revenue
from our dashboards while focusing on
newsfeed and community features.

19. LESSONS
LEARNED
TOOLS
LEANCAMP | @serenestudios | @andreasklinger

20. TOOLS
TL;DR: All Suck.

21. TOOLS
TL;DR: All Suck.
…but are useful for certain parts:

22. TOOLS
TL;DR: All Suck.
…but are useful for certain parts:
Google Analytics = Traffic Analysis
Kissmetrics, Mixpanel = People Analysis
Your own Database = drilldown
Be consistent throughout all tools:
- When/Where to measure a final goal
- Naming of goals
- Logging and Naming of user types
(eg. user, Designer, Admin, Visitor)

23. TOOLS
Google
Analytics
- Referrals
- Optimize Online Marketing,
- Bird eye view Website
- Suck for funnel analysis (no ad-hoc eg)
- Suck for consistancy (goals are easy to break)
Don’t do funnel testing in GA
- only track the last step
Dashboards rock.
Don’t go too crazy with customization
Unfortunately single
- it will break.
player only (yet)

24. TOOLS
KissMetrics
- Insights on Conversions
- Quick reports on features/pages
- Suck (yet) at dashboards
use it for funnel optimization.
Use it for quick flexible insights.
Use it to find users groups
“View this people” feature is genius If you are a SaaS use it for everything ;)
Don’t expect too much
and also test mixpanel.

25. TOOLS
LOOKK
Butler
- SQL = quick, Tables = cheap
- You can drilldown to your
actionable KPIs
- Dashboards for team
members
- Suck at bird-eye view, not ad-
hoc flexible
Questions:
Who are the most popular
designers if you only count
people who vote for more than
one designer?
Who are our most influential
users?

26. TOOLS
LOOKK
Butler
Team dashboards are easy to do

27. ThANKS.
PLEASE
ShARE
YOuR
LESSONS
@SERENESTuDIOS
@ANDREASKLINGER
Presentation slides: http://slideshare.net/andreasklinger