This document discusses the importance of establishing proper metrics and analytics frameworks when developing games. It recommends determining goals first before identifying relevant signals, questions, and metrics to track. Examples of metrics frameworks like HEART and AARRR are provided. Care must be taken to select actionable metrics and avoid biases. Analytics should be integrated from the start to help validate assumptions and drive optimal growth, engagement, and revenue.

1. Getting the
metrics
right!
Saurav Sahu
Associate Product Manager
Mech Mocha Game Studios
saurav@mechmocha.com

2. Analytics
vs Art
Saurav Sahu
Associate Product Manager
Mech Mocha Game Studios
saurav@mechmocha.com

3. Crafting deeply culturalised and
highly addictive games for
mobile-first Indians

4. Motivation
Why track data?
Helps you validate your assumptions and ideas
Helps you drive games for maximum growth, engagement or revenue

5. Motivation
Also, helps you to minimize
(Observed Behavior - Expected Behavior)

6. ● What to track?
● How to analyze the data?
Making a new game or a new feature?

7. ● What to track?
● How to analyze the data?
Making a new game or a new feature?

8. Personal Experience
Once, we had a question we could not find
an answer to.
We realized…[Next Slide]

9. Analytics should not be an afterthought
● Poorly chosen metrics lead to blind spots.
● Pre-planning of analyses/results we expect to obtain in future, contributes a long
way.

10. Metrics
Image: “Get All The Info You Can” © Chris Slane

11. Choosing the right metrics is very important
Actionable vs Vanity Metrics
Figure out the metrics which will help you understand player behavior better.

12. No matter how user-centered a metric is,
it is unlikely to be useful in practice unless it explicitly
relates to a goal,
and can be used to track progress towards that goal.
“HEART Framework” © Google

13. Goals
“HEART Framework” © Google
You need Goals!

14. Goals -> Signals/Qns -> Metrics
“HEART Framework” © Google

15. Goals -> Signals/Qns -> Metrics
What are my goals?
- Want users to
progress
“HEART Framework” © Google

16. Goals -> Signals/Qns -> Metrics
Identify signals that
indicate success
- Completion of
missions
- Neither too difficult
or too easy
What are my goals?
- Want users to
progress
“HEART Framework” © Google

17. Goals -> Signals/Qns -> Metrics
Identify signals that
indicate success
- Completion of
missions
- Neither too difficult
or too easy
What are my goals?
- Want users to
progress
Figuring out the specific
metrics
- %Conversion for
missions
- %Drop Offs
- # of Attempts
“HEART Framework” © Google

18. Goals -> Signals/Qns -> Metrics -> Evaluation
Figuring out the specific
metrics
- %Conversion for
missions
- %Drop Offs
- # of Attempts
Visual Representation
- Conversion Funnel
- Lapsed Players by
Level
- Attempt
Distribution
“HEART Framework” © Google

19. Attempt Distribution (For Level Difficulty)
Attempts
Users
Completing
Level 4
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Normal Distribution : Source Google

20. Attempt Distribution (For Level Difficulty)
Attempts
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Users
Completing
Level 4
Anova Statistics : Source Google

21. Attempt Distribution (For Level Difficulty)
Users
Completing
Level 4
Attempts
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Hardcore Players
Average Players
Noob Players
Anova Statistics : Source Google

22. One can use framework
AARRR
Acquisition
Activation
Retention
Referral
Revenue
“Pirate Metrics” © Dave McClure

23. One can use framework
HEART (by GOOGLE)
“HEART Framework” © Google

24. Sample Framework
FTUE Progression
Monetisation
Social Aggregate
EngagementAcquisition Misc.
Or Customize...

25. Area Goal Signals/Questions Metrics Evaluation
Engagement
Onboarding
in Tutorial
(FTUE)
completing tutorial
milestones
Tutorial Start, Tutorial
Completed
Tutorial Conversion Funnel
Tutorial Conversion Over Time -
day/week/month
% New Users who complete Tutorial
by Days since first launch
are the attempts to
complete as
expected?
"Attempts" property in 'Tutorial
Completed' event
Attempts Distribution for each
milestone and overall
good understanding
of control, effective
learning
Session Start, Session End,
Level Start, Level Complete
D0 Session Length
D0 Progression
D1 Retention%
... ... ... ... ...
Example
*not exhaustive

26. why?
● Because frameworks gives us a structure & a kickstart.
● Helps us clearly understand the goals of our game/feature. Sets up the expectation
beforehand.
● Lets us ensure value from analytics to game design, monetization, art, etc.
● If we can’t measure, we can’t fix
Listing down all metrics does not make sense if you are not gonna use them in future. It’s
a waste of resource, storage, processing, everything

27. Moving on...

28. ● What to track?
● How to analyze the data?
Making a new game or a new feature?
● Exploratory, Investigative, Predictive and so on
● Dashboards, Alerts, Reports

29. ● What to track?
● How to analyze the data?
Making a new game or a new feature?
Things to watch out for

30. Confirmation Bias
You see what you want to see
Things to watch out for

31. Metrics move in directions you can’t
imagine
Going Deeper (5 Ws)
We introduced a new feature
(liked by all testers)
Source: Hacker News
Things to watch out for

32. Ratio, Percentages (%,) Percentiles
are better than Averages
Anscombe's quartet
Source: Wikipedia
Things to watch out for

33. Unfair Comparisons
Suppose, in your game ‘Tournament’ is a mode you introduce at level 5.
You would want to know, How does playing tournament affect retention?
Dr Roseboom’s Game Analytics Insight Compendium © DeltaDNA
Things to watch out for

34. Unfair Comparisons
Dr Roseboom’s Game Analytics Insight Compendium © DeltaDNA
You need to exclude
players who didn’t even
reach Level 5
Things to watch out for

35. Summary
Measure in a proactive way
● Analytics should be integrated from the onset of a production
● Frameworks helps you in covering all areas & ensure that you are tracking quality
metrics over quantity
● Beware of the biases

36. When it comes to Games,
Creativity and Intuition are what matters in the end.
But, Data can go a long way to amplify and sharpen it.

37. saurav@mechmocha.com
Image © Séa