I gave this talk at the Institute of Mathematics and its Applications Conference for Early Career Mathematicians at Warwick University on November 2016. The talk details my experiences of working in the mobile games industry as a data scientist, drawing on my experience working at both Sega Hardlight and Exient Malta studios. It was the final talk of the day and it was well received.

1. Massively Multiplayer Data:
Challenges in Mobile Game
Analytics
Jak Marshall,
Sega Hardlight

2. Take-aways of this talk
An introductory overview of a very modern, complex, and lucrative industry that
doesn’t usually get a lot of press in academic circles.
Present challenges relating to free to play apps and games.
Discuss the current best practices in the industry.
Tempt both ardent academics and those considering industrial work that the
games industry offers a lot of ‘low hanging fruit’ and real hardcore challenges also.
Talk about videogames! Ask me anything!

3. About Me (Fun Version)
Lifelong Gamer
Content Creator (Lunch Time Game Review)
Tabletop Gamer (board and card games, mostly)
Blogger (103% Complete Gaming Blog)
LU Comedy Society
Vegan Runner

4. About Me (Career)
(2006 - 2010): Msci Maths and Stats at Lancaster University
(2008): Study Abroad at UC Berkeley
(2010): Data Intern at Unilever (Next Generation Methods, Port Sunlight
(2010): First Cohort of Stor-i DTC MRes
(2011 - 2016): PhD (currently writing up corrections)
(2015): Exient Malta : Lead Games Data Analyst
(2016): Sega Hardlight : Games Data Analyst

5. Softography (that I can tell you about)

6. Sega Hardlight
Created in 2012, based in the West Midlands.
Leamington Spa (very close to Warwick, home of JLR and Aston Martin).
Owned by and a part of the Sega group of companies (Did I mention SONIC!?).
Speciality in Mobile Games Development.
Leamington Spa is also known as Silicon Spa for the surprisingly large amount of
game developers and tech companies in the area.
This is because the Oliver Twins (Codemasters) set up there originally.

8. Games: Narrowing it down

9. Games: Narrowing it down

10. Mobile Games: Narrowing it down (by business plan)
Paid: Traditional,
consumer pays an
upfront cost and gets
the complete game.
Subscription: Pay
regular installments to
have access to game.
No upfront fee most of
the time.
Free to Play: No upfront
cost or subscription tiers,
in-app purchases and in-
app advertising drive
revenue.

11. Mobile Games: Narrowing it down (by business plan)
Paid: Traditional,
consumer pays an
upfront cost and gets
the complete game.
Subscription: Pay
regular installments to
have access to game.
No upfront fee most of
the time.
Free to Play: No upfront
cost or subscription tiers,
in-app purchases and in-
app advertising drive
revenue.

12. How does Free to Play work generally?
Economies of Scale
Since the price of entry is $0 (and sometimes incentivised!), the volume of users
using the app is large.
Not just games either! Most of us adopted Dropbox, Spotify, Skype etc because it
was free to install and access to most of the functionality of the product was given
away for nothing.
Even if only 10% of users ever purchase additional features and add-ons, that’s
10% of a large volume of users, so this pricing model still covers costs, in theory.

13. Success for a mobile app
Some definitions:
Churned user: Someone who has ‘permanently’ stopped using the app.
Mean Lifetime value (LTV) of a user: The mean amount of $revenue that an
individual user generates for the business by using the app and buying in-app
goods before they churn forever.
Cost per Install (CPI): The mean cost of acquiring a single user.
What we want to see: User Volume * (LTV - CPI) > Overheads + Dev Costs

14. A business model built on miserly users!?
Yes! But even those who do not pay any of their own
money still have real money value to the business.
1. They contribute to the virality of app, reducing the
cost of user acquisition depending on the k-factor of
that user (more on that later).
2. Passive and incentivised advertising costs nothing to
the user but provides revenue to the business if
consumed.
3. Their sheer numbers incentivise others to spend!
We’ll talk about ‘virality’ in some more detail/

15. What on earth is Virality/k-factor?
It’s actually very
costly to get users
to download your
app/game!
$4 to get a loyal
user on board.
Loyal doesn’t
necessarily mean
‘paying customer’!

16. What on earth is Virality/k-factor?
Virality for apps means that your
current users acquire even more
users for you.
Back in the ‘bad old days’ this meant
that your Facebook feed was awash
with invites to play Candy Crush
Saga.
As annoying as that was, it was a
monster success for King,

17. But how is Virality defined?
Open problem! One of many!
The k-factor method counts the average number of ‘free users’ you get from each
existing user.
If k = 0, that means nobody is inviting their friends to the party. If k > 1, then you
typically have a ‘viral game’ as the user base grows exponentially.
The Catch! -- Attribution is very hard and messy. How do you know why a user
decided to join your game? Especially when companies ‘lie’ about it.
You can track invites and clicks, but you can’t track word-of-mouth and response
to TV/Cinema ads very accurately! We needs better models for this...

18. Truth: Games Data Analytics is a Mess
1. Academic research in
this area is very
fragmented.
2. Many mobile studios
are currently made up
from former console or
tech start-up talent.

19. 1.Academic Work
Analysts like myself don’t publish their work and we’re not unionised yet.
We ‘pull’ inspiration from the (free!) literature as we need it but we don’t ‘push
back’. Not great links between academia and industry (Let’s change that?).
Operational pressures means that the industry can’t formalise its learning.
There are very few academic practitioners like there are in medicine and energy
for example. There’s not a prominent ‘games data’ themed journal or portal.
This is very sad because game development, publishing and live operations offer
a vast amount of potential projects for Stats, OR, and CS students and faculty.
Even sadder because I know how many of you are game nerds!

20. 2a. Former console talent
A great proportion of mobile studios consist of former console devs, making
‘finished’ box products (Think Assassin’s Creed, Tomb Raider, Bok-tai)
Traditional console studios never included analysts that worked on the games
themselves.
Analysts in the industry usually worked in publishing and market intelligence
side of things out of studio.
The presence of analysts in a games studio is an idea which isn’t even a
decade old at this point. Even big studios with centralised Business
Intelligence Departments are flying by the seat of their pants at this point in
time.

21. 2b. Technology companies who also make games.
These companies are awash with very bright analysts,
computer scientists and tech heads.
They’re ahead of the field in terms of solving a lot of
complicated technical problems and optimising processes.
However, they lack any real pedigree in game design
experience and user experience knowledge.
Probably not the kind of company that are interested in the
‘science of fun’ and more interested in simply grinding
money out of people in a soulless kind of way.

22. They found that Mariah Carey gets more installs
than Kate Upton through AB Testing though!

23. One more thing… you have to do this on the fly.
You have to collect this data while your
game is running.
If you have a complex real-time tactical
experience, you have to be able to
grab everything you need while the
game is happening, to a potentially
huge concurrent roster of players on
a server.
A large proportion of it will be available
to sample at the time that the
analytics events you want will fire.

24. So now we get to what I do for a living!
1. Providing regular ‘health checks’ on the games that are live for players right
now: Identifying risks and areas for improvement in our KPIs.
2. Overall data strategy: What we decide to track, how we track it and which
technology we use. The studio look to me to provide that direction.
3. Insights: Deep dives, post mortems, market analysis… that’s on me too!
4. Optimisation: Experimenting on players to learn what makes them play for
longer and spend more.

25. Let’s talk briefly about ‘passive reporting’
1. Providing regular ‘health checks’ on the games that are live for players right
now: Identifying risks and areas for improvement in our KPIs.
2. Overall data strategy: What we decide to track, how we track it and which
technology we use. The studio look to me to provide that direction.
3. Insights: Deep dives, post mortems, market analysis… that’s on me too!
4. Optimisation: Experimenting on players to learn what makes them play for
longer and spend more.

26. Top level Reporting - Always available to all
Non-technical execs want to be
able to look at ‘high-level
stats’ without adult
supervision.
This tends to be non-game
specific for cross title
comparisons
There’s a culture of ‘let’s track
what everyone else is
tracking’ but there’s certainly
room for improvement here.

27. Trying to improve our workflow and toolchain.
1. Providing regular ‘health checks’ on the games that are live for players right
now: Identifying risks and areas for improvement in our KPIs.
2. Overall data strategy: What we decide to track, how we track it and which
technology we use. The studio look to me to provide that direction.
3. Insights: Deep dives, post mortems, market analysis… that’s on me too!
4. Optimisation: Experimenting on players to learn what makes them play for
longer and spend more.

28. The tech I’m using.
It’s also a matter of evangelising the need to
think about analytics from design to servicing.

29. ‘Big Ticket’ Analysis - Deep Dive Reporting.
1. Providing regular ‘health checks’ on the games that are live for players right
now: Identifying risks and areas for improvement in our KPIs.
2. Overall data strategy: What we decide to track, how we track it and which
technology we use. The studio look to me to provide that direction.
3. Insights: Deep dives, post mortems, market analysis… that’s on me too!
4. Optimisation: Experimenting on players to learn what makes them play for
longer and spend more.

30. Example of Recent Deep Dive: Churn Detection.
What are the key predictors of churn? (leaving a game, never to return)
It largely depends on the game and the audience.
Took a lot of inspiration from some pioneer’s on Kaggle working in World of Warcraft.
https://www.kaggle.com/thibalbo/d/mylesoneill/warcraft-avatar-history/wow-dataset-exploratory-
analysis
It’s important that we can predict which of our users are at risk of leaving us so
we can decide when and how to intervene.
What we’ve found so far.
Social obligation reduces churn: Online ‘teams’ can create a sense of loyalty.
Social proof also reduces churn: Just seeing that your friends are playing puts you at lower risk.

31. But how do we intervene!?
1. Providing regular ‘health checks’ on the games that are live for players right
now: Identifying risks and areas for improvement in our KPIs.
2. Overall data strategy: What we decide to track, how we track it and which
technology we use. The studio look to me to provide that direction.
3. Insights: Deep dives, post mortems, market analysis… that’s on me too!
4. Optimisation: Experimenting on players to learn what makes them play for
longer and spend more.

32. AB Testing Approach (Boo!)
It seems to be the industry standard.
You randomly allocate users into separate groups and then change the game
experience for people in that group.|
Candy Crush changes the order and difficulty of its levels all the time to optimise
the trade off between selling extra moves and people getting fed up with it.
It’s slow, inefficient, and doesn’t control for an awful lot of things. It’s simple.
It’s also tricky when you have multiplayer games and clever clogs on forums

33. Bayes Optimisation: The Multi-Armed Bandit Future
You want to expose the fewest number of users as
possible to potential variants of the game
experience while still learning about the impact of
potential changes you want to make.
It’s very expensive to lose players, particularly if they
are influential spenders
Bandits seem to be the best way to balance the delivery
of product insights and the overheads associated
with doing so.
Gaming is behind the curve on this front, as it is with a
lot of its data practice, so the service providers that
deliver Bayesian versions of ‘off-shelf’ tools and
backend services stand to do well. (I’m doing my

34. The vital take-aways from all of this.
Mobile Gaming remains a growth industry with plenty of money flying around.
Large outfits such as Sega, Nintendo, and even King are far from yielding the
additional revenue to be gained from the proper application of analytics.
If you love games and you want to help make them, you can try and make these
changes from the belly of the beast like I’m doing.
If you’re more academically inclined there’s a whole lot of interesting problems
and applications up for grabs if we can establish good links between

35. Thanks for listening. I’ll now take questions.
jak.marshall@sega.net