The Games Industry Analytics Forum returned for its third gathering of the year in San Francisco and its 10th meet-up, on Monday August 10th.
Featuring presentations and expert panel discussions, the GIAF is a unique opportunity for practitioners looking to generate insight and value from big data game analytics; one of the most important trends in games.
Talks at this GIAF:
Nurturing the player journey
Kady Srinivasan, Sr Director of Player Engagement - Mobile at Electronic Arts
Lean analytics
Will Perone, CTO at Wicked Fun
Analytics architecture at IMVU
Jon Watte, VP of Technology at IMVU
Using data to prove the value of haptics
Nick Thomas*, Head of Gaming at Immersion Corporation.
The event is free for game analytics practitioners. For more info on future GIAF events, visit www.deltadna.com/GIAF
*Slides from Immersion are not included in this slide deck at present
World Economic Forum Metaverse Ecosystem By Utpal Chakraborty.pdf
GIAF USA: Summer 2015
1.
2. UPCOMING TALKS
Curating the Mobile Player Journey
Kady Srinivasan- Sr Dir, Player Engagement Mobile at EA
Lean Analytics
Will Perone – CTO at Wicked Fun
IMVU Data Architecture
Jon Watte - VP Technology and Analytics at IMVU
Using Data to Prove the Value of Haptics
Nick Thomas - Head of Gaming at Immersion Corp
3.
4. Curating the Mobile Player
Journey
Kady Srinivasan
Sr Dir, Player Engagement EA Mobile
5. Agenda
• Player Retention
• Managing the Player Journey
• Player Responsive Model
• 4 Areas to support Player Responsive Model
• Madden Mobile Rewards
2
6. Player Retention Key to Long Live Service
Games
Increasingly live service oriented games indicate heavy emphasis on engagement and
retention over acquisition
0%
10%
20%
30%
40%
50%
60%
ACQUISITION BRAND
AWARENESS
RETENTION
Average Company
Player retention is an integral part of product operations and EAM is beginning to invest
Source: Return on Behavior
%ofDollarsSpent
3
7. Managing the Player Lifecycle – Key to Retention
MTX
OFFER
TUTORIAL
YES
1. CONVERSION/ENGAGEMENT
CYCLE
MTX
CONVERSION
POINT 2
2. UPSELL/RETAIN CYCLE
UPGRADE
3. AFFINITY X PROMO
NEW
PLAYERS
NO
Presenting relevant “need based” offers at key decision points in a player journey
ENGAGEMENT
OFFER
INSTALLAFFINITY
Opportunity: 4X increase in Retention and 2X increase in Engagement*
*Source: Urban Airship
8. Curating Player Lifecycle – Four Key Areas
Player Value
(Who)
Decision Points
Mapping (When)
Desired Action
(Why)
Customized
Offers (What)
• WHO: PLAYER SEGMENTATION: Classify player and
expected value from player (i.e. Whales, Spenders,
Social Influencers)
• WHEN: DECISION POINTS MAPPING: Understand in-
game and out of game behavior and decision points
• WHY: DESIRED ACTION: Identify desired action we
want the player to take
• WHAT: CUSTOMIZED OFFERS: What offers fulfill need
states
TECHNOLOGY
&
DATA
PLATFORM
A scalable model of meeting player expectations supported by technology and 360 view
of player
Who -> When -> Why -> What
5
9. Who: Smart Segmentation
Following Best Practice
• RFM Segmentation
• Based on Predictive Propensity
• Multi channel Re-targeting and Customized Offers
• In most games, players treated equal
• Messaging not based on predictive
propensity
• All offers created equal
• No multi channel marketing
Generic Email
Generic
Promotions
(Ex: 7 day
Sales)
BEHIND THE CURVE
• Highly engaged: 3 sessions/day, > 10 mins per session,
plays on iPAD
• User has low probability of churn but high propensity to monetize
• Show 10% interstitial offer on 2nd day, 50% on 4th day, push notification on 5th
day, IAN on 10th day
• High monetizer: 1 sessions/day, > 10 mins per session
• User is fickle, convert early and high
• Show new and exciting content ad on 2nd day, show how to use premium
content, offer 50% off on 2nd purchase
• Fickle user: < 3 sessions/week, < 10 mins per session
• User is fickle, retain
• New and exciting content, push notes on Day 3, 5 and 7
• Reminders to come back and play for new content through mobile notifications
• If not engaging, X promo or show GMS ad
CURRENT STATE
Step function changes in targeted and segmented messaging in the last year
6
10. Who: Smart Segmentation V2.0
0
0.5
1
1.5
2
Push Note Response Predictors
PNResponse RevenuePOST_PNResponse
Predictors informed “personas” created from game play variables
0.0%
0.5%
1.0%
1.5%
2.0%
2.5%
3.0%
$-
$2.00
$4.00
$6.00
$8.00
$10.00
Friends Explorers Offerhogs Rushers
DARPU & PN Response Rate by Persona
DARPU PN_response %
Personas
FRIENDS EXPLORERS OFFER HOGS RUSHERS
7
11. Key: = PN = PN / IAM = IAM Green Text = New Campaign
When: Key Points During Player Journey
1d Churn PN
3d Lapsed
PN
5d Lapsed
PN
Job
Complete PN
Tutorial
Complete
IAM
~3x Status
Quo Event-
based Notes
7/30/60d
Lapsed
Spending
Friend
Visit IAM
7/30/60d
Lapsed
Spender
7/30d Single
Transactor
~3x Status
Quo Event-
based
Granpa
Winback
(Email
Only)
14/30d Lapsed
PN
Merch Engine
IAM / PN
Promo
Reminder PN
Predictive
Churn iAM / PN
Lapsed Sage
7/30d PN
New
Player
Mature
Player
Elder
Player
Player journey enhanced to drive greater touch points at relevant intervals
8
12. What: Need Based Offers
RETARGET
LAPSED PLAYERS
FIRST TIME BUY
OFFER
PERSONALIZED
PUSH NOTE
REPEAT
SPENDER
OFFER
eCRM PII
COLLECTION
NETWORK AFFINITY AD
These messages are triggered at appropriate points in the player journey
Right Message, Right Player, Right Time
REPEAT MTX
OFFER
9
13. Degree of customization
What: Need Based Offers V 2.0
Provide compelling and differentiated offerings to influence the desired action
Stage One
All
Players
• Relevant and Personalized
Messages
• Promotions, Offers and
Discounts
• Limited and Unique
Products/New Affinity Titles
• Personalized
Products/Store
• Personalized Game Play
Differentiated
Experience Driven By:
Relevant
Messages
Promotions Unique
Products
Personalize
d Products
Personalize
d Game
Play
Stage Two Stage Three Stage Four Stage Five
130
14. 140
Summary
Managing the player journey – massive opportunity but fraught with complexity
Needs tight integration between Analytics, Marketing and Product Management to
accelerate
How can you unlock this potential?
16. Who Am I?
• Designing and Programming games since age 11
• Making mobile games since 2005
• Making social games since 2007
• Worked for Glu, Funzio, Kixeye
• Started 3 companies - Andrograde, RubyCoins, Wicked Fun
17. State of Wicked Fun
• Venture backed
• 16 employees
• Cross-platform real time multiplayer game
• Pre-revenue
• Iterating the product fast
• Iterating UA/Marketing fast
18. A Conundrum
Be as lean as possible
Iterate as fast as possible
Execute the highest quality bar possible
19. Different Needs for Analytics
Business Intelligence
– Industry standard events easy
– Hard to dig into the ‘why’
Design
– Want complex game balancing metrics
– Want in-game economy related analytics
Engineering
– Want performance/bug/crash monitors
20. Wicked Fun’s Solution
• No dedicated PM/Analyst
• Focus on a few core metrics per department
• Don’t waste time building analytics in-house
• Empower each team to do their own analytics
• Balance quantitative and qualitative analysis, don’t let
analytics run your life
– Analytics can mislead you on what to focus on
21. Design
• Use hierarchical events for balancing
• Not all off the shelf analytics platforms support hierarchical events, may have
to manually do it
• Total in-game currency, Net currency over time
• What are people buying in game
• Certain things still require engineering to get involved (db
queries, adding events to code)
22. Engineering
• Use analytics as crash report statistics and performance/net
monitoring tools
– Ping, packet size, function times & counts…
– Set up special API keys so you can turn it off if event traffic too high
• Set up a different analytics API key per platform (mobile, web,
PC…)
– The platform itself will color user behavior even if the game is exactly
the same
23. BI
• Retention (D1, D7)
– Connected to # people who convert to paying
– How long does it take for people to buy?
• Total funnel is hard to set up but worth it
– View ->Install ->Register ->Tutorial ->Screen flow
– Higher dropoff in your marketing or in your game?
• What is the first thing people usually buy?
• Organics vs Paid Installs -> Virality
– Key to cheap UA
24. Industry Challenges
There is still no good solution for cross-platform marketing,
attribution and analytics
Supporting infrastructure for real time multiplayer cross-
platform games is still in its infancy
Developers want real time analytics data feed to personalize UX
to customers
28. Avatars Lean Start-up
User Generated
Content
Continuous
Deployment
Virtual Currency
Social
Entertainment
Social VR Multi-platform
Real-time at
Web Scale
30. Fun Facts
400 database shards
for customers
Haskell uses 10% as
much CPU as PHP
20 million item catalog
(works in VR, too!)
Top selling UGC has
sold > 1 million!
IMVU creators pay the
rent with UGC
End-to-end analytics
reduces payment fraud
Recommenders give
users what they want
Price Elasticity
optimizes revenue
12,000 new items
per day
700,000 items sold
every day300,000 metrics
10 second interval
32. JOIN IN THE CONVERSATION PARTICIPATE IN THE NEXT GIAF
Analytics for Games events@deltaDNA.com
www.deltaDNA.com/GIAF
Notas del editor
\/ \/ \/ \/ \/ \/ \/
US (games)
iPad US - #13 top grossing
iPhone US - #32 top grossing
Android - #44 top grossing
Australia
Who cares about Australia. We do
iPad - #1 top grossing
iPhone - #1 top grossing
Android -#3 top grossing
Overall, not only games
Ad-hoc analytics, daily fires, dashboards, Insights
Deep dive analysis - reports that take few weeks to complete;
Predictive analytics, Machine learning, statistical modelling
Data Pipeline, platform for machine learning and modelling
Insights
Data Science
Data Engineering
7 people; 4 in London office
We Are Hiring !
jobs@productmadness.com
Events are generated on server-side. This way we control data quality.
We are processing 350 Million events per day
They got ingested into Amazon Cloud to S3, with the help of Python and Spark.
And then got copied to Amazon Redshift - Columnar parallel database.
Currently we have 12 nodes, with total capacity of 24TB
Once the data is in there - we do all heavy aggregations and transformations.
We have moved from Hadoop more than a year ago and haven’t looked back since.
We perform most of interactive analysis in Python Notebooks.
For trivial things we are using re.dash, which is similar to Mode and Periscope.
It is Web-based SQL client with integrated plotting and collaboration functionality.
You can even create dashboards with re.dash, but for production dashboards we prefer to use Tableau or our own D3.js-based application.
All our web applications are using Python backend, Flask framework. We use scikit-learn for machine learning and predictive analytics.
As you have probably guessed, we like Python.
What we do:
AB tests
bread and butter of data science teams
yet controversial
and often misunderstood
Customer Lifetime Value modelling
knowing how much your customer worth, shortly after you acquite them is a holy-grail of User Acqusition
can easily spend next 40 minutes talking about customer lifetime value modelling, but ..
So - segmentation
In this presentation I will not be taking in details about classification algorithm, dimensionality reduction or machine learning.
Instead, we will be looking at segmentation from Product Marketing perspective.
Successful segmentation is the product of a detailed understanding of your market and will therefore take time
Segmentation is not a two-weeks task you assign to your analytics department
Customers have different needs and means.
Some players play for fun, others got a kick from competition.
We all know that players have very different willingness to pay.
Most of you know how rare it is to find a Normal Distribution among our players - our games are played by outliers. If you remove outliers from any analysis - you will probably miss the point of it.
Segmentation can help to understand those differences
Which can help to deliver on those needs
And drive higher profitability
A segment is a group of customers who display similarities to each other...
They may react similarly in a product/service offering
They may provide comparable values (profitability) to the company
They may bear the same needs or behave in alike ways
Customers move in and out of segments over time
There is no one right way to segment (not should there be):
Many different approaches and techniques. I will cover few techniques in the following slides.
Mix of art, science, common sense, experience and practical knowledge
You need to take business needs into account, but also what data is available and can be used, operationally.
Don’t aim to build one holistic taxonomy to meet all needs,
So what are different types of segmentation?
How do you approach a problem like this?
Multiple way to segment users
And there are different use cases for segmentation.
You can segment on: geography and basic demographics.
In our case - Australian players are very important, and usually behave quite differently from the rest of the world.
You can segment based on stage in a player’s Lifecycle - new players behave differently to someone who have been playing your game for the last two years. Also, knowing users who are showing signs of disengagement is very important.
You can also segment on Behaviour, Needs (if you can identify them, possibly based on observed behaviour) and, of course, based on Player Value.
But different parts of the business are interested in different segmentations.
Product Managers and Marketing teams might be very interested in Behavioural Segments. But CEO may be more interested to track retention metric for your most valuable players (whales).
The actual segmentation might be hybrid.
This is the segmentation of the Land-based Slot players.
First layer - by frequency of play, e.g. engagement
Second layers - bahavioural
But of course, it is important to understand why segmentation is useful for a business.
What decisions can it help to make?
And how it can affect daily operations and possibly product?
Clients of Segmentation
Strategy and Finance
Product development
Marketing operations
Strategy and Finance
When we looked at data after launch, amount of coints spent has actually dropped on the day of the launch!
But was it even a real drop?
But for a specific segment, that day was very positive.
Business knowledge:
- high-level segments goals
- product/marketing strategy
Data knowledge:
- how to access 360 view?
- what are segments definitions?
Stats/Analytical skills
- how to profile various segments?
ETL
- recalculated daily or real-time
- regular reviews
Integration with back-office and game
- segmentation engine + ETL
Dashboards
Reporting
Marketing:
- day-2-day campaigns for segments
- reporting (monthly and daily)
Product
- review feature success for segment
Analytics and data engineering
- ongoing support and refinement
What are business objectives and therefore customer characteristics we should use to profile the market?
What approach should we take to ensure segments accurately represent the market and actionable?
What criteria should we use to prioritise segments and select targets?
How can we ensure segmentation is operational and can be deployed?
How to do ‘land’ the segmentation within the organization and ensure it gains traction?
K-means
Hierarchical Clustering
Decision Trees
.. and many more
I’m going to take a look at what’s going on in game analytics. And why.
I’m going to look at four aspects – what’s going on in the market we serve – in the games industry itself – and how that’s shaping what gets done in analytics.
And I’m going to look at changes in the tools and tech we have at our disposal, to do the work we do.
I’m going to look at frontier zones – areas what gets done is evolving. Liminal zones where the ocean and fog meet.
Finally, I’m going to look at failure prevention in games analytics. (I didn’t want to leave you sad by titling it failure.) This is a glass half full take on things.
I’ve only picked out a few things under each heading, and there might be more that you think of. Tell me in question time, or afterwards in the bar.
You’re probably wondering. How does she know all this stuff? Has she wired my studio up for telemetry? Or is she making it all up?
Well it’s neither. I do what you do – talk to people, read trade press, listen in on social media, look at changes in vendors’ service offerings, go to events, and use this info as tell-tales to see which way the wind is blowing.
Last month I curated and chaired the Data Science and Analytics track at the games AI conference in Vienna, and I’m bringing back a few shiny snippets from that to share with you. Apparently crows don’t actually like shiny things, which I found out after researching this cool picture, but I do.
It makes sense that what gets done in games analytics would be influenced by what’s going on in the games industry itself. So what’s going on in games? It depends a lot on what facet of the market you live in.
The market as a whole is still growing. Could be worse.
But there are two trends that make life difficult. They have to do with distribution – both on mobile appstores and on steam. One trend is the increasing number of games on the market. The other is the stickiness of the top 10 lists.
Visibility in the face of competition, both long-tail and top 10, is a huge challenge.
Best sellers tend to hang about like low cloud over England in summer. This isn’t by accident. It’s to do with store managers wanting to optimise their returns, and giving successful titles visibility via multiple internal channels. Also also, on mobile appstores, it’s about sophisticated use of the advertising ecosystem by top sellers.
Also - the game needs to be good. But that isn’t enough.
What’s happening as a result? It’s making people pay even more serious attention to distribution and visibility. Some are choosing to go with publishers rather than self-publish.
Ouch.
Really. Ouch. It’s bleeping expensive. This isn’t the kind of spend to do casually.
This means having a good grip on your i/o for acquisition. This can get complicated. But the key point is that different players come in from different sources, which have different costs. You need to balance that view of your costs, with predictions about likely revenue. These predictions will become more accurate the longer a run of real data you have, but by that time your media buying window may have closed.
Since it’s so nailbitingly pricey to acquire players, there is an increasing focus on understanding how to keep them. This has always been of interest. But the truism that usually cheaper to retain a customer than to get a new one is being taken more seriously, now that competition for attention is fiercer than ever.
The areas that service providers are focussing on is often directional. App Annie has recently begun to offer competitor intelligence on how players interact with other games. After offering to integrating player metrics with their store performance data, for free.
There’s lots to say about tech enablers. I’m only going to give a light kick to two aspects of it here – but ask me other things in the question time, or afterwards.
One thing I’m seeing is the need for speed, for taking certain decisions. And with that an interest in streaming architectures and algorithms, particularly Spark streaming. There’s a good piece of work from nucl.ai on this, from one of our London games firms.
I’d like to give a shout out to some open sourced work by Mind Candy, on using probabilistic data structures for stream-based metrics. This is very like the material covered in Ilya Katsov’s ‘Highly scalable’ blog, but it includes links to the source code. Most of these approaches use hashes to enable constant-space scalability, at the expense of perfect accuracy. Agrawal from Berkeley is the author to watch here, if this is your bag. He’s got a whole book on it.
Also on the tech enablers front, there’s something almost unbearably hot in the machining learning world, that hasn’t yet become standard operating procedure in game analytics: deep learning.
Actually this is a better picture. There’s huge excitement about deep learning as it enables the system to learn the features which are important – and not only that – learn a hierarchy of features, with lower level features being more general, and higher level features being category specific. This has resulted in big progress in speech recognition, and visual processing. The visual processing work is particularly interesting as it dovetails well with work on neurology of visual processing, and on mathematical modelling of processing channels.
I have heard of people using it in analysis of play data, for player segmentation, but it’s been more along the lines of ‘I’ve tried everything but the kitchen sink and here’s this cool thing I will try too - I’m not quite sure what it’s good for but hey why not’.
I’d say adoption is at the garage tinkering stage. Hence the picture of the messy machine shop. Which looks a bit like my desk.
I’m not sure what problems in game analystics are the right shape, and have a strong analogy to signal processing. I think that’s tbd.
By frontier zone, I mean places where practice is evolving, interesting, and not settled. You could I guess count the use of deep learning as a discovery technique as a frontier zone. Here are a few more.
I’m not talking about Dark Side of the Moon, as the final frontier. But about being able to segment players (and game elements) in a way that informs design. There’s a particularly good piece of work in this direction from nucl.ai
There has been a fair amount published on churn prediction detection – game play gives off a number of signals that can be useful in this respect. There was some interesting work at CIG2014 on that last summer, from Wooga in collaboration with University of Lausanne. What there’s been less of, is work that integrates churn detection with churn prevention.
Here’s a really good piece of work from the track I chaired. What’s good about it combines analysis for prediction, for analysis to guide interventions. And it worked really well for them.
Another frontier zone I’d wave a hand to is a meeting of minds between games user research and quantitive product management. Games user research is a discipline with its own set of conferences. I went to one in July, in London, and there’s going to be a big one, ChiPlay in London in October. In the whole day there was only one talk that had any numbers in it.
It may not be obvious but these guys could make a great music together if they’d only learn each other’s languages.
Finally here’s something everyone cares about. My poster child for this once again comes from someone else’s work: Meta Brown. She wrote Data Mining for Dummies. She’s not a dummy though. She has an advanced degree in nuclear engineering from MIT. She’s done lots of big ticket analytics project work, and consulting. She gave a talk earlier this summer at the Imperial College Data Science Institute. I’ve never laughed so hard in a data science talk. I can’t begin to imitate her dry midwestern sense of humour, but there was some good stuff she said that I think bears repetition.
This is like the kind of koan you can chant while walking around whacking yourself on the head with a board. It’s that good.
If you don’t get clear agreement about what success looks like – and to whom – success is going to be elusive. Hugely so.
I’m not talking here about collecting basic metrics, but about going offroad, or on a deep dive in search of treasure.
The key message here is to agree the business problem – and understand it as well as possible – before touching a drop of data. I do see people enjoying just diving in to see what’s there, and that’s fun for a side proejct, but being prepped property makes it easier to explore further.
And as you get more complicated – you need more process support. That means Crisp-DM, or some cousin. According to Meta.
Don’t go mad for big data when small data will do. Use it for what it’s good for.
Here I’ve talked about what I think’s going on at the moment. As to what next, let’s take it to the bar.