This document discusses Electronic Arts' use of Docker containers to improve the scalability and deployment of their mobile game server infrastructure. It describes how EA was facing challenges with manual and complex deployment processes. With Docker, EA was able to break their application into microservices, package them as containers, and use tools like Jenkins and Fleet for automated builds, deployment, and orchestration. This resulted in faster and more reliable deployments while improving scalability to meet the demands of live service games.

1. It's in the game: The path to micro-services
at Electronic Arts with Docker
Andrew Hately
IBM CTO Cloud
Architecture
Scott Porter
EA Cloudcell Server Lead
Melbourne Firemonkeys

2. GAMING MARKET GROWING ACROSS ALL PLATFORMS AND
REGIONS
Mobile PC/F2P Console
NORTH AMERICA LATAM EUROPE ASIA/ROW
GROWTH YoY 12%
REVENUES 2015est.
21B
GROWTH YoY 19%
REVENUES 2015est.
4B
GROWTH YoY 7%
REVENUES 2015est.
17B
GROWTH YoY 9%
REVENUES 2015est.
40B
17% 46% 37%41% 16% 43%
195M 182M 553M 817M
33% 24% 43% 51% 44% 5%
2015 est. Revenues Consist of Software Gaming Market

3. MOBILE FUELING INDUSTRY GROWTH
$3.3 $4.3
$6.1
$9.9
$17.0
$27.8
$34.0
2009 2010 2011 2012 2013 2014 2015
Mobile games industry has grown 10x in less
than a decade
Mobile Gaming Revenue, WW $B

4. Mobile Game Server architecture/requirements
• Mobile Gaming Back-end is a very typical Web application
• Frequent updates to game content - infrequent updates to game engine
• Daily peak users 150,000 to 250,000 simultaneous users and 1-2 million
daily
• Need to scale-up/down around release and content pack releases
• Stateless front-ends with memory caching and both SQL and no-SQL DB
• 50TB per day of data - Increasing use of analytics on user experience,
recorded game telemetry, replays, etc.
Firewall
Load
Balancer
NGNIX
Game Server REDIS
Memcache
MySQL cluster
PHP
PHP
PHP
PHP
PHP
Object Storage

5. Demo – Docker for
Gaming

7. • Our current Ops complexity
• Partnership with IBM
• Why we believe Docker can help
• How we integrated Docker

8. Scott Porter
Cloudcell Server Lead
EA Melbourne Firemonkeys
scott@ea.com
@sl_porter

9. EA MELBOURNE FIREMONKEYS
• Moble studio based in Melbourne,
Australia
• Formed in 2012 from two EA Mobile
studios, Iron Monkey and Firemint
• Focused on developing the highest
quality mobile titles

10. • Firemonkeys in house Client/Server
Tech Stack
• Mobile focused connected features
• Originally developed by Firemint for
Flight Control

11. Integrated into all Firemonkeys live titles

12. • Live Service Games
• Bi monthly client releases
• No downtime
What do we do?

13. • 13.5k Peak Requests/s
• ~900M Requests/day
• ~430k Peak Concurrent Players
• 5M – 8M Daily Active Players
• > 100 TB Binary Data
• 170 Hosts (60 Live Hosts)
• Team of 7
Current Scale

14. • Baremetal Hosts
• Nginx, PHP, Memcache, MySQL, Redis
• Batched client requests, short-lived, stateless
• Requests can touch any datastore
Architecture

15. Game Logic Data StorageLoad Balancing
Architecture
External Services
StatsD / Diamond /
Graphite / Graphana
Object
Storage
Datadog

16. • Automated Development / Manual Live process
• In house database migration and code
deployment system
• Bash & rsync
• Rollbacks are handled via code duplication
Current Deployment Methodology

17. • Asynchronous processes (workers) manually
configured ( services, cron jobs )
• Brittle.
• SaltStack is awesome!
• But still slow, complex, no HA
Current Orchestration Methodology

18. • Moved binary data to SoftLayer Object Storage
(Open Stack Swift)
• Currently provision for Peak
• Shard live environments per game title
• Interested in transitioning to microservices
Current Scalability

19. • Improved deployments
• Growing ecosystem
• Scalable
• Easy transition, run side by side with existing stack
Why Docker?

20. • Focused on the Game Logic Layer
• Decomposition
• Nginx
• Admin
• Application
• Workers
Packaging Cloudcell

21. • Generic Containers
• Run multiple containers, not processes
• Tradeoff considerations
Packaging Cloudcell

22. Code
Building Images
Jenkins
Docker Registry
Base Images
nginx
php-fpm
php-worker
Built Images
php-fpm-nfs-0001
php-fpm-nfs-0002
php-worker-a-1
nfs-0002.tar
worker-a-1.tar
nfs-0001.tar
Dev Env
Jenkins
Application Code
(SVN)

23. • Fleet
• Unit files per service
• Nginx
• Admin
• Application
• Each worker type
Orchestration

24. Load Testing
Host
Baremetal
nginx
Application
Worker A Load Generation
LTG LTG
LTG LTG
LTG
LTG
Host
Docker
Docker
nginx
php-fpm-nfs-0001
php-worker-a-1
etcd
Fleet

25. • Overall 5 – 10% overhead running Load Test with Docker
Load Testing Results
PLACEHOLDER
NEED A REAL
GRAPH HERE!

26. • Docker Private Registry
• Jenkins
• Studio Jenkins ships code to build machine
• Build machine builds and commits images to
Registry
• Generates Fleet configuration from templates
• Executes Fleet deployments
Deployments

27. • Config Editor
• Bridges Cloudcell Application deployments to
environment state
• Provides a simple UI for building, deployments,
and state via Jenkins and Fleet
Deployments

28. Deployments
Docker Build
Machine Host
Docker
etcd
Jenkins
Docker Registry
Config Editor nginx
php-fpm-nfs-0001
php-worker-a-1
Fleet

29. Docker enhanced architecture on SoftLayer
CentOS 7 host
Discovery/Config
Fleet managed Docker services
nginx-
confd.service
NGINX-confd
container
Orchestration Docker
php-
fpm@1.service
ea/php-
fpm:latest 1
php-
fpm@2.service
ea/php-
fpm:latest 2
php-fpm-discovery
@2.service
php-worker-
A@2.service
ea/php-
worker:latest
2
php-worker-
B@1.service
ea/php-
worker:latest
1
php-worker-A-discovery
@2.service
php-worker-B-discovery
@1.service
php-worker-
A@1.service
ea/php-
worker:latest
1
php-worker-A-discovery
@1.service
php-fpm-discovery
@1.service
Redis
Externaldataservices
MySQLMemcached
LoadBalancer

30. Configuration, orchestration and deployment
host
servi
ce
servi
ce
servi
ce
servi
ce
host
servi
ce
servi
ce
servi
ce
servi
ce
host
servi
ce
servi
ce
servi
ce
servi
ce
host
servi
ce
servi
ce
servi
ce
servi
ce
Configure topology, maintain state.
Build images with code, deploy to or scale target
environments.
Run containers in staging,
load test, and production.
Capture metrics.

31. • Centralised Logging
• Monitoring - processes inside containers
Future Work

32. — Scott Porter
“Using IBM and Docker container
solutions, we have been able to
build a gaming platform that is
massively scalable, highly
performant and meets our
deployment velocity needs.”
33

33. Thank you!
@cloudtroll
hately@us.ibm.com
Andrew Hately Scott Porter
@sl_porter
sporter@ea.com