For Killzone Mercenary and Killzone ShadowFall, Guerrilla Games and SCE Online Technology Group switched from an inflexible, classic 3-tier architecture to one using well understood tools and AWS technologies. This switch allowed them to deliver more title-specific features without losing the benefits of sharing code between titles easily. This session covers problems in the previous architectures, how they fundamentally changed how they built game servers, some of the problems they faced while rebuilding from the ground up, and rabbit holes they got stuck down—both generally, and specific to their use of AWS Elastic Beanstalk, Amazon DynamoDB, and other AWS services. They show how they were able to react quicker to the changing needs of the title, make last-minute feature changes, whip up servers at a moment's notice, and switch out implementations when a new AWS feature came along that would deliver performance or cost improvements.

1. (re)Building Killzone’s Servers
How We Used AWS in Killzone: Shadow Fall and
Killzone: Mercenary
Tim Darby, Sony Computer Entertainment Europe
November 14, 2013
© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

2. Bio

3. Aims

4. What is Killzone?

5. A brief history of Killzone’s servers

6. Killzone
Killzone
(2004)
•
Serving core multiplayer
experience
•
Centrally developed backend tech
•
No title-specific backend code
•
C/C++
•
Regional real-time gaming servers
•
Centralized core servers
Killzone :
Liberation
Killzone
2
Killzone
3
Killzone : Mercenary
Killzone : ShadowFall
(2006)
(2009)
(2011)
(2013)

7. KZ : Liberation
Killzone
(2004)
•
Original real time backend plus …
•
Java 3-tier backend for community
features
•
Shared tech again
•
Customization at presentation layer, plus
configuration
•
Plugins for score processing and
tournament progression in ‘singleton’
•
Leaderboards sat on the DB
Killzone :
Liberation
Killzone
2
Killzone
3
Killzone : Mercenary
Killzone : ShadowFall
(2006)
(2009)
(2011)
(2013)

8. Killzone 2
Killzone
(2004)
•
Applying lessons learned from Killzone :
Liberation
•
Materialized views added to leaderboard
backend
•
Battle Replay using shared file system
(NFS)
•
Plugins in ‘singleton’ servers for
Tournament progression
•
Additional webapp delivering XML for
website
Killzone :
Liberation
Killzone
2
Killzone
3
Killzone : Mercenary
Killzone : ShadowFall
(2006)
(2009)
(2011)
(2013)

9. Killzone 3
Killzone
(2004)
•
Iteration on previous backend
(java and C++)
•
Redone leaderboards from
Materialized Views to MV + TSDB
•
Plugins to middle tier
•
C++ side matchmaking server with
title-specific MM rules plugin
•
And it wasn’t just us feeling the
constraints…
Killzone :
Liberation
Killzone
2
Killzone
3
Killzone : Mercenary
Killzone : ShadowFall
(2006)
(2009)
(2011)
(2013)

10. A Busy Year
September 2013
Killzone
(2004)
Killzone :
Liberation
Killzone
2
Killzone
3
Killzone : Mercenary
Killzone : ShadowFall
(2006)
(2009)
(2011)
(2013)
November 2013

11. Re-arch
Killzone
(2004)
• Separately versioned
shared components
• Independent game
projects
• Maven + Jenkins
• Dynamic Scaling
• Use of AWS services
Killzone :
Liberation
Killzone
2
Killzone
3
Killzone : Mercenary
Killzone : ShadowFall
(2006)
(2009)
(2011)
(2013)

12. Mercenary
Killzone
(2004)
•
Relatively straightforward REST API
•
Frontend, Scheduler, Admin Elastic
Beanstalk applications
•
Multiple Beanstalk Environments per app
(staging, or dev envs)
•
Both “SingleInstance” and “ELB”
Beanstalk Env types used
•
PSN functionality used for real-time
elements
–
Matching, NAT Traversal, P2P
Killzone :
Liberation
Killzone
2
Killzone
3
Killzone : Mercenary
Killzone : ShadowFall
(2006)
(2009)
(2011)
(2013)

13. ShadowFall
Killzone
(2004)
• Straight REST, with addition of
Comet style push messaging
• Similar topology for Elastic
Beanstalk elements to KZM
• Addition of real-time servers
• Hybrid deployment – part Sony
hosting, part AWS
Killzone :
Liberation
Killzone
2
Killzone
3
Killzone : Mercenary
Killzone : ShadowFall
(2006)
(2009)
(2011)
(2013)

14. Use of AWS Services

15. Scheduled Jobs
•
Scheduler webapp simple spring-quartz
•
Generates jobs, posts to Amazon SQS
•
Main webapp runs SQS receivers
•
Scheduled job receivers spawn worker threads
per job
•
Wrappers around these jobs provide
–
–
•
Amazon CloudWatch duration/error tracking
Distributed job management
Admin app provides ability to manually
schedule/fire jobs, and view distributed job
state info

16. Score Posting #1
•
Score posts pushed to SQS for async
processing
•
Listeners on component for game
history store triggered on successful
write to Amazon DynamoDB
•
Individual listeners perform many
secondary tasks
•
Failure modes in secondary tasks not
easily recovered

17. Score Posting #2
•
Changed game history listeners to
post to secondary game jobs SQS
queue
•
Added separate receiver threads for
secondary game jobs SQS queue
messages
•
Receivers then run secondary jobs
•
Failure modes in secondary jobs now
result in non-confirmation of SQS
message, and re-presentation by SQS

18. Leaderboards
•
SimpleDB, transitioned to DynamoDB
when available
•
DynamoDB required secondary indices
in some form for multiple sort columns on
a single board
•
Single Table vs. Multiple Table
implementations
•
Fast ranking a problem for both
•
Fast ranking provided by reverse index
rank calculator or statistical predictor

19. Issues and Lessons Learned

20. Issues
•
Old-style monitoring of dynamically
scaled environments
•
Manual scaling management on
DynamoDB
•
DynamoDB, leaderboards, and hotspots
•
Surprise! Changes to console
•
Limits of AWS CloudFormation

21. Lessons Learned
•
Appreciate the change in mindset going
from SQL to NoSQL
•
Swappable implementations generate
flexibility
•
Flexibility around architecture adds
options
•
Dynamic scaling/management doesn’t
necessarily stop at server boxes
•
Still no silver bullet for leaderboards

22. Summary

23. Questions?

24. Please give us your feedback on this
presentation
MBL305
As a thank you, we will select prize
winners daily for completed surveys!