Deploying a Low-Latency Multiplayer Game Globally: Loadout
•
6 recomendaciones•6,756 vistas
This is a deep-dive straight into the guts of running a low-latency multiplayer game, such as a first-person shooter, on a global scale. We dive into architectures that enable you to split apart your back-end APIs from your game servers, and Auto Scale them independently. See how to run game servers in multiple AWS regions such as China and Frankfurt, and integrate them with your central game stack. We’ll even demo this in action, using AWS CloudFormation and Chef to deploy Unreal Engine game servers.
![AWS Gaming Solutions
[Game] Online Problems Continue; Features
Disabled as [Game Studio] Works on Fixes
“We are working hard to get everyone online
as soon as possible,” [Game Studio] says.
Days later, [Game Studio] leaving
disconnected players stranded
[Game] has a great online mode – if you
can get online, that is
“This is total bull**** I paid for a preorder and
I can’t even connect WTF [Game Studio]??”](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Destacado
Destacado (20)
Similar a Deploying a Low-Latency Multiplayer Game Globally: Loadout
Similar a Deploying a Low-Latency Multiplayer Game Globally: Loadout (20)
Más de Amazon Web Services
Más de Amazon Web Services (20)
Último
Último (20)
Deploying a Low-Latency Multiplayer Game Globally: Loadout
- 1. 1 GDC 2015 – Amazon Developer Day Deploying a Low-Latency Multiplayer Game Globally Nate Wiger Principal, Gaming Solutions @nateware nateware@amazon.com James Gwertzman CEO, PlayFab @playfabnetwork http://playfab.com
- 2. AWS Gaming Solutions [Game] Online Problems Continue; Features Disabled as [Game Studio] Works on Fixes “We are working hard to get everyone online as soon as possible,” [Game Studio] says. Days later, [Game Studio] leaving disconnected players stranded [Game] has a great online mode – if you can get online, that is “This is total bull**** I paid for a preorder and I can’t even connect WTF [Game Studio]??”
- 4. AWS Gaming Solutions http://bit.ly/verizon-latency http://bit.ly/superdata-latency
- 6. AWS Gaming Solutions 100+ms 100+ms 100+ms 100+ms
- 7. AWS Gaming Solutions How To Be Awesome 1. Game servers near players 2. Everything else as HTTP APIs 3. Data replication = bad 4. Local caches = good 5. Nobody really “plays everywhere”
- 8. AWS Gaming Solutions Loosely Coupled Pods Tokyo Oregon Frankfurt Virginia
- 9. AWS Gaming Solutions VPC Subnet VPC Subnet Game API Pods Availability Zone A Availability Zone B VPC Subnet VPC Subnet Auto Scaling group WEB VPC Subnet WEB JOBS Cognito SNS Mobile Push SES
- 10. AWS Gaming Solutions Region ① Login via HTTP API ② Download Game Assets ③ Matchmaking to Game Server EC2 Game Flow EC2 EC2
- 11. AWS Gaming Solutions Region ① Login via HTTP API ② Download Game Assets ③ Matchmaking to Game Server ④ Connect to Server ⑤ Hack Apart Your Friends ⑥ Game Over Game Flow EC2 EC2
- 12. AWS Gaming Solutions Region ① Login via HTTP API ② Download Game Assets ③ Matchmaking to Game Server ④ Connect to Server ⑤ Hack Apart Your Friends ⑥ Game Over ⑦ Write via HTTP API Game Flow EC2 EC2
- 13. AWS Gaming Solutions VPC Private Subnet VPC Public Subnet Game Server Pods Availability Zone A Availability Zone B VPC Public Subnet VPC Private Subnet GAME GAME GAME GAME GAME GAME
- 14. AWS Gaming Solutions Make. Network. Faster. • C3/C4 instance family (or R3 for more memory) • In a VPC • HVM AMI – not PV • Enable Enhanced Networking (SR- IOV)http://bit.ly/ec2-enhanced-net • Linux: Tweak Networking Stack http://bit.ly/linux- tuning
- 15. AWS Gaming Solutions Enhanced Networking (SR-IOV) Before: Hypervisor After: Hardware
- 16. AWS Gaming Solutions VPC Private Subnet VPC Private Subnet RabbitMQ + Elastic Load Balancing Availability Zone A Availability Zone B 10.1.0.13 10.2.0.16 TCP 5672 rabbitmq-node1 rabbitmq-node2 TCP 4369 & 25672
- 17. AWS Gaming Solutions RabbitMQ + Elastic Load Balancing • Clustering Tutorial https://www.rabbitmq.com/clustering.html • Set Queue HA policy to "all" https://www.rabbitmq.com/ha.html • Create Internal load balancer – Listen Port: TCP 5672 • VPC Security Group – For load balancer: TCP 5672 – Between EC2 nodes: TCP 4369 & 25672 • Set up Client to Heartbeat
- 19. AWS Gaming Solutions VPC Private Subnet VPC Public Subnet Redis Pub / Sub Availability Zone A Availability Zone B VPC Public Subnet VPC Private Subnet GAME GAME GAME GAME GAME GAME Auto Scaling group
- 20. AWS Gaming Solutions VPC Private Subnet VPC Public Subnet CloudFormation + Chef Availability Zone A GAME GAME GAME Auto Scaling group
- 22. AWS Gaming Solutions Unreal Engine with CloudFormation UnrealServer unreal_linux.template.json true my-bucket-name Unreal-123.zip sshkeyname 1.2.3.4/24 MyPass123
- 26. AWS Gaming Solutions Where My Servers At? Tokyo Oregon Frankfurt Virginia ? ?
- 27. AWS Gaming Solutions VPC Subnet Server Registration Availability Zone A Availability Zone B VPC Subnet Auto Scaling group WEB WEB Oregon Tokyo VPC Subnet JOBS
- 28. AWS Gaming Solutions Server Registration & Scaling • HTTPS POST /api/servers/register • Include an HMAC (RFC 2104) • Send Server Status – Public IP – # Players – Game Modes • Matchmaking Service – Maintains server list – Removes servers
- 29. AWS Gaming Solutions VPC Private Subnet VPC Public Subnet Game Server Logs Availability Zone A S3 Bucket Amazon CloudWatch Logs GAME GAME GAME
- 30. AWS Gaming Solutions VPC Private Subnet VPC Public Subnet Game Server Logs Availability Zone A Amazon Kinesis S3 Bucket Amazon Redshift GAME GAME GAME
- 32. AWS Gaming Solutions Mini Pods Everywhere China Oregon Frankfurt VirginiaChina Leaderboard
- 33. AWS Gaming Solutions PlayFab: Loadout Architecture
- 34. AWS Gaming Solutions What is PlayFab? PlayFab is a complete live game operations platform. Focus on the fun – we’ll do the boring bits. + Tools for operating & monetizing live games Complete set of cloud-based backend services Eco-system of pre-integrated third-party services .. and more +
- 36. AWS Gaming Solutions Some Loadout Stats • Launched 1/31/14 • Over 1 billion API Calls • At Peak (Feb 2014): – 300k DAU – 120 fixed game servers – 110 EC2 game servers
- 37. AWS Gaming Solutions PlayFab Core Architecture Tenants • No scheduled downtime. • Automatic scaling for load. • Backend changes should be invisible to developers. • Use buffering to smooth over spikes or failures.
- 38. AWS Gaming Solutions AWS cloud: PlayFab Web Services US EU Hardware Game Servers Hardware Game Servers Game Client Amazon Route 53 (3.playfabapi.com) Amazon EC2 (API handling) Matchmaker Instance Instance Game Server Monitor DynamoDB Amazon RDS Amazon S3 Elastic Load Balancing Amazon RedshiftReports service Instance LogsArchitectureOverview Matchmaker (Secondary) Instance US EU SA AUS Amazon EC2 Game Servers Amazon EC2 Game Servers Amazon EC2 Game Servers Amazon EC2 Game Servers
- 39. AWS Gaming Solutions List of PlayFab Web APIs Client Server Authentication Title-Wide Data Management Shared Group Data Authentication AddUsernamePassword GetCatalogItems AddSharedGroupMembers AuthenticateSessionTicket LoginWithAndroidDeviceID GetStoreItems CreateSharedGroup Account Management LoginWithFacebook GetTitleData GetSharedGroupData GetUserAccountInfo LoginWithGameCenter GetTitleNews RemoveSharedGroupMembers SendPushNotification LoginWithGoogleAccount AddUserVirtualCurrency UpdateSharedGroupData Player Data Management LoginWithIOSDeviceID Player Item Management Server-Side Game Logic GetLeaderboard LoginWithPlayFab ConsumeItem GetLogicServerUrl GetLeaderboardAroundUser LoginWithSteam GetUserInventory GetUserData RegisterPlayFabUser RedeemCoupon GetUserInternalData SendAccountRecoveryEmail SubtractUserVirtualCurrency GetUserReadOnlyData Account Management UnlockContainerItem GetUserStatistics GetAccountInfo StartPurchase UpdateUserData GetPlayFabIDsFromFacebookIDs PayForPurchase UpdateUserInternalData GetUserCombinedInfo ConfirmPurchase UpdateUserReadOnlyData LinkFacebookAccount PurchaseItem UpdateUserStatistics LinkGameCenterAccount Friend List Management Title-Wide Data Management LinkSteamAccount AddFriend GetCatalogItems UnlinkFacebookAccount GetFriendsList GetTitleData UnlinkGameCenterAccount RemoveFriend SetTitleData UnlinkSteamAccount SetFriendTags Player Item Management UpdateEmailAddress IOS-Specific APIs AddUserVirtualCurrency UpdatePassword RegisterForIOSPushNotification GetUserInventory UpdateUserTitleDisplayName ValidateIOSReceipt GrantItemsToUsers Player Data Management Matchmaking APIs SubtractUserVirtualCurrency GetFriendLeaderboard GetCurrentGames Matchmaking APIs GetLeaderboard GetGameServerRegions NotifyMatchmakerPlayerLeft GetLeaderboardAroundCurrentUser Matchmake RedeemMatchmakerTicket GetUserData StartGame Steam-Specific APIs GetUserReadOnlyData Android-Specific APIs AwardSteamAchievement GetUserStatistics AndroidDevicePushNotificationRegistration UpdateUserData ValidateGooglePlayPurchase UpdateUserStatistics Analytics
- 40. AWS Gaming Solutions Example of Web API Call API Request: Response from Web Services:
- 41. AWS Gaming Solutions ELB Logging for Title Tracking Game Client Amazon Route 53 (3.playfabapi.com) API Request: Elastic Load Balancing Logs Amazon RedshiftAmazon S3 bucket Reports service Logs Instance Logs include title-specific endpoint for analytics AWS wildcard DNS (*.playfabapi.com) Log event:
- 42. AWS Gaming Solutions Loadout API Usage Report (Oct 2014) API NAME TOTAL CALLS AVERAGE BACKEND TIME (S) AVERAGE RESPONSE TIME (MS) AVERAGE REQUEST TIME (MS) TOTAL REQUEST TIME (M) AVERAGE RECIEVED (BYTES) AVERAGE SENT (BYTES) TOTAL ERRORS adminapi_userinfo 93,081,685 0.03 0.04 0.04 59.83 0 452 1979 gameserver_usercustomdatareadonly 1,582,348 0.06 0.04 0.04 1.06 42 902 290 matchmaker_userinfo 1,138,842 0.20 0.04 0.04 0.81 0 10046 319 matchmaker_playerjoined 1,138,247 0.01 0.04 0.04 0.79 64 2 93 matchmaker_playerleft 1,136,797 0.01 0.04 0.04 0.80 64 2 400 matchmaker_authuser 736,607 0.01 0.03 0.04 0.52 0 48 107 gameserver_usercustomdata 297,735 0.19 0.04 0.04 0.20 0 4080 0 gameserver_usercustomdatainternal 297,469 0.08 0.04 0.04 0.20 0 237 2
- 43. AWS Gaming Solutions Instance Making a Singleton Highly Available Health Check Auto Scaling group Auto Scaling service Instance Matchmaker (Primary) Instance Amazon Route 53) Elastic Load Balancing Matchmaker (Secondary) 1.Auto Scaling is set to keep two instances up. 2.Elastic Beanstalk Health Check terminates bad instance. 3.Amazon Route 53 fails over all traffic to secondary server. 4.Auto Scaling detects failure and starts new instance.
- 44. AWS Gaming Solutions Monitoring Service Health Amazon EC2 (API handling) Elastic Load Balancing CloudWatch Health Check Email Alerts On-call Engineer Mobile Alerts
- 45. AWS Gaming Solutions Handling Spikes Without Losing Data Amazon EC2 API Server Server SSD Amazon Redshift Local Reports service Local event logs Instance Events: logins and purchases Reports service Logs Customer’s game management dashboard Analytics Server Logs RabbitMQ message queue Logs Instance Logstash service Logstash Instance ElasticSearch Instance Instance
- 46. AWS Gaming Solutions Storage in DynamoDB, Amazon RDS, Amazon S3, Amazon Redshift DynamoDB Amazon RDS Amazon S3 Player data: • Accounts • Profile • Inventory • Save data • Player stats • Player currency values • Auditing • Log files • Game replays Amazon Redshift • Analytics data
- 47. AWS Gaming Solutions Managing Tables in DynamoDB Amazon DynamoDB • Player data: accounts, profile, inventory, save data User: UserID: Hash TitleID: range key - DisplayName - Linked accounts - … Custom user data: UserID: Hash TitleID: range key - Key/value 1 - Key/value 2 - … Inventory: UserID: Hash ItemID: range key - titleID - Catalog Item - … …
- 48. AWS Gaming Solutions Physical Deployment AZ 1 AZ 2 PlayFab Web Services PlayFab Web Services N. California AZ 1 AZ 2 PlayFab Web Services PlayFab Web Services Oregon AZ 1 PlayFab Web Services Beijing • Original deployment: us-west-1 • Expanded to us-west-2 in July • Coming soon: cn-north-1 Amazon EC2 Game Server Singapore Game Client Instance
- 49. AWS Gaming Solutions Managing Cross-Region Data Player accounts are shared across all regions: • Cross-regional wrapper for user.get sets order of databases and regions to check • user.get then checks for player account data • High availability, transparent to customers and players // Returns the first matching player record, searching DynamoDB and SimpleDB // in both regions in the specified order. // Updates the player record in the store and region from which it was loaded. // Creates a player record in the default store and region.
- 51. AWS Gaming Solutions AWS cloud: PlayFab Web Services US EU Hardware Game Servers Hardware Game ServersGame Client Amazon Route 53 (3.playfabapi.com) Amazon EC2 (API handling) Matchmaker Instance Instance Game Server Monitor DynamoDB Amazon RDS Amazon S3 Elastic Load Balancing Amazon RedshiftReports service Instance LogsArchitectureOverview Matchmaker (Secondary) Instance US EU SA AUS Amazon EC2 Game Servers Amazon EC2 Game Servers Amazon EC2 Game Servers Amazon EC2 Game Servers
- 52. 52 Nate Wiger Principal, Gaming Solutions @nateware nateware@amazon.com