講師: Zhao Fei, Solutions Architect, Amazon Web Services Architect 架构之道 Database 数据之道 Operation 运维之道 Security 安全之道

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Zhao Fei Solution Architect
2018-3-8
Chinese gaming customer cases sharing

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Agenda
Architect 架构之道
Database 数据之道
Operation 运维之道
Security 安全之道

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Architect

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Oasis Games: A Global Player
Over 45games in different genres
More than 40countries
In 16languages
Mobile / Browser / Steam / PS4

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Legacy Oasis Payment Platform
Panel Pay Pages

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Phase 1: Infrastructure replacement
 Computing
 Storage
 Network
S3
Region
Availability
Zone A
Tools
1. ApacheBench (http://httpd.apache.org/docs/2.2/en/programs/ab.html): simple HTTP web
server load test tool
2. Siege (https://www.joedog.org/siege-home/) : HTTP load test and benchmark tool
3. JMeter (http://jmeter.apache.org/) : very famous load test tool based on JAVA, originated
from 1998
4. The Grinder (http://grinder.sourceforge.net/): agent based load test framework, Java.
5. Gatling (http://gatling.io/) : Scala load test framework based on Akka and Netty
6. Tsung (http://tsung.erlang-projects.org/) : Erlang load test tool, support HTTP, websocket,
databse, MQTT, TCP
7. Bees (https://github.com/newsapps/beeswithmachineguns) : AWS oriented opensource
load test tool
8. Vegeta (https://github.com/tsenart/vegeta) : Go based opensource HTTP load test tool
9. RedLine13 (https://www.redline13.com/) : AWS Advanced Tehchnology Partner
10.Loader.io (https://loader.io/): load test service base on Cloud server.
11.Goad (https://goad.io/) : load test using AWS Lambda, based on Go. The result are
delivered to AWS SQS.

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Phase 2: External services
CloudFront acceleration
S3
Route53 Geo IP
CloudFront
CDN
ELB
S3
EC2EC2EC2
Region
Availability
Zone A
Availability
Zone B
TIPS
• One regions server is available
depends on game types
• Use CF for HTTP(s)
• Use Proxy Server for TCP
• Design your game service allowing
min latency ex) buffering, async
simple AI
• Check all latency in every region.

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Good architect, but..
Availability Zone
Amazon
Route 53
User
Amazon S3
Amazon
Cloudfront
Availability Zone
Elastic Load
Balancing
RDS DB Instance
Read Replica
App
Instance
App
Instance
App
Instance
RDS DB Instance
Read Replica
App
Instance
App
Instance
App
Instance
RDS DB Instance
Standby (Multi-AZ)
RDS DB Instance
Active (Multi-AZ)

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
0
20
40
60
80
100
120
2013 2014 2015 Jul, 2016
Total Orders (million)
1st Challenge :Growth & Slow
0
2
4
6
8
10
12
14
16
2013 2014 2015 Jul, 2016
Anaylze Query Cost (s)

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
2nd Challenge: Distributed data accessing
EU
Beijing
(Panel)
US
Faster
Data
Source
cross-region
replicated

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• MySQL tuning
• AWS Aurora
• Vertical scale (upgrade)
• Split cold/hot data
• Convert JOIN to staging tables
Review
Remaining issues:
• Not fully managed (not easily
scalable)
• Analysis limited to certain time
frame
• Adding future logic (index) is
very difficult
• Analysts in Beijing still wastes
a lot of waiting time

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Configurations User-Generated
Volume: small, not scaling much large and growing
Entity: some a few
Beijing: read-write read-only
End-nodes: read-only read-write
Data types

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• The cause of scalability:
• only UG data
• Relationship enforcement:
• not needed in UG data
• Transaction protection:
• lock logic still needed
• Analysis Query:
• we need DB, not relations
All the data should be R ?
Try
DynamoDB

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Database refactor
Availability Zone
Amazon
Route 53Players
Amazon S3
Amazon
CloudFront
Availability Zone
ELB
DynamoDB
App
Instance
App
Instance
App
Instance
App
Instance
RDS DB Instance
Standby (Multi-AZ)
RDS DB Instance
Active (Multi-AZ)
Users &
Transactions Data
Products Data
Streaming to
Lambda
Incremental
backup
Static contents
Business Regions( US, Europe)
• Migrate UG data to
DynamoDB
• Keep Configuration in RDS

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Cross region data synchronize
Availability Zone
Amazon
Route 53Player
s
Amazon
S3
Amazon
CloudFro
nt
Availability Zone
ELB
DynamoD
B
App
Instance
App
Instance
App
Instance
App
Instance
RDS DB
Instance Standby
(Multi-AZ)
RDS DB Instance
Active (Multi-AZ)
Users &
Transactions
Data
Products Data
Business Regions( US,
Europe)
Streaming to
Lambda
Incremental
backup
Static contents
Analysis Region( Asia)
Data Loaders
Cross region
Read Replica
SQSSNS
• RDS cross-region
replica for Configuration
• SNS: At least once
delivery to SQS
• SQS:
Reliable,transactional

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• Limitation of DynamoDB:
• not SQL-able
• limited amount of GSI/LSI
• cost unfriendly if you choose to use GSI
• DynamoDB ≠ a choice for analysis data source
What about analysis query?
Let’s add
the R back

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
RDB redefined: Redshift
• Have a copy of user data in
Redshift
• Only deploy in panel node
Analysis Region( Asia)
Data Loaders
Redshift
Data warehouse
App servers
ELB
Analysts
Amazon
S3
Cold data backup

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Phase3: Managed services
Availability Zone
Amazon
Route 53Players
Amazon S3
Amazon
CloudFront
Availability Zone
ELB
DynamoDB
App
Instance
App
Instance
App
Instance
App
Instance
RDS DB Instance
Standby (Multi-AZ)
RDS DB Instance
Active (Multi-AZ)
Users &
Transactions Data
Products Data
Business Regions( US, Europe)
Streaming to
Lambda
Incremental
backup
Static contents
Analysis Region( Asia)
Data Loaders
Cross region
Read Replica
Redshift
Data warehouse
App servers
ELB
Analyst
s
Amazon S3
Cold data backup
SQSSNS

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Summary: 3 Levels
Infrastructure
Replacement
External
Services
FaaS
using Lambda
DB & Storage
Alternative
Clustered
Analytics
DevOps
Toolset
Managed Services, as many as you can

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Database

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
FUN+: King of Avalon

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS
Availability Zone
Availability Zone
EC
2
EC
2
Amazon
S3
Amazon
Redshift*
Player Internet
gateway
Elastic Load
Balancing
Public Subnet
Public Subnet
Private Subnet
Private Subnet
Web Servers
Web Servers
APP Servers
APP Servers
DB Router
DB Router
Amazon
Route 53
Muliti-AZ
SQL
instance
Amazon
CloudFront endpoints
Muliti-AZ
SQL
instance
........
AWS Data
Pipeline
Amazon EFS
KOA: Asynchronous Online Gaming

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
BINLOG DATA DOUBLE-WRITELOG FRM FILES
TYPE OF WRITE
MYSQL WITH REPLICA
EBS mirrorEBS mirror
AZ 1 AZ 2
Amazon S3
EBS
Amazon Elastic
Block Store (EBS)
Primary
Instance
Replica
Instance
1
2
3
4
5
AZ 1 AZ 3
Primary
Instance
Amazon S3
AZ 2
Replica
Instance
ASYNC
4/6 QUORUM
DISTRIBUT
ED WRITES
Replica
Instance
AMAZON AURORA
780K transactions
7,388K I/Os per million txns (excludes mirroring, standby)
Average 7.4 I/Os per transaction
MySQL IO profile for 30 min Sysbench run
27,378K transactions 35X MORE
0.95 I/Os per transaction (6X amplification) 7.7X LESS
Aurora IO profile for 30 min Sysbench run
My SQL VS Aurora

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
MySQL shard consolidation
Master
Read
Replica
Shared distributed
storage volume
M S
M M
M
S S
S
MySQL shards Aurora cluster

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Aurora migration options
Source database From where Recommended option
RDS
EC2, on-premises
EC2, on-premises, RDS
Console based automated
snapshot ingestion and catch
up via binlog replication.
Binary snapshot ingestion
through S3 and catch up via
binlog replication.
Schema conversion using
SCT and data migration via
DMS.

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Operation

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Tap4Fun
A gaming company in Chengdu
Island Empire Galaxy Empire Global Threat Galaxy LegendSpartan WarsKing’s Empire

28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
More and more servers..

29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Operation based on AWS API and Tags

30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Simplify provisioning
CMDB system replace Excel
Simply and safely
Automatic backup
Resources management even after
termination

31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
About tags
Organizations, technical, business, automation, cost, security
Case sensitive
Development or development
Numbers
Static VS Dynamic

32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Security

33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Attack at any time
Highest : 1.5billion qps

34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DRT
1. Always-on monitoring engages the AWS DDoS
Response Team (DRT)
2. DRT proactively triages DDoS attack
3. DRT creates AWS WAF rules (prior
authorization required)
Proactive DRT engagement

35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Attack at any where
Auth
Server
Game
Server
Authentication

36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Protection Consolidation

37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

38. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
GameLift

39. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!