DraftKings is the largest Daily Fantasy Sports platform on which millions of users compete in 11 different fantasy sports for a chance to win real money. The traffic on the DraftKings website and native apps is highly volatile and directly correlates with the interest in sporting events, with scoring events in those games frequently driving a five-fold spike in traffic on a minute-to-minute basis. In this session, you learn how DraftKings runs a highly elastic server farm and scales its Windows-based infrastructure using a combination of AWS Lambda, Amazon CloudWatch, Auto Scaling groups, and predictive analysis techniques.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
The Gronk Effect
E f f i c i e n t l y H a n d l i n g H u g e S p i k e s i n T r a f f i c U s i n g P r e d i c t i v e
A n a l y t i c s a t D r a f t K i n g s
W I N 2 0 2
N o v e m b e r 2 7 , 2 0 1 7

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
WHAT’S IN STORE?
In this presentation, you will learn strategies for scaling infrastructure based upon predicted demand as
opposed to current demand
Current Demand
The CPU on your server is currently at 70%
Predicted Demand
Two hours from now, your peak traffic rate will be 1 million requests per minute
These strategies are suitable for use cases where demand changes too rapidly to respond reactively

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
WHAT DOES DRAFTKINGS DO?
A bit of context to help understand our technical problem
DraftKings is a sports media entertainment company and the largest provider of Daily Fantasy Sports
What’s Daily Fantasy Sports?
• Daily Fantasy Sports (DFS) are similar to season long
fantasy sports, with the key exception that the entire
contest starts and ends in a much shorter time frame.
• Contests range from a single day to one week,
depending on the sport
• Users pick a team of players based on a salary cap,
and those players earn points based on their in-game
performance.
• Players test their skills and knowledge playing
privately with friends or with other fans globally.

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
WHAT IS DRAFTKINGS’ SCALE?
At our size, efficient scaling of infrastructure translates into significant monetary savings.
1MM
API
Requests/Min
8MM
Registered
Users
22K
Entries/Min
60+
Microservices

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
WHAT IS DRAFTKINGS’ TECH STACK?
DraftKings is a large deployer of Windows machines within the AWS Cloud
Why Windows?
Warning: Highly subjective opinions ahead
• Incredible IDE with Visual Studio
• Flexibility of C#—benefits of a statically typed language with dynamic capabilities
• Founding team had a strong network of C# developers
Other Technologies in Use:
• Amazon Aurora
• Amazon MySql RDS
• ELB
• Swift, Objective C, Java, React

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
THE GRONK EFFECT
Gronk Effect (Noun): The request pattern to an API when users synchronize requests by watching the same
sports event.

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
OPTION 1
Always Scale to Previous Max
Peak
OPTION 2
Scale On Demand
OPTION 3
Scale to Predicted Short Term
Peak
OUR SCALING PROBLEM
How do you scale for traffic that spikes 5x minute to minute?

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Support 5x increases in
HTTP traffic in 2 minutes
timeframe
Lots of user-specific data
limits caching
effectiveness
Reactive scaling is too
slow to handle traffic
spikes (6 minutes)
CHALLENGES

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
N F L T r a f f i cM L B T r a f f i c
TRAFFIC PATTERN BY SPORT
MLB Peak

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
SCALING SOLUTION
Scale to predicted peak

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Manual
• Time consuming
• Error prone
Scheduled
• Managed by centralized process; not
scalable as number of services grows
• Over-provisioning of environment
leading to increased costs
DRAFTKINGS’ PREVIOUS SCALING STRATEGIES

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Prediction Based Scaling
• Time to scale up no longer a
concern
• Reduced costs by lowering the
average number of running
EC2 instances
• Scaling actions are
decentralized, scales as
number of services grow
SOLUTION

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
PREDICTIVE SCALING SYSTEM OVERVIEW
Predict Traffic
Compute
Scaling Needs
Scale Out
Step 1 Step 2 Step 3

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
PREDICT TRAFFIC
Pace UpdaterAurora RDS
Instance
Traffic Predictor
Aerospike cluster
CloudWatch
Event CRON
Load data
from
database
Publish
prediction

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
COMPUTE SCALING NEEDS
Max HTTP Traffic
Prediction
NOAA .NET
Webservice
Amazon CloudWatch
Required Scale
Out metric
Current Production Scale

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
COMPUTE SCALING NEEDS—DETAILS

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
SCALE OUT
CloudWatch alarm on
service load metric
Stepped scaling policy
increases desired
count
Scale Out
SNS Topic
.NET Lambda function
executes safety checks
Triggers scale
out policy
Instance
launched
lifecycle hook
Invokes
Lambda
function
Function
completes
lifecycle hook

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
A u t o S c a l i n g p o l i c yC l o u d W a t c h a l a r m
SCALE OUT—DETAILS

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
{
"LifecycleHookName": "scale_out",
"AccountId": "248797271221",
"RequestId": "d8857c21-8662-45eb-80cc-90f5516950e1",
"LifecycleTransition":
"autoscaling:EC2_INSTANCE_LAUNCHING",
"AutoScalingGroupName": "Titan-Service",
"Service": "AWS Auto Scaling",
"Time": "2017-10-10T22:22:47.808Z",
"EC2InstanceId": "i-0fe90523624cd5d71",
"LifecycleActionToken": "2c5492b6-29b8-4ec6-af46-
57e05c87f77b"
}
SCALE OUT—DETAILS

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
SCALE OUT—DETAILS

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
SCALE PREDICTOR—HIGH LEVEL PATTERN
Latest state
available for
consumption
by scaling
service
• AWS Lambda function
recomputes every 2 minutes
• Prediction method: Kalman Filter
used at DraftKings
• Method of prediction should be
customized to your traffic
patterns
Traffic
Predictor
State
Storage
• JSON-formatted object stored in
database or on Amazon S3
• Constantly updated by prediction
engine
• Aerospike used at DraftKings

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
UNDER THE HOOD—TRAFFIC PREDICTOR
Target:
Prediction of
upcoming
traffic
Incoming
Data Stream
Kalman
Filter
Kalman Filter is an intelligent state updater
• Algorithm considers previous states, changes since previous
state, and prior historical traffic pacing to constantly update
traffic projections
• Kalman Filters used a wide range of applications, including
projecting missile trajectories and projecting the positions of
self-driving vehicles
Historical
Benchmarks

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ADAPT MODEL TO FIT YOUR TRAFFIC PATTERNS
Kalman Filter ARIMA Model
Choose the right tool based on your enterprise traffic patterns
• Daily Fantasy Sports traffic pattern will differ greatly from other types of businesses
• Choose prediction methods and model inputs that suit your needs—many possible estimation frameworks
Hidden Markov
Model
Hazard Model Others
ARCH Model

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
{
"PacerVersion": "Tomahawk",
"PredMaxRequestRate1hr": 5894,
"PredMaxRequestRate2hr": 6143,
"PredMaxRequestRate3hr": 7294,
"PredMaxRequestRate4hr": 8483,
"PredMaxRequestRate4hr": 10945,
"PredMaxRequestRate5hr": 15827,
"PredMaxRequestRate6hr": 26113,
"PredMaxRequestRate7hr": 8094,
"PredMaxRequestRate8hr": 9344,
"PredMaxRequestRate9hr": 11995,
"PredMaxRequestRate10hr": 7566,
"PredMaxRequestRate11hr": 8732,
"PredMaxRequestRate12hr": 9585,
"UpdateDate": "2017-10-08 10:23:45"
}
UNDER THE HOOD—MAINTAINING STATE
Kalman filter stores its projections as JSON-formatted
object
• Includes traffic projections for upcoming 12 hours
• Updates every 2 minutes
• Stored in database or S3—available for query by scaling
service

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!