3. Game day & Chaos
Engineering
Concepts on AWS
•Famous Disasters
•Catastrophes common thread
•Ingredients for Catastrophes
•Classess of issues
•Fragility vs Resilience
•Resilience vs Antifragility
• GameDay Concept
• AWS GameDay with the AWS Well-Architected Framework
• Example GameDay Plan
• Chaos Experiments
• Resource Exhaustion
• The Network is Not Reliable
• Datastore saturation
• DNS Unavailability
4. Game day & Chaos
Engineering
Concepts on AWS
• Chaos Engineering
• Examples of inputs for chaos experiments:
• How Does Chaos Engineering Differ from Testing?
• Software Tools for Chaos Engineering
• Chaos Toolkit
• Simian Army
• Chaos Monkey
•Chaos Monkey Basic Demo on AWS
6. Game day & Chaos
Engineering
Concepts on AWS
In February 2017, Cloudflare faced a major
software bug that led to sensitive customer
data like passwords, cookies and
authentication tokens to get leaked from
customer websites.
Bitcoin Unlimited suffered a serious
memory leak which caused several nodes
to fall from 800 to about 300. This is
almost 70 percent of the nodes run by
Bitcoin Unlimited at the time.
7. Game day & Chaos
Engineering
Concepts on AWS
Less than a week into 2016, HSBC became the first
bank to suffer a major IT outage. Millions of the
bank's customers were unable to access online
accounts. Services only returned to normal after a
two-day outage.
The bank’s chief operating officer Jack Hackett
blamed a “complex technical issue” with its internal
systems.
In June 2015 about 600,000 payments failed
to enter the accounts of RBS overnight –
including wages and benefit payments. Many
took several days to come through. The
bank’s chief admin officer said a “technology
fault meant we could not ingest a file from a
third-party provider”
19. Game day & Chaos
Engineering
Concepts on AWS
GameDay
Concept
• GameDay is a hands-on competition with live data. By setting up a “fake-real”
environment for the teams to work in, players are free to try things out in a live
environment with no actual risk. It’s unusual to have such a “carefree experience
working in a live environment. Ironically, though, the one-day time frame is actually
“kind of punitive. In a real situation you’d probably have more time to deal with each
problem, but the high-pressure environment adds to the excitement, and AWS
GameDay’s goal is to entertain and make the learning experience as fun as possible.
20. Game day & Chaos
Engineering
Concepts on AWS
• How Are You Managing Keys?
• How Are You Planning for Disaster Recovery?
• How Does Your System Withstand Component Failures?
Looking at GameDay through the lens of the Well-Architected
Framework, it was obvious that there were many opportunities
for improvement. The AWS review team prioritized the findings
into two sets: critical and recommended. Most of the findings
were classified as recommended—these don’t pose an
immediate risk and will be incorporated into our roadmap.
However, the three elements that were identified as critical
needed to be addressed immediately.
21. Game day & Chaos
Engineering
Concepts on AWS
Target
In any GameDay, an exact target or targets should be specified. Without it, it’s
impossible to bring in the right people to run and observe the GameDay.
Time And Place
Things to include:
• Precise date
• War room for in-person attendance (make sure it fits enough people)
• Dial-in information - including conferencing link, phone number and code to join
Goals Of The GameDay
When planning, we need a goal for the GameDay. This is to ensure that we create
relevant test cases. Sometimes the goal is to replay as many previous production
impacts as possible, to test whether or not the current systems are more or less
resilient.
22. Game day & Chaos
Engineering
Concepts on AWS
Whiteboarding
• With so many great minds present and aligned on the goal of the GameDay, it’s
the perfect time to whiteboard out a system’s architecture. This session helps
paint a clear picture of what we’re about to break, and makes obvious some
areas worth testing.
Test Cases Scoping
• Test cases are developed to help answer the question, “What could go wrong?”
or “Do we know what will happen if this breaks?” As a team looks at the
architecture that’s on the whiteboard, you will start to identify areas of concern.
23. Game day & Chaos
Engineering
Concepts on AWS
Make a plan
• Role play and scope definition
• Create the simulation environment
• Set a deadline
• Create the GameDay Environment
• Activate AWS CloudTrail ; Gameplay recording and auditing
• Simulate activity
24. Game day & Chaos
Engineering
Concepts on AWS
Resources on computers are finite. A machine/VM/container will
inevitably hit a resource limit at some point, and the application will be forced to
handle the lack of a resource. Commonly, this is CPU, Memory, or I/O.
We can reproduce CPU exhaustion by conducting a chaos experiment. Running
this experiment will consume CPU cycles, leaving the application with the same
amount of customer-facing work, and less CPU to do it with. As always, we
advocate starting small on a single instance, then increasing the blast radius as
confidence grows. Common reactions to CPU exhaustion are an increase in
errors and latency and a reduction in successful requests to customers.
Attack: CPU / Memory / Disk
Scope: Single instance
Expected Results: Rate of good responses goes down, errors increase at all
layers, brownout mode entered (if implemented), alerts fire (if configured at
single-instance level), load balancer routes traffic away (if applicable)
25. Game day & Chaos
Engineering
Concepts on AWS
Network
Latency
Network dependencies are a fact of life in a distributed system, and as
distributed systems are growing in adoption AND complexity, chaos engineering becomes
an optimal way to test for potential failures on the path to increasing resilience.
Attack: Network Blackhole / Latency
Scope: single instance
Expected Results: Traffic to dependency goes to 0 (or gets slow), startup completes
without errors, application-level metrics in steady-state are unaffected, traffic to
fallback systems shows up and is successful, dependency alerts and pages may fire (if
scoped to single-instance)
26. Game day & Chaos
Engineering
Concepts on AWS
All interesting applications have some sort of storage, and managing the
relationship between application and datastore is critical to overall system health.
There are a variety of ways that an application may overwhelm a data store (poor
queries, lack of indices, bad sharding, upstream caching decisions, etc), but all of
them result in what appears to be an unresponsive data layer.
It’s important to understand how datastore saturation manifests in your
application. There are a few ways of modeling this with a Chaos Experiment. You
can blackhole your datastore, making it appear completely unavailable. You can
add latency to requests to your datastore, making it appear slow. Finally, you can
consume I/O bandwidth to simulate a congested path to the datastore.
Attack: Network Blackhole / Latency / IO
Scope: single instance
Expected Results: Traffic to datastore is reduced or slower, application-level
metrics in steady-state are unaffected to the degree possible, traffic to fallback
systems shows up and is successful, timeouts and concurrency limits kick in
when appropriate, alerts and pages may fire (if scoped to single-instance)
27. Game day & Chaos
Engineering
Concepts on AWS
The best way forward is to induce a DNS outage and understand how
your application behaves. If you blackhole DNS traffic on a single instance, it will
appear to that instance as if DNS is unavailable. The fixes will vary depending on
the issue, but common solutions are to pass around IP addresses instead of
hostnames for internal addressing and the use of a backup DNS provider.
Attack: DNS blackhole
Scope: single instance
Expected Results: Inbound traffic may drop, traffic to external systems may
fail, startup may not complete successfully
28. Game day & Chaos
Engineering
Concepts on AWS
What happened? Was that expected? What do we do next?
After tests are run, it’s good to take some time to wind down, then have
a follow up recap. This should be done relatively soon after the
GameDay (days, not weeks), as the experience is still fresh for everyone.
How long does it take to detect an event?
Simulate failure situation
Validate assumptions
Prove your architecture
Knowledge of Procedures
How is the communication channel or chain of command during the
game?
29. Game day & Chaos
Engineering
Concepts on AWS
Chaos Use
Cases
Simulating the failure of an entire region or datacenter.
Partially deleting Kafka topics over a variety of instances to recreate an issue that
occurred in production.
Injecting latency between services for a select percentage of traffic over a
predetermined period of time.
Function-based chaos (runtime injection): randomly causing functions to throw
exceptions.
Code insertion: Adding instructions to the target program and allowing fault injection
to occur prior to certain instructions.
Time travel: forcing system clocks out of sync with each other.
Executing a routine in driver code emulating I/O errors.
Maxing out CPU cores on an Elasticsearch cluster.
30. Game day & Chaos
Engineering
Concepts on AWS
The primary difference between Chaos Engineering and these other
approaches is that Chaos Engineering is a practice for generating new information,
while fault injection is a specific approach to testing one condition.
Tests are typically binary, and determine whether a property is true or false.
Strictly speaking, this does not generate new knowledge about the system, it just
assigns valence to a known property of it. Experimentation generates new
knowledge, and often suggests new avenues of exploration.
31. Game day & Chaos
Engineering
Concepts on AWS
01
02
05
03
VaryReal-World Events
Canary Analysis
Hypothesize about
SteadyState
Automate
Experiments toRun
Continuously
04
Run Experiments in
Production
