Building Resilient Serverless Apps with the Circuit Breaker Pattern

Patterns and Practices
for building resilient
serverless applications
presented by Yan Cui
@theburningmonk

@theburningmonk theburningmonk.com

“the capacity to recover quickly from difﬁculties; toughness.”
resilience
/rɪˈzɪlɪəns/
noun

“the capacity to recover quickly from difﬁculties; toughness.”
resilience
/rɪˈzɪlɪəns/
noun
it’s not about
preventing failures!

everything fails, all the time

we need to build applications that can withstand failures

don’t run your application on one server…

entire data centers can
go down…

run your application in multiple AZs and regions

Failures on load: exhaustion of resources

latency
reqs/s
Failures on load: exhaustion of resources
CPU saturation

Failures in distributed systems
Service A Service B Service C
user

user
HTTP 502

user
You suck!

microservices death stars circa 2015

Yan Cui
http://theburningmonk.com
@theburningmonk
AWS user for 10 years

Yan Cui
@theburningmonk
http://bit.ly/yubl-serverless

Yan Cui
@theburningmonk
Developer Advocate @

Yan Cui
@theburningmonk
Independent Consultant
advisetraining delivery

Lambda execution environment

Serverless - multiple AZ’s out of the box
Total resources created:
1 API Gateway
1 Lambda

Serverless - multiple AZ’s out of the box
Total resources created:
1 API Gateway
1 Lambda
don’t pay for idle
redundant resources!

Load balancing

Data replication in different AZ’s
DynamoDB
Global Tables

There are throttling everywhere!

Beware of timeout mismatch
API Gateway 
Integration timeout  
Default: 29s
Lambda 
Timeout
Max: 15 minutes

Lambda 
Timeout
Max: 15 minutes
SQS 
Visibility timeout 
Default: 30s
Min: 0s
Max: 12 hours

Lambda 
Timeout
Max: 15 minutes
SQS 
Default: 30s
Min: 0s
Max: 12 hours
set VisibilityTimeout to
6x Lambda timeout

Ofﬂoad computing operations to queues

better absorb
downstream problems

need way to replay
DLQ events

https://www.npmjs.com/package/lumigo-cli

great for ﬁre-and-forget tasks

“what if the client is waiting for a response?”

“Decoupled Invocation”

task id created at result
xxx xxx <null>
xxx xxx <null>
… … …
task results
not ready…

xxx xxx <null>
xxx xxx <null>
… … …
task results
not ready…
202

xxx xxx <null>
xxx xxx <null>
… … …
task results
reporting for duty!

xxx xxx <null>
xxx xxx <null>
… … …
task results
working hard…
not ready…

xxx xxx <null>
xxx xxx <null>
… … …
task results
202
working hard…

xxx xxx <null>
xxx xxx { … }
… … …
task results
done!

xxx xxx <null>
xxx xxx { … }
… … …
task results
200
{ … }

wait…

a distributed
transaction!

a distributed
transaction!
needs rollback

no distributed
transactions

do the work here

retry-until-success

24 hours data retention

24 hours data retention
need alerting to ensure
issue are addressed quickly

retry-until-success
needs to deal with
poinson messages

what if you can’t avoid distributed transactions?

The Saga pattern
A pattern for managing failures where each action
has a compensating action for rollback

The Saga pattern
https://www.youtube.com/watch?v=xDuwrtwYHu8

The Saga pattern
Begin transaction
Start book hotel request
End book hotel request
Start book ﬂight request
End book ﬂight request
Start book car rental request
End book car rental request
End transaction

The Saga pattern
model both actions and
compensating actions as
Lambda functions

The Saga pattern
use Step Functions as the
coordinator for the saga

The Saga pattern
Input

The Saga pattern

retry-until-success
needs to deal with
poinson messages
Mind the poison message

6, 3, 1, 1, 1, 1, …

6, 3, 1, 1, 1, 1, …
only count the “same” batch

have to fetch
from the stream

have to fetch
from the stream
do it before they expire
from the stream!

how do you prevent building up an insurmountable backlog?

Load shedding
implement load shedding
prioritize newer messages with
a better chance to succeed

Load shedding
excess load is sent to DLQ

Load shedding
process with a delay

Mind the partial failures
LambdaSQS

LambdaSQS Poller

LambdaSQS Poller
Delete

LambdaSQS Poller
Error

LambdaSQS Poller
Error
DLQ

LambdaSQS Poller
Error
DLQ
batch fails as a unit

https://lumigo.io/blog/sqs-and-lambda-the-missing-guide-on-failure-modes

Mind the retry storm
Service A

Mind the retry storm
Service A
retry
retry
retry
retry

retry storm

circuit breaker pattern
After X consecutive timeouts, trip the circuit

When circuit is open, fail fast

but, allow 1 request through every Y mins

but, allow 1 request through every Y mins
If request succeeds, close the circuit

where do I keep the state of the circuit?

in-memory
Service A
isOpen: false
isOpen: false
isOpen: false
isOpen: false

in-memory
Service A
isOpen: true
isOpen: false
isOpen: true
isOpen: false

in-memory
PROS
simplicity

in-memory
PROS
simplicity
no dependency on external service
requires another circuit
breaker to protect…
cost & maintenance
overhead (IAM, infra, etc.)

in-memory
PROS
simplicity
CONS
takes longer & more requests to stop all trafﬁc

in-memory
PROS
simplicity
CONS
takes longer & more requests to stop all trafﬁc
new containers would generate more trafﬁc

external service
Service AisOpen: false

external service
Service AisOpen: true

external service
PROS
minimizes no. of total requests to trip circuit
new containers respect collective decision
CONS
complexity
dependency on an external service

which approach should I use?

which approach should I use?
It depends. Maybe start with the simplest solution ﬁrst?

Lambda autoscaling
Burst concurrency limits: 
3000 – US West (Oregon), US East (N.
Virginia), Europe (Ireland), 1000 – Asia Paciﬁc
(Tokyo), Europe (Frankfurt), 500 – Other
Regions
Burst: 500 new instances / each minute

Lambda autoscaling
Regions
Burst: 500 new instances / each minute 
Standard burst concurrency limits when over the
provisioned capacity

Lambda autoscaling
Regions
Burst: 500 new instances / each minute 
Adjustable provisioned capacity based on
CloudWatch metrics
Standard burst concurrency limits when over the
provisioned capacity

Lambda limitations & throttling
Concurrent executions: 1000* 
Timeout: 15 minutes 
Burst concurrency: 500 - 3000 
Burst: 500 new instances / minute
* Can be increased with support ticket

Lambda limitations & throttling
good for spikey
trafﬁc, up to a point
Concurrent executions: 1000* 
Timeout: 15 minutes 
Burst concurrency: 500 - 3000 
Burst: 500 new instances / minute
* Can be increased with support ticket

“what if my trafﬁc is more spiky than that?”

Scenario: predictable spikes
Holidays, weekends, 
celebrations 
(Black Friday)
Planned launch of 
resources 
(new series available)
Sport events

scheduled auto-scaling

scheduled auto-scaling
the burst limits still
apply, factor the timing
into account

Scenario: unpredictable spikes
Trafﬁc generated by user
actions 
 
Jennifer Aniston’s ﬁrst post

“if Lambda scaling is the problem…”

Client only needs an acknowledgement

https://lumigo.io/blog/the-why-when-and-how-of-api-gateway-service-proxies

multi-region, active-active

us-east-1
API Gateway Lambda DynamoDBRoute53

eu-west-1
us-east-1
us-west-1

eu-west-1
us-east-1
us-west-1
GlobalTable

eu-central-1
us-east-1
us-east-1
SQS Lambda DynamoDB Lambda API Gateway
SNS
SNS

us-east-1
eu-central-1
us-east-1
SNS
SNS

https://lumigo.io/blog/amazon-builders-library-in-focus-5-static-stability-using-availability-zones

us-east-1
eu-central-1
us-east-1
SNS
SNS
Ddedupe

us-east-1
us-east-1
SNS
eu-central-1
SNS
eu-central-1
Global Table

Multi-region architecture - beneﬁts & tradeoffs
Protection against 
regional failures
Higher complexity Very hard to test

MUST KILL SERVERS!
RAWR!!
RAWR!!

“the discipline of experimenting on a system in order to build conﬁdence in the
system’s capability to withstand turbulent conditions in production”
principlesofchaos.org

“You don't choose the moment, the moment chooses you!
You only choose how prepared you are when it does.”
Fire Chief Mike Burtch

identify weaknesses before they manifest in system-wide, aberrant behaviors
GOAL

learn about the system’s behavior by observing it during a controlled experiments
HOW

learn about the system’s behavior by observing it during a controlled experiments
HOW
game days
failure injection

MUST KILL SERVERS!
RAWR!!
RAWR!!
ahhhhhhh!!!!
HELP!!!
OMG!!!
F***!!!

phew!

STEP 1.
deﬁne steady state
i.e. “what does normal look like”

STEP 2.
hypothesis that steady state continues in control and experimental group
e.g. “the system stays up if a server dies”

STEP 3.
inject realistic failures
e.g. “slow response from 3rd-party service”

STEP 4.
try to disprove hypothesis
i.e. “look for difference between control and experimental group”

DON’T START
EXPERIMENTS
IN PRODUCTION

“Corporation X lost millions due to a
chaos experiment went wrong and
destroyed key infrastructure,
resulting in hours of downtime and
unrecoverable data loss.”

Chaos Engineering doesn't cause problems. It reveals them.
Nora Jones

CONTAINMENT
run experiments during ofﬁce hours

CONTAINMENT
let others know what you’re doing, no surprises

CONTAINMENT
avoid important dates

CONTAINMENT
make the smallest change possible

CONTAINMENT
make the smallest change possible
have a rollback plan before you start

by Russ Miles @russmiles
source https://medium.com/russmiles/chaos-engineering-for-the-business-17b723f26361

chaos monkey kills an
EC2 instance
latency monkey induces
artiﬁcial delay in APIs
chaos gorilla kills an AWS
Availability Zone
chaos kong kills an entire
AWS region

there are no servers to kill!
SERVERLESS

improperly tuned timeouts

missing error handling

missing fallbacks

“what if DynamoDB has an elevated error rate?”

hypothesis: the AWS SDK retries would handle it

result: function times out after 6s
(hypothesis is disproved)

TIL: the js DynamoDB client defaults to 10 retries
with base delay of 50ms

TIL: the js DynamoDB client defaults to 10 retries
with base delay of 50ms
delay = Math.random() * (Math.pow(2, retryCount) * base)
this is Marc Brooker’s
fav formula!

action: set max retry count + fallback

outcome: a more resilient system

“what if service X has elevated latency?”

hypothesis: our try-catch would handle it

TIL: most HTTP client libraries have default timeout of 60s.
API Gateway has an integration timeout of 29s.
Most Lambda functions default to timeout of 3-6s.

https://bit.ly/2Wvfort

Beware of timeouts
API Gateway 
Integration timeout  
Default: 29s
Lambda 
Timeout
Max: 15 minutes
SQS 
Default: 30s
Min: 0s
Max: 12 hours

https://theburningmonk.com/hire-me
AdviseTraining Delivery
“Fundamentally, Yan has improved our team by increasing our
ability to derive value from AWS and Lambda in particular.”
Nick Blair
Tech Lead

lambdabestpractice.com bit.ly/complete-guide-to-aws-step-functions
20% off my courses
aws-delhi-may2020

@theburningmonk
theburningmonk.com
github.com/theburningmonk

Building Resilient Serverless Apps with the Circuit Breaker Pattern

Recomendados

Recomendados

Más contenido relacionado

Similar a Building Resilient Serverless Apps with the Circuit Breaker Pattern

Similar a Building Resilient Serverless Apps with the Circuit Breaker Pattern (20)

Más de Yan Cui

Más de Yan Cui (20)

Último

Último (20)

Building Resilient Serverless Apps with the Circuit Breaker Pattern