Looking in from the outside, serverless seems so simple! And yet, many companies are struggling on their journey to serverless. In this talk, I highlight a number of mistakes companies are making when they adopt serverless.

1. MIND
THE
POTHOLES
MIND
THE
POTHOLES

2. What do you mean
by ‘serverless’?

3. “Serverless”

5. Gojko Adzic
It is serverless the same way
WiFi is wireless.
http://bit.ly/2yQgwwb

6. Serverless means…
don’t pay for it if no-one uses it
don’t need to worry about scaling
don’t need to provision and manage servers

7. in other words, it’s a lot like taking a cab

8. Ownership
Fuel
Navigate
To get there!
Focus on
getting there!

9. HW Ownership
OS
Runtime & Scale
Code
Focus on
getting there!
Physical
Servers
Virtual
Machines
Containers Serverless

10. Nano Services Self Managed Cost Paradigm
ChangeAsync
Dynamic agile env

11. “why are we failing at this?”

12. hidden dangers

13. monolith microservices serverless

14. monolith microservices serverless
observability
distributed
systems
bounded
context

15. monolith microservices serverless
observability
distributed
systems
bounded
context

16. monolith microservices serverless
observability
distributed
systems
bounded
context
event driven

17. monolith serverless
missing learnings
from microservices

18. monolith serverless
missing learnings
from microservices
poor decisions

19. Yan Cui
http://theburningmonk.com
@theburningmonk
AWS user for 10 years

20. http://bit.ly/yubl-serverless

21. Yan Cui
http://theburningmonk.com
@theburningmonk
Developer Advocate @

23. Yan Cui
http://theburningmonk.com
@theburningmonk
Independent Consultant
advisetraining delivery

24. https://theburningmonk.com/workshops
Amsterdam, March 19-20 Helsinki, May 4-5 Stockholm, May 14-15
Dublin, June 16-17 London, September 24-25
early bird
until Feb 11

25. #1
not letting go of legacy
thinking

26. “we’re doing serverless,
but why aren’t thing
going faster?”

27. Socio Technical

28. there are no silver bullets

30. centralised team
Team A Team B Team C Team D …

31. “but the developers don’t understand AWS and how
our infrastructure is set up”

32. “but the developers don’t understand AWS and how
our infrastructure is set up”
let’s solve this
problem instead!

33. what got you here won’t get you there

34. if (path == “/user” && method == “GET”) {
return getUser(…);
} else if (path == “/user” && method == “DELETE”) {
return deleteUser(…);
} else if (path == “/user” && method == “POST”) {
return createUser(…);
} else if ….
Monolithic Functions

35. GET /user
POST /user
DELETE /user
Single-Purposed Functions

36. author: yan.cui
feature: user-api
user-api-dev
Monolithic Single-Purposed
author: yan.cui
feature: user-api
user-api-dev-get-user
author: yan.cui
feature: user-api
user-api-dev-create-user
author: yan.cui
feature: user-api
user-api-dev-delete-user

37. author: yan.cui
feature: user-api
user-api-dev
Monolithic Single-Purposed
author: yan.cui
feature: user-api
user-api-dev-get-user
author: yan.cui
feature: user-api
user-api-dev-create-user
author: yan.cui
feature: user-api
user-api-dev-delete-user
find related
functions by prefix

38. author: yan.cui
feature: user-api
user-api-dev
Monolithic Single-Purposed
author: yan.cui
feature: user-api
user-api-dev-get-user
author: yan.cui
feature: user-api
user-api-dev-create-user
author: yan.cui
feature: user-api
user-api-dev-delete-user
discoverability
(without having to dig into the code)

39. author: yan.cui
feature: user-api
user-api-dev
Monolithic Single-Purposed
author: yan.cui
feature: user-api
user-api-dev-get-user
author: yan.cui
feature: user-api
user-api-dev-create-user
author: yan.cui
feature: user-api
user-api-dev-delete-user
what does it do?

40. author: yan.cui
feature: user-api
user-api-dev
Monolithic Single-Purposed
author: yan.cui
feature: user-api
user-api-dev-get-user
author: yan.cui
feature: user-api
user-api-dev-create-user
author: yan.cui
feature: user-api
user-api-dev-delete-user
dynamodb:GetItem
dynamodb:PutItem
dynamodb:DeleteItem

41. author: yan.cui
feature: user-api
user-api-dev
Monolithic Single-Purposed
author: yan.cui
feature: user-api
user-api-dev-get-user
author: yan.cui
feature: user-api
user-api-dev-create-user
author: yan.cui
feature: user-api
user-api-dev-delete-user
dynamodb:GetItem
dynamodb:PutItem
dynamodb:DeleteItem
no least privilege…

42. author: yan.cui
feature: user-api
user-api-dev
Monolithic Single-Purposed
author: yan.cui
feature: user-api
user-api-dev-get-user
author: yan.cui
feature: user-api
user-api-dev-create-user
author: yan.cui
feature: user-api
user-api-dev-delete-user
require(x)
require(y)
require(z)

43. author: yan.cui
feature: user-api
user-api-dev
Monolithic Single-Purposed
author: yan.cui
feature: user-api
user-api-dev-get-user
author: yan.cui
feature: user-api
user-api-dev-create-user
author: yan.cui
feature: user-api
user-api-dev-delete-user
require(x)
require(y)
require(z)

45. more dependecies equals
slower cold start

46. author: yan.cui
feature: user-api
user-api-dev
Monolithic Single-Purposed
author: yan.cui
feature: user-api
user-api-dev-get-user
author: yan.cui
feature: user-api
user-api-dev-create-user
author: yan.cui
feature: user-api
user-api-dev-delete-user
require(x)
require(y)
require(z)
worse cold start
performance

47. keep functions simple, and single-purposed

48. #2
one account that rules
them all

49. mind the shared limits

50. no. of DynamoDB tables
no. of API Gateway regional APIs
no. of API Gateway edge-optimized APIs
no. of Kinesis shards
no. of IAM roles
no. of S3 buckets
no. of CloudFormation stacks
no. of SNS subscription filters
no. of SSM parameters
…
Resource Limits

51. DynamoDB read & write
API Gateway requests/second
Lambda concurrent executions
SSM parameter ops/second
…
Throughput Limits

53. compartmentalise security breaches

54. One account per Team per Environment

56. #3
do first, research later

57. https://einaregilsson.com/serverless-15-percent-slower-and-eight-times-more-expensive/

61. the platforms need to do better at educating users on
how to choose between different services

62. SNS vs SQS vs Kinesis vs MKS?
the platforms need to do better at educating users on
how to choose between different services

63. ordering
replay events
Kinesis SQS SNS
by shard
none (standard)
global (FIFO)
none
up to 7 days none none
mode
retry
batched batched (up to 10) singular
retried until
success
(customizable)
retry + DLQ retry + DLQ
concurrency 1 per shard auto-scaled fan-out!!!
subscribers many one-to-one many
EventBridge
many
none
none
singular
retry + DLQ
fan-out!!!

64. https://medium.com/theburningmonk-com/all-my-posts-on-serverless-aws-lambda-43c17a147f91

65. https://www.jeremydaly.com/newsletter/

66. #4
not using a deployment
toolkit

68. https://lumigo.io/blog/comparison-of-lambda-deployment-frameworks/

69. don’t write your own deployment framework

70. #5
console-driven
development

73. #6
one repo per function

74. github
repo
github
repo
github
repo
github
repo
github
repo
github
repo
github
repo
github
repo
github
repo

75. github
repo
github
repo
github
repo
github
repo
github
repo
github
repo
github
repo
github
repo
github
repo

76. monorepo?

77. github
repo

78. one repo per service?

79. github
repo
github
repo
github
repo
github
repo
user-api
timeline-api
relationship-api
search-api

80. https://lumigo.io/blog/mono-repo-vs-one-per-service/

88. github
repo
github
repo
github
repo
github
repo
user-api
timeline-api
relationship-api
search-api

89. CI/CD pipeline per service

90. functions are deployed together, as a stack

91. unencrypted secrets
in env vars
#7

92. secrets should NEVER be in plain text in env variables

93. SSM Parameter Store
Secret 1
Secret 2
IAM
Environment:
SECRET_1: …
SECRET_2: …
Environment:
SECRET_1: …
SECRET_2: …

94. SSM Parameter Store
Secret 1
Secret 2
IAM
Environment:
SECRET_1: …
SECRET_2: …
Environment:
SECRET_1: …
SECRET_2: …
yay!

97. SSM Parameter Store
Secret 1
Secret 2
IAM
fetch at cold start,
cache,
invalidate every x mins

98. https://github.com/middyjs/middy

100. SSM Parameter Store
Secret 1
Secret 2
IAM
switch to Higher
Throughput if you need
more than 40 ops/s

101. not following least
privilege principle
#8

105. missing DLQs
#9

106. async sync
S3
SNS
SES
CloudFormation
CloudWatch Logs
CloudWatch Events
Scheduled Events
CodeCommit
AWS Config
http://amzn.to/2v7Kc3b
Cognito
Alexa
Lex
API Gateway
pulling
DynamoDB Stream
Kinesis Stream
SQS

107. async sync
S3
SNS
SES
CloudFormation
CloudWatch Logs
CloudWatch Events
Scheduled Events
CodeCommit
AWS Config
http://amzn.to/2vs2lIg
Cognito
Alexa
Lex
API Gateway
pulling
DynamoDB Stream
Kinesis Stream
SQS
Lambda handles retries
(twice, then DLQ)

108. configure DLQ for async functions so you don’t lose failed events

111. DLQ Lambda Destinations
payload payload, context(s), and response

113. DLQ Lambda Destinations
payload payload, context(s), and response
DeadLetterErrors no error metrics!

114. too much/too little
concurrency
#10

115. “Lambda generates too much load for the downstream system”

116. one invocation
per message
SNS
Lambda

117. Downstream
System
SNS
Lambda

118. ordering
replay events
Kinesis SQS SNS
by shard
none (standard)
global (FIFO)
none
up to 7 days none none
mode
retry
batched batched (up to 10) singular
retried until
success
(customizable)
retry + DLQ retry + DLQ
concurrency 1 per shard auto-scaled fan-out!!!
subscribers many one-to-one many
EventBridge
many
none
none
singular
retry + DLQ
fan-out!!!

119. if you want…
maximum
throughput
SNS
precise control
over throughput
Kinesis

120. if you want…
maximum
throughput
SNS
precise control
over throughput
Kinesis
how quickly it scales out

121. if you want…
maximum
throughput
SNS
precise control
over throughput
Kinesis
how quickly it scales out
SQS DynamoDB
Streams

122. ordering
replay events
Kinesis SQS SNS
by shard
none (standard)
global (FIFO)
none
up to 7 days none none
mode
retry
batched batched (up to 10) singular
retried until
success
(customizable)
retry + DLQ retry + DLQ
concurrency 1 per shard auto-scaled fan-out!!!
subscribers many one-to-one many
EventBridge
many
none
none
singular
retry + DLQ
fan-out!!!

123. cold starts
#11

124. “cold starts only happen to the first request”

125. function invocationconcurrent execution
i.e. a container

126. function invocationconcurrent execution
i.e. a container
class instance method call

127. Lambda scales the number of concurrent executions
based on traffic

128. existing “containers” are reused where possible

129. time
invocation

130. time
invocation
invocation

131. time
invocation
invocation

132. time
invocation
invocation
invocation
invocation

133. time
invocation
invocation
invocation
invocation
invocation
invocation

134. time
invocation
invocation
invocation
invocation
invocation
invocation

135. time
invocation
invocation
invocation
invocation
invocation
invocation
invocation

136. time
invocation
invocation
invocation
invocation
invocation
invocation
invocation invocation

137. time
invocation
invocation
invocation
invocation
invocation
invocation
invocation invocation

138. time
invocation
invocation
invocation
invocation
invocation
invocation
invocation invocation

143. time
invocation
invocation
ping
invocation
invocation
invocation
ping ping

144. Lambda warmers don’t work when you have > 1
concurrent executions

145. FREQUENCY DURATION

146. FREQUENCY DURATION
dictated by user traffic,
out of your control

147. cold starts is generally not an issue if you have a
steady traffic pattern

148. time
req/s

149. time
req/s
El Classico

150. time
req/s
lunch dinner

151. FREQUENCY DURATION
optimize this!

152. minimise the duration of cold starts so they
fall within acceptable latency range

153. time
req/s
lunch dinner
Provisioned
Concurrency

154. time
req/s
lunch dinner
Provisioned
Concurrency
On-Demand
Concurrency

155. https://lumigo.io/blog/provisioned-concurrency-the-end-of-cold-starts/

156. there are no silver bullets

157. reserved concurrency is a powerful tool IFF you
have a cold start problem
don’t use it by default

158. RDS connection
handling
#12

159. default RDS configs are bad for Lambda

160. default RDS configs are bad for Lambda
idle connections are
not closed
too many connections
per “container”
max open connection
is too low

161. https://www.jeremydaly.com/manage-rds-connections-aws-lambda/

162. set “wait_timeout” and “interactive_timeout” to 10 mins
(default is 8 hours!)

163. increase “max_connections” setting

164. set client socket pool size to 1

167. (lack of) observability
#13

168. happened system repaireduser impact
reduce MTTR

169. Identify & Resolve
Issues
Understanding
costs
Visibility

170. Identify & Resolve
Issues
Understanding
costs
Visibility

171. happened system repaireduser impact
MTTDiscovery

173. “What alerts should I have?”

174. It depends on what you’re building…

175. But, this is a good starting point

176. Lambda
error rate %
throttle count
DLR error count
iterator age
regional concurrency

177. Lambda
error rate %
throttle count
DLR error count
iterator age
regional concurrency
API Gateway
p90/95/99 latency
success rate %
4xx rate %
5xx rate %

178. API Gateway
p90/95/99 latency
success rate %
4xx rate %
5xx rate %
SQS
message age
Lambda
error rate %
throttle count
DLR error count
iterator age
regional concurrency

179. API Gateway
p90/95/99 latency
success rate %
4xx rate %
5xx rate %
SQS
message age
Step Functions
failed count
throttle count
timed out count
Lambda
error rate %
throttle count
DLR error count
iterator age
regional concurrency

180. SQS
message age
Step Functions
failed count
throttle count
timed out count
API Gateway
p90/95/99 latency
success rate %
4xx rate %
5xx rate %
Lambda
error rate %
throttle count
DLR error count
iterator age
regional concurrency

181. monitor and alert on message flow rate for
event processing pipelines

182. “Can’t you codify these?”

185. https://theburningmonk.com/hire-me
AdviseTraining Delivery
“Fundamentally, Yan has improved our team by increasing our
ability to derive value from AWS and Lambda in particular.”
Nick Blair
Tech Lead

186. Production-Ready Serverless

187. bit.ly/prod-ready-sls-ams-2019
Backspace Escape Rooms
Transformatorweg 30A
20% off with
aws-ug-ams-0109

188. https://theburningmonk.com/workshops
Amsterdam, March 19-20 Helsinki, May 4-5 Stockholm, May 14-15
Dublin, June 16-17 London, September 24-25
early bird
until Feb 11

189. @theburningmonk
theburningmonk.com
github.com/theburningmonk