The document discusses using Amazon SNS, SQS, and AWS Lambda for building high throughput, bursty data applications. It provides examples of how media companies, banks, and retailers can use these serverless technologies to process events and scale automatically without managing infrastructure. The document also shares case studies of how Letgo and Enel migrated to using SNS, SQS and Lambda for improved performance and reduced costs when handling large volumes of bursty data and events.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Building High Throughput, Bursty Data Apps
with Amazon SQS, SNS & AWS Lambda
Robert Chow
Principal Product Manager
Amazon Web Services
A P I 3 0 6
Cristian García
VP of Infrastructure
Letgo
Luca Di Lieto
Solutions Architect –
2BEAT Product Owner
Enel

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Scenarios
Publishing at a media company
Processing financial transactions at a bank
Order processing at a retailer

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Serverless technologies
No infrastructure to
provision or manage
Scales automatically
by unit of consumption
Don’t pay for idle
(pay per invocation)
Built-in security
highly available compute

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS modern application development
SERVERLESS COMPUTE AND DATASTORES
AWS
Lambda
AWS
Fargate
Amazon
API Gateway
Amazon
SNS
Amazon
SQS
AWS
Step Functions
SERVERLESS INTEGRATION
DEVELOPER TOOLS
SECURITY AND COMPLIANCE
Amazon Aurora
Serverless
Amazon
S3
Amazon
DynamoDB
AWS
AppSync
AWS
IAM
AWS Firewall
Manager
Amazon
Cognito
Amazon
Inspector
Amazon
VPC
Amazon
GuardDuty
Amazon
Macie
Amazon
CloudFormation
AWS
Cloud9
AWS
CloudTrail
AWS
CloudWatch
AWS
X-Ray
AWS Serverless
App RepositoryFPO
AWS
CodePipeline

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SNS, SQS and AWS Lambda
Amazon SNS
(Simple Notification Service)
Amazon SQS
(Simple Queue Service)
AWS Lambda

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Event-driven architecture
Lambda
function
topicAmazon
SNS
User generated
events
Application or job
triggered events
Amazon
RDS
traditional server
Amazon
SQS
queue Lambda
function
Message
filters
Amazon
DynamoDB
Amazon
S3
Amazon
EC2
SNS Message Filtering
1. 2.
SQS as an event source for Lambda

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Event distribution
Event Sources
Lambda
function
topicAmazon
SNSApplication or job
triggered events
traditional server
Amazon
SQS
Message
filters
Amazon
EC2
Message Filters
Fan-Out

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Event distribution and bursty loads
Throughput
Fan-Out ratio
Lambda
function
topicAmazon
SNSApplication or job
triggered events
traditional server
Amazon
SQS
Message
filters
Amazon
EC2
Filters

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Event processing
Amazon
SQS
queue Lambda
function
Amazon
RDS
1. Message inserted into to a
queue
2. Lambda polls queue and
invokes function
3. Lambda removes message
from queue
Lambda Processing
Message Processing
AWS
Lambda

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Event processing and bursty loads
Amazon
SQS
queue Lambda
function
Amazon
RDS
1. Message inserted into to a
queue
2. Lambda polls queue and
invokes function
3. Lambda removes message
from queue
AWS
Lambda

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Lambda automatic scaling
• 5 concurrent functions to start + 60/minute
• Message batching
• Long polling
Queue

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Lambda cold starts
Your Function
Warm Start

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SNS or SQS as event sources for AWS Lambda
Comparison SNS -> Lambda SNS -> SQS -> Lambda
Scenario Downstream consumers are rate
not constrained
Downstream consumers are rate
constrained
Message
Deletion
Once delivered, message is
deleted (even if processing is not
successful)
Lambda deletes message upon
successful processing
Retention/
Retries
Delivery retry - up to 13 hours Message retention up to 14 days.

15. Cristian García
VP of Infrastructure

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Letgo
• 100M downloads
• 400M products
• 6B messages
• 65% listing growth in 2018

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The Letgo project
• >5Pb of monthly data
• >72K requests/sec
• Data oriented
• Squad modeled

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Transactions and usage
• Exponential growth and usage of many
AWS services
• Delivering close to 1 trillion monthly
SQS/SNS messages
• But…
• Not everything was a bed of roses in 2016...
• Because we used REST API’s for our
microservice communications

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
This seems good when it runs, but…

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
…but not as good we thought

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
REST API constraints
Infrastructure:
• High latency average
• Handle sudden traffic spikes
• Data loss in case a service is down
• Not affordable cost escalation
Coding:
• Complexity coding for HTTP having
better options 
• Not microservice-able

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Domain-driven migration challenge
• Each service publishes to its own SNS
topic, other services interested on the
data can subscribe to the topic
• In case service B needs the data and
changes from service A, just needs to
subscribe one of its own queues to the
service
• Usage of database projections for
storing data subsets
• Stage 1
• Stage 2
• Stage 3

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Domain events

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Outage in consumer tier

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why message filtering improved our services?
Without:
Filtering via code, high discarded
messages ratio (>90%), waste of
compute resources
With:
Only the correct message is sent to
to queue, then the consumer can
scale just for this kind of events

26. Money talks Speed talks
Filtered messages: real world results

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cost evolution

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tips and tricks
• Compute: Squeeze your instance!
• Scaling: Take a look at ApproximateNumberOfMessagesVisible
• AWS Lambda: Means instant muscle, so use it carefully!
• Dead_letter: 5 attempts + redrive policy
• Data loss: The messages will be there

29. Luca Di Lieto
Solutions Architect – 2BEAT Product Owner

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Enel group overview

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
ENEL Smart Meters
2016
Open Meter:
Smart Meters
2.0
2001
First digital meter
installed
2006
Entire Italian Customer base
32 mn digital meters
2018
44 mn digital meters
10% Smart Meters 2.0
(~4 million)
2020
Strategic Plan
48 mn digital meters
36% Smart Meters 2.0
(~17 million)

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Home
Energy Box
In Home
Device
Street-level sensors
and actuators for
Remote Control
Low Voltage
Manager
BEAT
Central System
3G/4G/Fiber Optic
Narrow-Band Power Line Communication (PLC)
W-MBUS 169 MHz Radio Communication
Market
SYSTEM ARCHITECTURE
Enel Open Meter
Smart Meters
near site mgt
Heat, Gas,
Watermeter
Meter

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
KEY FEATURES AND BENEFITS
Smart Meters 2.0

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
2BEAT
PROCESS, ANALYZE AND FORWARD ENERGY
CONSUMPTION PROFILES

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Consumption profiles
PREVIOUS GENERATION SMART METERS 2.0 / 2BEAT
• Records consumption at the end of
month
• Consumption is mainly relative to a
single metric (Active Energy) and to
three different bands
• 32 mn messages per month
• 3.2 GB data per month
• Records one measure every 15
minutes
• A number for each metric (3
different metrics)
• Load profile for each user per day
• 288 mn messages per day
• 288 GB data per day
• 96 mn raw data points per month • 12.5 bn raw data points per day

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Architecture
PREVIOUS GENERATION SMART METERS 2.0 / 2BEAT
• Based on traditional technologies like
Application Servers and RDBMS
• Pay for the peak due to architectural
constraints; there isn’t flexibility on
cost
• Quite monolithic
• Limitations on scalability
• Administration and deploy require in-
depth skills
• Redesign to make the application
idem-potent and not monolithic
• Pay per use
• Reengineering with microservices
based on AWS serverless components
• Use of scalable solutions
• Reduce administration and
deployment effort

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Current architecture
Legacy
Application
Amazon Elastic
Beanstalk
Amazon
Dynamo DB
Amazon Kinesis Data
Stream/Kinesis Data
Firehose
Devices
EXTERNAL
SYSTEM
Amazon SQS
Amazon SQS
Amazon S3
Amazon Redshift
Spectrum

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Current architecture
Legacy
Application
Get concentrators data
Amazon Elastic
Beanstalk
Publish msg to external
systems
Amazon
Dynamo DB
Amazon Kinesis Data
Stream/Kinesis Data
Firehose
Store messages as
json flat files
Load from S3 and perform
SQL analisys
Devices
EXTERNAL
SYSTEM
Send JSON message obtained
from SQS and entities events
Store data for
processing purposes.
60 Days
Amazon SQS
Amazon SQS
Amazon S3
288 mn req/day
Avg. 10k msg/s
Peak 20k msg/s
288 mn req/day
10k msg/s
(Twenty EC2
Instances)
Read Capacity: 20k item/s
Write Capacity: 20k item/s
Retention 60 days: 17 TBs of data
Record size: 1 KB
Storage:
518 TBs
for 5 yearsNote: Estimation for 32 mn
meters; currently it manages
5 mn meters.
Re-execution of a
request
Amazon Redshift
Spectrum
Data processing

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Evolution: Go fully serverless
AWS Elastic Beanstalk is a platform
based on EC2 instances and implies
a different administration compared
to serverless; it was a valuable
solution when we started the project
The availability of AWS Lambda
polling SQS gives us the opportunity
to go fully serverless.

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Legacy
Application
AWS Lamba
Amazon
Dynamo DB
Amazon Kinesis Data
Stream/Kinesis Data
Firehose
Devices
EXTERNAL
SYSTEM
Amazon SQS
Amazon SQS
Amazon S3
Amazon Redshift
Spectrum
To-be architecture

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SQS as an event source for AWS Lambda benefits
• Easier to operate
A fully serverless solution is well accepted by our Operations team. A mixed solution
is deviation from company standards.
• Easier to deploy
We deploy it with a company CI/CD infrastructure based on AWS CloudFormation
• Easier to scale
Scales quickly with AWS Lambda
• AWS Lambda is in the Enel “approved” AWS services portfolio
• Simplified monitoring and logging using Amazon CloudWatch

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Future Plans
The legacy application supports
proprietary protocol for device
communication that requires long
lived connection pulling data from
devices. Requires a complete
redesign with more suitable
approaches.
We are evaluating redesigning it by
leveraging on AWS IoT Core and
AWS Greengrass.

43. Summary

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Summary
• Amazon SNS, SQS, and AWS Lambda for high-throughput, bursty
applications
• Fully managed and serverless
• Scale up or scale down, no pre-provisioning resources required, don’t pay for idle periods
• Amazon SNS, SQS, and AWS Lambda for event driven designs
• Native integration with AWS event sources
• Pub/sub to decouple publishers from subscribers
• New features improve handling of high-throughput, bursty traffic
• Use message filtering to reduce processing load (and cost) of publishers and subscribers
• Use Amazon SQS as an Event Source for AWS Lambda when you want more flexibility and
resilience for bursty traffic

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Related breakouts
Best Practices for Queue Processing in Serverless Applications
Aria West, Level 3, Starvine 2, T2
Wednesday, Nov 28, 2018 : 4:45 PM - 5:45 PM
Choosing the Right Messaging Service for your Distributed App
MGM, Level 1, Grand Ballroom 122, T2
Wednesday, Nov 28, 2018 : 3:15 PM - 4:15 PM
Inside AWS: Technology Choices for Modern Application Development
Aria East, Level 2, Mariposa 5, T1
Thursday, Nov 29: 2:30 PM - 3:30 PM
Securing Data in Serverless Applications and Messaging Services
Mirage, Martinique A, T2
Thursday, Nov 29, 2018 : 3:15 PM - 4:15 PM

46. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Robert Chow
Principal Product Manager
Amazon Web Services
Cristian García
VP of Infrastructure
Letgo
Luca Di Lieto
Solutions Architect – 2BEAT
Product Owner
Enel

47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.