Learn all about real-time data processing using AWS Lambda.

2. WIFI: awsDevDay | PASS: CodeHappy
U P N E X T :
Real-Time Data Processing
Using AWS Lambda

3. T H A N K S T O O U R F R I E N D S A T :

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Chris Marshall, Solutions Architect
10/10/2017
SMC303
Real-Time Data Processing
Using AWS Lambda

5. Agenda
What’s Serverless Real-Time Data Processing?
Serverless Processing of Real-Time Streaming Data
Example Kinesis/Lambda Pipeline
Streaming Demo
Customer Story: Fannie Mae-Distributed Computing with Lambda

6. What’s Serverless Real-Time
Data Processing?

7. AWS Lambda
Bring your own code
• Node.js, Java, Python,
C#
• Bring your own libraries
(even native ones)
Simple resource model
• Select power rating from
128 MB to 1.5 GB
• CPU and network
allocated proportionately
Flexible use
• Synchronous or
asynchronous
• Integrated with other
AWS services
Flexible authorization
• Securely grant access to
resources and VPCs
• Fine-grained control for
invoking your functions

8. Amazon
S3
Amazon
DynamoDB
Amazon
Kinesis
AWS
CloudFormation
AWS
CloudTrail
Amazon
CloudWatch
Amazon
Cognito
Amazon
SNS
Amazon
SES
Cron
events
DATA STORES ENDPOINTS
CONFIGURATION REPOSITORIES EVENT/MESSAGE SERVICES
Lambda Event Sources
… more on the way!
AWS
CodeCommit
Amazon
API Gateway
Amazon
Alexa
AWS
IoT
AWS Step
Functions

9. Amazon
DynamoDB
Amazon
Kinesis
Amazon
S3
Amazon
SNS
ASYNCHRONOUS PUSH MODEL
STREAM PULL MODEL
Lambda Real-Time Event Sources
Amazon
Alexa
AWS
IoT
SYNCHRONOUS PUSH MODEL
Mapping owned by Event Source
Mapping owned by Lambda
Invokes Lambda via Event Source API
Lambda function invokes when new
records found on stream
Resource-based policy permissions
Lambda Execution role policy permissions
Concurrent executions
Sync invocation
Async Invocation
Sync invocation
Lambda polls the streams
HOW IT WORKS

10. Serverless Real-Time Data Processing Is..
Capture Data
Streams
IoT Data
Financial
Data
Log Data
No servers to
provision or
manage
EVENT SOURCE
Node.js
Python
Java
C#
Process Data
Streams
FUNCTION
Clickstream
Data
Output
Data
DATABASE
CLOUD
SERVICES

11. Serverless Processing of
Real-Time Streaming Data

12. Amazon Kinesis
Amazon Kinesis Offering: Managed services for streaming data
ingestion and processing.
• Amazon Kinesis Streams: Build applications that process or
analyze streaming data.
• Amazon Kinesis Firehose: Load massive volumes of
streaming data into Amazon S3, Amazon Redshift, and
Elasticsearch.
• Amazon Kinesis Analytics: Analyze data streams using SQL
queries.
Easy to use: Focus on quickly launching data streaming
applications instead of managing infrastructure.
Real-Time: Collect real-time data streams and promptly
respond to key business events and operational triggers.
Real-time latencies.

13. Processing Real-Time Streams: Lambda + Amazon Kinesis
Streaming data sent to Amazon
Kinesis and stored in shards
Multiple Lambda functions can be
triggered to process same Amazon
Kinesis stream for “fan out”
Lambda can process data and store
results ex. to DynamoDB, S3
Lambda can aggregate data to
services like Amazon Elasticsearch
Service for analytics
Lambda sends event data and
function info to Amazon CloudWatch
for capturing metrics and monitoring
Amazon
Kinesis
AWS
Lambda
Amazon
CloudWatch
Amazon
DynamoDB
AWS
Lambda
Amazon
Elasticsearch Service
Amazon
S3

14. Processing Streams: Set Up Amazon Kinesis Stream
Streams
Made up of Shards
Each Shard ingests/reads data up to 1 MB/sec
Each Shard emits/writes data up to 2 MB/sec
Each Shard supports 5 read transactions/sec
Data
All data is stored and is replayable for 24 hours (default)
Retention window can be configured up to 7 days
Partition key used to distribute PUTs across shards
Even partition key distribution optimizes throughput
Best Practice
Determine an initial size/shards to plan for expected maximum demand
ü Leverage “Help me decide how many shards I need” option in Console
ü Use formula for Number Of Shards:
max(incoming_write_bandwidth_in_KB/1000, outgoing_read_bandwidth_in_KB / 2000)

15. Processing Streams: Create Lambda functions
Memory
CPU allocation proportional to the memory configured
Increasing memory makes your code execute faster (if CPU bound)
Increasing memory allows for larger record sizes processed
Timeout
Increasing timeout allows for longer functions, but longer wait in case of errors
Retries
With Amazon Kinesis, Lambda retries until the data expires
(i.e. 24 hours)
Permission model
Execution role defined for Lambda must have permission to access the stream
Best Practice
Write Lambda function code to be stateless
Instantiate AWS clients & database clients outside the scope of the function handler to take
advantage of connection re-use.

16. Processing Streams: Configure Event Source
Amazon Kinesis mapped as event source in Lambda
Batch size
Max number of records that Lambda will send to one invocation
Not equivalent to effective batch size
Effective batch size is every 1 second – Calculated as:
MIN(records available, batch size, 6MB)
Increasing batch size allows fewer Lambda function invocations with more data
processed per function
Best Practices
Set to “Trim Horizon” for reading from start of
stream (all data)
Set to “Latest” for reading most recent data (LIFO) (latest data)
Set to “At timestamp” to pick up at a specific time

17. Processing streams: How It Works
Polling
Concurrent polling and processing per shard
Lambda polls every 1s if no records found
Will grab as much data as possible in one GetRecords call (Batch)
Batching
Batches are passed for invocation to Lambda through
function parameters
Batch size may impact duration if the Lambda function
takes longer to process more records
Sub batch in memory for invocation payload
Synchronous invocation
Batches invoked as synchronous RequestResponse type
Lambda honors Amazon Kinesis at least once semantics
Each shard blocks in order of synchronous invocation

18. Processing streams: Tuning throughput
If put / ingestion rate is greater than the theoretical throughput, your
processing is at risk of falling behind
Maximum theoretical throughput
# shards * 2MB / Lambda function duration (s)
Effective theoretical throughput
# shards * batch size (MB) / Lambda function duration (s)
… …
Source
Amazon Kinesis
Destination
1
Lambda
Destination
2
FunctionsShards
Lambda will scale automaticallyScale Amazon Kinesis by splitting or merging shards
Waits for responsePolls a batch

19. Processing streams: Tuning Throughput w/ Retries
Retries
Will retry on execution failures until the record is expired
Throttles and errors impacts duration and directly impacts throughput
Best Practice
Retry with exponential back-off of up to 60s
Effective theoretical throughput with retries
( # shards * batch size (MB) ) / ( function duration (s) * retries until expiry)
… …
Source
Amazon Kinesis
Destination
1
Lambda
Destination
2
FunctionsShards
Lambda will scale automatically
Receives errorPolls a batch
Receives error
Receives success

20. Processing streams: Common observations
Effective batch size may be less than configured during low throughput
Effective batch size will increase during higher throughput
Increased Lambda duration -> decreased # of invokes and GetRecords calls
Too many consumers of your stream may compete with Amazon Kinesis read
limits and induce ReadProvisionedThroughputExceeded errors and metrics
Amazon
Kinesis
AWS
Lambda

21. Processing streams: Monitoring with Cloudwatch
• GetRecords: (effective throughput)
• PutRecord: bytes, latency, records, etc
• GetRecords.IteratorAgeMilliseconds: how old your
last processed records were
Monitoring Amazon Kinesis Streams
Monitoring Lambda functions
• Invocation count: Time function invoked
• Duration: Execution/processing time
• Error count: Number of Errors
• Throttle count: Number of time function throttled
• Iterator Age: Time elapsed from batch received &
final record written to stream
• Review All Metrics
• Make Custom logs
• View RAM consumed
• Search for log events
Debugging

22. Example Kinesis/Lambda Pipeline

23. Kinesis/Lambda Demo
amzn.to/bigdata

24. Kinesis/Lambda Demo
amzn.to/bigdata
Quiz: If I set my batch size to 100, each Lambda call…
A) Will get exactly 100 records
B) Will get 100 records or less
C) Will get an average of 100 records
D) Will get 95 ReadProvisionedThroughputExceeded
errors

25. Kinesis/Lambda Demo
amzn.to/bigdata
Quiz: If I set my batch size to 100, each Lambda call…
A) Will get exactly 100 records
B) Will get 100 records or less
C) Will get an average of 100 records
D) Will get 95 ReadProvisionedThroughputExceeded
errors

26. Kinesis/Lambda Demo
amzn.to/bigdata
I think this session…
A) Was really useful
B) Was a too technical
C) Was not deep enough
D) When is lunch?
E) This guy is totally confusing

27. Kinesis/Lambda Demo
amzn.to/takeselfie

28. SELECT
STREAM COUNT(*) AS MUSTACH_COUNT,
STEP(ROWTIME BY INTERVAL '1' SECOND)
FROM SOURCE_STREAM
WHERE HAS_MUSTACH = TRUE;
End-to-End Architecture
Amazon
Kinesis
Stream
Amazon
Kinesis
Analytics
Amazon
Cognito
Amazon
Kinesis
Stream
Amazon
DynamoDB
Amazon
Lambda
Amazon
S3
JavaScript
SDK
Amazon
Rekognition
Amazon
Kinesis
Firehose
Amazon
S3

29. Kinesis
Ingestion
Stream
Kinesis
Analytics
Kinesis
Aggregate
Stream
Lambda
Function
DynamoDB
TableAmazon
Cognito
SELECT ROWTIME, userId, COUNT(*)
FROM STREAM
GROUP BY userId, FLOOR(ROWTIME to SECOND)
S3 Bucket
HTML, JavascriptAggregated DataRaw Device and
Quadrant Data
Demo architecture

30. The demo application
CREATE OR REPLACE STREAM DESTINATION_SQL_STREAM (UNIQUE_USER_COUNT INT, ANDROID_COUNT INT, IOS_COUNT INT, WINDOWS_PHONE_COUNT INT,
OTHER_OS_COUNT INT, QUADRANT_A_COUNT INT, QUADRANT_B_COUNT INT, QUADRANT_C_COUNT INT, QUADRANT_D_COUNT INT, WINDOW_TIME TIMESTAMP);
CREATE OR REPLACE STREAM DISTINCT_USER_STREAM (COGNITO_ID VARCHAR(64), DEVICE VARCHAR(32), OS VARCHAR(32), QUADRANT char(1), DT
TIMESTAMP);
CREATE OR REPLACE PUMP "DISTINCT_USER_PUMP" AS
INSERT INTO "DISTINCT_USER_STREAM"
SELECT STREAM DISTINCT
"cognitoId",
"device",
"os",
"quadrant",
FLOOR("SOURCE_SQL_STREAM_001".ROWTIME TO SECOND)
FROM "SOURCE_SQL_STREAM_001";
CREATE OR REPLACE PUMP "OUTPUT_PUMP" AS
INSERT INTO "DESTINATION_SQL_STREAM"
SELECT STREAM
COUNT("DISTINCT_USER_STREAM".COGNITO_ID) AS UNIQUE_USER_COUNT,
COUNT((CASE WHEN "DISTINCT_USER_STREAM".OS = 'Android' THEN COGNITO_ID ELSE null END)) AS ANDROID_COUNT,
COUNT((CASE WHEN "DISTINCT_USER_STREAM".OS = 'iOS' THEN COGNITO_ID ELSE null END)) AS IOS_COUNT,
COUNT((CASE WHEN "DISTINCT_USER_STREAM".OS = 'Windows Phone' THEN COGNITO_ID ELSE null END)) AS WINDOWS_PHONE_COUNT,
COUNT((CASE WHEN "DISTINCT_USER_STREAM".OS = 'other' THEN COGNITO_ID ELSE null END)) AS OTHER_OS_COUNT,
COUNT((CASE WHEN "DISTINCT_USER_STREAM".QUADRANT = 'A' THEN COGNITO_ID ELSE null END)) AS QUADRANT_A_COUNT,
COUNT((CASE WHEN "DISTINCT_USER_STREAM".QUADRANT = 'B' THEN COGNITO_ID ELSE null END)) AS QUADRANT_B_COUNT,
COUNT((CASE WHEN "DISTINCT_USER_STREAM".QUADRANT = 'C' THEN COGNITO_ID ELSE null END)) AS QUADRANT_C_COUNT,
COUNT((CASE WHEN "DISTINCT_USER_STREAM".QUADRANT = 'D' THEN COGNITO_ID ELSE null END)) AS QUADRANT_D_COUNT,
ROWTIME
FROM "DISTINCT_USER_STREAM"
GROUP BY
FLOOR("DISTINCT_USER_STREAM".ROWTIME TO SECOND);

31. Serverless Data Processing with
Distributed Computing
10101101
11001010

32. Serverless Distributed Computing: Map-Reduce Model
Why Serverless Data Processing with Distributed
Computing?
Remove Difficult infrastructure management
ü Cluster administration
ü Complex configuration tools
Enable simple, elastic, user-friendly distributed data
processing
ü Eliminate complexity of state management
ü Bring Distributed Computing power to the masses

33. Serverless Distributed Computing: MapReduce
Input Bucket
1
2
Driver
job state
Mapper Functions
map phase
S3
event
source
mapper
output
3 Coordinator
4
Reducer step 1
reducer output
5
recursively
create
n‘th reducer
step
ResultFinal Reducer
reduce phase
6

34. Fannie Mae
Distributed Computing with Lambda

35. Fannie Mae’s Serverless HPC Performance
Lambda service configuration:
• Initial burst rate = 2,000, incremental rate = 100 per
minute, throttle limit = 15,000.
• Lambda ramps up automatically from 2,000 to 15,000
concurrent executions.
Application Result:
• One simulation run of ~ 20 million mortgages takes 2
hours, >3 times faster than the existing process.
• The performance does not degrade during the ramp up
period.
• Lambdas’ CPU efficiency is close to 100%. Actual
elapsed time is consistent with the estimated elapsed
time based on Lambda billing time.
Number of New
Lambda Invocations
every 5 Mins
Maximum
Concurrent
Lambdas =
15,000

36. Complex Serverless HPC Reference
Architecture
Breakdown complex workload into multiple simple ones:
…
Reducer
Final Reducer Result
Input Bucket Mapper Functions
Reducers
Reducersmapper
output
mapper
output
reducer
output

37. Real-time Data Processing with
Lambda: Next Steps

38. Data Processing with AWS: Next steps
ü Learn more about AWS Serverless at
https://aws.amazon.com/serverless
ü Explore Real-time Clickstream Anomaly Detection with
Amazon Kinesis Analytics on the AWS Big Data Blog at
https://aws.amazon.com/blogs/big-data/real-time-
clickstream-anomaly-detection-with-amazon-kinesis-
analytics/
ü Explore the AWS Lambda Reference Architecture on GitHub:
§ Real-Time Streaming: https://github.com/awslabs/lambda-refarch-
streamprocessing
§ Distributed Computing Reference Architecture (serverless MapReduce)
https://github.com/awslabs/lambda-refarch-mapreduce

39. Data Processing with AWS: Next steps
ü Create an Amazon Kinesis stream. Visit the Amazon Kinesis
Console and configure a stream to receive data Ex. data from
Social media feeds.
ü Create & test a Lambda function to process streams from Amazon
Kinesis by visiting Lambda console. First 1M requests each month
are on us!
ü Read the Developer Guide and try the Lambda and Amazon
Kinesis Tutorial:
§ http://docs.aws.amazon.com/lambda/latest/dg/with-
kinesis.html
ü Send questions, comments, feedback to the AWS Lambda Forums

40. Thank You!

41. Don’t Forget Evaluations!