This is a hands-on workshop where every participant will not only learn how to architect and implement a serverless application on Amazon Web Services using nothing but serverless resources for all layers in theory, but actually do it in practice, with all the necessary support from the speakers. Serverless computing allows you to build and run applications and services without thinking about servers. Serverless applications don't require you to provision, scale, and manage any servers. You can build them for nearly any type of application or backend service, and everything required to run and scale your application with high availability is handled for you. Building serverless applications means that developers can focus on their core product instead of worrying about managing and operating servers or runtimes. This reduced overhead lets developers reclaim time and energy that can be spent on developing great products which scale and that are reliable.

1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Markus Kaiser, Solutions Architect
Daniel Geske, Solutions Architect
Wild Rydes: Dawn of a new unicorn
Serverless data processing on AWS

2. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
What to expect
1. Serverless architectures
2. Wild Rydes scenario
3. Modules and relevant AWS services
• Setup
• Real-time data streaming
• Stream aggregation
• Stream processing
• Data lake
• Extra credit (optional)
4. Summary and clean-up

3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Serverless
Build and run applications and
services without thinking of servers
Fully Managed
Developer Productivity
Continuous Scaling

4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark

5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Help Wild Rydes Disrupt Transportation!
So how does this magic work?

6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Wild Rydes is Backed by Leading Investors
THE BARN
ACCELERATOR
TENDERLOIN
CAPITAL
PENGLAI COMMUNICATIONS
AND POST NEW CENTURY
TECHNOLOGY CORP LIMITED

7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Your Task: Process and visualize data in real-time
Welcome to Wild Rydes Inc.,
Employee #3!

8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Serverless data processing on AWS
http://github.com/aws
-samples/aws-
serverless-workshops

9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Setup
Configure your AWS Cloud9 IDE and setup pre-requisites like an AWS
Account.

10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Real-time Streaming Data
Create a stream in Kinesis and write
to and read from the stream to
track Wild Rydes unicorns on a live
map. In this module you’ll also
create an Amazon Cognito identity
pool to grant the live map access to
your stream.
https://dataprocessing.wildrydes.com/streaming-data.html

11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Amazon Kinesis Data Streams
• Easy administration & low cost
• Secure, durable storage
• Build real time applications
with framework of choice

12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Amazon Kinesis Data Streams
Producers Consumers
Kinesis Agent
Apache Kafka
AWS SDK
LOG4J
Flume
Fluentd
AWS Mobile SDK
Kinesis Producer
Library
Get* APIs
Kinesis Client Library
+ Connector Library
Apache Storm
Amazon EMR
AWS Lambda
Apache Spark
Amazon
Kinesis

13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Stream Aggregation
Build an Kinesis Data Analytics application to read from the
stream and aggregate metrics like unicorn health and distance
traveled each minute.
https://dataprocessing.wildrydes.com/streaming-aggregation.html

14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Amazon Kinesis Analytics
• Connect to streaming source
• Write SQL code to process
streaming data
• Continuously deliver SQL results
• Fully managed, automatic elasticity

15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Stream Processing
Persist aggregate data from application to a backend database
stored in DynamoDB and run queries against those data.
https://dataprocessing.wildrydes.com/stream-processing.html

16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Data Lake
Use Kinesis Data Firehose to flush the raw sensor data to an S3
bucket for archival purposes. Using Athena, you’ll run SQL queries
against the raw data for ad-hoc analysis.
https://dataprocessing.wildrydes.com/data-lake.html

17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Amazon Kinesis Data Firehose
• Zero administration and
seamless elasticity
• Direct-to-data store integration
• Serverless, continuous data
transformations

18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Amazon Athena
No loading of data, analyze data directly from
Amazon S3
Standard SQL
• Support for complex data types (arrays, structs)
• Support for partitioning of data by any key
Query data in its raw format
• Athena supports multiple data formats:
Text, CSV, TSV, JSON, weblogs, AWS service logs
• or convert to an optimized form like ORC or Parquet
for the best performance and lowest cost

19. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Extra credit
AWS SDK
Amazon Kinesis Data Analytics
AWS Lambda
Amazon Athena

20. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Summary

21. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Serverless Data Processing Architecture

22. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Pricing
AWS Service Price* Example
Kinesis Firehose Data Volume: $0.033 per GB 1350 GB: $44.55
S3 Data Volume: $0.0245 per GB
PUT Requests: $0.0054 per 1.000
27 GB: $0.66
45.000 PUT: $0.243
Athena Scanned Data: $5 per TB 5.4 TB: $27.00
AWS Service Price* Example (Monthly)
Kinesis Streams Shard: $0.018 per hour
PUT Units: $0.0175 per Mio
2 x Shard: $27.00
270M PUTs: $4.725
Kinesis Analystics KPU: $0.132 per hour 1 x KPU: $99
Lambda Requests: $0.20 per Mio
Execution: $0,00001667 per GB/s (negligible)
1.000 Unicorns, 10s Status Msg, Msg Size 1KB
eu-central-1
30 days retention, 1:10 compression, 200 queries

23. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Clean up
1. Amazon Athena
2. Amazon Kinesis Data Analytics
3. Amazon Kinesis Data Firehose
4. Amazon Kinesis Data Streams
5. Amazon S3
6. AWS Lambda
7. Amazon DynamoDB
8. AWS IAM
9. Amazon Cognito
10.Cloud9 IDE

24. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Go build
Markus Kaiser, Solutions Architect
@markuskaiser
Daniel Geske, Solutions Architect
@btx94