Introduction to Real-Time Streaming Analytics - Amazon Kinesis State Of Union | AWS Floor28

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
SSID: Guest
Password: Cube@11999
Amazon Kinesis State Of Union
Introduction to Real-Time Streaming Analytics
Roy Ben-Alta,
Head Of Amazon Kinesis Business Development
AWS Data Analytics & Machine Learning

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential
What to Expect from the Session
• Overview of Real-Time Streaming Analytics
• Amazon Kinesis
• Use Cases

Overview of Real-Time Streaming Analytics

#Real-time

All data originates in real-time!

{
"payerId": "Joe",
"productCode": "AmazonS3",
"clientProductCode": "AmazonS3",
"usageType": "Bandwidth",
"operation": "PUT",
"value": "22490",
"timestamp": "1216674828"
}
Metering Record
127.0.0.1 user-identifier frank [10/Oct/2000:13:55:36 -0700] "GET
/apache_pb.gif HTTP/1.0" 200 2326
Common Log Entry
<165>1 2003-10-11T22:14:15.003Z mymachine.example.com evntslog -
ID47 [exampleSDID@32473 iut="3" eventSource="Application"
eventID="1011"][examplePriority@32473 class="high"]
Syslog Entry
“SeattlePublicWater/Kinesis/123/Realtime
” – 412309129140
MQTT Record
<R,AMZN,T,G,R1>
NASDAQ OMX Record
Smart Buildings
Beacons
Smart Textiles
Health Monitors

But, analytics to gain insights is usually done
much, much later.

#WhyWait

Insights are perishable.

Batch analytics operations take too long
BusinessValue
Time To Action
Data
originated
Analytics
performed
Insights
gleaned
Action
taken
Outdated
insights
Can be harmful
actions
PositiveNegative
Decision
made
Poor decision

Streaming technology is necessary to detect and act
on real-time perishable insights.

#Streaming

Stream New Data in Seconds
Get actionable insights quickly
Streaming
Ingest video & data as
it’s generated
Real-time analytics/ML,
alerts, actions
Process data
on the fly

Timely Decisions Require New Data in Minutes
Source: Perishable insights, Mike Gualtieri, Forrester
Data loses value quickly over time
Real time Seconds Minutes Hours Days Months
Valueofdatatodecision-making
Preventive/Predictive
Actionable Reactive Historical
Time critical decisions Traditional “batch” business intelligence
Information half-life
in decision-making

Most Common Uses of Streaming
Industrial
Automation
ML
Smart Home
Smart City
Data
Lakes/CDC/Pipeline
IoT
Analytics
Log
Analytics

Overview
Amazon Kinesis

Amazon Kinesis is a Foundational Service Used
Across Amazon
AWS
metering

Amazon Kinesis is a Foundational Service Used
Across Amazon
Amazon Go
video analytics
Amazon.com
online catalog
Amazon
CloudWatch
logs
Amazon
S3 events
AWS
metering

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Speed Matters

Streaming with Amazon Kinesis
Easily collect, process, and analyze video and data streams in real time
Capture, process, and
store video streams
Kinesis Video Streams
Load data streams into
data stores
Kinesis Data
Firehose
SQL
Analyze data streams
with SQL
Kinesis Data
Analytics
Capture, process, and
store data streams
Kinesis Data Streams

Amazon Kinesis Video Streams
Capture, process, and store video streams for analytics/machine learning
• Stream video from millions of devices
• Easily build vision-enabled apps
• Secure
• Durable, searchable storage
• Serverless
Amazon AI
Services
Apache
MxNet
TensorFlow
OpenCV
Custom
Video
Processing
Kinesis
Video Streams

Video analytics in the cloud and at the edge
with AWS DeepLens and Kinesis Video Streams

Amazon Kinesis Data Streaming
Collect, process, and analyze data streams in real time
Amazon
Elasticsearch
Service
SQL
EMR/Spark/
Amazon
SageMaker
Custom code
on EC2
Amazon S3
Amazon
Redshift
Splunk
Ingest
store data
streams
Kinesis Data
Streams
Kinesis Data
Analytics
Aggregate,
filter,
enrich data
Kinesis Data
Firehose
Egress
data
streams
AWS Lambda
• Real-time
• Fully-managed
• Scalable
• Secure
• Cost-effective

Amazon Kinesis Data Streams Overview

Kinesis Data Streams Producers and Consumers
Producers Consumers
Kinesis Agent
Apache Kafka
AWS SDK
LOG4J
Flume
Fluentd
AWS Mobile SDK
Kinesis Producer Library
Get* APIs
Kinesis Client Library
+ Connector Library
Apache Storm
Amazon EMR
AWS Lambda
Apache Spark
Amazon
Kinesis
Apache Kafka

Security and Compliance
• Supports VPC Endpoint powered by AWS PrivateLink
• Supports server-side encryption and
client-side encryption
• Using SSL and HTTPS
• Integrated with AWS Identity and Access
Management (IAM)
• FedRAMP, HIPAA, Soc, PCI, GDPR

Cost-Effective
• Pay-as-you-go pricing
• No upfront cost and no minimum fees
• Based on two dimensions (For KCL1.X) :
Shard-Hour: $0.015
PUT Payload Units (25K), per million units: $0.014
• Extended data retention, per shard hour: $0.020

Kinesis Data Streams: Standard egress
limitations
Shard 1
Shard 2
Shard 3
Shard n
Kinesis Data Stream
Consumer
Application A
GetRecords()
Data
GetRecords():
5 transactions per second, per shard
Data:
2MB per second, per shard
Data
Producer
up to 1 MB
or 1000
records per
second, per
shard
With only one
consumer
application,
records can be
retrieved every
200 ms.

Kinesis Data Streams: Standard egress
limitations
Shard 1
Shard 2
Shard 3
Shard n
Kinesis Data Stream
Consumer
Application
A
GetRecords()
Data
Data
Producer
Consumer
Application
B
Consumer
Application
C
Consumer
Application
D
Consumer
Application
E
With more
consumer
applications,
propagation delay
increases.
For example, with
5 consumer
applications, each
can only retrieve
records once per
second, and less
than 400 KBps.
<= 400 KBps

Kinesis Data Streams: Enhanced fan-out
Polling no longer necessary. Messages are pushed to the consumer as they arrive.
Shard 1
Kinesis Data Stream
Data
Producer
Consumer
Application A
SubscribeToShard()
Uses HTTP/2
• Up to 5 mins connection
• Data pushed to consumer
persist

Each consumer application gets dedicated 2MB per second egress, per shard
Shard 1
Kinesis Data Stream
Data
Producer
Consumer
Application B
Consumer
Application A
RegisterStreamConsumer()
EFO Pipe
SubscribeToShard()
Data: up to 2MB per second
RegisterStreamConsumer()
SubscribeToShard()Data: up to 2MB per second
EFO Pipe

When to use enhanced fan-out:
• Multiple consumer applications for the same Kinesis Data Stream
• Default limit of 5 registered consuming applications. More can
be supported with a service limit increase request.
• Low-latency requirements for data processing
• Messages are typically delivered to a consumer in less than 70
ms
Developing EFO Applications:
• Use Kinesis Client Library (KCL) 2.0

Pricing (US East - Ohio):
• $0.013 per GB of data retrieved
• $0.015 per consumer-shard hour

Kinesis Data Streams vs. Apache Kafka
Kafka Kinesis Data Streams
Cost $$ $ (pay for what you use)
Ease of use Advanced setup required Get started in minutes
Management
Overhead
High Low
Scalability Difficult to scale Scale in seconds with one click
Throughput Infinite
Scales with shards, supports up to 1mb
payloads
Durability Configurable 3x by default
Infrastructure You manage AWS manages
Write-to-Read Latency <100 ms is achievable <100 ms is achievable with HTTP2
Open Sourced? Yes No

Using Kafka with Kinesis Data Streams
Download the Kafka-Kinesis Connector Library
Kafka
cluster
Kafka
Kinesis
connector
S3 Bucket
(Archived Data/
Original data)
Redshift
(Data Warehousing)
S3 Bucket
(Transformed Data)
EC2
(Custom app)
EMR – Spark,
Pig, Hive, etc.
Athena
Kinesis Data Firehose
Transformation of
incoming data
Elasticsearch

Kinesis Data Firehose—How it Works
Ingest Transform Deliver
Amazon S3
Amazon Redshift
Amazon Elasticsearch Service
AWS IoT
Amazon Kinesis Agent
Amazon Kinesis Streams
Amazon CloudWatch Logs
Amazon CloudWatch Events
Apache Kafka

Kinesis Data Firehose: Record format
conversion
• Convert to Parquet or ORC before delivery to
S3
• Compresses file sizes and optimal format for
Hadoop usage
Kinesis Data
Firehose
Amazon S3

conversion
Kinesis Data
Firehose
Amazon S3
Glue Data
Catalog
Data
Producer
schema
convert to
columnar format
JSON data
/failed

conversion

conversion
When to use record format conversion:
• Delivering to S3 for use by Hadoop systems
• Columnar formats are more efficient.
• Compression leads to lower storage costs (vs.
uncompressed data).

Amazon Kinesis Data Analytics – How it Works

Customer Examples
50 billion daily
ad impressions,
sub-50 ms responses
Online stylist
processing 10 million
events/day
Facilitate
communications
between 100+
microservices
IoT predictive
analytics
Analyze billions of
network flows in
real-time
Near-real-time home
valuation
(Zestimates)
Live clickstream
dashboards refreshed
under 10’s
1 billion events per
week from
connected devices
100 GB/day
clickstreams from
250+ sites
Real-time
game events
analytics
IoT analytics

Buzzing@Hearst

Hearst’s Serverless Data Pipeline

Where are the anomalies in Netflix’s
network?

iRobot
Robot usage
• Number of robots, missions, sq ft cleaned
• Patterns of usage
User features
• Categorization
• Engagement
• Suggestions
System monitoring
• Important for serverless

Amazon Kinesis Solution Accelerators

AWS Answers
• Provides clear answers to common questions about architecting,
building, and running applications on the Amazon Web Services Cloud
• Each answer includes:
• Marketing page describe the problem and answer
• CloudFormation template with pre-built solution
• Implementation guide including details on how to customize

Solution Benefits
• Enable you to test out a solution in minutes
• Serverless and elastic
• Provide end to end template from real-time analytics to
integration with batch analytical tools
• Advanced analytics including machine learning (anomaly
detection)
• Best practice integrations with databases (DynamoDB), data lakes
(S3/Athena/Redshift), and monitoring (CloudWatch)
• Use the solutions as a starting point.
Customize to meet your needs.

Real-Time Insights on AWS Account Activity
https://aws.amazon.com/answers/account-management/real-time-insights-account-activity/

Real-Time IoT Device Monitoring with Kinesis Data Analytics
https://aws.amazon.com/answers/iot/real-time-iot-device-monitoring-with-kinesis/

Real-Time Web Analytics with Kinesis Data Analytics
https://aws.amazon.com/answers/web-applications/real-time-web-analytics-with-kinesis/

Monitor and Analyze VPC Network Traffic
Amazon
CloudWatch
Logs
Subscription
Amazon Kinesis
Data Analytics
Amazon S3
bucket for data
in Parquet/ORC
format
Amazon Kinesis
Data Firehose
Amazon
Athena
Amazon
CloudWatch
dashboardsmetrics & alarms
Amazon VPC
flow logs
Coming soon to AWS Big Data Blog

Thank You!
benaltar@

Introduction to Real-Time Streaming Analytics - Amazon Kinesis State Of Union | AWS Floor28

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Introduction to Real-Time Streaming Analytics - Amazon Kinesis State Of Union | AWS Floor28

Similar a Introduction to Real-Time Streaming Analytics - Amazon Kinesis State Of Union | AWS Floor28 (20)

Más de Amazon Web Services

Más de Amazon Web Services (20)

Introduction to Real-Time Streaming Analytics - Amazon Kinesis State Of Union | AWS Floor28