Nowadays, it’s common for a web server to be fronted by a global content delivery service, such as Amazon CloudFront, to accelerate delivery of websites, APIs, media content, and other web assets. Website administrators and developers want to generate insights in order to improve website availability through bot detection and mitigation, by optimizing web content based on the devices and browser used, by reducing perceived latency by caching a popular object closer to its viewer, and so on. In this session, we dive deep into building an end-to-end serverless analytics solution to analyze Amazon CloudFront access logs, both at rest and in transit, using Amazon Athena and Amazon Kinesis Analytics, respectively, and we generate visualization insights using Amazon QuickSight. Join a discussion with AWS solution architects to learn more about the various ways to generate insights to improve the overall perceived experience for your website users.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
Serverless Anaylsis & Visualization of
Website Access Logs
J o s e p h P e l l e , A W S S o f t w a r e D e v e l o p m e n t M a n a g e r
R a j e e v S r i n i v a s a n , A W S S o l u t i o n A r c h i t e c t
S a i S r i p a r a s a , A W S B i g D a t a C o n s u l t a n t
C T 4 0 8
N o v e m b e r 3 0 , 2 0 1 7

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What Are We Going to Cover?
• Benefits of Analyzing Amazon CloudFront Access Logs
• Service Introduction
• Amazon CloudFront Reports and Operational Metrics
• Custom Report Generation using Batch Analysis
• Bot detection and mitigation using Batch & Online Analysis

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Increased Service and Application Availability Meet Compliance & Audit Requirements
Reduce Website Latency Content Optimization Bot Detection & Mitigation
Benefits of Analyzing Access Logs

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
SERVICE INTRODUCTION
A W S L a m b d a
A m a z o n Q u i c k S i g h tA m a z o n K i n e s i s
A m a z o n C l o u d F r o n t
A m a z o n W A F
A m a z o n A t h e n a A W S G L U E

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon CloudFront
Global Content Delivery Network
with Massive Capacity and Scale
Optimized for
Performance and Scale Built-in Security Features
Deep Integration with Key AWS Services Robust Real-Time Reporting Static and Dynamic Object and Video Delivery

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon CloudFront
Amazon
CloudFront
Route 53
Amazon S3 Bucket Origin
Amazon S3 Bucket
for Access Logs
Amazon EC2 Origin
Access Logs
Private Datacenter

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon CloudFront + AWS WAF

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon CloudFront Access Logs
Amazon CloudFront
Edge Location
Amazon CloudFront
Edge Location
Amazon CloudFront
Edge Location
Amazon CloudFront
Edge Location
Website B
Website A
Log File for
Distribution A
Log File for
Distribution B
Amazon S3 Bucket
for Access Logs

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon CloudFront Access Logs
• S3 access control list (ACL) for the bucket must grant FULL_CONTROL to write log files to the bucket
• File Name Format
bucket-name.s3.amazonaws.com/optional-prefix/distribution-ID.YYYY-MM-DD-HH.unique-ID.gz
• File name includes the date and time of the period in which the requests occurred
• Delivers access logs for a distribution up to several times an hour
• CloudFront can save one or more files for a time period
• Disable logging: Amazon CloudFront doesn't delete the ACLs for both the bucket or the log files

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon CloudFront: Customer Stories
Social Media, Digital
Advertising, EdTech,
Finance
Enterprise
E-commerce
Media & Entertainment Gaming

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Lambda: Stream ProcessingCapture Data
Streams
IoT Data
Financial
Data
Log Data
No servers to
provision or manage
EVENT SOURCE
Node.js
Python
Java
C#
Process Data Streams
FUNCTION
Clickstream
Data
Output
Data
DATABASE
CLOUD
SERVICES

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Kinesis
Amazon Kinesis
Streams
Amazon Kinesis
Firehose
Amazon Kinesis
Analytics

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Kinesis
Easy administration: Create a stream, set capacity level with shards. Scale to match your data throughput rate and
volume.
Build real-time applications: Process streaming data with Kinesis Client Library (KCL), Apache Spark/Storm, AWS
Lambda...
Low cost: Cost-efficient for workloads of any scale.

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Firehose
Capture and submit streaming
data to Firehose
Analyze streaming data
using your favorite BI toolsFirehose loads streaming data continuously into Amazon S3,
Amazon Redshift and Amazon Elasticsearch
Zero administration: Capture and deliver streaming data to Amazon S3, Redshift, Elasticsearch
without writing an app or managing infrastructure
Direct-to-data store integration: Batch, compress, and encrypt streaming data for delivery in as little
as 60 seconds
Seamless elasticity: Seamlessly scales to match data throughput without intervention

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Delivery Format
• Multiple records concatenated based on buffering options
• Object path: app-name/YYYY/MM/DD/HH
• Object name: DeliveryStreamName-DeliveryStreamVersion-YYYY-MM-DD-HH-MM-SS-RandomString
Amazon Firehose
Delivery Stream
• Delivery stream can deliver to single S3 bucket
• Buffer size (1 MB to 128 MB)
• Buffer interval (60 to 900 seconds)
• Condition satisfied first triggers data delivery
Failure and Error Handling
Pause and retry for up to 24 hours

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Kinesis Analytics
Apply SQL on streams: Easily connect to a Kinesis Stream or Firehose Delivery Stream and apply SQL skills
Build real-time applications: Perform continual processing on streaming big data with sub-second processing
latencies
Easy scalability : Elastically scales to match data throughput

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Athena
Start Querying Instantly
Serverless, no ETL
Pay Per Query
Only pay for data scanned
Open, Powerful, Standard
Built on Presto, runs standard SQL.

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon QuickSight

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon QuickSight: Data Sources
S P I C E
S P I C E
Amazon
Athena
Amazon
S3

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon QuickSight: Data Sources
Amazon
Redshift
Redshift
Spectrum
S P I C E
S P I C E

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Glue
Integrated
Data Catalog
Automated
Data Discovery
Code
Generation
Developer
Endpoints
Flexible
Job Scheduler

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Glue: Components
Ø Hive Metastore-compatible with enhanced functionality
Ø Crawlers automatically extract metadata and create tables
Ø Integrated with Amazon Athena, Amazon Redshift Spectrum
Ø Run jobs on a serverless Spark platform
Ø Provides flexible scheduling
Ø Handles dependency resolution, monitoring, and alerting
Ø Auto-generates ETL code
Ø Build on open frameworks: Python and Spark
Ø Developer-centric: editing, debugging, sharing
Data Catalog
Job Authoring
Job Execution

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon CloudFront
Reports and Operational Metrics

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• 5 ready-for-you reports
• Provides insight from viewer requests
• Usage, Cache Stats, Popular Objects, Viewer
Data, Top Referrers
• Built on top of Access Logs and Billing Data
• Use for an accurate historical view
• Up to 60-day history
• Data in CSV format
Amazon CloudFront Reports
Amazon
CloudFront
Edge Locations Amazon
CloudFront Console
Amazon S3
Log Bucket

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• 6 Operational Metrics
• Near-real time operational view
• Use to watch what is happening now
• 1 minute, 5 minute, 1 hour, 1 day granularity
• Available in Amazon CloudFront Console and
Amazon CloudWatch
• Can add Amazon CloudWatch Alarms and get SNS
notifications
• Monitor requests, bytes downloaded, bytes
uploaded, total error rate, 4xx Error Rate, 5xx
Error Rate
Amazon CloudFront Metrics
Amazon
CloudFront
Edge Locations
Amazon
CloudFront Console
Amazon
CloudWatch

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Operational Monitoring
• Set alarm on 4xx Error Rate
• Monitor the spike in Amazon CloudFront or
Amazon CloudWatch

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Operational Analysis

28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Operational Analysis

29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Operational Analysis

30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Custom Report Generation
Batch Analysis

31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Deeper Analysis of Access Logs
#Version: 1.0
#Fields: date time x-edge-location sc-
bytes c-ip cs-method cs(Host) cs-uri-
stem …
2017-01-18 10:00:00 AMS1 110891
194.xxx.xxx.xxx GET
d1234.cloudfront.net
/assets/some-static.jpg 200 …

32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Setup the Reporting Pipeline
Amazon Athena
Amazon CloudFront
Access Logs
Amazon S3
AWS GlueAmazon
QuickSight
• Amazon CloudFront Access Logs are
available in S3
• Use AWS Glue to discover your data
sources, including your log bucket
• Build a data catalog of data and make it
available to Amazon Athena
• Use Amazon QuickSight to analyze the
data
• Amazon Athena will query your S3 bucket
• Simple and serverless!
AWS Glue Catalog
Data Source
Discovery
Query Data

33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What are the Benefits?
Amazon Athena
Amazon S3
AWS GlueAmazon
QuickSight
• Deeper insight than default reports
• Customized reports
• Drill into 4xx, 5xx
• Show error rates for region
• Look at top X popular objects, not top 50
• Optimize your cache by correlating user
agents and cache behavior
• Edge Location reporting
• Sales and Marketing
• Mine your Viewer and Geo data
• Compare to popular objects
AWS Glue Catalog
Data Source
Discovery
Query Data
Amazon CloudFront
Access Logs

34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Bot Detection & Mitigation

35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Example: Blacklisting Bad Bots
Block unwanted user agent headers and use transforms to stop evasion:
Host: www.example.com
User-Agent: bAdBoT
Accept: image/png,image/*;q=0.8,*/*;q=0.5
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referrer: http://www.InTeRnEtkItTiEs.com/
Connection: keep-alive
AWS
WAF
RAW request headers
CloudFront
Check: Header “User-Agent”
Transform: To lower
Match Type: Contains
Match: “badbot”
Action: BLOCK
Rule
String match condition
Scraper bot

36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon
CloudWatch
AWS WAF
Bot Detection Using Batch Analysis
Amazon
CloudFront
Users

37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon
CloudWatch
AWS WAF
Bot Detection Using Batch Analysis
Amazon S3
Amazon
CloudFront
Users

38. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS
Lambda
Amazon S3
Amazon
CloudWatch
AWS WAF
Bot Detection Using Batch Analysis
Users
Amazon
CloudFront

39. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon S3
Amazon
Kinesis Firehose
AWS
Lambda
Amazon S3
Amazon
CloudWatch
AWS WAF
Bot Detection Using Batch Analysis
Users
Amazon
CloudFront

40. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon
Athena
Amazon S3AWS
Lambda
Amazon S3
Amazon
CloudWatch
AWS WAF
Bot Detection Using Batch Analysis
Users
Amazon
CloudFront
Amazon
Kinesis Firehose

41. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon
CloudWatch
Amazon S3 Amazon
Kinesis Firehose
AWS
Lambda
AWS WAF
Bot Detection Using Online Analysis
Users
Amazon
CloudFront

42. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon
CloudWatch
Amazon S3 Amazon
Kinesis Firehose
AWS
Lambda
Amazon
Kinesis Analytics
AWS WAF
Bot Detection Using Online Analysis
Users
Amazon
CloudFront

43. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon
CloudWatch
Amazon S3 Amazon
Kinesis Firehose
AWS
Lambda
Amazon
Kinesis Analytics
Amazon
Kinesis Streams
AWS
Lambda
AWS WAF
Bot Detection Using Online Analysis
Users
Amazon
CloudFront

44. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon
CloudWatch
Amazon S3
Amazon
Kinesis FirehoseAWS
Lambda
Amazon
Kinesis Analytics
Amazon
Kinesis Streams
AWS Lambda
AWS WAF
Bot Detection & Mitigation
Users
Amazon
CloudFront

45. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon
CloudWatch
Amazon S3 Amazon
Kinesis Firehose
AWS
Lambda
Amazon Kinesis
Analytics
Amazon Kinesis
Streams
AWS
Lambda
Amazon
Athena
Amazon S3
AWS WAF
Bot Detection & Mitigation
Users Amazon
CloudFront

46. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon
CloudWatch
Amazon
S3
Amazon
Kinesis FirehoseAWS
Lambda
Amazon
Kinesis Analytics
Amazon
Kinesis Streams
AWS
Lambda
Amazon
Athena
Amazon
S3
Amazon
QuickSight
AWS WAF
Using Batch & Online Analysis for
Bot Detection & Mitigation
Users
Amazon
CloudFront

47. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Best Practices
Partitioning
• Reduce the amount of data scanned files necessary for queries
Compression file sizes
• Splittable files allow Athena’s execution engine to split the reading of a file by
multiple readers to increase parallelism
Columnar formats for analytics
• Optimize column-based reads
• Use Apache Parquet and Apache ORC

48. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Best Practices
Use environmental variables to pass operational parameters to your function
Control the dependencies in your function’s deployment package
Test your Lambda functions with different batches and record sizes
Use most-restrictive permissions when setting IAM policies

49. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Best Practices
Set up CloudWatch alarms
• Set up CloudWatch alarms to monitor input/output bytes and input/output records
• Monitor MillisBehindLatest to track how far behind the application is in reading from
the streaming source
Monitor CloudWatch alarms
• Getting the optimal number of Kinesis Analytics applications reading from the
Kinesis Data Streams and Kinesis Firehose Delivery Stream

50. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!
Pl e ase c o m p l e te yo ur su r v e y