Learning Objectives: - Learn how to accelerate common data transformations from a variety of data - Learn how to efficiently orchestrate transformation jobs - Learn best practices and methodologies in data preparation for analytics

1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Hemant Borole
AWS Professional Services Consultant
Marie Yap
AWS Enterprise Solutions Architect
Data Transformation Patterns
Using Amazon Glue to transform data in your Data Lake

2. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Agenda
• Our journey starts with the Data Lake
• How Glue connects everything
• ETL in AWS
• Glue Job and Scheduling
• Glue Transforms
• Demo on different Glue Transforms
• Tips and Best Practices

3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
A quick recap on Data Lake

4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Traditionally, analytics used to look like this
OLTP ERP CRM LOB
Data Warehouse
Business Intelligence Relational data
TBs-PBs scale
Schema defined prior to data load
Operational reporting and ad hoc
Large initial capex + $10k-$50k / TB / year

5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Data Lakes extend the traditional approach
Relational and non-relational data
TBs-EBs scale
Schema defined during analysis
Diverse analytical engines to gain insights
Designed for low cost storage and analytics
OLTP ERP CRM LOB
Data Warehouse
Business
Intelligence
Data Lake
100110000100101011100
101010111001010100001
011111011010
0011110010110010110
0100011000010
Devices Web Sensor
s
Social
Catalog
Machine
Learning
DW
Queries
Big data
processing
Interactive Real-time

6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Snowball Snowmobile Kinesis
Data Firehose
Kinesis
Data Streams
S3
Most ways to bring data in
Unmatched durability and availability at EB scale
Best security, compliance, and audit capabilities
Run any analytics on the same data without movement
Scale storage and compute independently
Store at $0.023 / GB-month; Query for $0.05/GB scanned
Redshift
EMR
Athena
Kinesis
Elasticsearch Service
Data Lakes on AWS

7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Extract Transform and Load
• Sources
• Raw
• Relational, semi
or unstructured
Extract
• File - Format,
Compression,
Partitioning
• Data
Transform • Stage or
Processed
• Destination(s)
• Visualization
Load

8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
There are many ways to do analytics
Interactive Analytics
Big Data Processing
Data Warehousing
Amazon
Athena
Amazon
EMR
Amazon
Redshift*
Amazon ES
Operational Analytics
Amazon
Kinesis
Analytics
Real time Analytics
Amazon
QuickSight
Dashboard and Visualization

9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
There has to be something that
Glues them together

10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Glue

11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Glue the services together through Data Catalog
Amazon
Athena
Amazon
EMR
AWS Glue
Jobs
Amazon
Redshift
Spectrum

12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Extract Transform Load in AWS
Load
•RDS/Databases
•EDW/Redshift
•NoSQL, DynamoDB
•Machine Learning
(SageMaker)
•S3 (Processed output bucket)
Transform
• Amazon Athena
• Amazon Redshift
• Amazon EMR
• AWS Glue
Extract
• Files
• RDS/Database
• EDW
• Glue Data Catalog
• S3

13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Knowing the JOB

14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Terminologies
JOB - It performs the ETL work in Glue
JOB BOOKMARKS – saves the state across multiple job
runs

15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How does a bookmark work?
START OF BOOKMARK
datasource0 =
glueContext.create_dynamic_frame.
from_catalog
(database = source_database,
table_name = source_tables[0],
transformation_ctx = "datasource0")
END OF BOOKMARK
job.commit()

16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Job Bookmark options

17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Job Scheduling
Compose jobs globally with event-
based dependencies
 Easy to reuse and leverage work across
organization boundaries
Multiple triggering mechanisms
 Schedule-based: e.g., time of day
 Event-based: e.g., job completion
 On-demand: e.g., AWS Lambda
 More coming soon: Data Catalog based
events, S3 notifications and Amazon
CloudWatch events
Logs and alerts are available in
Amazon CloudWatch
Marketing: Ad-spend
by
customer segment
Event Based
Lambda Trigger
Sales: Revenue by
customer segment
Schedule
Data
based
Central: ROI by
customer
segment
Weekly
sales
Data
based

18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
So how do I transform?

19. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Glue is based on Apache Spark

20. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Writing your transformations
https://docs.aws.amazon.com/glue/latest/dg/dev-endpoint.html

21. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Transforming Distributed, Immutable Data Structures

22. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
But why do we have a DynamicFrame?
Limitations of DataFrame for ETL Operations
• Expensive in loading large data sets
• Infer the schema
• Load the data
• Does not handle unstructured data types well
• Does not easily deal with Error Handling
DynamicFrame
• Do not need to know schema upfront (Schema per record)
• Supports ETL specific Transformations

23. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Problem Solving – Unstructured Data in DynamicFrame
df_order = glueContext.create_dynamic_frame.from_catalog
(database=“retail”, table_name=“orders”)

24. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Solution - ResolveChoice
Option 1 – cast
Eliminate the string values by replacing it with null.
Output Schema – ordereid:int
Option 2 – project
Project all the data to one data type. In this case it will convert string to int.
order_rc_cast = df.order.resolveChoice(specs = [(‘ordereid’,’project:int’)])
Output Schema – ordereid:int
order_rc_cast = df.order.resolveChoice(specs = [(‘ordereid’,’cast:int’)])

25. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Solution - ResolveChoice
Option 3 – make_cols
Splits the column into 2, one for each type
Output Schema – ordereid_int: int
ordereid_string: string
Option 2 – make_struct
Use struct to represent each of the data types
order_rc_cast = df.order.resolveChoice(specs = [(‘ordereid’,’make_struct’)])
Output Schema – ordered: struct
int: int
string: string
order_rc_cast = df.order.resolveChoice(specs = [(‘ordereid’,’make_cols’)])

26. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Ok now we know
DynamicFrame, lets get to the
ETL Job Programming

27. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Basics of Glue ETL Script
1. Create GlueContext, the entry point
2. Create DynamicFrame from Data Catalog
Other options:
DynamicFrame.fromDF
from_options
3. Transform Data by modifying DynamicFrame
4. Write DynamicFrame into the Target (sink)
Other Options:
from_catalog
from_jdbc_conf
Optionally, you can add your job bookmarks

28. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Using Glue’s Built-in
Transforms (Demo)

29. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Tips and Best Practices
• Useful for debugging
• pause bookmark
• spigot()
• errorsAsDynamicFrame( )
• Take advantage of bookmarks
• Know how much resource you need (DPU)
• Partition/Compress/Format your data to optimize transforms
• https://docs.aws.amazon.com/athena/latest/ug/glue-best-
practices.html

30. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank You!!!
Documentation:
https://docs.aws.amazon.com/glue/latest/dg/getting-started.html
Built in Glue Transforms:
https://docs.aws.amazon.com/glue/latest/dg/built-in-transforms.html
Examples:
https://github.com/aws-samples/aws-glue-samples