A sharing in a meetup of the AWS Taiwan User Group. The registration page: https://bityl.co/7yRK The promotion page: https://www.facebook.com/groups/awsugtw/permalink/4123481584394988/

1. Serverless Data Platform
Scott （考特）
July 22nd, 2021 (Thur)
AWS User Group

2. Scott (考特)
Shu-Jeng, Hsieh
● Sr. Data Engineer, the 104
● AWS Community Builder

3. Agenda
Prologue
What is a lakehouse?
The lakehouse built on AWS
Summary

4. Prologue

5. 104 Corporation Region
Production
Research Environment
Production
Research Environment
Analysts Scientists

6. Amazon EMR
Analysts
Developers
PMs
Managers

7. ● a SAP tool to ETL
● SPSS for insight &
stats modeling
Analysts
● license fee
● outdated front-end
technology
Scientists
● got power from the
AWS computing
● explore potential
ML applications
● unavailability of data
○ external
○ internal
● data difference
Data Engineers
● open-sourced
tools to ETL
● productionize
scientists’
invention
● maintain & evolve
existing services
● explore potential
ML applications
Other BUs
● Make wishes
● repetitive tasks on
data expansion
● unavailability of
data

8. What is a lakehouse?

9. Data Lake
Unstructured
Semi-structured
structured
Machine
Learning
Data
Science
Data Warehouse
Data Warehouse
Data Warehouse
BI
BI
BI

10. External Data Operational Data
BI Reports
Data Warehouse
ETL
Data Warehouses
- Built for BI and reporting
- No support for video, audio, text
- No support for data science, ML
- Limited support for streaming
- Closed & proprietary formats

11. structured, semi structured, and unstructured data
BI Reports
Data Warehouse
ETL
Data Lakes
- Poor BI support
- Complex to set up
- Poor performance
- Unreliable data swamps
Data Prep and
Validation
Real-time
Database
Data Lake
Machine
Learning
Data
Science

12. structured, semi structured, and unstructured data
BI
Lakehouse
Machine
Learning
Data
Science
Streaming
Analytics
One platform for every use case
Data Lake for all your data

13. The lakehouse built on
AWS

15. CloudFormation CDK
● With elapsed time,
YAML/JSON became larger
● Difficult to work with large
YAML/JSON files
○ High error ratio when
copying/pasting
○ It’s a text file, not
programming language
● Infrastructure AS code
● No abstraction
● IDE integration
○ Multiple languages
○ Syntax check,
autocompletion, etc.
● Higher level abstraction
○ Simplified statements
○ 500 lines of CFN to 30
lines of CDK code
● Infrastructure IS code

18. Why S3
S3 Standard S3 INT S3 S-IA S3 O-IA S3 Glacier
Frequent Infrequent
Access Frequency

19. ● Efficient, columnar data representation.
● Utilizes the record shredding and assembly algorithm.
● Supports schema evolution.

20. Dataset
Size on
Amazon S3
Query
Run
Time
Data
Scanned
Cost
Data stored as CSV files 1 TB
236
seconds
1.15 TB $5.75
Data stored in
Apache Parquet Format
130 GB
6.78
seconds
2.51 GB $0.01
Savings
87% less when
using Parquet
34x
faster
99% less data
scanned
99.7% savings

21. The lakehouse built on
AWS
CDC pipeline

22. Debezium Architecture

23. Database Migration Service
Source Database Target Database
AWS DMS
Source
Endpoint
Target
Endpoint
Replication Instance
Replication
Task

24. IBM® Db2®

25. ● CDC listening supported for some dbs
● C-language-based
● Configuration-oriented
● Abundant supported sources and targets
AWS DMS

26. The lakehouse built on
AWS
Big data tools

27. - Spark APIs
1. RDD
2. DataFrame
3. Dataset
- Various modes
1. Local
2. Standalone
3. Mesos
4. k8s
- Supported interfaces
1. Java/Scala
2. Python
3. R
4. SQL
- Unified interfaces for
batch and streaming
Worker
JVM
Executor
Worker
JVM
Executor
Worker
JVM
Executor
Driver

28. Apache Hudi
Apache Iceberg
Delta Lake

29. Delta Lake (open source) Apache Iceberg Apache Hudi
Transaction (ACID) Y Y Y
MVCC Y Y Y
Time travel Y Y Y
Schema Evolution Y Y Y
Data Mutation Y (update/delete/merge/ merge into) N Y (upsert)
Streaming Sink and source for Spark struct streaming
Sink and source (wip) for Spark
struct streaming, Flink (wip)
DeltaStreamer
HiveincrementalPuller
File Format Parquet Parquet, ORC, AVRO Parquet
Compaction/Cleanup Manual API available Manual and Auto
Integration DSv1, Delta connector DSv2, InputFormat DSv1, InputFormat
Multiple language support Scala/Java/Python Java/Python Java/Python
Storage Abstraction Y Y N
API dependency Spark-bundled Native/Engine bundled DeltaStreamer
Data ingestion Spark, Presto, Hive Spark, Hive DeltaStreamer

30. The lakehouse built on
AWS
Databricks

31. ACID transactions on Spark
Delta Lake
Scalable metadata handling
Streaming and batch unification
Schema enforcement
Time travel
Upserts and deletes

32. https://github.com/databrickslabs/terraform-provider-databricks

33. Databricks
AWS account
User
AWS account Data plane
in user account
Control plane network in Databricks account
Workspace web application, APIs, and other core services
AWS VPC
endpoint service
AWS VPC
endpoint service
Back-end VPC endpoint
for secure cluster connectivity
relay
Back-end VPC endpoint
for
REST APIs
AWS PrivateLink connection (back-end) AWS PrivateLink connection (back-end)
User on-premise
or VPN network
User transit
AWS account Front-end VPC endpoint
For user access to web
App and REST APIs
User request
to web app
or REST APIs
VPC
AWS PrivateLink
Connection (front-end)

34. Languages
Scala Rust Python ruby Golang
**
**
Services
Connectors
Databases
Databricks
dafka-
delta-
ingest
Airbyte *
* Currently in development ** coming soon

35. The lakehouse built on
AWS
Data Platform

36. CDK Application
Construct
utility components
Construct
S3-related
Construct
Databricks
Construct
DMS instances
Construct
DMS endpoints
Construct
DMS tasks

37. High-level
Architecture

38. The lakehouse built on
AWS
During the build time

39. Glue ETL
1. Serverless Spark
2. DynamicFrame
3. Self-describing, no schema required initially
4. Some feature functions
a. ResolveChoice
b. Unbox => similar to from_json
c. Spigot => similar to TABLESAMPLE

40. Glue crawlers
Native client JDBC MongoDB client

41. Glue workflow
Mohit, S., 2020, Simplify data pipelines with AWS Glue automatic code generation and Workflows

42. Developer
Endpoint
Glue scripts
https://docs.aws.amazon.com/glue/latest/dg/dev-endpoint-workflow.html

43. New log
data arrives
Start
Glue crawler
Needed stats
by requirements
Glue crawler
that deals with
metadata
Glue job
that executes
the ETL
Start
Glue job
CloudWatch
event
Before Jun 20, 2019….

44. Glue workflow
New log
data arrives
Start
Glue workflow
Needed stats
by requirements
Glue job
that executes
the ETL
Glue crawler
that deals with
metadata

45. CDK Application
Construct
S3 Event
Notification
Construct
S3 Event
Notification

46. Version
Supported Spark
and Python
Minimum DPU Default DPU
AW Glue 0.9
● Spark 2.2.1
● Python 2.7
2 10
AWS Glue 1.0
● Spark 2.4.3
● Python 2.7
● Python 3.6
2 10
AWS Glue 2.0
● Spark 2.4.3
● Python 3.7
2 10

47. AWS Glue Delta Lake
Spark version Spark 2.2.1 & 2.4.3 Always the latest
UI experience
More clicks to job
logs
On hover
Reliability & quality - ACID transactions
● Dynamic Partition Pruning
● Adaptive Query Execution
● accelerator -aware scheduler
● Unified table creation SQL syntax
● Shuffled hash join improvement

48. Summary

49. Amazon
S3
AWS
Snowball
AWS
Snowmobile
Amazon Kinesis
Video Streams
Amazon Kinesis
Data Firehose
Amazon Kinesis
Data Streams
Amazon
Kinesis
Amazon
Redshift
Amazon
EMR Amazon
Athena
Amazon
Elasticsearch
Service
AI services
● Any type of data
● Security, compliance, and
audit capabilities across
data lake
● Empower all personas
● Democratize ML with SQL
● Unified analytics

50. Open Data Lake (S3)
Data Management & Governance
Data Science
& Machine Learning
Real-time Data
Applications
BI & SQL
Analytics
Data Engineering
Structured Semi-structured Unstructured Streaming
Lakehouse Platform

52. Cheng-Wei
Bill Chih-Yi
Daniel Luffy Neil
TItan
Nick Scott

53. Ask
Me
Anything