104 DataFlow 從地上到雲端的無伺服器之路

1. DataFlow
從地上到雲端的無伺服器之路路
Senior Data Engineer
Gavin Lin
1

2. Why a data engineer become a
Data, Cloud, DevOps, System and AI
engineer ?
2

3. AGENDA
• Overview
• Why we upgrade
• What we want
• How we migrate
• Conclusions
3

4. Data
7.5
Millions
50
Millions
300
Billions
Resumes Jobs Behaviors
4Special thank Neil.Lee

5. Routine works
• Data processing
• Data mining (Marketing)
• Machine learning (Matching)
• Item-Item: Users who liked this
item also liked
• User-Item: Users who are similar
to you also liked
5Captain monkey - Sean.Chang

6. Why we upgrade
HDFS
Pig
v1.1.0
v0.11.0
• Resource utilization
• High-Availability
• Performance issue
6

7. What we want
• Resource Management. YARN
• High Availability. Zookeeper, Hadoop2
• Performance. Spark, Flink
• Streaming. Kafka
• Exploration. Kibana, Zeppelin, Jupyter
• Multi programming languages. Java, Python, Scala
7

8. Version II in IDC
HDFSv2
YARN
SparkFlink
Zookeeper
Kafka
Notebooks
HBase
Pig
ELK
8
Streaming Computing Exploration

9. Upgrade plan
Migration plan
9

10. Let’s go to AWS
Bottom-up &Top-down
10

11. Engineer == Everything Engineer
11

12. How to migrate
ASF: 350+ Projects
AWS
12

13. An easy and expensive version
HDFS
YARN
SparkFlink
Zookeeper
Kafka
Notebook
HBase
Pig
ELK
13
Amazon
EMR
m5.large(0.124 USD) x 64台 x 24⼩小時 = 190 USD/day
128 cores, 512 G ram

14. “XXX as a Service” first
why not ?
14
Open Mind

15. Streaming
15
Raw
Parquet
LogServer
AWS
Batch
Amazon
ECS
container
event
(time-based)
Amazon Kinesis
Firehose
Dataset/Model
Amazon
EMR
Exploration
Amazon
DynamoDB
Amazon
ElastiCache
container container
containercontainer container
containercontainer
containercontainer
Application
Load Balance
ServingETL, ML
Amazon
CloudWatch

16. Full-Controllable or not
Amazon
Kinesis
Apache
Kafka
16
Unit Stream Topic
Distribution Shards Partitions
Thoughtput
2 MB read/shards
1 MB write/shards
Based on cluster size
Fault tolerance Handled by AWS Replica
Transformer AWS Lambda Connectors/Processors
Framework Support Spark and Flink

17. Storage
17
Raw
Parquet
LogServer
AWS
Batch
Amazon
ECS
container
event
(time-based)
Amazon Kinesis
Firehose
Dataset/Model
Amazon
EMR
Exploration
Amazon
DynamoDB
Amazon
ElastiCache
container container
containercontainer container
containercontainer
containercontainer
Application
Load Balance
ServingETL, ML
Amazon
CloudWatch

18. Cost / Performance
Amazon
S3
Amazon
EMR
Type Objects Block device
Throughput Middle High
Cost 0.025 USD/GB 0.12x3 USD/GB (EBS gp2)
Maintenance
No
(Policy / Lifecycle)
Yes
Libraries
Hadoop-aws
(3.x.x is better)
ALL
18
Storing Apache Hadoop Data on the Cloud - HDFS vs. S3
Top 5 Reasons for Choosing S3 over HDFS

19. Computing
19
Raw
Parquet
LogServer
AWS
Batch
Amazon
ECS
container
event
(time-based)
Amazon Kinesis
Firehose
Dataset/Model
Amazon
EMR
Exploration
Amazon
DynamoDB
Amazon
ElastiCache
container container
containercontainer container
containercontainer
containercontainer
Application
Load Balance
ServingETL, ML
Amazon
CloudWatch

20. Exploration
Online Analytical Processing
• Interactive interface for complexity and repeated
SQL query
20
Amazon
EMR
Warn man - ifegn.chen

21. ETL / ML
Orchestration
21
AWS Batch
https://kubernetes.io/blog/2018/06/28/airflow-on-kubernetes-part-1-a-different-kind-of-operator/

22. SUBMITTED
PENDING
Directed Acyclic Graph
22
RUNNABLE
STARTING
RUNNING
FAILED
SUCCEEDED
M, R family
Queue1
P family
Aggregation
One-hot
TrainingQueue2
One-hot
ETL
Submit Job
Dependencies
Job Definition 1
Job Definition 2
AWS Batch
Tool man - scott.hsieh
A typical machine learning workflow.

23. Artifact Docker image
JAR,Wheel
(Docker image)
Parallelism Independent Cluster
Scheduling YES No
Our scenario ETL / ML Exploration
Computing depends on
scenarios.
Amazon
EMR
AWS
Batch
23

24. Event-Driven
24
Raw
Parquet
LogServer
AWS
Batch
Amazon
ECS
container
event
(time-based)
Amazon Kinesis
Firehose
Dataset/Model
Amazon
EMR
Exploration
Amazon
DynamoDB
Amazon
ElastiCache
container container
containercontainer container
containercontainer
containercontainer
Application
Load Balance
ServingETL, ML
Amazon
CloudWatch

25. Event eeee….
25
Amazon S3
alarm
event
(time-based)
event
(event-based)
AWS
CloudFormation
AWS Batch
Amazon
EMR
IAM
AWS
Config
Amazon Kinesis
Batch
Streaming
Auditting/Monitoring Processing
Notification
Justin - scott.liao

26. Serving(POC)
26
Raw
Parquet
LogServer
AWS
Batch
Amazon
ECS
container
event
(time-based)
Amazon Kinesis
Firehose
Dataset/Model
Amazon
EMR
Exploration
Amazon
DynamoDB
Amazon
ElastiCache
container container
containercontainer container
containercontainer
containercontainer
Application
Load Balance
ServingETL, ML
Amazon
CloudWatch

27. AWS - DataFlow
Raw
Parquet
LogServer
AWS
Batch
Amazon
ECS
container
event
(time-based)
Amazon Kinesis
Firehose
Dataset/Model
Amazon
EMR
Exploration
Amazon
DynamoDB
Amazon
ElastiCache
container container
containercontainer container
containercontainer
containercontainer
Application
Load Balance
Serving
27
ETL, ML
Amazon
CloudWatch

28. Conclusions
28

29. • “XXX as a Service” first - why not
• Full-Controllable or not
• Cost / Performance (pay-as-you-go)
• Computing environment depends on scenario.
29

30. Speed UP
• Official Document
• Support
• Official
• Community
30

31. Thanks you all,
and my teammates.
31