Apache Airflow는 복잡한 데이터 처리 파이프라인의 전체적인 프로세스를 자동화하기 위한 워크플로우 관리 플랫폼이며 오픈 소스 커뮤니티에서 활발하게 기여하고 있는 top-level 프로젝트 입니다. AWS는 최근에 Amazon Managed Workflow for Apache Airflow (MWAA) 서비스를 정식 출시하였고, 본 강연에서는 Apache Airflow 및 MWAA를 소개하고 어떻게 AWS 서비스와 연동하여 데이터 처리 워크플로우를 구축할 수 있는지 데모를 통해 알려 드립니다.

1. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
K O R E A | M A Y 1 1 - 1 2 , 2 0 2 1

2. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
아름답고 유연한 데이터 파이프라인 구축을
위한 Amazon Managed Workflows for
Apache Airflow
유다니엘
솔루션즈 아키텍트
AWS

3. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data Pipeline Problem
Apache Airflow
Amazon Managed Workflows for Apache Airflow (MWAA)
Demo
Airflow Integrations
MWAA vs Step Function
Agenda

4. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
복잡한 데이터 파이프라인

5. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
오케스트레이션

6. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Apache Airflow

7. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Apache Airflow

8. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
사용 사례
ETL AI/ML DevOps

9. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
주요 구성 요소
스케줄러 워커 메타 데이터베이스
웹서버

10. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
셀프 구축 및 운영의 어려움
셋업 보안 유지관리
업그레이드
스케일링

11. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Managed Workflows
for Apache Airflow (MWAA)

12. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA가 제공하는 혜택 (1)
셋업

13. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA가 제공하는 혜택 (2)
스케일링

14. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA가 제공하는 혜택 (3)
보안

15. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA가 제공하는 혜택 (4)
업그레이드

16. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA가 제공하는 혜택 (5)
유지관리

17. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA 아키텍처
Airflow Scheduler
Airflow Worker(s) Airflow Web Server
Meta Database
Private
Network
Public
Network
Service VPC
Customer VPC

18. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA 사용 방법
MWAA 환경 생성 DAGs 및 Plugin을
S3에 복사
Airflow UI에 접속

19. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA 환경 생성

20. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA 환경 생성

21. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA 환경 생성

22. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA 환경 생성

23. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA 환경 생성

24. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA 환경 생성

25. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA 환경 생성

26. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA 환경 생성

27. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA 환경 생성

28. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA 환경 생성

29. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA 환경 생성

30. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA 환경 생성

31. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA 환경 생성

32. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA 환경 생성

33. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo

34. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
데이터 파이프라인
DW workflow
Glue ETL
(데이터 수집)
ODS workflow
Athena
(테이블 생성)
Athena
(데이터 변환)
EMR Spark
(데이터 변환)
Glue Catalog
S3
Copy
RDS
S3
Redshift

35. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
데이터 파이프라인
DW workflow
Glue ETL
(데이터 수집)
ODS workflow
Athena
(테이블 생성)
Athena
(데이터 변환)
EMR Spark
(데이터 변환)
Glue Catalog
S3
Copy
RDS
S3
Redshift

36. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Apache Airflow 기본 개념
Operator
• Task 수행 방법 및 Task 수행을 위한 도구 보유
• e.g. Python Operator, Bash Operator, Postgres Operator, Email Operator
Task
• Operator가 수행하는 작업
• e.g.
- Python Operator로 일부 API를 호출하여 데이터를 수집하는 작업 구성
- MySQL Operator로 데이터베이스에 데이터를 쓰기 위한 작업 구성
DAG
• Task 스케줄링, Task 수행 순서, Task 실패 시 재시도 등을 정의하는 지침
• e.g.
- Task A가 성공적으로 완료되면 Task B를 실행
- Task 하나가 실패하면 전체 프로세스를 중지하고 알림 발송

37. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Apache Airflow 기본 개념
Connection
• 외부 시스템에 대한 연결 혹은 자격 증명 정보를 안전하게 저장하는 방법
• e.g.
- Postgres Connection = Postgres DB에 대한 연결 정보
- AWS Connection = AWS Access key
Variables
• 임의의 정보를 Key-Value 형식으로 저장하고 Task에서 사용 가능한 변수
• e.g.
- S3 bucket name
- EMR cluster ID
XComs
• Task 간의 메시지를 교환하기 위한 메커니즘

38. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
워크플로우
01.
RDS
데이터 수집
02.
ODS
테이블 생성
03.
DW 구성
04.
Redshift
데이터 로딩
05.
알림 발송

39. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
워크플로우
01.
RDS
데이터 수집
S3
Operator
Glue
Operator
AWS Connection
Variables

40. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
워크플로우
01.
RDS
데이터 수집
02.
ODS
테이블 생성
S3
Operator
Glue
Operator
AWS Connection
Variables
Athena
Operator
AWS Connection
Variables

41. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
워크플로우
01.
RDS
데이터 수집
02.
ODS
테이블 생성
03.
DW 구성
S3
Operator
Glue
Operator
AWS Connection
Variables
Athena
Operator
AWS Connection
Variables
S3
Operator
EMR
Operator
AWS Connection
Variables
Athena
Operator

42. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
워크플로우
01.
RDS
데이터 수집
02.
ODS
테이블 생성
03.
DW 구성
04.
Redshift
데이터 로딩
S3
Operator
Glue
Operator
AWS Connection
Variables
Athena
Operator
AWS Connection
Variables
Postgres
Operator
S3ToRedshift
Operator
AWS Connection
Redshift Connection
Variables
S3
Operator
EMR
Operator
AWS Connection
Variables
Athena
Operator

43. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
워크플로우
01.
RDS
데이터 수집
02.
ODS
테이블 생성
03.
DW 구성
04.
Redshift
데이터 로딩
05.
알림 발송
Email
Operator
Slack
Operator
S3
Operator
Glue
Operator
AWS Connection
Variables
Athena
Operator
AWS Connection
Variables
Postgres
Operator
S3ToRedshift
Operator
AWS Connection
Redshift Connection
Variables
S3
Operator
EMR
Operator
AWS Connection
Variables
Athena
Operator

44. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.

45. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Airflow 시스템 통합

46. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS 워크플로우 서비스
Amazon MWAA AWS Step Functions

47. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
유용한 자료
Apache Airflow Documentation
https://airflow.apache.org
AWS MWAA Documentation
https://docs.aws.amazon.com/mwaa/index.html
AWS MWAA Blog
https://aws.amazon.com/blogs/big-data/category/application-
integration/amazon-managed-workflows-for-apache-airflow-amazon-mwaa/

48. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
여러분의 소중한 피드백을 기다립니다.
강연 종료 후, 강연 평가에 참여해 주세요!
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.

49. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
감사합니다
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.