에이슬립은 AI 모델로 수면을 정확하게 분석하여 맞춤형 수면 서비스를 제공하는 SaaS 서비스를 제공합니다. 본 세션에서는 SaaS 서비스 인증 및 권한제어, 사용량 측정 등 필수적인 기능을 AWS 기반 서버리스 서비스를 통해 제공하여 데이터베이스 부하 없이 대규모 트래픽에도 견딜 수 있는 아키텍처를 소개합니다. 매드업은 디지털 마케팅 자동화 솔루션을 개발 및 운영하면서 대용량 데이터를 수집, 가공해서 데이터웨어하우스를 만드는 도전이 있습니다. 본 세션에서는 기존의 Amazon Athena에서 어떻게 Amazon Redshift Serverless를 통해 데이터가 필요한 조직에 정확하게 공급했는지 그 사례와 장단점을 소개합니다.

1. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SEOUL | MAY 4, 2023

2. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SaaS
Backend Developer

3. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
/ Liam Park
System Softerware Engineer
Backend Engineer
AWS infra, Node JS And Python Developer
TmaxTibero Software Engineer
Carillon Backend Engineer
Asleep, Senior Backend Engineer

4. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
1. Sleeptrack API
2. API
3. Sleeptrack API - / /
4. Sleeptrack API –
5.

5. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
1. Sleeptrack API

6. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Sleeptrack API

7. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Sleeptrack API

8. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
2. API

9. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Saas API
01
AUTOMATION
API
02
MINIMAL
DEPENDENCY
Live
.
03
ACCURACY
API
.
04
SCALABILITY
.

10. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
client ECS
server log
RDS
cloud
watch
API
Gateway
update per
request
ELB
1. ECS API
2. API DB
3.
DB ,

11. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
3. Sleeptrack API
/ /

12. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Athena
• SQL S3
• csv, json, parquet
•
•
RDS
DocumentDB
DynamoDB
S3
Glue Athena

13. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Athena
1. SERVERLESS
2. Athena
3. ( / / )
Spark/Flink Presto/Hive Athena RedShift
운영 효율성 Low Low High Medium
Learning Curve High Medium Low Low
운영 비용 High Medium Low Medium

14. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
API
Firehose
API Gateway
API Firehose
01 02 03 04
Access log Parquet Data Athena
Parquet Data Athena
Firehose S3
JSON
JSON Access Log
Parquet Data

15. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
API
Client API
Gateway
ECS
S3
access log
Firehose
RDS Athena
Glue Schema
ELB
(1) / /

16. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Firehose
API
Gateway
Firehose

17. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Access Log
Firehose
S3

18. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Access Log
Firehose
S3

19. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Parquet Data
•
• Athena S3
•
23MB
JSON Data
$25
JSON Data
500KB
Parquet Data
$5
Parquet Data
Before After

20. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Athena
: 2023 5 4 2
Create
Parquet
access log Glue Schema
Partitioned
by Datetime
parquet Schema For
Parquet
Athena
CTAS Query
Add Next Partition

21. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DB
• Athena Jenkins DB
Parquet Athena
insert
hourly
Client API
Gateway Jenkins
aggregate
query
ECS RDS
ELB

22. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
4. Sleeptrack API

23. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis SQL Application
• SQL
•
Kinesis Stream
Kinesis
SQL Application
Firehose
S3

24. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SQL Application

25. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
API
Firehose
API Gateway API
Firehose
01 02 03
SQL Application Lambda
SQL Application
lambda
Firehose SQL
Application

26. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
API –
Client API
Gateway
Firehose
SQL
Application
Lambda RDS
update per
minute

27. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SQL Application
Firehose
SQL
Application

28. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SQL Application
SQL Application
Lambda
RDS

29. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SQL Application
SQL Application
Lambda
RDS

30. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
5.

31. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DB
API query string, header
access log API access log
SQL Application

32. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DW
Jaeyeon Baek
Tech Lead
MADUP

33. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
❏ Adtech(Advertising technology)
❏ Tech Lead at Data Platform Team
❏ Cloud GDE
❏ HashiCorp Ambassador 2023
❏ : jybaek.tistory.com

34. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
❏
❏ / /
❏ Datashare
❏ Redshift Serverless
❏ Wrap up
Agenda

35. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
❏
❏ / /
❏ Datashare
❏ Redshift Serverless
❏ Wrap up
Agenda

36. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
,
,

37. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
05.06
install : 2,522 install : 2,599
install : 2,871
05.01
install : 2,501
06.06
06.06
. install 04 20
05.02
install : 2,872

38. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DW
manifest
delete
copy
commit
download
•data freshness
•transaction size
•distkey, sortkey

39. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
❏
❏ / /
❏ Datashare
❏ Redshift Serverless
❏ Wrap up
Agenda

40. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
,
/ /
EKS , Pod
.
MWAA
Redshift .
ECS Fargate
JupyterHub
( ) .

41. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
/ /
•
• Amazon S3 ,
Redshift COPY bz2
▪ raw data bucket:
▪ compressed data bucket : Redshift COPY
○
• MWAA DAG manifest
Amazon Redshift Delete & COPY
Internal
Micro
Services
Amazon S3
Amazon Simple
Queue Service
Amazon EKS
Amazon MWAA
Amazon Athena Amazon Redshift

42. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
/ /
average
30%
• Average 30%, Maximum 90%
• Counting only COPY queries.
Redshift dc2.large

43. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
/ /
average
100%
Normal Query
• DM Load
•

44. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
/ /
Redshift
ETL Data Scientist
Data Mart
ETC

45. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
/ /
dc2.large to RA3
AQUA(Advanced Query
Accelerator) 10
Amazon Managed Storage
. ra3.xlplus 32TB
Datashare
Amazon Redshift

46. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
❏
❏ / /
❏ Datashare
❏ Redshift Serverless
❏ Wrap up
Agenda

47. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Datashare
Datashare
AWS AWS
.
,
Amazon Redshift

48. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Datashare
Datashare …
•inbound shares / outbound shares
▪
•producer cluster / consumer cluster
▪
•same AWS account or different AWS accounts
▪ .
•Encryption
▪ producer, consumers
▪ .

49. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Datashare
Datashare
Amazon Redshift Managed Storage
Producer Cluster
(write data)
consumer cluster
(read shared data)
consumer cluster
(read shared data)
Amazon Redshift Amazon Redshift Amazon Redshift

50. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
❏
❏ / /
❏ Datashare
❏ Redshift Serverless
❏ Wrap up
Agenda

51. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Redshift Serverless
DW , Amazon Redshift Serverless
(2022-07 GA, )
(RPUs) ,
Amazon Redshift Serverless

52. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Redshift Serverless
Datashare Redshift Serverless
Amazon Redshift Managed Storage
Producer Cluster
(write data)
consumer cluster
(read shared data)
consumer cluster
(read shared data)
Amazon Redshift Amazon Redshift Serverless Amazon Redshift Serverless

53. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Redshift Serverless
Datashare Redshift Serverless
Amazon Redshift Managed Storage
Producer Cluster
(write data)
consumer cluster
(read shared data)
consumer cluster
(read shared data)
Amazon Redshift
Amazon Redshift Serverless Amazon Redshift Serverless

54. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
1 transaction
Redshift Serverless
-
File (1) Table (1)
x10
DELETE & COPY
File (2) Table (2)
DELETE & COPY
File
(32)
Table (32)
…
…
DELETE & COPY

55. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Redshift Serverless
타입 노드 스펙 월 비용 Delete & Insert 평균 시간
dc2.large x 8
- vCPU : 2
- Memory : 15 GIB
- Storage : 0.16 TB
1,728 USD 224s
ra3.xlplus x 2
- vCPU : 4
- Memory : 32 GIB
- Storage : 32 TB
1,840.32 USD 201s
ra3.xlplus x 3
- vCPU : 4
- Memory : 32 GIB
- Storage : 32 TB 2,760 USD
145s
Serverless x 32 RPU - RPU 시간 당 0.438 USD
매일 6시간 30일 사용
2,522.88 USD
102s
Serverless x 64 RPU - RPU 시간 당 0.438 USD
매일 6시간 30일 사용
5,045.76 USD
95s
-
32 (80MB) x 10

56. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Redshift Serverless
DW
Amazon MWAA Amazon
Redshift
Redshift Serverless
Redshift Serverless
RPUs
Data Mart
Data Scientist

57. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
❏
❏ / /
❏ Datashare
❏ Redshift Serverless
❏ Wrap up
Agenda

58. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Wrap up
•
▪ : DB scale-up.
▪
•Redshift Serveless RPUs
▪
▪ / /
• .
▪
▪
▪

59. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Wrap up
Redshift
•
▪ Athena
•Athena
▪ ( Redshift )
▪ (DAG)
▪ ETL
•Athena
▪ Hive , , , and so on..

60. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Wrap up
jybaek@madup.com :-)
?

61. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
감사합니다
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.