AWS Batch를 통한 손쉬운 일괄 처리 작업 관리하기 - 윤석찬 (AWS 테크에반젤리스트)

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
2018

n and ff Predictable peaksWASTE
O L S E L
( , ) , , T G

1. https://aws.amazon.com/ko/solutions/case-studies/novartis/
2. https://aws.amazon.com/blogs/aws/experiment-that-discovered-the-higgs-boson-uses-aws-to-probe-nature/
3. https://www.slideshare.net/AmazonWebServices/bdt311-megarun-behind-the-156000-core-hpc-run-on-aws-and-experience-of-
ondemand-clusters-for-manufacturing-production-workloads-aws-reinvent-2014
1 ,, N8D
6 7E ED R C2
1 6 H E
D R 7E R C2
1 0 5 3 9 D D
9 3 7 R C2

• S3/EC2/Spot Fleet
• Auto-Scaling
• SNS/SQS/CloudWatch
• S3/ECS
• SQS/CloudWatch

t 12
a eW
v Dm A
So E m
1 6357 ,
r C
B
A -
53 76 SnS
187: SnSI p
eW C
B 21 06 6:
B
B

g P m
,
(
o h
) u
d MW C
A m
s
, /
c il
s
il
( C
l l a
m S l
t E
e
e
C U v
n
L I E
2/ z )
L g P
E

21
) )Batch Queue (2)
Batch Queue (1)
Batch Queue (0)
priority
(
Container Property
Compute
Resources
DependsOn
Container Property
Container Property
( ) )
( )
( )
( )
…(
Define the
Batch Job
Create
Compute
Environment(s)
Create Job
Queue Submit Job Job Scheduling
Monitor Job and
Outputs

IAM Role for
Batch Job
Input Files
Queue of
Runnable Jobs
S3 Events Trigger
Lambda Function
Submits Batch Job
Compute
Environments (ECS)
Job Definition
Batch Execution
Application
Image (ECR)
Batch Scheduler
Batch Job
Input
Batch Job
Output
Output Files

(
• B 2 CEA
A 2 B
aws batch submit-job --cli-input-json file://submit_job.json
( )
SUBMITTED
PENDING
RUNNABLE
STARTING
RUNNING
SUCCEEDED
FAILED

()( ( )
• - C A ,
A
aws batch register-job-definition --cli-input-json file://jdef.json

) ( (
• ,
aws batch create-job-queue --cli-input-json file://job_queue.json

) ( )( (
• . "
" , O
aws batch create-compute-environment --cli-input-json file://job_env.json

https://aws.amazon.com/blogs/compute/using-aws-cloudformation-to-create-and-manage-aws-batch-resources/

$ aws batch create-compute-environment --compute-environment-name c4Spot50 --type MANAGED
--state ENABLED --compute-resources
type=SPOT,instanceTypes=c4,minvCpus=0,maxvCpus=10000,desiredvCpus=2500,bidpercentage=50,s
ubnets=subnet-220c0e0a,subnet-1a95556d,subnet-978f6dce,instanceRole=ecsInstanceRole,
securityGroupIds=secGrp01,secGrp02,secGrp03,spotIamFleetRole=arn:aws:iam::012345678910:ro
le/aws-ec2-spot-fleet-role,serviceRole=arn:aws:iam::012345678910:role/AWSBatchServiceRole
$ aws batch create-compute-environment --compute-environment-name C4-RI --type MANAGED --
state ENABLED --compute-resources
type=EC2,instanceTypes=c4.2xlarge,minvCpus=0,maxvCpus=6000,desiredvCpus=1000,subnets=subn
et-220c0e0a,instanceRole=ecsInstanceRole,
securityGroupIds=secGrp01,spotIamFleetRole=arn:aws:iam::012345678910:role/aws-ec2-spot-
fleet-role, serviceRole=arn:aws:iam::012345678910:role/AWSBatchServiceRole
.2 12
1 12

F A
{
...
"SubmitJob": {
"Type": "Task",
"Resource": ”arn…",
"Next": "GetJobStatus"
},
...
}

Annotation
Variant
Calling
QC
Alignment
• https://aws.amazon.com/ko/blogs/compute/building-high-throughput-
genomics-batch-workflows-on-aws-introduction-part-1-of-4/
• https://github.com/aws-samples/aws-batch-genomics

3 .
• M
•
• 1
• M
• C
•
Job A Job C
Job B:0
Job B:1
Job B:n
…
$ aws batch submit-job --job-name BigBatch --job-
queue ProdQueue --job-definition monte-carlo:8 --
array-properties “size=10000” ...
{
"jobName": ”BigBatch", "jobId": "350f4655-
5d61-40f0-aa0b-03ad787db329”
}
Job Name: BigBatch
Job ID: 350f4655-5d61-40f0-aa0b-03ad787db329
Job Name: BigBatch
Job ID: 350f4655-5d61-40f0-aa0b-03ad787db329:0
Job Name: BigBatch
Job Name: BigBatch
…

Job Depends on Array Job
“Job-B”
B:0
…
B:1
B:99
“Job-A”
$ aws batch submit-job –cli-input-json file://./Job-A.json
<Job-A.json>
{
"jobName": ”Job-A",
"jobQueue": "ProdQueue",
"jobDefinition": ”job-s3-list-validate:1",
}
$ aws batch submit-job –cli-input-json file://./Job-B.json
<Job-B.json>
{
"jobName": ”Job-B",
"jobQueue": "ProdQueue",
"jobDefinition": ”job-s3-CPU-intensive:1",
”containerOverride": { ”vcpus”: 32, “memory”: 4096 }
”arrayProperties": { “size”: 100 }
"dependsOn": [
{"jobId": "<job ID for Job A>" }
]
}
)3 ( ,
J
A
S B

Two Equally-Sized Array Jobs,
a.k.a. “N-to-N”
“Job-A”
A:0
…
A:1
A:2
A:3
A:9999
B:0
B:1
B:2
B:3
B:n
“Job-B”
“Job-A”
“Job-C”
C:0
…
C:1
C:2
C:3
C:9999
D:0
D:1
D:2
D:3
D:9999
“Job-D”
“Job-B”
C is dependent on A and B
D has an N_TO_N dependency on C

“Job-A”
“Job-C”
C:0
…
C:1
C:2
C:3
C:9999
D:0
D:1
D:2
D:3
D:9999
“Job-D”“Job-B”
B:0
…
B:1
B:9
B:2
“Job-E”
Heavy
Network I/O
CPU
Intensive
Large
Memory
Setup Cleanup

• E n
• 1 - E u n
• / E , 2 ,-2 n 13 -1 St
• Mm a n
• G E
• P , 2 u
• P h l UI W oi W ,
U C
• zu d
• r p 2 : A - 5 n
• c r p 5B : A 5 t F

N A D
https://aws.amazon.com/ko/batch/use-cases/

9
2 E9CG9 89
, E B 9E
0- 9 8
4-
Deletion
E 6
96
.15E
.15 G E
99
8 99
68
.15E
.15 G E
99
8 99
9
9
9
8 9
8 9
8 9
9
9
9
.15E
.15 G E
.15E
0- -3
.15 G E
9
AWS re:Invent 2017: AWS Batch: Easy and Efficient Batch Computing on AWS (CMP323) https://www.youtube.com/watch?v=8dApnlJLY54

API Server
(ECS)
SWF
Workflow
Job Manager
(ECS)
Generate Variants
Solve Variants
start
Workflow
poll
for
Decision
submit Batch Job
poll for Activity
submit Batch Job
poll for Activity
task Completed
task Completed
§ :
§ 2 ) A: :
§ ( 2 ) A: : B )

Deploy an 8K HEVC pipeline using Amazon EC2 P3 instances with AWS Batch
https://aws.amazon.com/ko/blogs/compute/deploy-an-8k-hevc-pipeline-using-amazon-ec2-p3-
instances-with-aws-batch/

1.2 MILLION JOBS
SUBMITTED
300K INSTANCES
CHURNED
2 TEAMS USE IN
PRODUCTION
• 5 D
• , 0 2 c
• 6 0 c
600 CPU YEARS
SPENT
) ( 7 2 1

비용 최적화 (Cost-optimized)
• 인프라 설정 및 관리 운영 부담 최소화
• 언제나 원하는 시간에 따라 다양한 작업 실행 가능
• 별도 SW 구매 없이도 다양한 배치 작업에 맞는 작업 구성 관리 가능
비지니스 현장에서 발생하는 다양한 요구 사항에 대해 민첩하게 대응 가능한 배치
작업 아키텍처 제공 가능
자원 최적화 (Resource-optimized)
• 예산이 한정되어 있는 경우
• 다중 업무에 따른 사내 작업 순위와 컴퓨팅 환경을 공유해야 하는 경우
• 기존 구매 자원 (Reserved Instance) 등의 활용을 못하고 있는 경우
RI
시간 최적화(Time-optimized)
• 작업에 데드라인이 있는 경우, 작업 전체 시간을 줄이고 싶을 때
• 다양한 컴퓨팅 인스턴스와 지불 방식을 조합하여 빠르게 끝낼 수 있음

• / P
( T o S - a no r
• A: . SL T w
/ A dr P s r fc
• A: / W f
He T h o S
• / / : / / fc W
b lt P u Ws g k
• m
.) / P p i
CC B :C D EB B DBC A
F G B R !
/ D
FMS P RC EF

• FFB H IA A F F
• FFB H IA A A F C
• : FFB H IA A F : FF : F DF
•
FFB A H IA A A D F F F G D:G H F F F
•
• B A AD F F B G F A - F :D F A
FFB : F G A H H F : A
• A BGF A: /A F D : AH FA F /+
FFB H IA A A: A BGF D F : BD A F
• B . D : A F
FFB H IA A A: A BGF B D : A H F

AWS Batch를 통한 손쉬운 일괄 처리 작업 관리하기 - 윤석찬 (AWS 테크에반젤리스트)

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a AWS Batch를 통한 손쉬운 일괄 처리 작업 관리하기 - 윤석찬 (AWS 테크에반젤리스트)

Similar a AWS Batch를 통한 손쉬운 일괄 처리 작업 관리하기 - 윤석찬 (AWS 테크에반젤리스트) (20)

Más de Amazon Web Services Korea

Más de Amazon Web Services Korea (20)

Último

Último (20)

AWS Batch를 통한 손쉬운 일괄 처리 작업 관리하기 - 윤석찬 (AWS 테크에반젤리스트)