Kubernetes Jobによるバッチシステムのリソース最適化 / AbemaTV DevCon 2018 TrackB Session B6
•
5 recomendaciones•4,528 vistas
AbemaTVの動画トランスコードを支えるバッチシステムが抱えていた問題と、Kubernetes Jobを使った改善策についてお話します。
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Similar a Kubernetes Jobによるバッチシステムのリソース最適化 / AbemaTV DevCon 2018 TrackB Session B6
Similar a Kubernetes Jobによるバッチシステムのリソース最適化 / AbemaTV DevCon 2018 TrackB Session B6 (20)
Más de AbemaTV, Inc.
Más de AbemaTV, Inc. (9)
Último
Último (20)
Kubernetes Jobによるバッチシステムのリソース最適化 / AbemaTV DevCon 2018 TrackB Session B6
- 2. ! "
- 5. ! !
- 6. 1080p! 720p! 480p" 240p"360p" 180p" HLS MPEG2-TS 1080p! 720p! 480p" 240p"360p" 180p" MPEG-DASH Fragmented MP4 # $ Thumbnail % DRM packaging
- 8. Worker 1 Worker 2 Worker 3 Queue ∠
- 9. Task1Task2Task3 Worker Pod1 Worker Pod2 Worker Pod3 Node (GCE VM Instance) Enqueue task ID Store task details Pull tasks Transcode Request Kubernetes Task1 Task2 program: XXX type: master status: inqueue program: YYY type: mbr status: inqueue ∠
- 10. Task1Task2Task3 Worker Pod1 Worker Pod2 Worker Pod3 Node (GCE VM Instance) Enqueue task ID Store task details Pull tasks Transcode Request Kubernetes Task1 Task2 program: XXX type: master status: inqueue program: YYY type: mbr status: inqueue ∠
- 11. Worker Pod 1 Worker Pod 2 Worker Pod 3 vCPUx5, Mem 8GB vCPUx5, Mem 8GB vCPUx5, Mem 8GB メタデータ生成 必要リソース小 240pトランスコード 必要リソース中 1080pトランスコード 必要リソース大 Kubernetes Node (vCPUx16, Mem 25GB)
- 15. kind: Pod spec: containers: - name: transcoder image: transcoder resources: requests: memory: “500Mi" cpu: "1" limits: memory: “8Gi" cpu: "5"
- 16. Kubernetes Worker Pod 1 use: vCPUx1, Mem “500Mi” Worker Pod 2 use: vCPUx5, Mem “8Gi” Node (vCPUx16, Mem 25GB) … Kubernetes Worker Pod 2 use: vCPUx1, Mem “500Mi” Worker Pod 1 use: vCPUx5, Mem “8Gi” Node (vCPUx16, Mem 25GB) …
- 17. Kubernetes Worker Pod 1 use: vCPUx8, Mem 10GB Worker Pod 2 use: vCPUx4, Mem 8GB Worker Pod 3 use: vCPUx4, Mem 8GB Worker Pod 4 use: vCPUx8, Mem 10GB Node (vCPUx16, Mem 32GB) Node Evict
- 18. Kubernetes Worker Pod 1 use: vCPUx8, Mem 10GB Worker Pod 2 use: vCPUx4, Mem 8GB Worker Pod 3 use: vCPUx4, Mem 8GB Worker Pod 4 use: vCPUx8, Mem 10GB Node (vCPUx16, Mem 32GB) Node Evict
- 20. High-perf Workers vCPUx4 Mem 8GB. Low-perf Workers vCPUx1 Mem 2GB. Kubernetes ex) metadata, 180p, 240p ex) 720p, 1080p Trancode ∠ ∠Transcode Request
- 23. MBR Queue ∠ Transcode Queue ∠ DRM Queue ∠ Metadata Generator vCPUx1 Mem 2GB. MBR Transcoder vCPUx4 Mem 8GB. DRM Packager vCPUx2 Mem 4GB. Kubernetes
- 26. Job Pod
- 27. Task1 Task2 program: XXX type: master status: complete program: YYY type: mbr status: running Create JobStore task details Kubernetes Transcode 1080p vCPUx8, Mem 10GB DRM packaging vCPUx1, Mem 1GB Transcode 240p vCPUx2, Mem 2GB Transcode 480p vCPUx4, Mem 4GB Transcode Manager
- 28. Task1 Task2 program: XXX type: master status: complete program: YYY type: mbr status: running Create JobStore task details Kubernetes Transcode 1080p vCPUx8, Mem 10GB DRM packaging vCPUx1, Mem 1GB Transcode 240p vCPUx2, Mem 2GB Transcode 480p vCPUx4, Mem 4GB Transcode Manager
- 29. ∠
- 31. ! Video Metadata " Thumbnail ! MPEG2-TS MBR ! FMP4 MBR # DRM packaging 1080p 1080p 720p 720p 480p 480p 360p 360p 240p 240p 180p 180p MPEG2-TS MBR FMP4 MBR
- 32. ! Video Metadata " Thumbnail ! MPEG2-TS MBR ! FMP4 MBR # DRM packaging 1080p 1080p 720p 720p 480p 480p 360p 360p 240p 240p 180p 180p MPEG2-TS MBR FMP4 MBR
- 33. kind: Job spec: template: spec: containers: - name: transcoder image: abema-transcoder command: [“transcoder”, …] restartPolicy: OnFailure backOffLimit: 3
- 34. ! Video Metadata " Thumbnail ! MPEG2-TS MBR ! FMP4 MBR # DRM packaging 1080p 1080p 720p 720p 480p 480p 360p 360p 240p 240p 180p 180p MPEG2-TS MBR FMP4 MBR
- 35. kind: Job spec: template: spec: containers: - name: transcoder image: abema-transcoder command: [“transcoder”, …] parallelism: 3 completions: 3
- 36. Job MBR Manager Job 1080p Job 720p Job 480p Job 360p Job 240p Job 180p
- 37. import ( “k8s.io/client-go/kubernetes" "k8s.io/client-go/pkg/watch" ) wi, err := client.BatchV1().Jobs(..).Watch(..) defer wi.Stop() for { r, _ := <- wi.ResultChan() switch r.Type { case watch.Modified: // ... case watch.Deleted: // ... case watch.Error: // … } (k8s.io/client-go/kubernetes) Clientset.BatchV1().Jobs().Watch()
- 38. https://kubernetes.io/docs/concepts/workloads/controllers/ jobs-run-to-completion/#job-patterns A single Job to create a pod which then creates other pods, acting as a sort of custom controller for those pods. This allows the most flexibility, but may be somewhat complicated to get started with and offers less integration with Kubernetes. $ %
- 39. https://kubernetes.io/docs/concepts/workloads/controllers/ jobs-run-to-completion/#job-patterns A single Job to create a pod which then creates other pods, acting as a sort of custom controller for those pods. This allows the most flexibility, but may be somewhat complicated to get started with and offers less integration with Kubernetes. $ %
- 43. THANK YOU