Media Processing in the Cloud
•
1 recomendación•1,104 vistas
Talk from AWS Media & Entertainment Summit (10/7/2011). Also at http://aws.amazon.com/mediaevent/
Recomendados
Recomendados
Más contenido relacionado
Más de Deepak Singh
Más de Deepak Singh (20)
Último
Último (20)
Media Processing in the Cloud
- 1. Media Processing in the Cloud Deepak Singh P r i n c i p a l P r o d u c t M a n a g e r
- 3. media today
- 6. Image: Tonz
- 7. everything in between
- 8. streaming
- 9. devices
- 11. Image: NASA
- 13. 3D
- 14. so what?
- 15. lots and lots and lots and lots and lots and lots and lots and lots and lots of data
- 16. lots of compute
- 17. how can the cloud help?
- 18. let us orchestrate a processing application
- 19. I want to process data, for example encode movies, … Job Queue EC2 Instance Data Processing
- 20. … and store the results in S3. Job Queue EC2 Instance S3 Data Processing
- 21. And I want to be no@fied on comple@on Job Queue EC2 Instance Result e-mail S3 Data Processing
- 22. I like using SNS, because … Job Queue EC2 Instance Topic Result e-mail S3 Data Processing
- 23. I like using SNS, because … I can integrate other systems via SQS, or HTTP(S) web-‐hooks Job Queue EC2 Instance Topic Result e-mail S3 Data Processing SQS HTTP
- 24. I should be done ... right? Job Queue EC2 Instance Topic Result e-mail S3 Data Processing
- 25. Not quite – I’m also cost conscious … Don’t want to pay anything when there’s no work Autoscaling Job Queue Charge empty $0 Topic Result e-mail S3 Data Processing
- 26. Need a controller … Capacity Control (on-demand launch) Autoscaling Job Queue Charge $0 Topic Result e-mail S3 Data Processing
- 27. Measure … Detect work … Capacity Control (on-demand launch) CW Alarm CW Metric #msgs Autoscaling Job Queue Charge $0 Topic Result e-mail S3 Data Processing
- 28. When there’s a message queued … Capacity Control (on-demand launch) CW Alarm Autoscaling Policy add instance CW Metric if #msg > 0 #msgs Autoscaling Job Queue Charge $0 Topic Result e-mail S3 Data Processing
- 29. When there’s a message queued … start an EC2 instance Capacity Control (on-demand launch) CW Alarm Autoscaling Policy add instance CW Metric if #msg > 0 #msgs Autoscaling Job Queue EC2 Instance Topic Result e-mail S3 Data Processing
- 30. ... or 5000 EC2 instances Capacity Control (on-demand launch) CW Alarm Autoscaling Policy keep adding instances CW Metric if #msg > 0 #msgs Autoscaling Job Queue 5000 EC2 Instances Topic Result e-mail S3 Data Processing
- 31. When the work is done … Capacity Control (on-demand launch) CW Alarm Autoscaling Policy add instance CW Metric if #msg > 0 #msgs Autoscaling Job Queue EC2 Instance Topic Result e-mail S3 Data Processing
- 32. When the work is done … Terminate the EC2 instance Capacity Control (on-demand launch) CW Alarm Autoscaling Policy add instance CW Metric if #msg > 0 #msgs Autoscaling Job Queue TERM’D Topic Result e-mail S3 Data Processing
- 33. For full produc@on use, add IAM Users, Permissions, Policies, OperaFonal Alarm, etc. Capacity Control (on-demand launch) CW Alarm Autoscaling Policy add instance CW Metric if #msg > 0 #msgs Autoscaling Job Queue EC2 Instance Topic Result e-mail S3 IAM User IAM User IAM IAM Permissions IAM Permissions IAM Permissions IAM Policy Policy Data Processing
- 34. On Demand SQS Worker Example Capacity Control (on-demand launch) CW Alarm Autoscaling Policy add instance CW Metric if #msg > 0 #msgs Autoscaling Job Queue EC2 Instance Topic Result e-mail S3 Data Processing No glue-‐scrip.ng (or UI code) required … We let you focus on the business problem!
- 35. Nested templates: enable custom abstrac@ons Job Queue S3 Job Queue S3 Job Queue S3 Job Queue S3
- 36. One substack per e.g. Video format to encode Parent template/stack Parameterized by e.g. format/resolu1on Job Queue Encode for low-res, … S3 S3 Job Queue Encode for Job: high-res, … S3 S3 Render X Topic Job Queue S3 S3 Job Queue S3 S3
- 37. 4
- 42. on demand global secure
- 43. programmable
- 47. elastic
- 49. instance types
- 50. t1.micro standard (m1) high memory (m2) high CPU (c1)
- 51. high performance
- 53. MPI
- 55. Cluster Compute Instance
- 56. 2*Intel Xeon 5570 23 GB RAM 1.7 TB disk HVM
- 57. 10 gig E
- 58. Placement Group
- 59. ull- on f i ect bis Placement group
- 60. linpack
- 61. Cores 7040 R max 41.82 R peak 82.51
- 62. 231
- 63. 451
- 64. WIEN2K Parallel Performance H size 56,000 (25GB) Runtime (16x8 processors) Local (Infiniband) 3h:48 Cloud (10Gbps) 1h:30 ($40) 1200 atom unit cell; SCALAPACK+MPI diagonalization, matrix size 50k-100k Credit: K. Jorissen, F. D. Villa, and J. J. Rehr (U. Washington)
- 65. HPC is evolving
- 67. 2*Intel Xeon 5570 22 GB RAM 1.7 TB disk HVM 2*NVidia M2050
- 70. optimizing costs
- 71. on-demand
- 72. reserved
- 73. spot
- 76. 2. Orchestration
- 79. bootstrap
- 80. Cloud Init
- 81. chef/puppet
- 82. familiar tools
- 84. LSF
- 85. Condor
- 86. combining worlds
- 87. MIT Starcluster
- 88. $ starcluster start mycluster $ starcluster listclusters
- 93. 30,472 cores
- 94. $1279/hr
- 96. Big Data
- 98. S3 Input data
- 99. S3 Input data Code Elastic MapReduce
- 100. S3 Input data Code Elastic Name MapReduce node
- 101. S3 Input data Code Elastic Name MapReduce node Elastic cluster
- 102. S3 Input data Code Elastic Name MapReduce node HDFS Elastic cluster
- 103. S3 Input data Code Elastic Name MapReduce node Queries HDFS + BI Via JDBC, Pig, Hive Elastic cluster
- 104. S3 Input data Code Elastic Name Output MapReduce node S3 + SimpleDB Queries HDFS + BI Via JDBC, Pig, Hive Elastic cluster
- 105. 3. Applications
- 107. NASA JPL
- 110. Netflix needed to transcode 17,000 titles (80TB of data) to support the launch of Sony PS3. They provisioned 1200 Amazon EC2 instances and completed the transcoding process in just days. Source: Adrian Cockroft (Netflix)
- 111. Source: Adrian Cockroft (Netflix)
- 118. “Our tests have shown more than 90 percent scaling efficiency on clusters of up to 128 GPUs each”
- 119. 4. People
- 121. CPU, storage, Hardware memory Collections, datasets, Data management provenance parallelization, Software optimization Backup, redundant, Availability replicated Cost Small
- 122. where should we optimize?
- 124. removing constraints
- 125. undifferentiated heavy lifting
- 126. focus on value
- 127. faster
- 128. more
- 130. creating art
- 132. 4
- 133. 1. Infrastructure
- 134. 2. Orchestration
- 135. 3. Applications
- 136. 4. People
- 137. creating content
- 138. deesingh@amazon.com Twitter:@mndoci http://slideshare.net/mndoci http://mndoci.github.com Inspiration and ideas from Matt Wood & Larry Lessig Credit” Oberazzi under a CC-BY-NC-SA license