The document discusses rendering visual effects and animation at scale using Amazon Web Services (AWS). It provides examples of companies using AWS for rendering including theme parks, gaming, and manufacturing. It then discusses the workflow components of VFX/animation rendering and challenges of on-premise rendering capacity. The document outlines how AWS provides scalability and faster outputs through rendering in the cloud. It discusses licensing models for cloud rendering, shared file systems, managing cloud infrastructure, and benchmarks showing improved performance of cloud rendering over on-premise. Overall, the document examines the state of cloud rendering on AWS and trends toward more workloads moving fully to the cloud.

1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Usman Shakeel – Amazon Web Services
Cloud Rendering at scale
on AWS

2. Visual Effects and
Animation
1
Who is using AWS for Rendering?
3 Theme Parks
5 Gaming
Marketing2
4 Manufacturing
6 Life Sciences
7 Engineering and Architecture

3. Visual Effects and
Animation
1

4. VFX/Animation Rendering - workflow components
CompostingModeling Rendering
Asset Management
Collaboration and Task Management

5. The challenge of making a film

6. The challenge of making a film
On-premise capacity

7. The challenge of making a film
On-premise capacity
Rendering in the Cloud

8. The challenge of making a film
On-premise capacity
Rendering in the Cloud
Cloud provides you the capability to
scale fast and get the outputs faster
Initial project on-boarding
artwork

9. Classifying Studios By Consumption

10. They all ask us the same thing …
The ability to spin up thousands of cores on-demand
… without any upfront investment
… and leverage the most up-to-date configurations
A project based “disposable” infrastructure
… with a flexible licensing / utility / by the hour

11. They all tell us the same thing …
=< $0.01
per core/hour
Access to thousands of
cores whenever needed
No upfront investments in
infrastructure
Easier collaboration
Ecosystem of software
providers
Access to large memory
configs to do 6K/10K renders
Project based “disposable”
infrastructure

12. … when rubber meets the road !
Share FS Everywhere Latency Large Datasets Lots of instances
{Data/Content}

13. Rendering in the Cloud

14. Rendering in the Cloud - State of the Union
Scale at a very cheap price
EC2 Spot

15. Leveraging Spot successfully today requires some
effort
Build stateless, distributed, scalable applications
Choose which instance types fit your workload the best
Ingest price feed data for AZs and regions
Make run time decisions on which Spot pools to launch in based on
price and volatility
Manage interruptions
Monitor and manage market prices across AZs and instance types
Manage the capacity footprint in the fleet
And all of this while you don’t know where the capacity is
Serve your customers

16. Spot Fleet
Instead of writing all that code to manage Spot Instances,
simply specify:
• Target Capacity – The number of EC2 instances that you want
in your fleet.
• Maximum Bid Price – The maximum bid price that you are
willing to pay.
• Launch Specifications – # of and types of instances, AMI id,
VPC, subnets or AZs, etc.
• IAM Fleet Role – The name of an IAM role. It must allow EC2 to
terminate instances on your behalf.

17. Spot Fleet Example – Instance Weighting
Say your workload needs at least 60 GB of memory
Want capacity to complete 20 units of work
Choices:
• r3.2xlarge (61.0 GB, 8 vCPUs) = 1 unit of 20
• r3.4xlarge (122.0 GB, 16 vCPUs) = 2 units of 20
• r3.8xlarge (244.0 GB, 32 vCPUs) = 4 units of 20
An option to bid for all of these instance types:

18. AWS Cloud Scale is “Large”
•10s/100s/1000s/10000s cores on-demand in the Cloud
•A “Large” renderfarm (Actual example):
55,000 cores
•At re:Invent 2015:
~40,000 cores on
EC2 Spot Market
Rendering in the Cloud - State of the Union
Scale at a very cheap price

19. • BYOL
Vray, Solid Angle, Autodesk Maya 3DsMax
• SaaS
Thinkbox Deadline Cloud Wizard
• AWS Marketplace
• Elastic Licensing models
Thinkbox Deadline Usage Based Licensing
• Render nodes pull metered licenses from Cloud-based license server
• Usage is tracked per minute
• Bulk minutes will be available via Thinkbox’s online store
• Store will eventually host 3rd party licensing (Nuke, VRay, etc)
Pixar Renderman/Tractor (demo at SIGGRAPH)
Rendering in the Cloud - State of the Union
Licensing at Cloud Scale

20. Rendering in the Cloud - State of the Union
Hydrating the Cloud Renderfarm
S3 as the source of truth for your content/data
• On AWS Marketplace/SaaS
(Aspera, Signiant, File Catalyst, Expedat)
• S3 Multi-part Upload
• AWS import/export Snowball
Direct to Shared File Systems
• EFS throughput scales linearly to the storage
• Lustre can hydrate from an S3 bucket
• Avere can be fronted to S3 or an on-premise NAS
+ Direct Connect
EFS
S3
Multipart
AWS Snowball

21. Rendering in the Cloud - State of the Union
Shared FileSystem Everywhere (some ideas)
Shared Storage
On-prem Storage
AWS Direct Connect
Storage Cache
Amazon S3
Luster on EC2
Avere on EC2
EFS
AWS Direct Connect
Hydrate workers
EC2 Spot
Shared Storage
FXT on-prem

22. Rendering in the Cloud - State of the Union
Shared FS (Content/Data Share) Everywhere (some ideas)
Elastic File System (Amazon EFS)
• Designed to support Peta-Byte scale file systems
• Throughput scales linearly to storage
• Same latency spec across each AZ
• Thousands of concurrent NFS connections
• Works great for Large I/O sizes
• Pay for only what you use not what you provision
• Managed with multi-copy durability Amazon EFS

23. Rendering in the Cloud - State of the Union
Move the Graphic Artist to the Cloud …
Rendering is going Global
• NVIDIA GPU based EC2 instances
• Nice DCV
• Teradici PCoIP
• Frame, Otoy
• Windows and Linux (VNC+VirtualGL)
3D Modeler
Modeling Dumb Client
Remote Application
running on a G2 instance
G2

24. Rendering in the Cloud - State of the Union
Managing your “disposable” infrastructure
Launch a CloudFormation stack
with all the infrastructure
resources for a specific project
Autoscale the stack as
appropriate
AMI
CloudFormation
Template
CloudFormation
Terminate
Template

25. Rendering in the Cloud - State of the Union
The Crown Jewels
• AWS alignment with the latest MPAA cloud based application
guidelines for content security – August 2015
• VPC private endpoint for S3 – enables a true private workflow
capability
• Encryption & key management capabilities
• Glacier Vault for high-value media/originals

26. Rendering in the Cloud - A Sample Architecture
(All in Cloud Pipeline)
Shared Storage
Renderfarm
On-Prem Storage
Pipeline and License Manager
3D Modeler
Remote
App Visualization
AWS Direct Connect
Modeling Dumb Client
Storage Cache
Amazon S3
Avere on EC2
Scalable Renderfarm on EC2
Appstream or Teradici running on a G2 instance
Pipeline Manager running on EC2
G2
EC2 SPOT
EFS
Hydrate workers
EC2 Spot

27. Render Farm
Rendering in the Cloud - A Sample Architecture
(A Hybrid Pipeline)
Shared Storage
Renderfarm
On-Prem Storage
AWS Direct Connect
Storage Cache
Amazon S3
Avere on EC2
Scalable Renderfarm on EC2
EFS
Hydrate workers
EC2 Spot
On-premise
Renderfarm
Cloud renderfarm as an
extension of on-prem renderfarm
FXT on-prem
Pipeline and License
Manager (also manage
cloud renderfarm)

28. A Customer Example
A large scale test:
• Deploying ~40K cores
• In 20 min
• Leveraging Spot Fleet
• Hybrid rendering environment
using Avere (on-premise Data)
• EFS
Faster than on-premise
environment

29. Disney Animation Renderfarm
Renderfarm
Avere FXT
cluster
WDAS Data Center
Renderfarm
Avere FXT
cluster
Storage
Remote Data Center
Renderfarm
Avere FXT
cluster
Remote Data Center
San Francisco
Los Angeles
Burbank
Artists
Redundant 10Gb

30. Disney Animation Renderfarm
(on the Cloud)
Renderfarm
Avere FXT
cluster
WDAS Data Center
Renderfarm
Avere FXT
cluster
Storage
Remote Data Center
Renderfarm
Avere FXT
cluster
Remote Data Center
San Francisco
Los Angeles
Burbank
Artists
Redundant 10Gb
virtual private cloud
Avere vFXT
Oregon
Spot InstancesEFS

31. Spot Fleet Deployment

32. Another Customer Example

33. Cloud Rendering Benchmarks

34. Benchmarks: On Premise vs. The Cloud
0
20
40
60
80
100
120
stream triad disk read disk write
On Prem
r3.4xlarge
r3.8xlarge
m4.4xlarge
m4.10xlarge
cr1.8xlarge
Higher is better

35. Average Open Latency

36. Average Read Latency
0
100
200
300
400
500
600
700
100 500 800 1200 2400 4000
Time(µs)
Render Processes
Mid-TierA
Mid-TierB
Mid-TierC
Archive
EFS

37. Rendering in the Cloud vs. On-Premise
-
5,000
10,000
15,000
20,000
25,000
30,000
1 10 20 30 40 50 60 70 80 90
RenderTime(s)
Frame #
EC2/EFS
On Prem
Lower is better

38. Lessons Learned
• Use as many different instance types as you can. Especially older generations.
• Think about ways to modify your workload
• Use every availability-zone
• Check your limits, especially your EBS limit and VPC setup (address space)
• Resource-Oriented Bidding
• Diversified Allocation
• Benchmark your workload and set pricing accordingly
• Set ONLY realistic pricing that you will pay for
• Don’t be afraid to ask for help or pre-planning your run from AWS

39. Conclusion
Cloud Rendering on AWS
State of the Union is getting stronger …
Rendering Forecast
Partly cloudy with a chance of all in the cloud…
Technology Trends
•Containers for Rendering
•GPU based Rendering
•More vendors investing in Elastic Licensing models due to the push from their
customers
•All in-Cloud rendering with more robust application streaming solutions

40. Thank you!