Eric Durand once again takes us to a journey of Storage solutions for digital media, using the AWS Cloud.
This presentation was delivered at AWS Toronto, during the Media and Entertainment Symposium.
5. A Concept - the Content Lake
Inspired from Data Lake (Coined by James Dixon in 2010)
A single store of all of digital content that you create and
acquire in any form or factor
•Don’t assume any resolutions/formats (for now or future)
•It is up to the consumer (application consuming the content) to use the
appropriate infrastructure for processing
6. Amazon S3 : the Content Lake
• Durable, cost-effective and fast
• Highly scalable front-end
– Multi-part uploads (parallel writes)
– Range-gets (parallel reads)
• No need for capacity planning or
provisioning
• Use Amazon S3 with on-premises
storage in a hybrid model
• Secure
8. 1 PB raw storage
800 TB usable storage
600 TB allocated storage
400 TB application data
S3 capacity pricing—pay only for what you use
AWS Cloud
Storage
9. AWS Storage for Active Archive
S3 Standard
Active data Archive dataInfrequently accessed data
S3 Standard - Infrequent
Access
Amazon Glacier
Milliseconds 3-5 hoursMilliseconds
$0.03/GB/mo $0.007/GB/mo$0.0125/GB/mo
10. Traditional archiving approaches
• Storage arrays / disk arrays
• Tape silos / tape libraries
• Tape drives (LTO-X / DLT / etc.)
• Virtual tape libraries (VTLs)
• Tape out / vaulting
• Specialized software and
personnel
11. How can AWS help with your archival?
Metered usage:
Pay as you go
No capital investment
No commitment
No risky capacity planning
Avoid risks of physical
media handling
Control your
geographic locality for
performance and
compliance
12. A Closer Look: S3-IA and Amazon Glacier
S3 - IA
• Same durability and throughput as S3 Standard
• Instant access
• $0.01/GB on each data retrieval
Amazon Glacier
• Same 11 9s durability as S3 Standard
• 3-5 hour data retrieval latency
• Suitable for cold archive such as offsite tapes
S3 Standard - Infrequent
Access
Amazon Glacier
13. A Closer Look: S3-IA and Amazon Glacier Durability
4 9s durability
5 9s durability
S3 - IA Glacier
11 9s durability
14. - Transition Standard to Standard-IA
- Transition Standard-IA to Amazon Glacier
- Expiration lifecycle policy
- Versioning support
Data lifecycle management
T T+3 days T+5 days T+ 15 days T + 25 days T + 30 days T + 60 days T + 90 days T + 150 days T + 250 days T + 365 days
Data access frequency over time
15. Securing your data on AWS
• AWS alignment with the latest MPAA cloud
based application guidelines for content security
– August 2015
• VPC private endpoint for Amazon S3 – enables
a true private workflow capability
• Encryption & key management capabilities
• Amazon Glacier Vault for high-value
media/originals
17. Amazon EFS
File
Amazon EBS
Amazon EC2
Instance Store
Block
Amazon S3 Amazon Glacier
Object
Data Transfer
AWS Direct
Connect
ISV
Connectors
Amazon
Kinesis
Firehose
Storage
Gateway
S3 Transfer
Acceleration
AWS
Snowball
Amazon
CloudFront
Internet/VPN
AWS Storage is a Platform
18. AWS Snowball
Petabyte-scale data transport
E-ink shipping
label
Ruggedized case
“8.5G impact”
All data encrypted
end-to-end
Rain- and dust-
resistant
Tamper-resistant
case and
electronics
80 TB
10 GE network
19. Customer: Scripps Networks Interactive
Problem Statement:
• Need storage platform to manage active archive content
• Existing content repository too large to migrate via available
network based ingest methods
Use of AWS:
• S3 and Snowball for massively scalable ingest
• S3 for storage, Glacier for content archive
• Snowball to securely transport existing media content from on prem
storage and tape vault
Business Benefits:
• Petabyte-scale data transport without increased network costs
• Massive scalability and elasticity
• Lower TCO for active archive storage
Active archive transport and archival for digital content provider
20. AWS Snowball Pricing
Dimension Price
Usage Charge per Job $250.00 (80 TB)
Extra Day Charge (First 10 days* are free) $15.00
Data Transfer In $0.00/GB
Data Transfer Out $0.03/GB
Shipping** Varies
Amazon S3 Charges Standard storage and request
fees apply
* Starts one day after the appliance is delivered to you. The first day the appliance is received at your site and the last day the appliance is shipped out are also free
and not included in the 10-day free usage time.
** Shipping charges are based on your shipment destination and the shipping option (e.g., overnight, 2-day) you choose.
21. Amazon S3 Transfer Acceleration
Optimized Global Data Transfer
S3 Bucket
AWS Edge
Location
Uploader
Optimized
Throughput!
Typically 50%–400% faster
Change your endpoint, not your code
55 global edge locations
No firewall exceptions
No client software required
22. Customers: Frame.io, Hudl, Viocorp
Problem Statement:
• Needed to accelerate customer content ingest into their respective
applications running on AWS
• Existing ingest options were proprietary and too expensive
Use of AWS:
• S3 and S3 Transfer Acceleration for massively scalable ingest
• S3 for storage, Cloudfront and S3 Transfer Acceleration for ingest
Business Benefits:
• Global highly distributed data transport available on demand
• Massive scalability and elasticity
• Lower TCO for storage and data transport infrastructure
Accelerating media content uploads to their platforms
S3 Bucket
AWS Edge
Location
Uploader
Optimized
Throughput!
23. S3 Transfer Acceleration Pricing*
Dimension Price / GB
Data Transfer In from Internet** $0.04 (Edge location in US, EU, JP)
$0.08 (Edge location in rest of the world)
Data Transfer Out to Internet $0.04
Data Transfer Out to Another AWS Region $0.04
Amazon S3 Charges Standard data transfer charges apply
*Plus standard Amazon S3 data transfer charges apply
**Accelerated performance or there is no bandwidth charge
24. Media Workloads (redefined)
EBS
Instance
Store
Amazon EBS/EFS/EC2 Instance Store
Process
Partner/Affiliate/
Service Provider
User Delivery/ConsumptionVFX/Production
On-Prem Apps
Archive
Amazon Glacier (Life Cycle Policies)
c
c
Direct Connect
Content Access Transfer
Disposable Infrastructure
Auto-scaling
Workload specific
Amazon S3
EFS
25. Q&A
Learn more at: http://aws.amazon.com/s3/
http://aws.amazon.com/glacier/
http://aws.amazon.com/importexport/
eddurand@amazon.com