When evaluating and planning migrating your data from on premises to the Cloud, you might encounter physical limitations. Amazon offers a suite of tools to help you surmount these limitations by moving data using networks, roads, and technology partners. In this session, we discuss how to move large amounts of data into and out of the Cloud in batches, increments, and streams.

1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Marc Trimuschat
AWS Storage Services
November 2016
AWS Data Transfer Services
Data Ingest Strategies into the AWS Cloud
ENT210

2. Storage is the gravity
for cloud applications

3. Batches and Streams
Direct
Connect
Snowball,
Snowball Edge,
Snowmobile
3rd Party
Connectors
Transfer
Acceleration
Storage
Gateway
Kinesis Firehose
File
Amazon EFS
Block
Amazon EBS
(persistent)
Object
Amazon GlacierAmazon S3 Amazon EC2
Instance Store
(ephemeral)
Internet/VPN CloudFront

4. Internet/VPN ingest

5. What is Internet/VPN?
Globally available
Default method of ingesting content into Amazon S3
Simple standards-based (HTTP) connection
Use your existing internet connection
Available in a VPC for VPN connectivity
Acceleration through multipart upload
Data transfer into AWS is free
VPN connections using VPC virtual private gateway
•$0.05 per VPN connection-hour
•$0.048 per VPN connection-hour for connections to the Tokyo region

6. How does Internet/VPN ingest work?
Accelerate data transfer using
multipart upload
Ingest data directly into S3 buckets
with existing internet connectivity
S3 bucket
AWS Region
and
through the console or API
customer
gateway
endpoints
VPN
connection
Internet Internet through VPN +
VPC

7. Amazon S3 Transfer Acceleration

8. What is Transfer Acceleration?
Network- and protocol-based data transfer service
Acceleration of data ingress/egress with S3 buckets
Typically 50% to 300% faster
Feature of S3 enabled at the bucket level
Available in all S3 regions worldwide
No client/server software required
No code changes to your application
No firewall exceptions
Simple pricing model

9. Ingest & egress with Transfer Acceleration
S3 bucket
AWS edge
location
Uploader
Optimized
throughput!
Uses AWS 59 global edge locations
AWS determines best edge location
Data transfer optimized between
edge and customer, and edge and S3
Data is not stored on the edge cache

10. Customers: Frame.io, Hudl, Viocorp
Problem Statement:
• Needed to accelerate customer content ingest into their respective
applications running on AWS
• Existing ingest options were proprietary and too expensive
Use of AWS:
• S3 and S3 transfer acceleration for massively scalable ingest
• S3 for storage, CloudFront and S3 transfer acceleration for ingest
Business Benefits:
• Global highly distributed data transport available on demand
• Massive scalability and elasticity
• Lower TCO for storage and data transport infrastructure
Accelerating media content uploads to their platforms
S3 Bucket
AWS Edge
Location
Uploader
Optimized
Throughput!

11. Amazon
Route 53
Resolve
b1.s3-accelerate.amazonaws.com
HTTPS PUT/POST
upload_files.zip
HTTP/S PUT/POST
“upload_files.zip”
Service traffic flow
Client to S3 bucket example
S3 bucket
b1.s3-accelerate.amazonaws.com
EC2 proxy
AWS region
AWS edge location
Customer client
1
2
3
4
Data is not cached on the
AWS edge location
Fully managed file transfer acceleration
using all AWS edge locations

12. Rio De
Janeiro
Warsaw New York Atlanta Madrid Virginia Melbourne Paris Los
Angeles
Seattle Tokyo Singapore
Time[hrs]
500 GB upload from these edge locations to a bucket in Singapore
Public internet
How fast is S3 Transfer Acceleration?
S3 transfer acceleration

13. AWS Direct Connect

14. What is AWS Direct Connect?
Dedicated, 1 or 10 GE private pipes into AWS
Create private (VPC) or public virtual interfaces to AWS
Reduced data-out rates (data-in still free)
Consistent network performance
At least 1 location to each AWS region
Option for redundant connections
Uses BGP to exchange routing information over a VLAN

15. Physical connection
• Cross-connect at the location
• Single-mode optical fiber
- 1000Base-LX or 10GBASE-LR
• Potential onward delivery through Direct Connect partner
• Customer router

16. At the Direct Connect location
CORP
AWS Direct
Connect
Routers
Customer
Router
Colocation
DX Location
Customer
network
`
AWS backbone
network
Cross-
connect
Customer
router
Customer’s network
Demarcation

17. Dedicated port through Direct Connect partner
CORP
AWS Direct
Connect
Routers
Colocation
DX Location
Partner network
AWS backbone
network
Cross-
connect
Customer
router
Partner
network
Access
circuit
Demarcation
Partner
equipment

18. Hybrid cloud storage expansion:
Amazon EFS through Direct Connect
“Bursting”
File Workloads
Data Migration
into EFS
Amazon EFSOn-Premises AWS Direct Connect

19. AWS Storage Gateway

20. What is AWS Storage Gateway?
Works with your existing applications
Secure and durable storage in AWS
Low latency for frequently used data
Scalable and cost-effective on-premises storage - $.01/GB
written to AWS + S3/Amazon Glacier storage fees
Service connecting an on-premises software appliance
with cloud-based storage

21. Hybrid storage use cases and architectures for
AWS Storage Gateway
Enabling cloud workloads
Move data to AWS storage for Big Data, cloud bursting, or migration
Tiered cloud storage
Easily add AWS storage to your on-premises environment
Backup, archive, and disaster recovery
Cost effective storage in AWS with local or cloud restore

22. Storage Gateway hybrid storage solutions
Enables using standard storage protocols to access AWS storage services
Customer Premises
Storage
Gateway
Amazon EBS
snapshots
Amazon
S3
Amazon Glacier
AWS Identity and Access
Management (IAM)
AWS Key Management
Service (KMS)
AWS
CloudTrail
Amazon
CloudWatch
Enterprise
storage
Devices
Application
servers

23. Storage gateway – Files, volumes, and tapes
File gateway NFS (v3 and v4.1) interface **NEW!**
On-premises file storage backed by Amazon S3 objects
Volume gateway iSCSI block interface
On-premises block storage backed by Amazon S3 with EBS snapshots
Tape gateway iSCSI virtual tape library (VTL) interface
Virtual tape storage in Amazon S3 and Glacier with VTL management

24. Detail: AWS File Gateway for S3
NFS Interface ElasticityAmazon S3 Bucket
Easy Integration Cloud ScaleCloud Access

25. AWS File Gateway
Bursting Tiered Storage
STG213

26. AWS Snowball

27. What is AWS Snowball?
Petabyte-scale data transport
E-ink shipping
label
Ruggedized case
“8.5G impact”
All data encrypted
end-to-end
Rain- and dust-
resistant
Tamper-resistant
case and
electronics
80 TB
10 GE network

28. AWS storage migration expansion:
AWS Snowball
Transfer
Capacity
Integration
Regional
Availability
80TB model
HDFS support
3rd party API
HIPAA support
All EXCEPT:
Asia Pacific (Singapore)
Asia Pacific (Seoul)
China (Beijing)

29. How it works

30. How fast is Snowball?
• Less than 1 day to transfer 200TB via 3x10G connections with 3
Snowballs, less than 1 week including shipping
• Number of days to transfer 200TB via the Internet at typical utilizations
Internet Connection Speed
Utilization 1Gbps 500Mbps 300Mbps 150Mbps
25% 71 141 236 471
50% 36 71 118 236
75% 24 47 225 157

31. Customer: Scripps Networks Interactive
Problem Statement:
• Need storage platform to manage active archive content
• Existing content repository too large to migrate via available
network-based ingest methods
Use of AWS:
• S3 and Snowball for massively scalable ingest
• S3 for storage, Glacier for content archive
• Snowball to securely transport existing media content from on-
premises storage and tape vault
Business Benefits:
• Petabyte-scale data transport without increased network costs
• Massive scalability and elasticity
• Lower TCO for active archive storage
Active archive transport and archival for digital content provider

32. AWS storage migration expansion:
AWS Snowmobile

33. Hybrid cloud storage expansion:
AWS Snowball Edge
On-premises
Capacity
On-premises
Integration
On-premises
Compute
Clustered local storage
100TB capacity
NFS and S3-compatible
endpoint
AWS Lambda
support

34. Snowball Edge use cases
Offline
Staging
Local Tiering and
Compute
IoT
Local
Transformation

35. AWS Snowball Edge
Integrated
Storage and Compute
Applications
STG214

36. Storage Ecosystem Partners

37. Hybrid cloud storage ecosystem
Backup
AWS Storage Gateway VTL
Direct to Amazon S3
File
Systems
Object Storage
Block Storage

38. Backup to AWS approaches
Amazon S3
Amazon
Glacier
AWS
Direct
Connect
Internet
Amazon S3-IA
Application
servers
Cloud gateway
Local disk
Media
server
Cloud gateway
Application
servers
Backup SW cloud connector
Local disk
Media
server with cloud
connector

39. Bursting
Migration
Tiering
Accessing EFS
via Direct Connect
Storage
Gateway
Snowball
Edge
3rd Party
Ecosystem
AWS hybrid cloud storage choices

40. Related Sessions
• Snowball Edge: STG214, ENT211
• Snowmobile: STG214, ENT211
• Storage Gateway: STG213, ENT211

41. Thank you!

42. Remember to complete
your evaluations!