by Everett Dolgner, Business Development Manager AWS With a hybrid architecture approach to managing data on-premises and in the cloud, organizations can be more agile and responsive than ever before. Find out what your peers are doing with cloud and how data backup, recovery, management, and e-discovery capabilities can help maximize your use of AWS. We will also cover dynamic data indexing across on-premises and cloud storage and holistic data protection.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
March 2018
AWS Hybrid and Edge Storage

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Data movement
Online
Data security
and management
Complete Set of Data Building Blocks
Amazon
EFS
Amazon
EBS
AWS Snow family
AWS Storage
Gateway Family
AWS Direct Connect
Amazon EFS File Sync
Amazon S3
Transfer Acceleration
Storage Partners
Amazon Kinesis
Data Streams
Amazon Kinesis
Video Streams
Amazon
S3
Amazon
Glacier
AWS KMS
AWS IAM
AWS CloudWatch
AWS CloudTrail
AWS CloudFormation
AWS Lambda
Amazon Macie
AWS QuickSight
Offline

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Two Cloud Deployment Options
“All-In” Cloud
Fully deployed in
the cloud
Hybrid
Deployed on-
premises and the
cloud

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What is Hybrid?
At the most fundamental level, hybrid computing can be
viewed as having data that resides both on-premises and
in the cloud. This is often done to economically store
large data sets, utilize new cloud-native databases, move
data closer to customers, or to create a backup and
archive solution with cost-effective high availability.

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Why Hybrid?
• Legacy systems that don’t support cloud protocols
• Store large datasets economically
• Tape replacement for backup and archiving
• Durable, offsite storage
• First step to cloud migration
• Scale on demand

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
On-Premises
VPN connectivity
VGW
Internet Access
The Internet

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Direct Connect
on-premises
Colocation Facility
VGW Private VIF
Public VIF
VLAN B
VLAN A
AWS Direct
Connect POP
Customer or
Partner Cage
Service Provider
Network
Internet Access

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Snowball Edge
Petabyte-scale hybrid device with onboard compute and storage
• 100 TB local storage
• Local compute equivalent to an Amazon
EC2 m4.4xlarge instance
• 10GBase-T, 10/25Gb SFP28, and 40Gb
QSFP+ copper, and optical networking
• Ruggedized and rack-mountable

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Snowball Edge key features
S3-compatible endpoint
File interface (NFS)
Clustering
Run AWS Lambda functions
Faster data transfer
Encryption

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Snowball Edge Modes
Snowball Edge
Snowball Edge +
AWS Greengrass
Snowball Edge
Cluster
Import into Amazon S3 Yes Yes No
Export from Amazon
S3
Yes Yes No
Clustered (Durable)
Storage
No No Yes
Clustered Compute
(GG/Lambda)
No No Yes

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Storage Gateway hybrid storage solutions
Use standard storage protocols to access AWS storage services
AWS Storage
Gateway
Amazon EBS
snapshots
Amazon S3
Amazon Glacier
AWS Identity and Access
Management (IAM)
AWS Key Management
Service (KMS)
AWS
CloudTrail
Amazon
CloudWatch
Files
Volumes
Tapes
On-premises
AWSCloud

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Common Storage Gateway Capabilities
Standard storage protocols integrate with on-premises
applications
• NFS v3 / v4.1
• iSCSI
Local caching for low-latency access to frequently used data
Efficient data transfer with buffering and bandwidth
management
Cache Upload
buffer

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Common Storage Gateway Capabilities
Native data storage in AWS
Stateless virtual appliance for resiliency
Integration with AWS monitoring, management and security
Amazon EC2
Amazon S3 Amazon Glacier
Amazon EBS
snapshots
Amazon
CloudWatch
IAMAWS CloudTrail AWS Key Management
Service (KMS)

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Cache and upload buffer
Gateway Type Cache storage Upload buffer
Tape
Stored Volume
Cached Volume
File

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Cache and upload buffer
Cache: Applications write and read data from the
on-premises disks used as cache storage.
Upload buffer: Incoming data on-premises disk
staging area
• Queues data for upload to AWS
Cache
Upload buffer

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Cache
The cache represents the working set of data.
Customer Premises
Application
Server Cache
iSCSI
Read/Write
Storage
Gateway VM
• Hot data:
Recently read
or written
• Cold data:
Less recently
used data
• Dirty data:
Written but
not yet
uploaded
Cached data states
Dirty
Hot
Cold

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Enabling a spectrum of hybrid use cases
Enterprise Data Centers
Amazon
EBS
Amazon
S3
Amazon
Glacier
App. serversFile servers
Research Sites
AWS Storage Gateway
DevicesDatabasesMultimedia content
Analytics | File Services | Production Tiering | Data Processing
Data Distribution | Backup | DR | Archive | Migration
Amazon
EC2
AWS
Lambda
Amazon
CloudFront
Amazon
Athena
Amazon
EMR
Backup serversUsers
Remote Offices Small-Medium Businesses

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway
File storage maintained as objects in Amazon S3
• Data stored and retrieved from your Amazon S3 buckets
• One-to-one mapping from files-to-objects
• File metadata stored in object metadata
• Bucket access managed by IAM role you own and manage
• Use S3 Lifecycle Policies, versioning, or CRR to manage data
• Use CloudWatch Events to trigger Lambda functions for in-cloud processing
GlacierS3 Standard S3 Standard -
Infrequent
Access
Storage
Gateway
Service
Application
Server File Gateway
VM
HTTPSNFS v3/v4.1
Customer Premises
share
Region

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
s3://{bucket name}/{folder1}/{file1}
s3://{bucket name}/{folder1}/{file2}
s3://{bucket name}/{folder1}/{file3}
s3://{bucket name}/{folder1}/{file4}
s3://{bucket name}/{folder2}/{file1}
s3://{bucket name}/{folder2}/{file2}
File Gateway
One-to-one mapping from files-to-objects
nfs://{IP}/{share name}/{folder1}/{file1}
nfs://{IP}/{share name}/{folder1}/{file2}
nfs://{IP}/{share name}/{folder1}/{file3}
nfs://{IP}/{share name}/{folder1}/{file4}
nfs://{IP}/{share name}/{folder2}/{file1}
nfs://{IP}/{share name}/{folder2}/{file2}
File Gateway
VM
NFS client Objects in your
S3 bucket

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Hybrid Processing: File Gateway, Objects in S3 & AWS
services
Amazon S3 bucketNFS client File Gateway
VM
Refresh
Cache
Amazon
CloudWatch
Amazon
Athena
Amazon
QuickSight
Amazon
EMR
AWS
Lambda
Amazon
Rekognition

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway: Automated Hybrid Workflows with
CloudWatch Events
Customer Premises
File Gateway VM
File
share
Storage
Gateway
ServiceNFS
Client
File stored as
objects in S3
Cache
Devices
Users
Data flow Messaging flow Query-in-place operation

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway: Automated Hybrid Workflows with
CloudWatch Events
Customer Premises
File Gateway VM
File
share
Storage
Gateway
ServiceNFS
Client
File stored as
objects in S3
Cache
Devices
Users
Amazon
CloudWatch
File Upload
Notification
Amazon
Athena
CloudWatch Event
starts a query-in-
place job
Data flow Messaging flow Query-in-place operation

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway: Automated Hybrid Workflows with
CloudWatch Events
Customer Premises
File Gateway VM
File
share
Storage
Gateway
ServiceNFS
Client
File stored as
objects in S3
Cache
Devices
Users
Amazon
CloudWatch
File Upload
Notification
Amazon
Athena
CloudWatch Event
starts a query-in-
place job
Athena writes results set
back to S3
Athena
analyzes data
Data flow Messaging flow Query-in-place operation
RefreshCache

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Notify-when-uploaded
AWS Storage Gateway can send a notification through Amazon CloudWatch
Events when all files written to your file share up to that point in time have been
uploaded to Amazon S3.
aws storagegateway notify-when-uploaded --file-share-arn
arn:aws:storagegateway:us-east-1:123456789011:share/share-123456FF

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CloudWatch Events

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Refresh Cache Lambda

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Requester Pays
File Gateway can now access S3 buckets configured as
Requester Pays.
S3 Bucket
S3 Bucket
Rquestor
Pays
Owner pays
Individual requestor pays

28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Enabling cloud workloads
Move data to AWS storage for big data, cloud bursting, or migration
“Storage Gateway has the promise to transform the way we move
data into the cloud. The NFS interface lets us easily integrate data
files from analytical instruments, and the transparent S3 storage
lets us easily connect our cloud-based applications and leverage the
powerful storage capabilities of S3.
With Storage Gateway, we can now unleash the full power of AWS
on our instrument data.”

29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Enabling cloud workloads
“We have applications that don’t support S3 natively, and we can’t
make changes to the applications, so being able to use native
protocols is very important to us. We tried clients for end users, but that
required changes to their environment that were not acceptable.
Storage Gateway allows Celgene to get data to the cloud without
making any changes to the application or environment.”

30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Volume Gateway
On-premises volume storage backed by Amazon S3 and EBS snapshots
• Block storage in S3 accessed via the volume gateway
• Compression of data in-transit and at-rest
• Backup on-premises volumes to EBS snapshots
• Create on-premises volumes from EBS snapshots
• Up to 1PB of total volume storage per gateway
Application
Server
Amazon EBS
snapshots
Storage Gateway
Service
Volume
Gateway VM
Volume Stored in
Amazon S3
HTTPSiSCSI
Customer Premises Region

31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Volume Stored
volume Amazon EBS
snapshots
Customer Premises
iSCSI
Storage Gateway VM
Region
stored
volumeApp
Server
complete volume
stored in the cloud
and on-prem
Amazon
CloudWatch
Storage
Gateway
Service
Supporting storage
can be SAN, VMDK,
VHD
cache
upload
buffer

32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Volume Cached
volume Amazon EBS
snapshots
Customer Premises
App
Server
iSCSI
Storage Gateway VM
Region
cache
virtual
volume
upload
buffer
complete volume
stored in the cloud
virtual volume stored
on-prem backed by
cache and cloud
Amazon
CloudWatch
Storage
Gateway
Service

33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Windows server with SGW backend
SMB hosted onsite, blocks stored durably in Amazon S3
Private Datacenter
us-west-2
Storage
Gateway VM
Storage
Gateway
Volume
Windows
Clients
Windows Server
HTTPSiSCSISMB
Storage
Gateway
Service

34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
High availability Windows Server storage
Local HA with remote DR
us-west-1
Data Center
Windows
Clients Storage
Gateway VMMicrosoft DFS
HTTPSiSCSI
HTTPSiSCSI
SMB
Storage Gateway Service
us-east-1
Microsoft DFS
HTTPSiSCSI
Storage Gateway Service
us-east-1
us-west-1

35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Backup & Disaster Recovery
• Gateway volumes are continually
written to S3
• Point in time protection via EBS snaps
• Ability to recover to EBS from snaps
• Ability to clone the latest volume to
another GW in the event of a disaster
Region
Storage
Gateway
Service
Amazon EBS
snapshots
EC2 instance EC2 instance
Windows
Server
Availability Zone #2
EC2 instance
Windows
Server
DATA
Data
SGW
Application
Server
Volume
Gateway VM
Customer Premises
Data
Data
iSCSI
Data
Availability Zone #1
(Clone)

36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Tiering storage into Amazon S3 and EBS
Easily add AWS storage to your on-premises environment
“Storage Gateway is at the core of our disaster
recovery and business continuity (BCM) processes,
handling our Co-Lo'd OLTP and OLAP off site data
backups, as well as our in-office BCM. It works
transparently, in a lights out way, archiving off to a
separate AWS account with a simple grandfather-father-
son snapshot plan in place.”

37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Flexible storage
Easily expand storage capacity on-demand
“Storage Gateway allows us to store backups offsite,
create online archives of big files, and easily expand
our file servers. With cached volumes, we have the
most used files local, and all of the files are stored safely
in AWS. We use Storage Gateway whenever we need
a quick way to have big storage available.”

38. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Storage
Gateway
Service
Tape Gateway
Virtual tape storage in Amazon S3 and Amazon Glacier with VTL management
Virtual tape storage in S3 and Glacier accessed via tape gateway
Data compressed in-transit and at-rest
Up to 1 PB total tape storage per gateway, unlimited archive capacity
Supports leading backup applications
**3-5 hour tape retrieval from Amazon Glacier
Archived Tapes
stored in
Amazon Glacier
MEDIA
CHANGER
TAPE
DRIVE
Customer Premises
Virtual Tapes
stored in
Amazon S3
Backup
Server
HTTPSiSCSI
Tape
Gateway VM

39. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Tape Gateway Emulates a Physical Tape Device
Customer Premises
Tape Gateway VM
Region
Storage Gateway
Service
Backup
Server
When tapes are
“Ejected”, they are
moved to Glacier
Virtual Tapes are housed in S3
Upload
Buffer
cache
Media
Changer
Tape Drive
Tape Drive
Tape Drive
Tape Drive
Tape Drive
…
Tape Library
(S3)
Tape Shelf
(Glacier)
• Tape Gateway emulates a physical tape device with a media changer and 10 tape drives.
• Virtual Tapes are written to S3 – data is in S3 when tape is in virtual library drive or slot
• Exported virtual tapes are marked read-only and moved to Glacier

40. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Backup, archive, and disaster recovery
Cost-effective storage in AWS with local or cloud restore
“Tapes are a headache, prone with hardware
failures, offsite storage costs, and constant
maintenance needs. Storage Gateway
provided the most cost-effective and simple
alternative. We even got disaster recovery by
using a bi-coastal data center.”

41. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Example: Data Sanitization at AWS