You wouldn't use a scooter to help someone move; likewise, there is no one-size-fits-all data storage solution. AWS provides a wide variety of storage services to address the spectrum of needs, from casual users saving photos to mission-critical, specialized databases utilized at the largest private and public sector entities. This session will give you an overview of these storage offerings, provide you with the groundwork to match these to your use cases. Learn More: https://aws.amazon.com/government-education/

1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Dave Cuthbert, US Federal Solutions Architecture
June 13, 2017
Demystifying Storage on AWS

2. Typical end-user requirements
“This application needs 15 TB.”
“We need storage for 300,000 users.”
“The storage needs to be fast.”
Will this grow?
How fast?
Per server?
How much per user? Activity per day?
Throughput or latency? Equal load or hot spots?

3. Mission critical
How you end up buying
Everything else

4. What if we could purchase
storage just-in-time?

5. Traditional
Amazon EC2 Instance Amazon EBS volume
800 GB SSD (gp2)200 GB SSD (gp2)400 GB SSD (gp2)
Elasticity with Amazon Elastic Block Storage
Amazon EC2
2x200 GB SSD

6. Traditional
Amazon EC2 instance Amazon EBS volume
800 GB SSD (gp2)
1.2 TB HDD
Archival (sc1)
400 GB HDD
Throughput-Optimized (st1)
20 GB SSD
200 IOPS
Provisioned I/O (io1)
Elasticity with Amazon Elastic Block Storage
Amazon EC2
2x200 GB SSD

7. Nightmares of the storage administrator
“We’ve been amassing data on drives around the lab since
the 70s. I don’t know what’s on them and tracking them is
hard. If one of these goes missing, it could be a non-event or
we could all go to jail.”
“We just broke the 1 PB mark. Of course we’re taking
backups of this data. I’m pretty sure the tapes are good, but
we have no way to test this. We just don’t have the funds to
buy a 1 PB test bed.”

8. Encrypted volumes
Security and recovery
AWS Key Management Service
Data is encrypted over the
wire and at rest
vol-a123456b

9. Encrypted volumes
Security and recovery
Snapshots
AWS Key Management Service
Data is encrypted over the
wire and at rest
Snapshot of vol-a123456b
taken January 1, 00:00
vol-a123456b
Snapshot of vol-a123456b
taken January 1, 12:00
Snapshot of vol-a123456b
taken January 2, 00:00
vol-c789012c

10. Shared Storage

11. Traditional
Web-accessible storage
Amazon S3 and Amazon Glacier
Amazon S3 buckets
VPC endpoint
Amazon Glacier vault
httpd.conf
.htconfig
Nagios
Apache

12. Data lakes
Central Storage
Secure, cost-effective
storage in Amazon S3
Data Ingestion
Get your data into S3 quickly and securely
Kinesis Firehose, Direct Connect, Snowball,
Database Migration Service
Catalog & Search
Access and search metadata
Access & User Interface
Give your users easy and secure access
Processing & Analytics
Use of predictive and prescriptive analytics
to gain better understanding
DynamoDB
Elasticsearch Service
API Gateway
AWS Directory Service
Cognito
Athena, QuickSight, EMR, Amazon RedshiftProtect & Secure
Use entitlements to ensure data is secure and users’
identities are verified
IAM, CloudWatch, CloudTrail, KMS

13. Data lakes

14. S3 cross-region replication
Source
(Virginia)
Destination
(Oregon)
• Only replicates new PUTs. Once
S3 is configured, all new uploads
into a source bucket will be
replicated
• Entire bucket or prefix based
• 1:1 replication between any 2
regions
• Versioning required
Automated, fast, and reliable asynchronous replication of data across AWS Regions
Use cases:
• Compliance: Store data hundreds of miles apart
• Lower latency: Distribute data to regional customers
• Security: Create remote replicas managed by separate AWS accounts

15. Traditional
Networked file systems
Amazon Elastic File System
EFS share
/etc/exports
Nagios
nfsd
Direct Connect
On-premises
server

16. Storage Gateway
Extend on-premises storage to the cloud
Mirror my
disks to AWS
Volume Gateway
Stored Mode Block data backed by EBS
snapshots
On-premises
cache of disks
Volume Gateway
Cached Mode
Block data backed by S3
16 TB 16 TB
1 TB 32 TB
Move my
tapes to AWS
Tape data stored as Amazon
Glacier archives
Tape Gateway
On-premises
cache of files
160 GB
File Gateway
File data stored as S3 objects
8 EB
1 PB

17. A Word on Databases

18. Relational databases
For the most part, you know when you need a relational database
(RDBMS).
• You’ve been given data in relational format.
• You’re using an application that needs an RDBMS.
You can do this on EC2 natively:
Amazon EC2 instance
Root volume Data volume
Security patches
Snapshots
Schema and data

19. Relational databases
But wouldn’t it be nice to have someone else take care of the
undifferentiated heavy lifting?
Amazon EC2 instance
Root volume Data volume
Security patches
Snapshots
Schema and data
Amazon RDS does just this.
Amazon RDS instance

20. Caching

21. When your storage can’t keep up
DatabaseAmazon EC2 instances
Elastic Load Balancing

22. Instance stores and high memory instances
Amazon EC2 instances
Elastic Load Balancing
Database

23. Where to learn more

24. Resources
AWS Documentation:
https://aws.amazon.com/documentation/
AWS Discussion Forums:
https://forums.aws.amazon.com/
AWS Blog:
https://aws.amazon.com/blogs/aws/

25. Thank you!