Data Lifecycle Management

© 2019, Amazon Web Services, Inc. or its Affiliates.
Chandra Kapireddy
Specialist SA, Data & Analytics
Data Lifecycle Management
from ingest to archive

Agenda
• What’s new with data today
• How AWS can help
• Data storage services
Break (15 min)
• Data transfer ( & hybrid storage) services
• Data lakes, database and analytic services
• Q/A

What’s new with data today?

Data is a strategic asset
for every organization
The world’s most valuable
resource is no longer oil,
but data.

https://www.networkworld.com/article/3325397/idc-expect-175-zettabytes-of-data-worldwide-by-2025.html
Data is growing at an
exponential rate.
The IDC predicts:*
The collective sum of the world’s 2018 data
was 33 zettabytes.
By 2025, it will grow to 175 zettabytes.
Data types are also diversifying, requiring
storage and analysis of structured AND
unstructured data.

Governance
& control
There are more
people working
with data than ever
before.
How do I provide democratized access
to data to enable informed decisions
while at the same time enforce data
governance and prevent
mismanagement of the data?
Democratization
of data

There are more ways to analyze
data than ever before.
Hadoop Elasticsearch
Years ago
12 9 6 5
Presto Spark
Didn’t exist

Why streaming data?
Get actionable insights quickly
Source: Perishable insights, Mike Gualtieri, Forrester
Real time Seconds Minutes Hours Days Months
Valueofdatatodecision-making
Preventive/Predictive
Actionable Reactive Historical
Time critical decisions Traditional “batch” business intelligence
Information half-life
in decision-making

Thinking about data as an asset, not a cost
Stop
throwing
data away.
Make it
available to
more users.
Arm users
with more
data processing
technologies.
What you need to succeed

Storage
Processing
Analytics /
Machine
learning
Archive /
Retire
Creation /
Ingest
At every stage of your data’s
lifecycle, you need scalable
services that can flex with your
data growth.

How AWS can help

Typical storage workloads
Backup &
Restore
Non-disruptive
Easy place to start
Integrated with all
major vendors
Archive &
Compliance
Media workflows
Tape replacement
Public Sector,
FinServ,
Healthcare/Life
Sciences
Home
Directories
Simple to move
Less latency
sensitive
Significant cost
savings
Data Lakes
Variety of
analytics tools
Foundation for
AI/ML
Built for
streaming data
Data visualization
Business-
Critical
Applications
Integrated with
major vendors
Fully managed
infrastructure
Lift-and-shift
migrations

AWS Storage Solutions – Use Cases & Customers
Backup & Restore
• Tools for in-cloud and on-prem backup
• S3 Standard-IA and S3 One Zone-IA for
cost-effective backup storage
• Storage Gateway integrations for tape,
volume, and block backup in the cloud
Archive
• S3 Glacier for frequent retrievals
• S3 Glacier Deep Archive for cold archive at
$1 for 1TB/month
• VTL capabilities to replace physical tape
• Query archive data with S3 Glaicer Select
Data Lakes
• Run analytics applications without ETL
• Query data in place with S3 Select and S3
Glacier Select
• Integrations with FSx for Lustre to run HPC,
ML, and data media processing
Enterprise Applications
• EFS for file systems to run applications
• EBS for compute storage
• Seamless integration between partner
application and existing AWS systems
• Backup to S3 and S3 Glacier
Data Compliance
• S3 Object Lock to configure & enforce
write-once-ready-many (WORM) controls
• S3 Object Lock can be set to compliance or
governance modes
• S3 Glacier Vault Lock for WORM archive
Hybrid Storage
• Connect on-premises environments to the
AWS Cloud with AWS Storage Gateway
• Back up data, burst compute-intensive
workloads, or transfer files, virtual tapes,
and volumes in the AWS Cloud

Data
AWS cloud storage is core
…yielding bigger insights…
…helping you
innovate faster…
…gives you unique scale…
Building on or migrating
an application to AWS…
Most big data & data lakes
Most managed databases
Simplest enterprise applications
Easiest data warehousing
Singular query-in-place analytics
Greatest reliability
Highest security
Most manageable
Broadest compliance
Widest portfolio
Fastest innovation
Active archive
Disaster recovery
IoT
Artificial Intelligence
Advanced developer tools
Experienced consulting and support
Methodical migration services
The most data movement services

More services & storage classes for every use case
Block storage
Amazon Elastic Block Storage (EBS)
• General purpose SSD
• Provisioned IOPS SSD
• Throughput-optimized HDD
• Cold HDD
• Elastic Volumes
File storage
Amazon Elastic File System (EFS)
• EFS Standard
• EFS Infrequent Access (for cost savings)
Object storage
Amazon Simple Storage Service (S3)
• 6 storage classes for various access patterns
• Includes 2 levels or archiving
• Build data lakes for structured and
unstructured data
Amazon EFS AWS Storage
Gateway Family
Amazon S3
Amazon FSx
for Lustre
Amazon FSx
for Windows
File Server
Amazon
EBS
Amazon
EC2

More options for data transfer
AWS
Direct Connect
Amazon Kinesis
Firehose
AWS
Snowball
AWS Snowmobile
AWS
Storage
Gateway
Amazon S3
Transfer
Acceleration
AWS
DataSync
AWS Transfer
for SFTP
AWS
Snowball Edge
Storage Optimized
Amazon Kinesis
Data Streams
Amazon Kinesis
Video Streams
AWS
Snowball Edge
Compute Optimized

Gartner Magic Quadrant
Magic Quadrant for
Public Cloud Storage Services,
Worldwide – 2018
Positioned furthest for
completeness of vision
and highest for ability
to execute in each report
since inception in 2014
Magic Quadrant for Public Cloud Storage Services, July 2018 – Raj Bala, Julia Palmer
This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the
context of the entire document. The Gartner document is available upon request from Amazon Web Services.
Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise
technology users to select only those vendors with the highest ratings or other designation. Gartner research
publications consist of the opinions of Gartner’s research organization and should not be construed as statements
of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any
warranties of merchantability or fitness for a particular purpose.

Data storage services

Three types of storage

© 2019, Amazon Web Services, Inc. or its Affiliates.© 2019, Amazon Web Services, Inc. or its Affiliates.
Object storage

S3 Batch Operations
FPO
The most features to cost-effectively store,
manage, audit, secure, and query data
– at virtually any scale.
S3 Standard
S3 Standard-IA
S3 Intelligent-Tiering
S3 One Zone-IA
S3 Glacier
S3 Glacier
Deep Archive
Use S3 Storage Class Analysis to
learn access patterns and S3
Lifecycle policies to move
objects between classes
S3 Storage Classes
Configure access to S3 resources
and define user access. Block all
public access requests with S3 Block
Public Access.
Access management
Replicate objects to a region of your
choice to reduce latency and for
compliance.
Cross-Region Replication
Append up to 10 metadata tags to an
object. Use tags, buckets, and prefixes to
organize data. Audit and report on access
requests and activities.
Data management tools
Execute tasks and invoke AWS Lambda across
billions of objects – with a single API call or a few
clicks in the console.
S3-integrated analytics applications
AWS Lake Formation
S3 Select to query data in place
FSx for Lustre for HPC, ML, and media data
processing
Analytics & file systems integrations
Supported by the most secure, durable, and performant storage infrastructure
Security by design Compliance programs 11 9’s of durability Multi-AZ resiliency Limitless scalability
Amazon Simple Storage Service (S3)

Choose the storage class that fits best
≥ 3 AZs 1 AZ
99.99% 99.5%
Milliseconds Hours
Hours YearsFrequent Infrequent
0 Bytes 5 Terabytes
Reduce storage cost > 80% by choosing the
storage class option that best fits your use case
2 Regions
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
99.9%

Your choice of Amazon S3 storage classes
Access FrequencyFrequent Infrequent
• Active, frequently
accessed data
• Milliseconds access
• > 3 AZ
• $0.0210/GB
• Data with changing
access patterns
• > 3 AZ
• $0.0210 to $0.0125/GB
• Monitoring fee per Obj.
• Min storage duration
• Infrequently accessed
data
• > 3 AZ
• $0.0125/GB
• Retrieval fee per GB
• Min object size
S3 Standard S3 S-IA S3 Z-IA Amazon Glacier
• Re-creatable, less
accessed data
• 1 AZ
• $0.0100/GB
• Min object size
• Archive data
• Select minutes or
hours
• > 3 AZ
• $0.0040/GB
• Min object size
S3 INT

Amazon S3 Glacier Deep Archive
Lowest cost storage class for long-term
archiving and digital asset preservation
Fully managed
without tape
burden
$0.00099 per
GB-month
Designed for
99.999999999%
durability
Recover
data in 12
hours

S3 Storage Class Analysis and S3 Lifecycle Policy
Use S3 Storage Class Analysis to identify storage age
groups that are less frequently accessed
Set S3 Lifecycle Policy to tier storage to lower cost
storage classes and expire storage based on age of
object
Great for predictable workloads (object age
indicates access frequency)
Fine tune analysis by bucket, prefix, or object tag

S3 Intelligent-Tiering storage class
Automatically optimizes storage costs for data with
changing access patterns
Moves objects between two storage tiers:
• Frequent access tier optimized for frequent use of
data
• Lower cost infrequent access tier optimized for
less accessed data
Monitors access patterns and auto-tiers on granular
object level
No performance impact, no operational overhead
Milliseconds access, > 3 AZs, Monitoring fee per
Object, Minimum storage duration

File storage

Fully managed cloud file systems
Amazon EFS
Amazon FSx for
Windows File Server
Amazon FSx
for Lustre
LINUX-BASED WORKLOADS WINDOWS-BASED WORKLOADS
Fully managed Windows file servers
for business applications
Fully managed cloud-native file system
for Linux-based applications
Fully managed Lustre file system for
compute-intensive workloads
COMPUTE-INTENSIVE WORKLOADS
AWS provides file system options that help you easily address
the diverse needs of your file-based applications and workloads

Performance
Amazon Elastic File System
Scalable, elastic, cloud-native Linux file system
Shared access Highly durable
and available
Elastic and
scalable
Secure and
compliant
Storage classes

Amazon FSx for Windows File Server
Lift and shift your Windows file storage with fully managed windows file servers
Fast and flexible
performance
Broad
accessibility
Fully
managed
Native Windows
compatibility
Enterprise-ready

Amazon FSx for Lustre
Fully managed Lustre file system for compute-intensive workloads
Native file
system interface
Fully
managed
Seamless access to
your data repositories
Massively scalable
performance
Cost-optimized for
compute-intensive workloads
Secure
and compliant

Seamless integration with Amazon S3
Data stored in S3 is loaded to
Amazon FSx for processing
Output of processing
returned to S3 for retention
When your workload finishes, simply delete your file system
Link your Amazon S3 data set to your Amazon FSx for Lustre file system, then….

Block storage

AWS block storage offerings
EC2 instance
store
sc1st1
io1gp2
EBS
SSD-backed
volumes
EBS
HDD-backed
volumes
HDDSSD

What is Amazon EC2 instance store?
• Local to instance
• Non-persistent data store
• Data not replicated (by default)
• No snapshot support
• SSD or HDD
EC2 instances
Physical Host
Instance Store
or

What is Amazon EBS?
• Block storage as a service
• Create, attach volumes through an API
• Service accessed over the network
EC2
instance
EBS
volume

What is Amazon EBS?
• Volumes persist independent of EC2
• Select storage and compute based on
your workload
• Detach and attach between instances
within the same Availability Zone
EC2
instance
EC2
instance
Availability Zone
AWS Region
EBS
volume

What is Amazon EBS?
• Volumes attach to one instance
• Many volumes can attach to an instance
• Separate boot and data volumes
EC2
instance
EBS
volume
(boot)
EBS
volume
(data)
EBS
volume
(data)
Availability Zone
AWS Region

Amazon EBS volume types
Hard disk drives (HDD)Solid-state drives (SSD)

Choosing an Amazon EBS volume type
or
What is more important to your workload?
IOPS Throughput?

Choosing an Amazon EBS volume type
i3
gp2
Latency?
< 1 ms Single-digit ms
Which is more important?
Cost Performance
IOPS
≤ 80,000> 80,000
is more important
io1
Throughput
is more important
Small, random I/O Large, sequential I/O
st1
d2
≤ 1,750 MiB/s
Aggregate throughput?
> 1,750 MiB/s
Which is more important?
Cost Performance
sc1

Data transfer & hybrid storage
services

AWS data transfer & hybrid storage
Online managed
data transfer
Hybrid
storage
Offline
data transfer
Private
network
connections
to AWS
Load
streaming
data into
Amazon S3
Ship static
data into and
out of
Amazon S3
Access AWS
storage from
on-premises
Edge locations
for Amazon
S3 enabled
applications
Online
transfer of
active data
AWS
DataSync
AWS
Transfer
for SFTP
SFTP transfers
into Amazon S3
NEW NEW
Storage and
compute in
disconnected
environments
Network-based
services

Network-based services
AWS Direct Connect &
S3 Transfer Acceleration

AWS Direct Connect
Reduce
bandwidth Costs
Consistent
network
performance
Compatible with
all AWS services
Private
connectivity to
VPC
Elastic Simple
• Establish private connectivity between AWS and your data center
• Dedicated connection can be partitioned into multiple virtual interfaces
• Maintain network separation between public and private environments
BENEFITS

Amazon S3 Transfer Acceleration
AWS Edge
Location
Optimized
Throughput!
Leverages AWS Global Edge Locations
(Amazon CloudFront) and optimized
AWS network path
Optimized protocols
Change your endpoint, not your code
No firewall exceptions & no client
software required
Speeds up transfers for applications that use S3 API over long distances
On average, a 171%
improvement over regular
Amazon S3 cli commands when
uploading over long distances

AWS DataSync

What use cases need to transfer active data?
Migration of active
application data
Transferring data
for time sensitive
in-cloud analysis
Replication of data
for business
continuity
Online data transfer

AWS DataSync
Online transfer service that simplifies, automates, and accelerates
moving data between on-premises storage and AWS
Fast data
transfer
Cost-
effective
Enterprise
ready
Combines the speed and reliability of network acceleration software
with the cost-effectiveness of open source tools
Easy to use Secure and
reliable
Up to 10Gbps Fully managed
in-cloud w/
agents on-prem.
Data encryption
& validation
PCI & HIPAA;
Works w/ AWS
IAM & CloudTrail
Usage-based,
$0.04 per-GB
copied

Shared
file system
NFS TLS
How AWS DataSync works
On-Premise
Amazon S3
bucket
AWS Storage resources
AWS
DataSync
Agent deployed
on-premises for
fast access to
local storage
Region
Amazon EFS
file system
AWS DataSync
agent
Data transfer
over the WAN via
efficient purpose-
built protocol
Managed from the
console or AWS
Command Line
Interface (AWS CLI)
Service in AWS
writes or reads
data from AWS
storage services

AWS Transfer for SFTP

SFTP: It’s here; it’s everywhere
Protocol is deeply embedded in workflows across a variety of industries
Financial services
$
Retail
Healthcare … and more

AWS Transfer for SFTP
Fully managed service enabling transfer
of data over SFTP while stored in Amazon S3
Seamless migration
of existing workflows
Native integration
with AWS services
Simple
to use
Cost
effective
Fully managed
in AWS
Enterprise
ready

Snowball family

Why Snowball Edge & the Snow family?
Offline Transfer of large data volumes + Edge Computing, analytics &
machine learning in remote and harsh environments
Moving large datasets over slow links can take years
Remote locations with limited, intermittent or no WAN
Many industries need edge computing for environments
where data generation is decentralized, and data volumes
are significant

AWS Snowball
AWS Snowball Edge
Compute or Storage Optimized AWS Snowmobile
• 80-TB storage capacity
• 10GE networking
• Data encryption end-to-end
• Rugged 8.5-G impact case
• Rain and dust-resistant
• 42 or 100-TB storage capacity
• Rugged 8.5-G impact case
• Rain and dust resistant
• AWS Greengrass support for local
compute, messaging, and caching
• EC2/AMI support for edge
compute
• Optional GPU
• Exabyte-scale storage in a 45-ft
container
• Dedicated security personnel
• GPS tracking, alarm monitoring,
24/7 surveillance, and optional
additional security
AWS Snow Family

Snowball Edge options summary
Amazon S3 Compatible Storage 42TB 100TB
Compute Comparable to m5d.2xlarge Comparable to m4.4xlarge
AWS Services available Amazon Lambda, File Gateway, Amazon EC2 Lambda, File Gateway, Amazon EC2
Memory 208GB 32GB
vCPUs 52 24
Disk Space for Instances 7.62TB 1TB
Clustering Available Available
Typical job lifetime Can be long lasting About 1 month
Networking Max 100Gb Max 40Gb
GPU Optional Nvidia Tesla V100 No
Storage
Optimized
Compute
Optimized

AWS Storage Gateway family

AWS Storage Gateway
On-premises access to virtually unlimited cloud storage
Customer premises AWS Cloud
Amazon S3
S3 Glacier
S3 Glacier
Deep Archive
Amazon Elastic
Block Store (EBS)
Integrated with IAM, KMS, CloudTrail,
CloudWatch services
Appliance Configuration: VMware, Hyper-V, EC2, Hardware
Files
(NFS/SMB)
Volumes
(iSCSI)
Tapes
(iSCSI VTL)
AWS Backup
HTTPS
Gateway
Appliance
Storage
Gateway
Managed
Service

The AWS Storage Gateway family
File gateway
Store and access objects in Amazon S3 from file-based
applications with local caching
Volume gateway
Block storage on-premises backed by cloud storage with local
caching, Amazon EBS snapshots, and clones
Tape gateway
Drop-in replacement for physical tape infrastructure backed by
cloud storage with local caching
Three gateway types provide file, block, and tape storage interfaces

AWS Streaming Services

Streaming with Amazon Kinesis
Easily collect, process, and analyze video and data streams in real time
Capture, process,
and store video
streams
Load data streams
into AWS data
stores
Analyze data
streams in real time
Capture, process,
and store data
streams

Amazon Kinesis Data Firehose—How it works
Ingest Transform Deliver
Amazon S3
Amazon Redshift
Amazon Elasticsearch Service
AWS IoT
Amazon Kinesis Agent
Amazon Kinesis Streams
Amazon CloudWatch Logs
Amazon CloudWatch Events
Apache Kafka

AWS Partner Network (APN) for
data transfer, migration & storage

Backup & restore Archive Primary storage BC/DRData migration
AWS Partner Network (APN): Migration & storage

Data Lakes, Database and
Analytic Services

Store exabytes of structured and unstructured data
Load, transform, and catalog once
Make data available to many tools
Open formats and interfaces support innovation
Introducing AWS Lake Formation
Snowball
Snowmobile Kinesis
Data Firehose
Kinesis
Data Streams
Amazon S3
Amazon
Redshift
Amazon
EMR
Amazon
Athena
Amazon
Kinesis
Amazon
Elasticsearch
Service
Kinesis
Video Streams
AI Services
Amazon
QuickSight
Data lakes
Build and deploy a fully
managed data lake with a
few clicks
Centrally define security,
governance, and auditing
policies
Self-service discovery and
safe access to all data
from a single catalog

AWS Marketplace
Amazon Redshift
Data warehousing
Amazon EMR
Hadoop + Spark
Athena
Interactive analytics
Kinesis Analytics
Real-time
Amazon Elasticsearch service
Operational Analytics
RDS
MySQL, PostgreSQL, MariaDB,
Oracle, SQL Server
Aurora
MySQL, PostgreSQL
Amazon
QuickSight
Amazon
SageMaker
DynamoDB
Key value, Document
ElastiCache
Redis, Memcached
Neptune
Graph
Timestream
Time Series
QLDB
Ledger Database
S3/Amazon Glacier
AWS Glue
ETL & Data Catalog
Lake Formation
Data Lakes
Database Migration Service | Snowball | Snowmobile | Kinesis Data Firehose | Kinesis Data Streams | Data Pipeline | Direct Connect
Data Movement
AnalyticsDatabases
Business Intelligence & Machine Learning
Data Lake
Managed
Blockchain
Blockchain
Templates
Blockchain
Amazon
Comprehend
Amazon
Rekognition
Amazon
Lex
Amazon
Transcribe
AWS DeepLens 250+ solutions
730+ Database
solutions
600+ Analytics
solutions
25+ Blockchain
solutions
20+ Data lake
solutions
30+ solutions
RDS on VMWare
Databases and analytics services – built for builders

What are DMS and SCT?
AWS Database Migration Service (DMS) easily and
securely migrates and/or replicate your databases and
data warehouses to AWS
AWS Schema Conversion Tool (SCT) converts your
commercial database and data warehouse schemas to open-
source engines or AWS-native services, such as Amazon
Aurora and Redshift

When to use DMS*?
Migrate
• Migrate business-critical applications
• Migrate from Classic to VPC
• Migrate data warehouse to Redshift
• Upgrade to a minor version
• Consolidate shards into Aurora
• Archive old data
• Migrate from NoSQL to SQL, SQL to
NoSQL or NoSQL to NoSQL
Targets:
Amazon Dynamo
DB
Amazon Redshift
Amazon S3
Amazon Aurora
*DMS is a HIPAA certified service
Amazon S3
Sources:

When to use DMS?
Replicate • Create cross-regions Read Replicas
• Run your analytics in the cloud
• Keep your dev/test and production
environment sync

DMS + Snowball
Common use cases –
• Migrate large databases (over 5TB)
• Migrate many databases at once
• Migrate over slow network
• Push vs. Pull

Security & Compliance is a shared responsibility between AWS and the customer.
What about security?

Thank you

Data Lifecycle Management

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Data Lifecycle Management

Similar to Data Lifecycle Management (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Data Lifecycle Management