Se ha denunciado esta presentación.
Se está descargando tu SlideShare. ×

ENT306 Migrating Large Scale Data Sets to the Cloud

Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio

Eche un vistazo a continuación

1 de 37 Anuncio

ENT306 Migrating Large Scale Data Sets to the Cloud

Descargar para leer sin conexión

Data migration at petabyte scale is now a simple service from AWS. You can easily migrate large volumes of data from on-premises environments to the cloud, quickly get started with the cloud as a backup target, or burst workloads between your on-premises environments and the AWS Cloud. Learn about AWS Snowball, AWS Snowball Edge, AWS Snowmobile and AWS Storage Gateway, and understand which one is the right fit for your requirements. We will go through customer use cases, review the different applications used, and help you cut IT spend and management time on hardware and backup solutions.

Data migration at petabyte scale is now a simple service from AWS. You can easily migrate large volumes of data from on-premises environments to the cloud, quickly get started with the cloud as a backup target, or burst workloads between your on-premises environments and the AWS Cloud. Learn about AWS Snowball, AWS Snowball Edge, AWS Snowmobile and AWS Storage Gateway, and understand which one is the right fit for your requirements. We will go through customer use cases, review the different applications used, and help you cut IT spend and management time on hardware and backup solutions.

Anuncio
Anuncio

Más Contenido Relacionado

Presentaciones para usted (20)

Similares a ENT306 Migrating Large Scale Data Sets to the Cloud (20)

Anuncio

Más de Amazon Web Services (20)

Más reciente (20)

Anuncio

ENT306 Migrating Large Scale Data Sets to the Cloud

  1. 1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Everett Dolgner, Business Development Manager Storage at Amazon Web Services July 27, 2017 Migrating Large Scale Data Sets to the Cloud
  2. 2. Cloud Data Migration Direct Connect Snow* data transport family 3rd Party Connectors Transfer Acceleration Storage Gateway Kinesis Firehose The AWS storage portfolio Object Amazon GlacierAmazon S3 Block Amazon EBS (persistent) Amazon EC2 Instance Store (ephemeral) File Amazon EFS
  3. 3. AWS data transfer services Moving large batches offline Edge computing Augmenting on-prem with cloud Using a dedicated network Integrating existing software Moving over long distances Streaming data AWS Snowball & Snowmobile AWS Snowball Edge AWS Storage Gateway AWS Direct Connect 3rd party connectors S3 Transfer Acceleration Amazon Kinesis
  4. 4. The Snow Family: large batches or “edge” scenarios
  5. 5. AWS Snow Family Snowball Snowball Edge Snowmobile Petabyte-scale data migration Compute & storage for hybrid/Edge workloads Exabyte-scale data migration
  6. 6. AWS Snowball Petabyte-scale data transport • Rugged 8.5G impact case • Rain and dust resistant • Data encryption end-to-end • 80 TB capacity/10G network E-ink shipping label
  7. 7. How Snowball moves data into and out of AWS Create a job Connect the Snowball Copy data to the Snowball Your data moved to Amazon S3 In transit to you Delivered to you Delivered to AWS At AWS Job created Job completed
  8. 8. AWS Snowball Edge Petabyte-scale hybrid device with onboard compute and storage • 100 TB local storage • Local compute equivalent to an Amazon EC2 m4.4xlarge instance • 10GBase-T, 10/25Gb SFP28, and 40Gb QSFP+ copper, and optical networking • Ruggedized and rack-mountable
  9. 9. Collect data Create job Copy data Moved to S3 Snowball Edge: Hybrid capabilities beyond data migrationMIGRATIONCOLLECTION Create job Copy data Moved to S3
  10. 10. When to use AWS Snowball Cloud Migration Disaster Recovery Data Center Decommission Content Distribution
  11. 11. Snowmobile case study: DigitalGlobe Use case: Seeing a better world DigitalGlobe takes satellite imagery of the Earth 100 PB image library = 6 billion square kilometers 1 PB new image every year Architecture before AWS Snowmobile: Stored data in their own data center Needed elastic compute power to retrieve and analyze images Wanted to move data to the cloud, but no feasible solution AWS Snowmobile lets DigitalGlobe migrate 100 PB of data to the cloud
  12. 12. Snowmobile from space Picture taken by DigitalGlobe’s WorldView-3 satellite http://blog.digitalglobe.com/industry/digitalglobe-moves-to-the-cloud-with-aws-snowmobile/
  13. 13. Amazon S3 Transfer Acceleration: moving large files over long distances
  14. 14. Amazon S3 transfer acceleration S3 Bucket AWS Edge Location Uploader Optimized Throughput! Change your endpoint, not your code Leverages 59 global edge locations Optimized protocols No firewall exceptions No client software required
  15. 15. Getting started Enable S3 transfer acceleration for their bucket DNS resolution • <bucket>.s3-accelerate.amazonaws.com resolves to the nearest POP location for the client • uses the client IP and CloudFront latency-based routing Customers update application/destination URI to: <bucket-name>.s3-accelerate.amazonaws.com File is uploaded to: <bucket-name> S3 Management Console, AWS SDK, AWS CLI
  16. 16. Rio De Janeiro Warsaw New York Atlanta Madrid Virginia Melbourne Paris Los Angeles Seattle Tokyo Singapore Time[hrs] 500 GB upload from these Edge locations to a bucket in Singapore Public Internet How fast is S3 Transfer Acceleration? S3 Transfer Acceleration On average, we have seen 171% improvement over regular S3 when uploading over long distances
  17. 17. Use case: Media upload ” “S3 transfer acceleration reduces the average time it takes for us to ingest videos from our global user base by almost half. This gives our customers the ability to edit and share videos sooner where speed is a critical factor. - Brian Kaiser, CTO Typical Friday during football season – over 35 hours of video every minute is uploaded Data in: User > S3-TA > S3 > Transcode > Redshift Data out: S3 > CloudFront > User S3-TA = >20% increase in upload and encoding speeds
  18. 18. Storage Gateway: augmenting existing on-premises storage with AWS
  19. 19. Storage Gateway hybrid storage solutions Use standard storage protocols to access AWS storage services AWS Storage Gateway Amazon EBS snapshots Amazon S3 Amazon Glacier AWS Identity and Access Management (IAM) AWS Key Management Service (KMS) AWS CloudTrail Amazon CloudWatch Files Volumes Tapes On-premises AWSCloud
  20. 20. File Gateway On-premises file storage maintained as objects in Amazon S3 Customer Premises File Gateway Data stored and retrieved from your S3 buckets One-to-one mapping from files-to-objects File metadata stored in object metadata Bucket access managed by IAM role you own and manage Use S3 lifecycle policies, versioning, or CRR to manage data Amazon Glacier S3 Standard S3 Standard - Infrequent Access HTTPS NFS v3 / v4.1 Application Server
  21. 21. File gateway use cases S3 Bucket Storage Gateway NFS client RefreshCache Amazon EMR In-cloud workload S3 Bucket Amazon Athena Amazon QuickSight Snowball + Gateway NFS client Storage Gateway RefreshCache Storage Gateway S3 BucketRead-only NFS client Cross region replication File sharing Site A Site B NFS client
  22. 22. Enabling cloud workloads Move data to AWS storage for big data, cloud bursting, or migration “Storage Gateway has the promise to transform the way we move data into the cloud. The NFS interface lets us easily integrate data files from analytical instruments, and the transparent S3 storage lets us easily connect our cloud-based applications and leverage the powerful storage capabilities of S3. With Storage Gateway, we can now unleash the full power of AWS on our instrument data.”
  23. 23. Application Server Volume Gateway On-premises volume storage backed by Amazon S3 with EBS snapshots Block storage in S3 accessed via the volume gateway Compression of data in-transit and at-rest Backup on-premises volumes to EBS snapshots Create on-premises volumes from EBS snapshots Up to 1 PB of total volume storage per gateway Amazon EBS snapshots Storage Gateway bucket in Amazon S3 Customer Premises Volume Gateway iSCSI HTTPS
  24. 24. Tiering storage into S3 and EBS Easily add AWS storage to your on-premises environment “Storage Gateway is at the core of our disaster recovery and business continuity (BCM) processes, handling our Co-Lo'd OLTP and OLAP offsite data backups, as well as our in-office BCM. It works transparently, in a lights out way, archiving off to a separate AWS account with a simple grandfather-father- son snapshot plan in place.”
  25. 25. Durable offsite storage Easily store important data in a durable remote site “As our business expanded, it became clear that using on-premises backup processes to store and recover these records was massively inefficient, not scalable, and very costly. Using the AWS Cached Volume Storage Gateway, we can retrieve these records very quickly.”
  26. 26. Data center SMB server with SGW backend SMB hosted onsite, blocks stored durably in Amazon S3 Private Datacenter Storage Gateway us-west-2 Amazon S3 Storage Gateway Volume Windows Clients Windows Server HTTPSiSCSISMB
  27. 27. Private Data Center Storage Gateway Amazon S3 Storage Gateway Volume Windows Clients Remote Windows Server HTTPSiSCSI us-east-1 Amazon S3 Storage Gateway Volumes HTTPSiSCSI Storage Gateway DFS with Replication Windows Servers HTTPSiSCSI SMB us-west-2
  28. 28. Tape gateway Virtual tape storage in Amazon S3 and Amazon Glacier with VTL management Virtual tape storage in S3 and Amazon Glacier accessed via tape gateway Data compressed in-transit and at-rest Up to 1 PB total tape storage per gateway, unlimited archive capacity Supports leading backup applications **3-5 hour tape retrieval from Amazon Glacier Archived Tapes stored in Amazon Glacier MEDIA CHANGER TAPE DRIVE Customer Premises Tape Gateway Virtual Tapes stored in Amazon S3 Backup Server HTTPSiSCSI
  29. 29. Backup, archive, and disaster recovery Cost-effective storage in AWS with local or cloud restore “Tapes are a headache, prone with hardware failures, offsite storage costs, and constant maintenance needs. Storage Gateway provided the most cost-effective and simple alternative. We even got disaster recovery by using a bi-coastal data center.”
  30. 30. GlacierS3 Standard S3- Infrequent Access File Gateway PetroBank Application ServersLTO NAS Active archive migration from LTO Cost-effective storage in AWS with local data access AWS Direct Connect Self-service loading of data, reducing time-to-data by days or weeks Storage archive costs reduced by 90%
  31. 31. EFS with Direct Connect (DX): working over a dedicated network
  32. 32. Access your EFS file system via AWS Direct Connect Direct Connect EFS in your Amazon VPCOn-premises Servers
  33. 33. Three scenarios for working with file data across on-premises environments and Amazon EFS Bursting Migration  Move entire data set permanently to EFS  Access the data from applications running on EC2 instances  Move data set temporarily to EFS  Access the data from applications running on EC2 instances  Move data back on premises once processing finishes Backup and Disaster Recovery  Maintain copy of entire data set on EFS  Restore the data to on premises storage or (for DR) access the data from failed-over applications running on EC2 instances
  34. 34. Partner Tiering & Migration Solutions: working with what’s already there
  35. 35. Storage Partner Solutions Technology solutions vetted by the AWS Storage Competency Program aws.amazon.com/backup-recovery/partner-solutions/ Note: Represents a sample of storage partners Note: Dell-EMC, IBM and Veritas have solutions and are working towards competency requirements Backup and RecoveryPrimary Storage Archive BCDR Solutions that leverage file, block, object, and streamed data formats as an extension to on-premises storage Solutions that leverage Amazon S3 for durable data backup Solutions that leverage Amazon Glacier for durable and cost-effective long-term data backup Solutions that utilize AWS to enable recovery strategies focused on RTO and RPO requirements
  36. 36. Bursting Migration Tiering Offline Summary: Hybrid cloud storage services Backup Online Snowball, Snowball Edge, Snowmobile Storage Gateway, EFS over DX Storage Gateway, S3 Transfer Acceleration Snowball Edge Snowball Storage Gateway, 3rd Parties, EFS over DX Storage Gateway
  37. 37. Thank you!

×