Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Tech Talks On Site- Edição de Agosto- Armazenamento em AWS

177 visualizaciones

Publicado el

Apresentação utilizada no Tech Talks de Agosto

Publicado en: Tecnología
  • Sé el primero en comentar

  • Sé el primero en recomendar esto

Tech Talks On Site- Edição de Agosto- Armazenamento em AWS

  1. 1. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Mv – Marcus Vinicius Ferreira / André Rosa Public Sector Team Ago/2019 Hands-on: Storage in the CloudEBS, EFS, S3
  2. 2. Mv – Marcus Vinicius Ferreira mvferr@amazon.com SolutionsArchitect BR, Public Sector, Education Mv
  3. 3. André Rosa anrosa@amazon.com SolutionsArchitect BR, Public Sector, Partners André
  4. 4. AWS Topics • Why: Storage Motivation - Overview • What: Storage Services - The Block Storage "Family": EBS, Snapshots - The Object Storage "Family": S3, S3-IA, Glacier - The Transfer Storage "Family": Storage Gateway, Snowball, Direct Connect, DMS • How: Scenarios and Architectures - Databases - Web Applications - Analytics, Big Data - Backup and Recovery - Legacy Systems
  5. 5. Why: Storage Motivation
  6. 6. GB TB PB ZB EB Big Data: Unconstrained Growth • Unstructured data growth is explosive • 95% of the 1.2 zettabytes of data in the digital universe is unstructured • Logs, Machine data and IoT will only steepen the curve • 70% of this data is user- generated content • Videos resolution is always increasing: 1080p, 4K, 8K Source: IDC, The Internet of Things: Getting Ready to Embrace Its Impact on the Digital Economy, March 2016.
  7. 7. Gartner: User Survey Analysis: Key Trends Shaping the Future of Data Center Infrastructure Through 2011 IDC: Worldwide Business Analytics Software 2012–2016 Forecast and 2011 Vendor Shares Available for analysis Generated data 1990 2000 2010 2020 Key Insight: Most Data Falls on the Floor 90% of the data in a company is never analyzed High costs and complexity of traditional DW systems make it hard to justify the capital expense
  8. 8. Data is a strategic asset for every organization The world’s most valuable resource is no longer oil, but data.* *Copyright:The Economist, 2017, David Parkins “ ”
  9. 9. Two Facts of Life
  10. 10. Two Facts of Life
  11. 11. Amazon EFS File Amazon EBS Amazon EC2 Instance Store Block Amazon S3 Amazon Glacier Object Data Transfer AWS Direct Connect ISV Connectors Amazon Kinesis Firehose Storage Gateway S3 Transfer Acceleration AWS Storage is a Platform AWS Snowball Amazon CloudFront Internet/VPN
  12. 12. What: Storage Services
  13. 13. Example Data Center: Where Do We Put All of This on AWS? DB (Master) DB (Slave) Back-ups on tapes Web server Web server App serverApp server App server SAN NAS file server File system disks LDAP server
  14. 14. Example Data Center: Where Do We Put All of This on AWS? Web server Web server App serverApp server App server Amazon Elastic File System Elastic Load Balancing Elastic Load Balancing Amazon Elastic Block Store Amazon RDS (Master) Amazon RDS (Standby) Backups to Amazon S3 or Glacier AWS Directory Service
  15. 15. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Block vs File vs Object Block Storage Raw Storage Data organized as an array of unrelated blocks Host File System places data on disk e.g.: Microsoft NTFS, Unix ZFS File Storage Unrelated data blocks managed by a file (serving) system Native file system places data on disk Object Storage Stores Virtual containers that encapsulate the data, data attributes, metadata and Object IDs API Access to data Metadata Driven, Policy-based, etc
  16. 16. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Three types of storage File ObjectBlock Amazon EFS Amazon FSx Amazon EBS Amazon S3, Amazon Glacier
  17. 17. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Performance comparison of storage types File Object Block Latency Throughput
  18. 18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Object storage S3 Standard S3 Glacier Deep Archive S3 Glacier S3 Intelligent-Tiering S3 One Zone-IA S3 Standard-IA Block storage Provisioned IOPS SSD Cold HDD Throughput-Optimized HDD NEW! COMING SOON! File storage EFS Standard EFS Infrequent Access COMING SOON! Elastic Amazon EFS AWS Storage Gateway Family Amazon S3 NEW!Amazon FSx for Lustre Amazon FSx for Windows File Server NEW! Amazon EBS Amazon EC2
  19. 19. EBS, Snapshots, EFS, Ephemeral
  20. 20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWSblockstorageofferings EC2 instance store sc1st1 io1gp2 EBS SSD-backed volumes EBS HDD-backed volumes HDDSSD
  21. 21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWSblockstorageofferings EC2 instance store sc1st1 io1gp2 EBS SSD-backed volumes EBS HDD-backed volumes HDDSSD
  22. 22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EBS volume types HDDSSD Provisioned IOPS SSD io1 General Purpose SSD gp2 Throughput Optimized HDD st1 sc1 Cold HDD
  23. 23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. or Choosing an EBS volume type What is more important to your workload? IOPS Throughput?
  24. 24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. i3 gp2 Choosing an EBS volume type Latency? < 1 ms Single-digit ms Which is more important? Cost Performance IOPS ≤ 80,000> 80,000 is more important
  25. 25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EBS volume types: General Purpose SSD gp2 Throughput: Up to 160 MiB/s Latency: Single-digit ms Capacity: 1 GiB to 16 TiB Baseline: 100 to 10,000 IOPS; 3 IOPS per GiB Burst: 3,000 IOPS (for volumes up to 1,000 GiB) Great for boot volumes, low-latency applications, and bursty databases General Purpose SSD
  26. 26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. i3 gp2 io1 Choosing an EBS volume type Latency? < 1 ms Single-digit ms Which is more important? Cost Performance IOPS ≤ 80,000> 80,000 is more important
  27. 27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EBS volume types: Provisioned IOPS io1 Baseline: 100 to 20,000 IOPS Throughput: Up to 320 MiB/s Latency: Single-digit ms Capacity: 4 GiB to 16 TiB Ideal for critical applications and databases with sustained IOPS Provisioned IOPS
  28. 28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. IOPS 0 2 16 1,000 5,000 10,000 15,000 20,000 6 90.4 Volume Size (TiB) Scaling Provisioned IOPS SSD (io1) MAX PROVISIONED IOPS (Maximum IOPS:GB ratio of 50:1) Available Provisioned IOPS ~ 400 GiB
  29. 29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. i3 gp2 io1 Choosing an EBS volume type Latency? < 1 ms Single-digit ms Which is more important? Cost Performance IOPS ≤ 80,000> 80,000 is more important Throughput?
  30. 30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Throughput is more important Small, random I/O Large, sequential I/O i3 gp2 io1 st1 d2 Choosing an EBS volume type Latency? < 1 ms Single-digit ms ≤ 1,750 MiB/s Aggregate throughput? > 1,750 MiB/s Which is more important? Cost Performance IOPS ≤ 80,000> 80,000 is more important Which is more important? Cost Performance
  31. 31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EBS volume types: Throughput Provisioned st1 Baseline: 40 MiB/s per TiB up to 500 MiB/s Capacity: 500 GiB to 16 TiB Burst: 250 MiB/s per TiB up to 500 MiB/s Ideal for large-block, high-throughput sequential workloads
  32. 32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Throughput is more important Small, random I/O Large, sequential I/O i3 gp2 io1 sc1 st1 d2 Choosing an EBS volume type Latency? < 1 ms Single-digit ms ≤ 1,750 MiB/s Aggregate throughput? > 1,750 MiB/s Which is more important? Cost Performance IOPS ≤ 80,000> 80,000 is more important Which is more important? Cost Performance
  33. 33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. sc1 EBS volume types: Throughput Provisioned Cold HDD Baseline: 12 MiB/s per TB up to 192 MiB/s Capacity: 500 GiB to 16 TiB Burst: 80 MiB/s per TB up to 250 MiB/s Ideal for sequential throughput workloads, such as logging and backup
  34. 34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Throughput is more important Small, random I/O Large, sequential I/O i3 gp2 io1 sc1 st1 d2 Choosing an EBS volume type Latency? < 1 ms Single-digit ms ≤ 1,750 MiB/s Aggregate throughput? > 1,750 MiB/s Which is more important? Cost Performance IOPS ≤ 80,000> 80,000 is more important Which is more important? Cost Performance
  35. 35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Don’t know Choosing an EBS volume type your workload yet?
  36. 36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EBS volume types: General Purpose SSD gp2 Throughput: Up to 160 MiB/s Latency: Single-digit ms Capacity: 1 GiB to 16 TiB Baseline: 100 to 10,000 IOPS; 3 IOPS per GiB Burst: 3,000 IOPS (for volumes up to 1,000 GiB) Great for boot volumes, low-latency applications, and bursty databases General Purpose SSD
  37. 37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved What is EBS? EBS volume EC2 instance • Block storage as a service • Create, attach volumes through an API • Service accessed over the network
  38. 38. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved What is EBS? EBS volume Availability Zone AWS Region EC2 instance • Volume and instance must be in the same AZ
  39. 39. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved What is EBS? EBS volume EC2 instance • Volumes persist independent of EC2 • Detach and attach between instances • Volume and instance must be in the same AZ Availability Zone AWS Region
  40. 40. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved What is EBS? EBS volume EC2 instance • Volumes persist independent of EC2 • Detach and attach between instances • Volume and instance must be in the same AZ Availability Zone AWS Region
  41. 41. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved What is EBS? EBS volume • Volumes persist independent of EC2 • Detach and attach between instances • Volume and instance must be in the same AZ Availability Zone AWS Region
  42. 42. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved What is EBS? EBS volume EC2 instance • Volumes persist independent of EC2 • Detach and attach between instances • Volume and instance must be in the same AZ Availability Zone AWS Region
  43. 43. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved What is EBS? Availability Zone AWS Region EC2 instance EBS data volume EBS data volume • Volumes attach to one instance at a time • Many volumes can attach to an instance • Maximum Volume Size is 16TB EBS data volume
  44. 44. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved What is EBS? Availability Zone AWS Region EC2 instance EBS data volume EBS data volume • Volumes attach to one instance at a time • Many volumes can attach to an instance • Maximum Volume Size is 16TB16TB 16TB 8TB EBS data volume
  45. 45. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved What is EBS? Availability Zone AWS Region EC2 instance EBS data volume • Volumes attach to one instance at a time 16TB
  46. 46. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved EBS is designed for: What is EBS? 99.999% service availability 0.1% to 0.2% annual failure rate (AFR)
  47. 47. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved What is an EBS snapshot? EBS volume Availability Zone AWS Region Amazon S3 EBS snapshot Availability Zone Replica
  48. 48. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved What can you do with a snapshot? EBS volume Availability Zone AWS Region Amazon S3 EBS snapshot Availability Zone EBS volume Replica Replica
  49. 49. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved What can you do with a snapshot? EBS volume Availability Zone AWS Region Amazon S3 EBS snapshot EBS volume Availability Zone AWS Region EBS snapshot Replica Replica
  50. 50. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved EBS restrictions are... • Single-AZ • Attach to just one EC2 at a time
  51. 51. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Do it yourself – NFS architecture NFS Server Volume Volume NFS Server Volume Volume NFS Server Volume Volume NFS Clients NFS Clients NFS Clients http://bit.ly/amazonefstutorial
  52. 52. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Do it yourself http://bit.ly/amazonefstutorial
  53. 53. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Amazon EFS architecture NFS Clients NFS Clients NFS Clients Mount Target Single Namespace Mount Target Mount Target http://bit.ly/amazonefstutorial
  54. 54. S3, S3-IA, Glacier
  55. 55. 1 PB raw storage 800 TB usable storage 600 TB allocated storage 400 TB application data Traditional Storage System
  56. 56. 1 PB raw storage 800 TB usable storage 600 TB allocated storage 400 TB application data S3 unlimited capacity -- pay only for what you use! Amazon S3 ~ $0.021 / GB
  57. 57. Amazon S3: HTTP access 1. HTTP/HTTPS access 2. Unlimited amount of files 3. Unlimited growth... 4. Any type of data: backups, photos, videos, documents, logs 5. Cheap, unlimited storage [bucket name] Preview2.mp4 Tokyo Region (ap-northeast-1) Bucket Object https://s3-ap-northeast-1.amazonaws.com/[bucket name]/Preview2.mp4 https://s3-ap-northeast-1.amazonaws.com/[bucket name]/ Region code Bucket name Key
  58. 58. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. ObjectKeys An object key is the unique identifier for an object in a bucket. http://doc.s3.amazonaws.com/2006-03-01/AmazonS3.html Bucket Object/Key
  59. 59. A Closer Look: S3 Durability 4 9s durability 5 9s durability S3, S3-IA Glacier 11 9s durability 99.999999999 % VS.
  60. 60. VS. Understanding Durability designed for 99.99% durability Two copies on one site designed for 99.999% durability Copies on two sites designed for 99.999999999% durability GlacierStandard IA AWS Region
  61. 61. Understanding Durability Availability Zone Availability Zone Availability Zone S3 Standard S3 Standard-IA Glacier Availability Zone S3 One Zone-IA AWS Region AWS Region
  62. 62. S3 Standard S3 Standard – Infrequent Access Amazon Glacier Active data Archive dataInfrequently accessed data Milliseconds Minutes to HoursMilliseconds $0.021/GB/mo $0.004/GB/mo$0.0125/GB/mo Choice of storage classes on Amazon S3
  63. 63. Storage Gateway, Snowball, Direct Connect
  64. 64. AWS offers the most ways to move data to the cloud AWS Direct Connect A private connection between your data center, office, or colocation environment and AWS AWS Snow family (Snowball, Snowball Edge, Snowmobile) Secure, physical transport appliances that move up to Exabytes of data into and out of AWS AWS Storage Gateways Hybrid storage that seamlessly connects on- premises applications to AWS storage. Ideal for backup, DR, bursting, tiering or migration Amazon Kinesis Firehose Capture, trans- form, & load streaming data into S3 for use with Amazon business intelligence and analytics tools Amazon EFS File Sync Up to 5x faster file transfers than open source tools. Ideal for migrating data into EFS or moving between cloud file systems Amazon S3 Transfer Acceleration Up to 300% faster transfers into and out of S3. Ideal when working with long geographic distances APN competency partners Integrations between 3rd party vendors and AWS services. Ideal for leveraging existing software licenses and skills Networks Shipping Hybrid
  65. 65. Storage Gateway: Enterprise Backup Amazon S3 Amazon Glacier Internet Amazon S3-IA Application servers Storage Gateway Local disk Media server Gateway Application servers Cloud Connector/Native Integration Local disk Media server with cloud connector VPNVPN
  66. 66. Which On-Premise Backup Software? All of them! AWS Storage Gateway VTL Native S3 Integration
  67. 67. Enterprise Backup: Direct Connect Amazon S3 Amazon Glacier AWS Direct Connect Amazon S3-IA Application servers Storage Gateway Local disk Media server Gateway Application servers Cloud Connector/Native Integration Local disk Media server with cloud connector VPN 1 GB or 10 GB dedicated link
  68. 68. Amazon S3Transfer Acceleration Rio De Janeiro Warsaw New York Atlanta Madrid Virginia Melbourne Paris Los Angeles Seattle Tokyo Singapore Time[hrs] 500 GB upload from clients in these locations to a bucket in Singapore Public InternetAccelerated Transfer Up to 300% faster 171% on average
  69. 69. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
  70. 70. What is Snowball? Petabyte scale data transport E-ink shipping label Ruggedized case “8.5G Impact” All data encrypted end-to-end 50TB or 80TB 10G network Rain & dust resistant Tamper-resistant case & electronics
  71. 71. How fast is Snowball? • Less than 1 day to transfer 50TB via a 10G connection with Snowball, less than 1 week including shipping • Number of days to transfer 50TB via the internet at typical utilizations Internet Connection Speed Utilization 1Gbps 500Mbps 300Mbps 150Mbps 25% 19 38 63 126 50% 9 19 32 63 75% 6 13 21 42
  72. 72. How fast is Snowball? • Less than 1 day to transfer 250TB via 5x10G connections with 5 Snowballs, less than 1 week including shipping • Number of days to transfer 250TB via the Internet at typical utilizations Internet Connection Speed Utilization 1Gbps 500Mbps 300Mbps 150Mbps 25% 95 190 316 632 50% 47 95 158 316 75% 32 63 105 211
  73. 73. AWS Snow* Family Snowball Snowball Edge Snowmobile Petabyte-scale data migration Showball with Lambda inside Exabyte-scale data migration
  74. 74. How: Scenarios and Architectures
  75. 75. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved EBS for Databases: SQL, NoSQL, BigData EC2 Server Volume Volume Volume Volume Volume Volume
  76. 76. S3: Sharing web files 172.31.0.0/16 sa-east-1a sa-east-1b sa-east-1c
  77. 77. S3: Sharing web files: because of AutoScaling 172.31.0.0/16 sa-east-1a sa-east-1b sa-east-1c
  78. 78. EFS: Legacy Systems 172.31.0.0/16
  79. 79. How: Big Data and Analytics
  80. 80. S3 for Big Data • Scalability & Elasticity • Resize a running cluster based on how much work is needed to be done. • Durability and Availability • Fault tolerant for slave node (HDFS) • Backup to S3 for resilience against master node failures • Standard Interfaces • Hive, Pig, Spark, Hbase, Impala, Hunk, Presto, other popular tools Amazon EMR Cluster Amazon EMR Cluster Amazon EMR Cluster
  81. 81. Big Data is about large amount of files Stored logs structure (in Amazon S3) Raw log data (sample) Order_ID Customer_ID Order_date Total
  82. 82. AWS EMR Environment: Hadoop, Spark, et al. Master instance group Task instance groupCore instance group Amazon S3 Core instances:  Manage data and tasks  Can be added and removed Task instances (optional) are added or subtracted in response to work Amazon S3 as primary storage HDFS HDFS Terabytes of files
  83. 83. Netflix Uses S3 to Back its Various Clusters S3
  84. 84. Fraud Detection FINRA uses Amazon EMR and Amazon S3 to process up to 75 billion trading events per day and securely store over 5 petabytes of data, attaining savings of $10-20mm per year.
  85. 85. NASDAQ LISTS3 , 6 0 0 G L O B A L C O M P A N I E S IN MARKET CAP REPRESENTING WORTH $9.6TRILLION DIVERSE INDUSTRIES AND MANY OF THE WORLD’S MOST WELL-KNOWN AND INNOVATIVE BRANDSMORE THAN U.S. 1 TRILLIONNATIONAL VALUE IS TIED TO OUR LIBRARY OF MORE THAN 41,000 GLOBAL INDEXES N A S D A Q T E C H N O L O G Y IS USED TO POWER MORE THAN IN 50 COUNTRIES 100 MARKETPLACES OUR GLOBAL PLATFORM CAN HANDLE MORE THAN 1 MILLION MESSAGES/SECOND AT SUB-40 MICROSECONDS AV E R A G E S P E E D S 1 C L E A R I N G H O U S E WE OWN AND OPERATE 26 MARKETS 5 CENTRAL SECURITIES DEPOSITORIES INCLUDING A C R O S S A S S E T CL A S SE S & GEOGRAPHIES
  86. 86. High Level Architecture Overview
  87. 87. Labs https://www.qwiklabs.com/ https://bit.ly/ps-hands-on-efs https://bit.ly/ps-hands-on-ebs https://bit.ly/ps-hands-on-s3
  88. 88. Questions?
  89. 89. Summary • What: Storage Services - The Block Storage "Family": EBS, Snapshots - The Object Storage "Family": S3, S3-IA, Glacier - The Transfer Storage "Family": Storage Gateway, Snowball, Direct Connect • How: Scenarios and Architectures - Databases: EBS - Web Applications: S3 - Analytics, Big Data: S3 - Backup and Recovery: S3, Storage Gateway, Direct Connect, Snowball - Legacy Systems: EBS, EFS, Storage Gateway
  90. 90. ThankYou! https://aws.amazon.com/ebs/ https://aws.amazon.com/efs/ https://aws.amazon.com/s3/
  91. 91. Preencha a pesquisa de satisfação e ganhe crédito de U$30,00 em nossa console https://amazonmr.au1.qualtrics.com/jfe/form/SV_40Ex9lGFKy 2BifP

×