Builders' Day - Best Practises for S3 - BL
•
0 recomendaciones•364 vistas
Builders' Day Presentation
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Similar a Builders' Day - Best Practises for S3 - BL
Similar a Builders' Day - Best Practises for S3 - BL (20)
Más de Amazon Web Services LATAM
Más de Amazon Web Services LATAM (20)
Último
Último (20)
Builders' Day - Best Practises for S3 - BL
- 1. Best Practices for S3 Bruno Laurenti Solutions Architect
- 2. • Storage Classes • Bucket Settings & Features • Managing Data at Scale • Performance • Data Transfer Agenda
- 3. Enterprise applicationsAnalytics Archiving Backup & restore Origin storage for CDN Benefits of Amazon S3 Website hosting Mobile sync and storage
- 5. Decreasing prices and more storage options 1 2 Decreasing storage prices S3 Standard (2006) Glacier (2012) S-IA (2015) Z-IA (H1-2018) INT (Q4-2018) Accelerating innovation 2006 2018
- 6. Choose the storage class that fits best ≥ 3 AZs 1 AZ 99.99% 99.5% Milliseconds Hours Hours YearsFrequent Infrequent 0 Bytes 5 Terabytes Reduce storage cost > 80% by choosing the storage class option that best fits your use case 2 Regions 99.9%
- 7. Your choice of Amazon S3 storage classes Access FrequencyFrequent Infrequent • Active, frequently accessed data • Milliseconds access • > 3 AZ • From: $0.0210/GB • Data with changing access pattern • Milliseconds access • > 3 AZ • From: $0.0210 to $0.0125/GB • Monitoring fee per obj. • Min storage duration • Infrequently accessed data • Milliseconds access • > 3 AZ • From: $0.0125/GB • Retrieval fee per GB • Min storage duration • Min object size S3 Standard S3 Standard-IA S3 One Zone-IA S3 Glacier • Re-creatable less accessed data • Milliseconds access • 1 AZ • From: $0.0100/GB • Retrieval fee per GB • Min storage duration • Min object size • Archive data • Minutes to hours access • > 3 AZ • From: $0.0040/GB • Retrieval fee per GB • Min storage duration • Min object size S3 Intelligent- Tiering S3 Glacier Deep Archive • Archive data • Hours access • > 3 AZ • From: $0.00099/GB • Retrieval fee per GB • Min storage duration • Min object size
- 8. S3 Storage Class Analysis and S3 Lifecycle Policy Use S3 Storage Class Analysis to identify storage age groups that are less frequently accessed Set S3 Lifecycle Policy to tier storage to lower cost storage classes and expire storage based on age of object Great for predictable workloads (object age indicates access frequency) Fine tune analysis by bucket, prefix, or object tag
- 9. Set S3 Lifecycle Policy to tier and expire storage S3 Lifecycle Policy to tier to lower cost storage classes and expire storage S3 Storage Class Analysis results help set up a S3 Lifecycle Policy Policies are based on age of object and set by bucket, prefix, or object tag S3 Standard S3 S-IA S3 Glacier
- 10. S3 Intelligent-Tiering Automatically optimizes storage costs for data with changing access patterns Moves objects between two storage tiers: • Frequent Access Tier • Infrequent Access Tier Monitors access patterns and auto-tiers on granular object level Milliseconds access, > 3 AZ, Monitoring fee per Object, minimum storage duration
- 11. Ideal use cases for S3 Intelligent-Tiering Dynamic cost optimization with no performance impact and no operational overhead Big Data, Data Lakes Storage with changing access patterns used by multiple applications Enterprises Storage accessed by fragmented applications from various organizations Startups Constrained on resources and experience to optimize storage themselves Amazon S3
- 12. Bucket Settings & Features
- 13. Permissions Permissions • IAM Policies • S3 Bucket Policies • S3 ACLs As a general rule, AWS recommends using S3 bucket policies or IAM policies for access control. S3 ACLs is a legacy access control mechanism that predates IAM.
- 14. Use IAM policies if: • You need to control access to AWS services other than S3. IAM policies will be easier to manage since you can centrally manage all of your permissions in IAM, instead of spreading them between IAM and S3. • You have numerous S3 buckets each with different permissions requirements. IAM policies will be easier to manage since you don’t have to define a large number of S3 bucket policies and can instead rely on fewer, more detailed IAM policies. • You prefer to keep access control policies in the IAM environment. Use S3 bucket policies if: • You want a simple way to grant cross-account access to your S3 environment, without using IAM roles. • Your IAM policies bump up against the size limit (up to 2 kb for users, 5 kb for groups, and 10 kb for roles). S3 supports bucket policies of up 20 kb. • You prefer to keep access control policies in the S3 environment. Permissions
- 15. With a few clicks in the S3 management console, you can apply S3 Block Public Access to every bucket in your account – both existing and any new buckets created in the future – and make sure that there is no public access to any object S3 Block Public Access Set at the account or bucket-level
- 16. Versioning Protect your data from accidental deletion • Create a new version with every upload • Previous versions are retained, not overwritten • Protect from unintended user deletes • Making delete requests without a version ID removes access to objects, but keeps the data • Manage previous versions with lifecycle • Transition or expire objects a specified number of days after they are no longer the current version
- 17. S3 Object Lock Immutable S3 Objects • Write Once Read Many (WORM) Protection for S3 Objects • Object or bucket control of WORM & retention attributes Retention Management Controls • Define retention periods in your app or with bucket-level defaults • Objects Locked for the Duration of the Retention Period • Support for Legal Hold scenarios Data Protection and Compliance • Assessed for use in SEC 17a-4, CFTC, and FINRA environments • Extra protection against accidental or malicious delete
- 18. S3 Object Lock Modes Compliance Mode • Intended for Compliance • Deletes disallowed, even for root account • Assessed for SEC 17a-4 by Cohasset Associates Governance Mode • Intended for Data Protection • Enables privileged delete of WORM-protected objects • Protects against account compromise & rogue actors • Retention can be changed to Compliance Mode
- 19. S3 Object Lock Legal Hold
- 20. Amazon S3 Glacier Enhancements Restore Notifications Notifications fire when a S3 Glacier restore starts and completes Restore Speed Upgrade Upgrade an in-progress restore to a faster restore speed Direct PUT Direct access to S3 Glacier through the S3 PUT API CRR direct to Glacier Replicate data direct to S3 Glacier in a secondary AWS region
- 21. Object Tags Add up to ten tags to your objects to control access and drive actions For example: • Grant an IAM user permissions to read only objects with specific tags • Use tags to indicate which objects should be replicated • Apply tags to specify granular lifecycle policies • Filter metrics and reports based on tags photos/photo1.jpg project/projectX/document.pdf project/projectY/document2.pdf projectX
- 22. Cross-Region Replication Flexibility to replicate data: • At the bucket, prefix, or object level • From any region to any region • To any storage class • Across AWS accounts • Change the object owner in the destination region
- 23. Managing Data at Scale
- 24. Amazon S3 Batch Operations Take large-scale actions on Amazon S3 objects
- 25. Amazon S3 Batch Operations Take large-scale actions on Amazon S3 objects
- 26. S3 Batch Operations Perform API actions across thousands, millions, or billions of objects
- 27. S3 Batch Operations Choose Objects Select an Operation View Progress Perform API actions across thousands, millions, or billions of objects
- 28. Performance
- 29. Amazon S3 performance optimization • What we hear from you … § How do I get the highest request rates? § How do I saturate my compute, network, and storage resources? § How do I maximize my single-threaded throughput? § How do I achieve more predictable outlier performance? § How do I optimize data query performance against Amazon S3? • Where do you start? § Your object naming scheme! § We’ll review this first, then walk through a streaming video workflow use case as an example.
- 30. Amazon S3 performance increase Amazon S3 for data analytics BEFORE Compute W R I T E T I M E R E A D T I M E 5TB of 2MB objects S3 1with prefix
- 31. Amazon S3 performance increase Amazon S3 request performance increase NOW Compute W R I T E T I M E R E A D T I M E S3 41m 40s 13m 52s 5TB of 2MB objects 1with prefix0
- 32. Amazon S3 performance increase Amazon S3 request performance increase PARALLEL PROCESSING Compute W R I T E T I M E R E A D T I M E S3 12m 00s 7m 00s 41m 40s 13m 52s 5TB of 2MB objects 1with 0 prefix
- 34. Up to 3,500 PUT* tps Up to 5,500 GET tps ExampleAWSbucket Let’s look at how prefixes scale request rate performance BucketName/Prefix: ExampleAWSbucket/LogFiles/ ExampleAWSbucket/Logistics/ ExampleAWSbucket/… * PUT, POST, and DELETE are all included in the PUT tps tps = transactions per second Initial partition
- 35. 3,500 PUT tps 5,500 GET tps 3,500 PUT tps 5,500 GET tps 2nd partition Initial partition ExampleAWSbucket BucketName/Prefix: ExampleAWSbucket/LogFiles/ ExampleAWSbucket/Logistics/ ExampleAWSbucket/… Let’s look at how prefixes scale request rate performance /Log /… PUT, POST, and DELETE are all included in the PUT tps tps = transactions per second
- 36. 3,500 PUT tps 5,500 GET tps 3,500 PUT tps 5,500 GET tps 3,500 PUT tps 5,500 GET tps 2nd partition Files/ istics/ 3rd partition 3,500 PUT tps 5,500 GET tps … ExampleAWSbucket BucketName/Prefix: ExampleAWSbucket/LogFiles/ ExampleAWSbucket/Logistics/ ExampleAWSbucket/… Let’s look at how prefixes scale request rate performance /Log /… PUT, POST, and DELETE are all included in the PUT tps tps = transactions per second Initial partition
- 37. 3,500 PUT tps 5,500 GET tps 3,500 PUT tps 0 GET tps 0 PUT tps 5,500 GET tps Request rate performance is allocated proportionally PUT, POST, and DELETE are all included in the PUT tps tps = transactions per second
- 38. Data Transfer
- 39. Flexible Data Transfer More ways to get data into Amazon S3 Database Migration Service
- 40. AWS Transfer for SFTP Fully-managed service enabling transfer of data over SFTP, while stored in Amazon S3 Seamless migration of existing workflows Native integration with AWS services Simple to use Cost effective Secure and CompliantFully managed in AWS
- 41. AWS integrated AWS Transfer service that simplifies, automates, and accelerates data movement Transfers up to 10 Gbps per agent Pay as you go Secure and reliable transfers Replicate data to AWS for business continuity Transfer data for timely in-cloud analysis Migrate active application data to AWS Combines the speed and reliability of network acceleration software with the cost-effectiveness of open source tools Simple data movement to S3 or Amazon EFS AWS DataSync
- 42. Let’s Build!