As your data stores grow, managing and operating on your stored objects becomes increasingly difficult to scale. In this session, AWS experts demonstrate Amazon S3 features you can use to perform and manage operations across any number of objects, from hundreds to billions, stored in Amazon S3. Learn how to monitor performance, ensure compliance, automate actions, and optimize storage across all your Amazon S3 objects. We also provide relevant use cases that demonstrate the full range of Amazon S3 capabilities and options, such as copying objects across buckets to create development environments, restricting access to sensitive data, or restoring many objects from Amazon Glacier.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Introducing S3 Batch Operations (Preview):
Performing API Actions across Billions of
Objects in Amazon S3
S T G 2 1 2
Rob Wilson
Senior Technical Product Manager
Amazon Web Services
Matt Sidley
Senior Technical Product Manager
Amazon Web Services

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
Operating at scale in Amazon Simple Storage Service
(Amazon S3)
Overview of Amazon S3 Batch Operations (preview) NEW!
S3 Batch Operations console demo
Deep Dive on S3 Batch Operations functionality
Common use cases
Q&A

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Related breakouts
Thursday, November 29th
STG 213-L – AWS Storage Leadership Session: What's New in Amazon S3,
Amazon EFS, Amazon EBS, and more
1:45 – 2:45 | Mirage
Friday, November 30th
STG 203 – Best Practices for Amazon S3 and Amazon Glacier
11:30 – 12:30 | Mirage

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Features to help you operate at scale
Lifecycle management
S3 Intelligent―Tiering storage class NEW!
Cross-Region replication
Inventory reports
S3 Batch Operations (preview) NEW!

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Set S3 Lifecycle policy to tier and
expire storage
S3 Lifecycle Policy to tier to lower
cost storage classes and expire
storage
S3 Storage Class analysis results
help set up a S3 Lifecycle Policy
Policies are based on age of
object and set by bucket, prefix, or
object tag
S3 Standard S3 S-IA S3 Glacier

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Lifecycle management example policies
Lifecycle rules take action based on object age:
• Move all objects older than 60 days to S3 S-IA, move all objects older
than 180 days to S3 Glacier
S3 Standard S3 S-IA S3 Glacier

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Lifecycle management example policies
Lifecycle rules take action based on object age:
• Move all objects older than 180 days to S3 Glacier
S3 Intelligent-
Tiering
S3 Glacier

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S3 Intelligent-Tiering NEW!
automates cost savings
Automatically optimizes storage costs
for data with changing access patterns
Moves objects between two storage
tiers:
• Frequent access tier
• Infrequent access tier
Monitors access patterns and auto-
tiers on granular object level
Milliseconds access, > 3 AZ, monitoring
fee per object, minimum storage
duration

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Your choice of Amazon S3 storage classes
Access FrequencyFrequent Infrequent
• Active, frequently
accessed data
• Milliseconds access
• > 3 AZ
• From: $0.0210/GB
• Data with changing
access pattern
• Milliseconds access
• > 3 AZ
• From: $0.0210 to
$0.0125/GB
• Monitoring fee per obj.
• Min storage duration
• Infrequently accessed
data
• Milliseconds access
• > 3 AZ
• From: $0.0125/GB
• Retrieval fee per GB
• Min storage duration
• Min object size
S3 Standard S3 Standard-IA S3 One Zone-IA S3 Glacier
• Re-creatable less
accessed data
• Milliseconds access
• 1 AZ
• From: $0.0100/GB
• Retrieval fee per GB
• Min storage duration
• Min object size
• Archive data
• Minutes to hours access
• > 3 AZ
• From: $0.0040/GB
• Retrieval fee per GB
• Min storage duration
• Min object size
S3 Intelligent-
Tiering
N E W !

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cross-Region replication
Flexibility to replicate data:
• At the bucket, prefix, or object level NEW!
• From any region to any region
• To any storage class NEW!
• Across AWS accounts
• Change the object owner in the destination region

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon S3 Inventory
Provides a list of your objects and their metadata for analytics and
auditing
Displays each objects:
- Encryption status
- Replication status
- Retention date NEW!
- Size, and more
Answer operational and compliance questions quickly with Amazon
Athena and other query tools

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S3 Batch Operations
Perform API actions across thousands, millions, or billions of objects

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why we built S3 Batch Operations
We heard these questions from customers:
How do I copy 300 TB of data to another bucket?
I know how to tag my new objects, but how should I add tags to my
billions of existing objects?
When I need to restore all of this data from S3 Glacier, how do I do it?

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S3 Batch Operations is a managed solution
S3 Batch Operations includes:
• Automatic retries
• Scalability
• Progress visibility
• Management controls
• Notifications
• Auditing
So there’s no need to build and maintain an application to call APIs in
bulk

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S3 Batch Operations
Perform API actions across thousands, millions, or billions of objects

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S3 Batch Operations
Perform API actions across thousands, millions, or billions of objects

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S3 Batch Operations
Perform API actions across thousands, millions, or billions of objects

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S3 Batch Operations
Perform API actions across thousands, millions, or billions of objects

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Benefits of S3 Batch Operations

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S3 Batch Operations―List view

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S3 Batch Operations―Select operation

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S3 Batch Operations―Configure options

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S3 Batch Operations―Verify job details

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S3 Batch Operations―Clone jobs

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S3 Batch Operations―Update priority

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S3 Batch Operations―Cancel jobs

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Manifest creation
Your manifest lists the target objects for your S3 Batch Operations job
Manifest formats:
• S3 Inventory report
• CSV file
Input fields:
• Bucket
• Key name
• Version ID (optional)

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How does S3 Batch Operations execute the job?
S3 Batch Operations calls the existing API for the operation you specify
Offers the same or similar functionality as calling the API directly
• Includes AWS CloudTrail support
• Allows you to customize the API’s parameters
• Requires the same permissions as the existing API
• Makes it simple to use S3 Batch Operations with your existing applications

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Setting up permissions
S3 Batch Operations assumes an Identity and Access Management (IAM)
role to perform the job
S3 Batch Operations requires permissions for:
• Reading your manifest of objects
• Performing the action in the source and destination (if required) bucket
• Writing the optional completion report (recommended)

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S3 Batch Operations can be customized
Specify whether or not a confirmation step is required
• Allows you to view the job parameters and the number of objects in the manifest
• Jobs will not run until they are confirmed
Set different priorities for different types of jobs
• Allows you to control your resources, ensuring the most important jobs run first
• Higher integer values indicate higher priority
• Cannot be used for ordering jobs, only to set relative precedence
Specify the information to include in your completion report
• All operations performed
• All failed operations
• No completion report

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Manage your jobs
New S3 Batch Operations API calls for:
• Creating jobs―select your manifest, operation type, and job details
• Confirming and running jobs―check your job’s details before it runs
• Updating jobs―changes the job’s priority level
• Describing jobs―shows a job’s details and progress
• Listing jobs―lists jobs in a region by creation date and job state
• Canceling jobs―cancels the job

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Regional list view
S3 Batch Operations shows your job activity at the account level
You can view all jobs in a region
using the ListJobs API, which shows:
• Job ID and description
• Operation type
• Creation date
• Job state and job priority
• Job size and progress
Choose the region for your jobs based on:
• Object location for most operations
• Destination region for all copy jobs

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S3 Batch Operations execution details
Each job has an automatic failure threshold
• This prevents large jobs from continuing to run and accelerates error resolution
• If >50% of operations have failed after 1,000 operations were attempted, the job will fail
Your job’s speed adjusts automatically
• S3 Batch Operations schedules jobs to efficiently manage resources
• Operations are automatically retried if needed
• Your AWS Lambda functions can adjust the rate of execution

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Job progress and notifications
Amazon CloudWatch Events inform you when your job changes state
Detailed job progress at the object level is visible through:
• DescribeJob API
• ListJobs API (view up to 1,000 jobs at a time filtered by job states)
• Amazon S3 Management Console

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Completion reports
A record of all the work performed by S3 Batch Operations during a job
Lists:
• Bucket
• Key name
• Version ID
• Success/Failure
• Status code
• Code description
• Additional detail or result set

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Simplify recurring and one-time workloads
Recurring
• Apply new object tags to
drive lifecycle policies
• Perform bulk S3 Glacier
restores to run periodic
reports
• Standardize data formats
using AWS Lambda functions
• Copy data as part of a
workflow or for backups
One-time
• Change object access control
lists (ACLs)
• Move data between storage
classes using the copy API
• Encrypt data with the copy API
• Use a Lambda function to
create thumbnails for all the
images in your bucket
• Copy data to another bucket
• Bulk tag existing objects

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Replace tag sets

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Restore objects from S3 Glacier

46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Put new object access control lists (ACLs)

47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Copy objects

48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Invoke Lambda functions

49. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

50. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Rob Wilson and Matt Sidley

51. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.