Large-scale migration to the cloud can be complex and time-consuming. AWS solutions, like AWS Snowball, AWS Snowball Edge, and AWS Snowmobile, help customers move petabytes of data cost-effectively and without saturating a network connection for months at a time. In this chalk talk, hear about a real-world scenario with AWS customer, Photobox, Europe's leading provider of photobooks and other personalized items, which migrated over 10 PB of data to Amazon S3 and Amazon Glacier using Snowball Edge to build its data lake. The technical architects responsible for this migration are available to answer detailed questions. 

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Petabyte-scale Migrations to Amazon
S3: How Photobox Moved Europe’s
Largest Photo Library to AWS
S T G 3 9 3
Chris Astall
Chief Architect
Photobox Group
Giorgio Bonfiglio
Sr. TAM
AWS

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
Photobox’s journey to AWS
History of a large photo library
Moving to Amazon Simple Storage Service (Amazon S3)
Challenges in large-scale migrations
Re-designing the storage and serving architecture
What’s next?

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Group history

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Photobox’s journey to AWS
Started moving in 2012
the web platform was 100% on AWS by 2015
a lift and shift evolved in a cloud native containerized infrastructure
Photo storage wasn’t immediately migrated
no significant reasons to move
not expected to be cost effective
a migration would have been an hard technical challenge
on-premise equipment was still fairly new and supported

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Growth in photos
During 2017 uploads peaked at 150 millions a month and 1.2 millions in an hour

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Photobox’s photo library
5.7 billion individual photos / 9 Petabytes
• each of them stored in three different resolutions:
• full-size ~ 1.6 MB
• scaled photo < 100 KB
• thumbnail < 10 KB
Growing at a rate of 1.2 B/year
… expected to keep increasing
Peaking at 120k GET/s and 26k PUT/s

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Access patterns
Only 48% of photos uploaded are printed
Of those:
• 41% are printed on the day they are uploaded
• 83% within 30 days
• … and 95% within 180 days

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The on-premise architecture

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why moving?
Cloud first strategy
Photobox had already migrated the website into AWS and improved the operating model
Cost
About to face large capital investments for hardware expansion and replacement
Cloud storage cost kept dropping consistently
Ecosystem
AWS services can be leveraged to enhance the upload and serving workflow

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why moving? (continued)
Improved security
access control in AWS is more flexible and granular than on-premise
Improved resilience and flexibility
any given photo was stored in a single datacenter
uploads were “sharded” between the two datacenters (AMS and PAR)
storage arrays took months to scale up

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How to move?
Early dilemmas: network vs AWS Snowball Edge
with the existing 4 Gbps internet uplink the migration would have taken ¾ of a year
upgrading the production network was slow and expensive
There was no need to move everything
scaled photos and thumbnails can be rebuilt directly into AWS
… or on the fly
Snowball Edge proved to be a good fit
82 TB of useable storage per device
~110 of them needed to carry out the migration

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Integrity checking
After the first use photos are hardly ever accessed again
impossible to spot corrupted objects
the migration strategy/workflow must immediately fix any inconsistency
Hashing comes to the rescue!
in our use case the ETag of objects always matched their MD5 hash (careful: This is not always
true)
easy to query it from Amazon S3 Inventory and compare to what has been computed at the
very beginning of the process

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The migration workflow

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Challenges … at scale
Network congestion
the on-premise network had never been tested at 100%
traffic flows during a migration are “atypical”
Handling an huge amount of small objects
the overhead of every single operation becomes significant
both the data extraction and loading process had to be optimized with this in mind

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Challenges … at scale (continued)
Tracking migration status
needed to follow the status of every single object
… this results in a massive database and heavy data flow
Logistics in the physical world
in EU UPS doesn’t deliver or collect on weekends
next day delivery … doesn’t always happen next day

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Storage tiering
Photos are heavily accessed in the first 90 days
When they are needed access time should be in ms
A mix of Amazon S3 and Amazon S3 Infrequent Access helps achieving
the performance and cost targets
Why not Amazon Glacier?

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Storage architecture
How to choose the right file name?
we had to consider how AWS partitions data in Amazon S3 and our R/W throughput
we used an MD5 hash of the photo name to generate a random prefix
What is the right bucket design?
group by prefix
group by bucket name

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The serving architecture

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Evolution of the serving architecture
Serverless as a natural evolution
Low-management overhead
Can scale instantly
Multi-region is the next big step
Keeping the live user experience consistent in case of a single region outage
Amazon S3 One Zone-Infrequent Access empowers this model

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Evolution of the user journey
Amazon S3 is only the foundation
Now that the photo assets are in AWS they can be retrieved/used right away
A world of possibilities
Amazon Rekognition can be leveraged to radically change Photobox’s customer journey
Magic Book
Using AI to curating customer photos to enhance their experience

31. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Chris Astall
chris.astall@photobox.com
Giorgio Bonfiglio
bonfigg@amazon.com

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.