Pinterest is rolling out a phased platform migration from EC2-Classic to EC2-VPC. We used ClassicLink to link our EC2-Classic instances to VPCs, and we applied AWS best practices to configure VPC subnets and security groups. In this session, we share the lessons we learned along the way, and we also show you how to create a migration strategy and track migration costs.

1. AWS re:Invent© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Jayme Cox, Cloud Computing, Pinterest
Kaartik Viswanath, Product Manager, AWS
October 2015
NET307
Pinterest
The Road from EC2-Classic to EC2-VPC

2. Frequently asked Questions
1. When should I start adopting Amazon VPC?
2. Why should I adopt Amazon VPC?
3. How do I go about the process?

3. AWS re:Invent
Overview
3
How
what tools, processes, and procedures
Why
reasons to migrate to VPC
When
timelines and schedules
Lessons Learned
what to think about before migrating
1
2
3
4
What we will talk about

4. AWS re:Invent 4
•100 million pinners
•150,000 requests/sec at peak
•100% in Amazon EC2
What is Pinterest?
4
We help people discover things they
love, and inspire them to do those
things in their daily lives

5. AWS re:Invent 5
Why
How
When
Lessons

6. AWS re:Invent
Performance
Security
Access
Benefits of VPC
6

7. AWS re:Invent
Enhanced networking
● EC2-Classic: 250,000 pps
● EC2-VPC: 900,000 pps
● EC2-Classic: ~8.5 Gbit/sec
● EC2-VPC: ~9.9 Gbit/sec
Performance
7
make it fast

8. AWS re:Invent
Enhanced networking
● faster!
Internal ELB
● replace DNS roundrobin
● replace haproxy/nginx nodes
● health checks
Performance
8
make it fast

9. AWS re:Invent
Performance
Security
Access
Benefits of VPC
9

10. AWS re:Invent
Network level controls
● Private Amazon S3 endpoint
Security
10
make it secure

11. AWS re:Invent
Network level controls
● Private Amazon S3 endpoint
● Subnets and full IP address control
Security
11
make it secure

12. AWS re:Invent
Network level controls
● Private Amazon S3 endpoint
● Subnets and full IP address control
● Network-level Access Control Lists
Security
12
make it secure

13. AWS re:Invent
Performance
Security
Access
Benefits of VPC
13

14. AWS re:Invent
Access Options
● VPC Peering allows secure connections to multiple VPCs
Access
14
make it available

15. AWS re:Invent
Access Options
● VPC Peering allows secure connections to multiple VPCs
● AWS Direct Connect private connectivity to VPC
Access
15
make it available

16. AWS re:Invent
Access Options
● VPC Peering allows secure connections to multiple VPCs
● AWS Direct Connect private connectivity to VPC
● Virtual Private Gateway allows VPN access
Access
16
make it available

17. AWS re:Invent 17
Why
How
When
Lessons

18. AWS re:Invent
The Pinterest migration to VPC required that
we have zero downtime.
A hard cutover would have duplicated 40%
of our current infrastructure, which was too
expensive.
ClassicLink to the rescue!
18

19. AWS re:Invent
ClassicLink
● Phased migration on a service-by-service basis
Tools
19
make it go

20. AWS re:Invent
ClassicLink
● Phased migration on a service-by-service basis
● VPC address space must not overlap EC2-Classic
Tools
20
make it go

21. AWS re:Invent
ClassicLink
● Phased migration on a service-by-service basis
● VPC address space must not overlap EC2-Classic
● Automate linking classic EC2 nodes to VPC
Tools
21
make it go

22. AWS re:Invent
ClassicLink
● Phased migration on a service-by-service basis
● VPC address space must not overlap EC2-Classic
● Automate linking classic EC2 nodes to VPC
Service discovery
● What services do you need to migrate?
Tools
22
make it go

23. AWS re:Invent
ClassicLink
● Phased migration on a service-by-service basis
● VPC address space must not overlap EC2-Classic
● Automate linking classic EC2 nodes to VPC
Service discovery
● What services do you need to migrate?
● And what services do they depend on?
Tools
23
make it go

24. AWS re:Invent
ClassicLink
● Phased migration on a service-by-service basis
● VPC address space must not overlap EC2-Classic
● Automate linking classic EC2 nodes to VPC
Service discovery
● What services do you need to migrate?
● And what services do they depend on?
Tracking system
● Master ticket for each service
Tools
24
make it go

25. AWS re:Invent
ClassicLink
● Phased migration on a service-by-service basis
● VPC address space must not overlap EC2-Classic
● Automate linking classic EC2 nodes to VPC
Service discovery
● What services do you need to migrate?
● And what services do they depend on?
Tracking system
● Master ticket for each service
● Migration status of each service
Tools
25
make it go

26. AWS re:Invent
ClassicLink
● Phased migration on a service-by-service basis
● VPC address space must not overlap EC2-Classic
● Automate linking classic EC2 nodes to VPC
Service discovery
● What services do you need to migrate?
● And what services do they depend on?
Tracking system
● Master ticket for each service
● Migration status of each service
● Track migration rate
Tools
26
make it go

27. AWS re:Invent
ClassicLink
● Phased migration on a service-by-service basis
● VPC address space must not overlap EC2-Classic
● Automate linking classic EC2 nodes to VPC
Service discovery
● What services do you need to migrate?
● And what services do they depend on?
Tracking system
● Master ticket for each service
● Migration status of each service
● Track migration rate
● Track problems and areas to improve
Tools
27
make it go

28. AWS re:Invent
NAT gateways
● Private subnets require a NAT gateway for egress
● Possible network bottleneck
● Large subnets may need multiple NAT gateways
Design
28
each piece of the puzzle

29. AWS re:Invent
NAT gateways
● Private subnets require a NAT gateway for egress
● Possible network bottleneck
● Large subnets may need multiple NAT gateways
Subnets
● Allow for future growth in each subnet
● Balance size of subnet vs. network segmentation
● Use public subnets for high traffic Internet egress
Design
29
each piece of the puzzle

30. AWS re:Invent
NAT gateways
● Private subnets require a NAT gateway for egress
● Possible network bottleneck
● Large subnets may need multiple NAT gateways
Subnets
● Allow for future growth in each subnet
● Balance size of subnet vs. network segmentation
● Use public subnets for high traffic Internet egress
Security groups
● Plan out based on number of rules allowed
● Contiguous subnets allow for CIDR-based rules
● Plan for ClassicLink access from EC2 private address space
Design
30
each piece of the puzzle

31. AWS re:Invent
Gather information
● Create a template for service owners to fill in
Document
31
the migration process

32. AWS re:Invent
Gather information
● Create a template for service owners to fill in
● Document current setup
Document
32
the migration process

33. AWS re:Invent
Gather information
● Create a template for service owners to fill in
● Document current setup
● Document changes required by VPC
Document
33
the migration process

34. AWS re:Invent
Gather information
● Create a template for service owners to fill in
● Document current setup
● Document changes required by VPC
● Service dependencies
Document
34
the migration process

35. AWS re:Invent
Gather information
● Create a template for service owners to fill in
● Document current setup
● Document changes required by VPC
● Service dependencies
Service migration plan
● Service runbooks and dashboards
Document
35
the migration process

36. AWS re:Invent
Gather information
● Create a template for service owners to fill in
● Document current setup
● Document changes required by VPC
● Service dependencies
Service migration plan
● Service runbooks and dashboards
● Canary and Testing
Document
36
the migration process

37. AWS re:Invent
Gather information
● Create a template for service owners to fill in
● Document current setup
● Document changes required by VPC
● Service dependencies
Service migration plan
● Service runbooks and dashboards
● Canary and Testing
● Full rollout to VPC
Document
37
the migration process

38. AWS re:Invent
Gather information
● Create a template for service owners to fill in
● Document current setup
● Document changes required by VPC
● Service dependencies
Service migration plan
● Service runbooks and dashboards
● Canary and Testing
● Full rollout to VPC
● Rollback plan
Document
38
the migration process

39. AWS re:Invent
Stateful clusters
● Sync to replica hosts in VPC
Architect
39
for each type of service

40. AWS re:Invent
Stateful clusters
● Sync to replica hosts in VPC
● Promote VPC cluster to master
● Decom classic hosts
Architect
40
for each type of service

41. AWS re:Invent
Stateful clusters
● Sync to replica hosts in VPC
● Promote VPC cluster to master
● Decom classic hosts
Stateless
● Mixed pools
● Grow VPC
Architect
41
for each type of service

42. AWS re:Invent
Stateful clusters
● Sync to replica hosts in VPC
● Promote VPC cluster to master
● Decom classic hosts
Stateless
● Mixed pools
● Grow VPC
● Decom classic hosts
Architect
42
for each type of service

43. AWS re:Invent
Stateful clusters
● Sync to replica hosts in VPC
● Promote VPC cluster to master
● Decom classic hosts
Stateless
● Mixed pools
● Grow VPC
● Decom classic hosts
Frontend traffic
● Create new VPC ELB
Architect
43
for each type of service

44. AWS re:Invent
Stateful clusters
● Sync to replica hosts in VPC
● Promote VPC cluster to master
● Decom classic hosts
Stateless
● Mixed pools
● Grow VPC
● Decom classic hosts
Frontend traffic
● Create new VPC ELB
● Register VPC hosts
Architect
44
for each type of service

45. AWS re:Invent
Stateful clusters
● Sync to replica hosts in VPC
● Promote VPC cluster to master
● Decom classic hosts
Stateless
● Mixed pools
● Grow VPC
● Decom classic hosts
Frontend traffic
● Create new VPC ELB
● Register VPC hosts
● Migrated traffic via DNS
Architect
45
for each type of service

46. AWS re:Invent 46
Why
How
When
Lessons

47. AWS re:Invent
Timeline
● Build dependency tree to avoid overlap
● Schedule each service
● Meet weekly to track progress
Coordinate
47
each service

48. AWS re:Invent
Timeline
● Build dependency tree to avoid overlap
● Schedule each service
● Meet weekly to track progress
Tracking
● Tag each service before and after
● Track cost via tags
● Build tooling/dashboard
Coordinate
48
each service

49. AWS re:Invent
Timeline
● Build dependency tree to avoid overlap
● Schedule each service
● Meet weekly to track progress
Tracking
● Tag each service before and after
● Track cost via tags
● Build tooling/dashboard
Service owners
● Add to service owners’ roadmap
● Service owners fill out migration template
● Use master ticket to keep in sync
Coordinate
49
each service

50. AWS re:Invent 50
Why
How
When
Lessons

51. AWS re:Invent
Limit scope of changes
● Try to limit the number of changes during migration
● Do not change application versions, OS revs, etc.
Favorite things
51
to think about

52. AWS re:Invent
Limit scope of changes
● Try to limit the number of changes during migration
● Do not change application versions, OS revs, etc.
Network ACL
● Optional; you may only need security groups
● Tricky with the private S3 endpoint using public IPs
Favorite things
52
to think about

53. AWS re:Invent
Limit scope of changes
● Try to limit the number of changes during migration
● Do not change application versions, OS revs, etc.
Network ACL
● Optional; you may only need security groups
● Tricky with the private S3 endpoint using public IPs
Service discovery
● Difficult to do, so start before migration, it will pay off
● Building the service dependency map will smooth the migration
● Helps to track down port numbers used for security groups
● Build tooling to automate and report
Favorite things
53
to think about

54. AWS re:Invent
Subnets
● Spend time to get it right
● Public vs. Private
● Contiguous IP address space (CIDR)
Favorite things
54
to think about

55. AWS re:Invent
Subnets
● Spend time to get it right
● Public vs. Private
● Contiguous IP address space (CIDR)
Security groups
● Start with deny all
● Work towards open
Favorite things
55
to think about

56. AWS re:Invent
Subnets
● Spend time to get it right
● Public vs. Private
● Contiguous IP address space (CIDR)
Security groups
● Start with deny all
● Work towards open
amazonaws.com
● split DNS works in classic
● but in VPC it only returns public IP
Favorite things
56
to think about

57. AWS re:Invent
Long tail
● ClassicLink allows mixed environment
● Encourage service owners to migrate
Favorite things
57
to think about

58. AWS re:Invent
Long tail
● ClassicLink allows mixed environment
● Encourage service owners to migrate
Cost
● Someone cares about cost
● So track it from the beginning
Favorite things
58
to think about

59. AWS re:Invent
Questions?

60. AWS re:Invent
Thank you!

61. AWS re:Invent
Remember to complete
your evaluations!