In this session, we will walk through the Amazon VPC network presentation and describe the problems we were trying to solve when we created it. Next, we will discuss how these problems are traditionally solved, and why those solutions are not scalable, inexpensive, or secure enough for AWS. Finally, we will provide an overview of the solution that we've implemented and discuss some of the unique mechanisms that we use to ensure customer isolation, get packets into and out of the network, and support new features like VPC endpoints. AWS Speaker: Steve Seymour, Solution Architect - Amazon Web Services Customer Speaker: Kim Edwards – Network Engineering, Absa

1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Steve Seymour – Principal Solutions Architect, AWS
Kim Edwards – Network Engineering, ABSA
July 2017
A day in the life of a billion packets
@sseymour

2. Cloudfront
EC2
Instance
Direct
Connect
Availability Zone “a”
VPC subnet
172.31.0.0/24
EC2
Instance
Availability Zone “b”
VPC subnet
172.31.1.0/24
172.31.0.0/16
Your
Data Center
Your
Users
Edge to Instance

3. Cloudfront
EC2
Instance
Direct
Connect
Availability Zone “a”
VPC subnet
172.31.0.0/24
EC2
Instance
Availability Zone “b”
VPC subnet
172.31.1.0/24
172.31.0.0/16
Your
Data Center
Your
Users
Edge to Instance – Direct Connect

4. AWS Direct Connect
• Dedicated, private connection into AWS
• 1 Gbps or 10 Gbps connections
• Create private (VPC) or public virtual interfaces to AWS
• Consistent network performance
• Option for redundant connections
• Uses BGP to exchange routing information over a VLAN

5. AWS Direct Connect
AWS Region
Direct Connect
Location
16 Regions - 60 Direct Connect Locations

6. Cloudfront
EC2
Instance
Direct
Connect
Availability Zone “a”
VPC subnet
172.31.0.0/24
EC2
Instance
Availability Zone “b”
VPC subnet
172.31.1.0/24
172.31.0.0/16
Your
Data Center
Your
Users
Edge to Instance - CloudFront

7. The Amazon CloudFront Service
• Global Content Delivery Network with Massive Capacity and Scale
• Optimized for Performance and Scale
• Built in Security Features
• Self-Service Full Control Configurations
• Robust Real Time Reporting
Amazon
CloudFront
• Static and Dynamic Object and Video Delivery

8. Edge
location
AWS Region /
Regional Edge Cache
Regional Edge
Cache
North America
Cities: 19
PoPs: 27
Europe / Middle East / Africa
Cities: 15
PoPs: 24
Amsterdam, The Netherlands (2)
Berlin, Germany
Dublin, Ireland
Frankfurt, Germany (5)
London, England (4)
Madrid, Spain
Marseille, France
Milan, Italy
Munich, Germany
Paris, France (2)
Prague, Czech Republic
Stockholm, Sweden
Vienna, Austria
Warsaw, Poland
Zurich, Switzerland
Ashburn, VA (3)
Atlanta, GA (3)
Chicago, IL
Dallas/Fort Worth, TX (3)
Hayward, CA
Jacksonville, FL
Los Angeles, CA (2)
Miami, FL
Minneapolis, MN
Montreal, QC
Newark, NJ
New York, NY (3)
Palo Alto, CA
Philadelphia, PA
San Jose, CA
Seattle, WA (2)
South Bend, IN
St. Louis, MO
Toronto, ON
CloudFront Regional Edge Caches
Regional Edge Caches: 11
Oregon, N. Virginia, Ohio, Frankfurt,
London, Sao Paulo, Mumbai, Singapore,
Seoul, Tokyo, Sydney
Asia Pacific
Cities: 12
PoPs: 20
Chennai, India
Hong Kong, China (3)
Manila, the Philippines
Melbourne, Australia
Mumbai, India (2)
New Delhi, India
Osaka, Japan
Seoul, Korea (3)
Singapore (2)
Sydney, Australia
Taipei, Taiwan
Tokyo, Japan (4)
South America
Cities: 2
PoPs: 3
Rio de Janeiro, Brazil (2)
São Paulo, Brazil
CloudFront Global Content Delivery Network
88 Edge Locations - 77 PoPs, 11 Regional Edge Caches (20 in last 12 months)

9. Cloudfront
EC2
Instance
Direct
Connect
Availability Zone “a”
VPC subnet
172.31.0.0/24
EC2
Instance
Availability Zone “b”
VPC subnet
172.31.1.0/24
172.31.0.0/16
Your
Data Center
Your
Users
Edge to Instance – Global Network

13. Cloudfront
EC2
Instance
Direct
Connect
Availability Zone “a”
VPC subnet
172.31.0.0/24
EC2
Instance
Availability Zone “b”
VPC subnet
172.31.1.0/24
172.31.0.0/16
Your
Data Center
Your
Users
Edge to Instance - Region

16. Cloudfront
Direct
Connect VPC subnet
172.31.0.0/24
VPC subnet
172.31.1.0/24
172.31.0.0/16
Your
Data Center
Your
Users
Edge to Instance – Availability Zones
EC2
Instance
EC2
Instance
Availability Zone “a”
Availability Zone “b”

24. Cloudfront
Direct
Connect VPC subnet
172.31.0.0/24
VPC subnet
172.31.1.0/24
172.31.0.0/16
Your
Data Center
Your
Users
Edge to Instance – EC2 Instances
EC2
Instance
EC2
Instance
Availability Zone “a”
Availability Zone “b”

28. Unrestricted distribution
AWS and ABSA – Network Journey
ABSA Network Architecture and Engineering
5 July 2017
Kim Edwards

29. Unrestricted distribution
Why AWS?
29 | AWS and ABSA - Network Journey 5 July 2017
SPEED
We want to deploy
Infrastructure
Services and more...
FASTER!
1
COST
We don’t want to pay
for services when we
no longer need them.
2
FLEXIBILITY
We want to be able to
adapt to changing
requirements without
being locked into
hardware
3

30. Unrestricted distribution
First Deployment of “Bank-connected” VPC
30 | AWS and ABSA - Network Journey 5 July 2017
VPC
Subnet X.X.X.X/X
DMZ
Trusted Network
ABSA
InternetVPN
Internet VPN (IPSec VPN)
Static Routes only
No IGW in VPC
No Custom Route Table
No Expenditure
Used existing hardware and links
Very Restricted Access
Communications can only be initiated
from Bank
Requirements Vague
No Automation

31. Unrestricted distribution
First Deployment of “Bank-connected” VPC
31 | AWS and ABSA - Network Journey 5 July 2017
VPC
Subnet X.X.X.X/X
DMZ
Trusted Network
ABSA
InternetVPN
Internet VPN (IPSec VPN)
Static Routes only
No IGW in VPC
No Custom Route Table
No Expenditure
Used existing hardware and links
Very Restricted Access
Communications can only be initiated
from Bank
Requirements Vague
No Automation

32. Unrestricted distribution
Second Deployment of “Bank-connected” VPC’s
32 | AWS and ABSA - Network Journey 5 July 2017
Internet VPN’s
Connections for 2 VPC’s via 1 ISP
New Network for Developers
Brand new environment for innovation
VPC Peering
Communications between different VPC’s a first
Internet Gateway in one VPC
Security Architecture well-defined and implemented
Bi-directional Flow
Security Groups and NACL’s allow for more “open”
communications
Automated provisioning of AWS
“Infrastructure” begins!
DevOps Network
Trusted Network
ABSA
Internet VPN
VPC B
Subnet Y.Y.Y.Y/Y
VPC A
Subnet X.X.X.X/X
Internet VPN

33. Unrestricted distribution
Third Deployment of “Bank-connected” VPC’s
33 | AWS and ABSA - Network Journey 5 July 2017
ABSA
Network
ABSA
Network
Firewall
ISP Router ISP Router
Router Router
ISPAWS
MPLS
Firewall
Layer 2 handoff
BGP Session
IPSec VPN Tunnel IPSec VPN Tunnel
VPC
Subnet X.X.X.X/X
Public Virtual
Interface
Public Virtual
Interface
ABSA
Diverse Paths
• Route
diversity
provided
No Internet
Gateway
• New Trust
model
needed for
dedicated
links
First DX
Deployment
• Deployed
in one DC
• Used Public
VIF’s
IPsec VPN
• Traffic in
transit
encrypted
Static Routes
to ISP only
• Still no
BGP

34. Unrestricted distribution
What’s Next?
34 | AWS and ABSA - Network Journey 5 July 2017
ABSA Data Centre
ABSA
ABSA Data Centre
ABSA
ABSA Data Centre
ABSA
CloudConnect Layer
Shared services VPC
Subnet Z.Z.Z.Z/X
TRANSIT VPC
Subnet X.X.X.X/X
AZ 2
CSR 2
AZ 1
CSR 1
Spoke VPC
Subnet Y.Y.Y.Y/X
Spoke VPC
Subnet Y.Y.Y.Y/X
Spoke VPC
Subnet Y.Y.Y.Y/X
Target Architecture is now clearly defined
Build Dedicated CloudConnect Layer in all
Data Centers
Secure High Performance Network to be
deployed
Add 2 New Providers for DX Connectivity
Increase level of availability and DR for
Network
Deploy Transit VPC for Production
Enable transitive routing in AWS and dynamic
routing between Bank as well
Automate!
Provisioning to be as automated as possible

35. Unrestricted distribution
Lessons Learnt
35 | AWS and ABSA - Network Journey 5 July 2017
•Start Small
•Don’t wait for perfection before you begin
•Fail Fast

36. Unrestricted distribution
36 | AWS and ABSA - Network Journey 5 July 2017
Thank you!

37. Cloudfront
Direct
Connect VPC subnet
172.31.0.0/24
VPC subnet
172.31.1.0/24
172.31.0.0/16
Your
Data Center
Your
Users
Edge to Instance – VPC
EC2
Instance
EC2
Instance
Availability Zone “a”
Availability Zone “b”

38. VPC Requirements
Customer selected IP addresses
Route aggregation for external connectivity
Conformance with existing network designs

39. 172.31.0.0/18
192.168.0.0/16
Routing Table
• 192.168.0.0/16: stay here
• 172.31.0.0/18: AWS
172.31.1.0/24 172.31.2.0/24
172.31.1.7
172.31.1.8
172.31.1.9
172.31.2.12
172.31.2.51
Amazon Virtual Private Cloud

40. This Is Just Virtual Networking!
Subnet ~= VLAN
VPC ~= VRF (virtual routing and forwarding)
But…

41. Scaling Challenges
VLAN ID space is constrained
• 12 bits => 4096 total VLANs
VRF support is constrained
• Large routers => 1-2 thousand VRFs
Fixed ratio of VLANs:VRFs

42. Implementation Requirements
Scale to millions of environments the size of Amazon.com
Any server, anywhere in a region can host an instance
attached to any subnet in any VPC

43. Server:
Physical host in an
Amazon data center
Instance:
Amazon EC2
instance owned by a
customer
VPC:
Amazon Virtual
Private Cloud
owned by a
customer
VPC ID:
Identifier for a VPC
such as vpc-
1a2b3c4d
Mapping Service:
Distributed lookup
service. Maps VPC
+ Instance IP to
server
Concepts
Server 192.168.0.3
Server 192.168.0.4
…
Server 192.168.1.3
Server 192.168.1.4
…
10.0.0.2
10.0.0.3
10.0.0.4
10.0.0.4
10.0.0.2
10.0.0.5
10.0.0.3
Mapping Service

44. L2 Src: MAC(10.0.0.2)
L2 Dst: ff:ff:ff:ff:ff:ff
ARP Who has
10.0.0.3?
The switch floods the
ARP request out all
ports
L2 Src: MAC(10.0.0.3)
L2 Dst: MAC(10.0.0.2)
ARP 10.0.0.3 is at
MAC(10.0.0.3)
The switch snoops the
ARP response and
learns the port for
MAC(10.0.0.3).
L2 Src: MAC(10.0.0.2)
L2 Dst: MAC(10.0.0.3)
L3 Src: 10.0.0.2
L3 Dst: 10.0.0.3
ICMP/TCP/UDP/…
Layer 2 (L2): Ethernet
10.0.0.2
10.0.0.3
Ethernet Switch

45. L2 Src: MAC(10.0.0.3)
L2 Dst: MAC(10.0.0.2)
ARP 10.0.0.3 is at
MAC(10.0.0.3)
Src: 192.168.0.3
Dst: Mapping Service
Query:
Blue 10.0.0.3
Src: Mapping Service
Dst: 192.168.0.3
Reply:
Host: 192.168.1.4
MAC: MAC(10.0.0.3)
L2 Src: MAC(10.0.0.2)
L2 Dst: ff:ff:ff:ff:ff:ff
ARP Who has
10.0.0.3?
Layer 2 (L2): VPC
Server 192.168.0.3
Server 192.168.0.4
…
Server 192.168.1.3
Server 192.168.1.4
10.0.0.3
10.0.0.4
10.0.0.4
10.0.0.2
10.0.0.5
10.0.0.3
Mapping Service
10.0.0.2

46. Src: Mapping Service
Dst: 192.168.1.4
Mapping valid:
Blue 10.0.0.2 is at
192.168.0.3
Src: 192.168.1.4
Dst: Mapping Service
Validate:
Blue 10.0.0.2 is at
192.168.0.3
L2 Src: MAC(10.0.0.2)
L2 Dst: MAC(10.0.0.3)
L3 Src: 10.0.0.2
L3 Dst: 10.0.0.3
ICMP/TCP/UDP/…
Src: 192.168.0.3
Dst: 192.168.1.4
VPC: Blue
Server 192.168.0.3
Server 192.168.0.4
Server 192.168.1.3
Server 192.168.1.4
10.0.0.3
10.0.0.4
10.0.0.4
10.0.0.2
10.0.0.5
10.0.0.3
Mapping Service
10.0.0.2
Layer 2 (L2): VPC
…

47. Src: 192.168.0.4
Dst: Mapping Service
Query:
Grey 10.0.0.3
L2 Src: MAC(10.0.0.4)
L2 Dst: ff:ff:ff:ff:ff:ff
ARP Who has
10.0.0.3?
VPC Isolation
Server 192.168.0.3
Server 192.168.0.4
…
Server 192.168.1.3
Server 192.168.1.4
10.0.0.3
10.0.0.4
10.0.0.4
10.0.0.2
10.0.0.5
10.0.0.3
Mapping Service
10.0.0.2

48. 192.168.0.4 is not
hosting any instances
in VPC Blue.
Mapping Denied
Alarm Raised
L2 Src: MAC(10.0.0.4)
L2 Dst: ff:ff:ff:ff:ff:ff
ARP Who has
10.0.0.3?
Src: 192.168.0.4
Dst: Mapping Service
Query:
Blue 10.0.0.3
VPC Isolation
Server 192.168.0.3
Server 192.168.0.4
…
Server 192.168.1.3
Server 192.168.1.4
10.0.0.3
10.0.0.4
10.0.0.4
10.0.0.2
10.0.0.5
10.0.0.3
Mapping Service
10.0.0.2

49. Src: 192.168.1.4
Dst: Mapping Service
Validate:
Blue 10.0.0.4 is at
192.168.0.4
Src: 192.168.0.4
Dst: 192.168.1.4
L2 Src: MAC(10.0.0.4)
L2 Dst: MAC(10.0.0.3)
L3 Src: 10.0.0.4
L3 Dst: 10.0.0.3
ICMP/TCP/UDP/…
VPC: Blue
Src: Mapping Service
Dst: 192.168.1.4
Mapping invalid!
192.168.1.4 does not
deliver the packet to
the instance.
Alarm Raised.
VPC Isolation
Server 192.168.0.3
Server 192.168.0.4
…
Server 192.168.1.3
Server 192.168.1.4
10.0.0.3
10.0.0.4
10.0.0.4
10.0.0.2
10.0.0.5
10.0.0.3
Mapping Service
10.0.0.2

50. L2 Src: MAC(10.0.0.2)
L2 Dst: ff:ff:ff:ff:ff:ff
ARP Who has
10.0.0.1?
L2 Src: MAC(10.0.0.1)
L2 Dst: MAC(10.0.0.2)
ARP 10.0.0.1 is at
MAC(10.0.0.1)
L2 Src: MAC(10.0.0.2)
L2 Dst: MAC(10.0.0.1)
L3 Src: 10.0.0.2
L3 Dst: 10.0.1.3
ICMP/TCP/UDP/…
Layer 3 (L3): IP Routing
10.0.0.2
10.0.1.3
Ethernet Switch
Router
Ethernet Switch
L2 Src: MAC(10.0.1.1)
L2 Dst: MAC(10.0.1.3)
L3 Src: 10.0.0.2
L3 Dst: 10.0.1.3
ICMP/TCP/UDP/…

51. L2 Src: MAC(10.0.0.2)
L2 Dst: ff:ff:ff:ff:ff:ff
ARP Who has
10.0.0.1?
L2 Src: MAC(10.0.0.1)
L2 Dst: MAC(10.0.0.2)
ARP 10.0.0.1 is at
MAC(10.0.0.1)
Src: 192.168.0.3
Dst: Mapping Service
Query:
Blue 10.0.0.1
Src: Mapping Service
Dst: 192.168.0.3
Reply:
Host: Gateway
MAC: MAC(10.0.0.1)
Layer 3 (L3): VPC
Server 192.168.0.3
Server 192.168.0.4
…
Server 192.168.1.3
Server 192.168.1.4
10.0.1.3
10.0.0.4
10.0.0.4
10.0.0.2
10.0.0.5
10.0.0.3
Mapping Service
10.0.0.2

52. Src: Mapping Service
Dst: 192.168.0.3
Reply:
Host: 192.168.1.4
MAC: MAC(10.0.1.3)
Src: 192.168.1.4
Dst: Mapping Service
Validate:
Blue 10.0.0.2 is at
192.168.0.3
L2 Src: MAC(10.0.0.2)
L2 Dst: MAC(10.0.0.1)
L3 Src: 10.0.0.2
L3 Dst: 10.0.1.3
ICMP/TCP/UDP/…
L2 Src: MAC(10.0.1.1)
L2 Dst: MAC(10.0.1.3)
L3 Src: 10.0.0.2
L3 Dst: 10.0.1.3
ICMP/TCP/UDP/…
Src: Mapping Service
Dst: 192.168.1.4
Mapping valid:
Blue 10.0.0.2 is at
192.168.0.3
Layer 3 (L3): VPC
Server 192.168.0.3
Server 192.168.0.4
…
Server 192.168.1.3
Server 192.168.1.4
10.0.1.3
10.0.0.4
10.0.0.4
10.0.0.2
10.0.0.5
10.0.0.3
Mapping Service
Src: 192.168.0.3
Dst: Mapping Service
Query:
Blue 10.0.1.3
10.0.0.2
VPC: Blue
Src: 192.168.0.3
Dst: 192.168.1.4

53. Caching
Server 192.168.0.3
Server 192.168.0.4
…
Server 192.168.1.3
Server 192.168.1.4
…
10.0.0.2
10.0.0.3
10.0.0.4
10.0.0.4
10.0.0.2
10.0.0.5
10.0.0.3
Mapping Service
L2 Src: MAC(10.0.1.1)
L2 Dst: MAC(10.0.1.3)
L3 Src: 10.0.0.2
L3 Dst: 10.0.1.3
ICMP/TCP/UDP/…

54. 10.0.0.0/18
172.16.0.0/16
10.0.0.0/24 10.0.1.0/24
10.0.0.7
10.0.0.8
10.0.0.9
10.0.1.12
10.0.1.51
Getting Home – Or Anywhere, Really
VPC: Blue
Src: 192.168.0.3
Dst: ???
L3 Src: 10.0.0.7
L3 Dst: 172.16.14.17
ICMP/TCP/UDP/…

55. Src: 192.168.0.3
Dst: 192.168.4.3
L3 Src: 10.0.0.2
L3 Dst: 172.16.14.17
ICMP/TCP/UDP/…
L3 Src: 10.0.0.2
L3 Dst: 172.16.14.17
ICMP/TCP/UDP/…
Edges
Server 192.168.0.3
Server 192.168.0.4
…
Edge 192.168.4.3
Edge 192.168.4.4
10.0.1.3
10.0.0.4
10.0.0.2
Mapping Service
10.0.0.2
VPC: Blue
Host 10.0.0.4 è 192.168.0.4
Host 10.0.1.4 è 192.168.0.4
…
172.16.0.0/16 è Edge 192.168.4.3
…

56. Edges: VPN
Edge 192.168.4.3
VPC: Blue
Src: 192.168.0.3
Dst: 192.168.4.3
L3 Src: 10.0.0.2
L3 Dst: 172.16.14.17
ICMP/TCP/UDP/…
IPSEC Stuff
Src: 54.68.100.245
Dst: 205.251.242.54
L3 Src: 10.0.0.2
L3 Dst: 172.16.14.17
ICMP/TCP/UDP/…
VPN

57. Edges: Direct Connect
Edge 192.168.4.3
VPC: Blue
Src: 192.168.0.3
Dst: 192.168.4.3
L3 Src: 10.0.0.2
L3 Dst: 172.16.14.17
ICMP/TCP/UDP/…
802.1Q VLAN Tag
Src: 54.68.100.245
Dst: 205.251.242.54
L3 Src: 10.0.0.2
L3 Dst: 172.16.14.17
ICMP/TCP/UDP/…
AWS
Direct Connect

58. Edges: Internet (IGW)
Edge 192.168.4.3
VPC: Blue
Src: 192.168.0.3
Dst: 192.168.4.3
L3 Src: 10.0.0.2
L3 Dst: 176.32.96.190
ICMP/TCP/UDP/…
L3 Src: 10.0.0.2
L3 Dst: 176.32.96.190
ICMP/TCP/UDP/…
Internet
54.148.157.46

59. Edges: Recap
VPN
Edge 192.168.4.3
VPC: Blue
Src: 192.168.0.3
Dst: 192.168.4.3
L3 Src: 10.0.0.2
L3 Dst: 172.16.14.17
ICMP/TCP/UDP/…
IPSEC Stuff
Src: 54.68.100.245
Dst: 205.251.242.54
L3 Src: 10.0.0.2
L3 Dst: 172.16.14.17
ICMP/TCP/UDP/…
Direct Connect
Edge 192.168.4.3
VPC: Blue
Src: 192.168.0.3
Dst: 192.168.4.3
L3 Src: 10.0.0.2
L3 Dst: 172.16.14.17
ICMP/TCP/UDP/…
802.1Q VLAN Tag
Src: 54.68.100.245
Dst: 205.251.242.54
L3 Src: 10.0.0.2
L3 Dst: 172.16.14.17
ICMP/TCP/UDP/…
Internet
Edge 192.168.4.3
VPC: Blue
Src: 192.168.0.3
Dst: 192.168.4.3
L3 Src: 10.0.0.2
L3 Dst: 176.32.96.190
ICMP/TCP/UDP/…
L3 Src: 54.148.157.46
L3 Dst: 176.32.96.190
ICMP/TCP/UDP/…`

60. 172.31.0.0/18
172.31.1.0/24 172.31.2.0/24
172.31.1.7
172.31.1.8
172.31.2.12
172.31.2.51
VPC As A Platform

61. Cloudfront
EC2
Instance
Direct
Connect
Availability Zone “a”
VPC subnet
172.31.0.0/24
EC2
Instance
Availability Zone “b”
VPC subnet
172.31.1.0/24
172.31.0.0/16
Your
Data Center
Your
Users
Edge to Instance

62. Cloudfront
EC2
Instance
Direct
Connect
Availability Zone “a”
VPC subnet
172.31.0.0/24
EC2
Instance
Availability Zone “b”
VPC subnet
172.31.1.0/24
172.31.0.0/16
Your
Data Center
Your
Users
Edge to Instance

63. Thank you!
@sseymour