Edge to Instance - AWS Networking

1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Steve Seymour, Principal Solutions Architect
Networking Specialist, EMEA
June 2017
Edge to Instance - AWS Networking
Media & Entertainment Symposium
@sseymour

2. Cloudfront
EC2
Instance
Direct
Connect
Availability Zone “a”
VPC subnet
172.31.0.0/24
EC2
Instance
Availability Zone “b”
VPC subnet
172.31.1.0/24
172.31.0.0/16
Your
Data Center
Your
Users
Edge to Instance

3. Cloudfront
EC2
Instance
Direct
Connect
Availability Zone “a”
VPC subnet
172.31.0.0/24
EC2
Instance
Availability Zone “b”
VPC subnet
172.31.1.0/24
172.31.0.0/16
Your
Data Center
Your
Users
Edge to Instance - Cloudfront

4. Edge
location
AWS Region /
Regional Edge Cache
Regional Edge
Cache
North America
Cities: 19
PoPs: 27
Europe / Middle East / Africa
Cities: 15
PoPs: 24
Amsterdam, The Netherlands (2)
Berlin, Germany
Dublin, Ireland
Frankfurt, Germany (5)
London, England (4)
Madrid, Spain
Marseille, France
Milan, Italy
Munich, Germany
Paris, France (2)
Prague, Czech Republic
Stockholm, Sweden
Vienna, Austria
Warsaw, Poland
Zurich, Switzerland
Ashburn, VA (3)
Atlanta, GA (3)
Chicago, IL
Dallas/Fort Worth, TX (3)
Hayward, CA
Jacksonville, FL
Los Angeles, CA (2)
Miami, FL
Minneapolis, MN
Montreal, QC
Newark, NJ
New York, NY (3)
Palo Alto, CA
Philadelphia, PA
San Jose, CA
Seattle, WA (2)
South Bend, IN
St. Louis, MO
Toronto, ON
Asia Pacific
Cities: 12
PoPs: 20
Chennai, India
Hong Kong, China (3)
Manila, the Philippines
Melbourne, Australia
Mumbai, India (2)
New Delhi, India
Osaka, Japan
Seoul, Korea (3)
Singapore (2)
Sydney, Australia
Taipei, Taiwan
Tokyo, Japan (4)
South America
Cities: 2
PoPs: 3
Rio de Janeiro, Brazil (2)
São Paulo, Brazil
CloudFront Global Content Delivery Network
88 Edge Locations - 77 PoPs, 11 Regional Edge Caches (20 in last 12 months)

5. Cloudfront
EC2
Instance
Direct
Connect
Availability Zone “a”
VPC subnet
172.31.0.0/24
EC2
Instance
Availability Zone “b”
VPC subnet
172.31.1.0/24
172.31.0.0/16
Your
Data Center
Your
Users
Edge to Instance – Direct Connect

6. AWS Direct Connect
AWS Region
Direct Connect
Location
16 Regions - 58 Direct Connect Locations

7. Cloudfront
EC2
Instance
Direct
Connect
Availability Zone “a”
VPC subnet
172.31.0.0/24
EC2
Instance
Availability Zone “b”
VPC subnet
172.31.1.0/24
172.31.0.0/16
Your
Data Center
Your
Users
Edge to Instance – Global Network

11. Cloudfront
EC2
Instance
Direct
Connect
Availability Zone “a”
VPC subnet
172.31.0.0/24
EC2
Instance
Availability Zone “b”
VPC subnet
172.31.1.0/24
172.31.0.0/16
Your
Data Center
Your
Users
Edge to Instance - Region

14. Cloudfront
Direct
Connect VPC subnet
172.31.0.0/24
VPC subnet
172.31.1.0/24
172.31.0.0/16
Your
Data Center
Your
Users
Edge to Instance – Availability Zones
EC2
Instance
EC2
Instance
Availability Zone “a”
Availability Zone “b”

22. Cloudfront
Direct
Connect VPC subnet
172.31.0.0/24
VPC subnet
172.31.1.0/24
172.31.0.0/16
Your
Data Center
Your
Users
Edge to Instance – EC2 Instances
EC2
Instance
EC2
Instance
Availability Zone “a”
Availability Zone “b”

26. Cloudfront
Direct
Connect VPC subnet
172.31.0.0/24
VPC subnet
172.31.1.0/24
172.31.0.0/16
Your
Data Center
Your
Users
Edge to Instance – VPC
EC2
Instance
EC2
Instance
Availability Zone “a”
Availability Zone “b”

27. VPC Requirements
Customer selected IP addresses
Route aggregation for external connectivity
Conformance with existing network designs

28. 172.31.0.0/18
192.168.0.0/16
Routing Table
•192.168.0.0/16: stay here
• 172.31.0.0/18: AWS
172.31.1.0/24 172.31.2.0/24
172.31.1.7
172.31.1.8
172.31.1.9
172.31.2.12
172.31.2.51
Amazon Virtual Private Cloud

29. This Is Just Virtual Networking!
Subnet ~= VLAN
VPC ~= VRF (virtual routing and forwarding)
But…

30. Scaling Challenges
VLAN ID space is constrained
• 12 bits => 4096 total VLANs
VRF support is constrained
• Large routers => 1-2 thousand VRFs
Fixed ratio of VLANs:VRFs

31. Implementation Requirements
Scale to millions of environments the size of Amazon.com
Any server, anywhere in a region can host an instance
attached to any subnet in any VPC

32. Server:
Physical host in an
Amazon data center
Instance:
Amazon EC2
instance owned by a
customer
VPC:
Amazon Virtual
Private Cloud
owned by a
customer
VPC ID:
Identifier for a VPC
such as vpc-
1a2b3c4d
Mapping Service:
Distributed lookup
service. Maps VPC
+ Instance IP to
server
Concepts
Server 192.168.0.3
Server 192.168.0.4
…
Server 192.168.1.3
Server 192.168.1.4
…
10.0.0.2
10.0.0.3
10.0.0.4
10.0.0.4
10.0.0.2
10.0.0.5
10.0.0.3
Mapping Service

33. L2 Src: MAC(10.0.0.2)
L2 Dst: ff:ff:ff:ff:ff:ff
ARP Who has
10.0.0.3?
The switch floods the
ARP request out all
ports
L2 Src: MAC(10.0.0.3)
L2 Dst: MAC(10.0.0.2)
ARP 10.0.0.3 is at
MAC(10.0.0.3)
The switch snoops the
ARP response and
learns the port for
MAC(10.0.0.3).
L2 Src: MAC(10.0.0.2)
L2 Dst: MAC(10.0.0.3)
L3 Src: 10.0.0.2
L3 Dst: 10.0.0.3
ICMP/TCP/UDP/…
Layer 2 (L2): Ethernet
10.0.0.2
10.0.0.3
Ethernet Switch

34. L2 Src: MAC(10.0.0.3)
L2 Dst: MAC(10.0.0.2)
ARP 10.0.0.3 is at
MAC(10.0.0.3)
Src: 192.168.0.3
Dst: Mapping Service
Query:
Blue 10.0.0.3
Src: Mapping Service
Dst: 192.168.0.3
Reply:
Host: 192.168.1.4
MAC: MAC(10.0.0.3)
L2 Src: MAC(10.0.0.2)
L2 Dst: ff:ff:ff:ff:ff:ff
ARP Who has
10.0.0.3?
Layer 2 (L2): VPC
Server 192.168.0.3
Server 192.168.0.4
…
Server 192.168.1.3
Server 192.168.1.4
10.0.0.3
10.0.0.4
10.0.0.4
10.0.0.2
10.0.0.5
10.0.0.3
Mapping Service
10.0.0.2

35. Src: Mapping Service
Dst: 192.168.1.4
Mapping valid:
Blue 10.0.0.2 is at
192.168.0.3
Src: 192.168.1.4
Dst: Mapping Service
Validate:
Blue 10.0.0.2 is at
192.168.0.3
L2 Src: MAC(10.0.0.2)
L2 Dst: MAC(10.0.0.3)
L3 Src: 10.0.0.2
L3 Dst: 10.0.0.3
ICMP/TCP/UDP/…
Src: 192.168.0.3
Dst: 192.168.1.4
VPC: Blue
Server 192.168.0.3
Server 192.168.0.4
Server 192.168.1.3
Server 192.168.1.4
10.0.0.3
10.0.0.4
10.0.0.4
10.0.0.2
10.0.0.5
10.0.0.3
Mapping Service
10.0.0.2
Layer 2 (L2): VPC
…

36. Src: 192.168.0.4
Dst: Mapping Service
Query:
Grey 10.0.0.3
L2 Src: MAC(10.0.0.4)
L2 Dst: ff:ff:ff:ff:ff:ff
ARP Who has
10.0.0.3?
VPC Isolation
Server 192.168.0.3
Server 192.168.0.4
…
Server 192.168.1.3
Server 192.168.1.4
10.0.0.3
10.0.0.4
10.0.0.4
10.0.0.2
10.0.0.5
10.0.0.3
Mapping Service
10.0.0.2

37. 192.168.0.4 is not
hosting any instances
in VPC Blue.
Mapping Denied
Alarm Raised
L2 Src: MAC(10.0.0.4)
L2 Dst: ff:ff:ff:ff:ff:ff
ARP Who has
10.0.0.3?
Src: 192.168.0.4
Dst: Mapping Service
Query:
Blue 10.0.0.3
VPC Isolation
Server 192.168.0.3
Server 192.168.0.4
…
Server 192.168.1.3
Server 192.168.1.4
10.0.0.3
10.0.0.4
10.0.0.4
10.0.0.2
10.0.0.5
10.0.0.3
Mapping Service
10.0.0.2

38. Src: 192.168.1.4
Dst: Mapping Service
Validate:
Blue 10.0.0.4 is at
192.168.0.4
Src: 192.168.0.4
Dst: 192.168.1.4
L2 Src: MAC(10.0.0.4)
L2 Dst: MAC(10.0.0.3)
L3 Src: 10.0.0.4
L3 Dst: 10.0.0.3
ICMP/TCP/UDP/…
VPC: Blue
Src: Mapping Service
Dst: 192.168.1.4
Mapping invalid!
192.168.1.4 does not
deliver the packet to
the instance.
Alarm Raised.
VPC Isolation
Server 192.168.0.3
Server 192.168.0.4
…
Server 192.168.1.3
Server 192.168.1.4
10.0.0.3
10.0.0.4
10.0.0.4
10.0.0.2
10.0.0.5
10.0.0.3
Mapping Service
10.0.0.2

39. L2 Src: MAC(10.0.0.2)
L2 Dst: ff:ff:ff:ff:ff:ff
ARP Who has
10.0.0.1?
L2 Src: MAC(10.0.0.1)
L2 Dst: MAC(10.0.0.2)
ARP 10.0.0.1 is at
MAC(10.0.0.1)
L2 Src: MAC(10.0.0.2)
L2 Dst: MAC(10.0.0.1)
L3 Src: 10.0.0.2
L3 Dst: 10.0.1.3
ICMP/TCP/UDP/…
Layer 3 (L3): IP Routing
10.0.0.2
10.0.1.3
Ethernet Switch
Router
Ethernet Switch
L2 Src: MAC(10.0.1.1)
L2 Dst: MAC(10.0.1.3)
L3 Src: 10.0.0.2
L3 Dst: 10.0.1.3
ICMP/TCP/UDP/…

40. L2 Src: MAC(10.0.0.2)
L2 Dst: ff:ff:ff:ff:ff:ff
ARP Who has
10.0.0.1?
L2 Src: MAC(10.0.0.1)
L2 Dst: MAC(10.0.0.2)
ARP 10.0.0.1 is at
MAC(10.0.0.1)
Src: 192.168.0.3
Dst: Mapping Service
Query:
Blue 10.0.0.1
Src: Mapping Service
Dst: 192.168.0.3
Reply:
Host: Gateway
MAC: MAC(10.0.0.1)
Layer 3 (L3): VPC
Server 192.168.0.3
Server 192.168.0.4
…
Server 192.168.1.3
Server 192.168.1.4
10.0.1.3
10.0.0.4
10.0.0.4
10.0.0.2
10.0.0.5
10.0.0.3
Mapping Service
10.0.0.2

41. Src: Mapping Service
Dst: 192.168.0.3
Reply:
Host: 192.168.1.4
MAC: MAC(10.0.1.3)
Src: 192.168.1.4
Dst: Mapping Service
Validate:
Blue 10.0.0.2 is at
192.168.0.3
L2 Src: MAC(10.0.0.2)
L2 Dst: MAC(10.0.0.1)
L3 Src: 10.0.0.2
L3 Dst: 10.0.1.3
ICMP/TCP/UDP/…
L2 Src: MAC(10.0.1.1)
L2 Dst: MAC(10.0.1.3)
L3 Src: 10.0.0.2
L3 Dst: 10.0.1.3
ICMP/TCP/UDP/…
Src: Mapping Service
Dst: 192.168.1.4
Mapping valid:
Blue 10.0.0.2 is at
192.168.0.3
Layer 3 (L3): VPC
Server 192.168.0.3
Server 192.168.0.4
…
Server 192.168.1.3
Server 192.168.1.4
10.0.1.3
10.0.0.4
10.0.0.4
10.0.0.2
10.0.0.5
10.0.0.3
Mapping Service
Src: 192.168.0.3
Dst: Mapping Service
Query:
Blue 10.0.1.3
10.0.0.2
VPC: Blue
Src: 192.168.0.3
Dst: 192.168.1.4

42. Caching
Server 192.168.0.3
Server 192.168.0.4
…
Server 192.168.1.3
Server 192.168.1.4
…
10.0.0.2
10.0.0.3
10.0.0.4
10.0.0.4
10.0.0.2
10.0.0.5
10.0.0.3
Mapping Service
L2 Src: MAC(10.0.1.1)
L2 Dst: MAC(10.0.1.3)
L3 Src: 10.0.0.2
L3 Dst: 10.0.1.3
ICMP/TCP/UDP/…

43. 10.0.0.0/18
172.16.0.0/16
10.0.0.0/24 10.0.1.0/24
10.0.0.7
10.0.0.8
10.0.0.9
10.0.1.12
10.0.1.51
Getting Home – Or Anywhere, Really
VPC: Blue
Src: 192.168.0.3
Dst: ???
L3 Src: 10.0.0.7
L3 Dst: 172.16.14.17
ICMP/TCP/UDP/…

44. Src: 192.168.0.3
Dst: 192.168.4.3
L3 Src: 10.0.0.2
L3 Dst: 172.16.14.17
ICMP/TCP/UDP/…
L3 Src: 10.0.0.2
L3 Dst: 172.16.14.17
ICMP/TCP/UDP/…
Edges
Server 192.168.0.3
Server 192.168.0.4
…
Edge 192.168.4.3
Edge 192.168.4.4
10.0.1.3
10.0.0.4
10.0.0.2
Mapping Service
10.0.0.2
VPC: Blue
Host 10.0.0.4  192.168.0.4
Host 10.0.1.4  192.168.0.4
…
172.16.0.0/16  Edge
192.168.4.3
…

45. Edges: VPN
Edge 192.168.4.3
VPC: Blue
Src: 192.168.0.3
Dst: 192.168.4.3
L3 Src: 10.0.0.2
L3 Dst: 172.16.14.17
ICMP/TCP/UDP/…
IPSEC Stuff
Src: 54.68.100.245
Dst: 205.251.242.54
L3 Src: 10.0.0.2
L3 Dst: 172.16.14.17
ICMP/TCP/UDP/…
VPN

46. Edges: Direct Connect
Edge 192.168.4.3
VPC: Blue
Src: 192.168.0.3
Dst: 192.168.4.3
L3 Src: 10.0.0.2
L3 Dst: 172.16.14.17
ICMP/TCP/UDP/…
802.1Q VLAN Tag
Src: 54.68.100.245
Dst: 205.251.242.54
L3 Src: 10.0.0.2
L3 Dst: 172.16.14.17
ICMP/TCP/UDP/…
AWS
Direct Connect

47. Edges: Internet (IGW)
Edge 192.168.4.3
VPC: Blue
Src: 192.168.0.3
Dst: 192.168.4.3
L3 Src: 10.0.0.2
L3 Dst: 176.32.96.190
ICMP/TCP/UDP/…
L3 Src: 10.0.0.2
L3 Dst: 176.32.96.190
ICMP/TCP/UDP/…
Internet
54.148.157.46

48. Edges: Recap
VPN
Edge 192.168.4.3
VPC: Blue
Src: 192.168.0.3
Dst: 192.168.4.3
L3 Src: 10.0.0.2
L3 Dst: 172.16.14.17
ICMP/TCP/UDP/…
IPSEC Stuff
Src: 54.68.100.245
Dst: 205.251.242.54
L3 Src: 10.0.0.2
L3 Dst: 172.16.14.17
ICMP/TCP/UDP/…
Direct Connect
Edge 192.168.4.3
VPC: Blue
Src: 192.168.0.3
Dst: 192.168.4.3
L3 Src: 10.0.0.2
L3 Dst: 172.16.14.17
ICMP/TCP/UDP/…
802.1Q VLAN Tag
Src: 54.68.100.245
Dst: 205.251.242.54
L3 Src: 10.0.0.2
L3 Dst: 172.16.14.17
ICMP/TCP/UDP/…
Internet
Edge 192.168.4.3
VPC: Blue
Src: 192.168.0.3
Dst: 192.168.4.3
L3 Src: 10.0.0.2
L3 Dst: 176.32.96.190
ICMP/TCP/UDP/…
L3 Src: 54.148.157.46
L3 Dst: 176.32.96.190
ICMP/TCP/UDP/…`

49. 172.31.0.0/18
172.31.1.0/24 172.31.2.0/24
172.31.1.7
172.31.1.8
172.31.2.12
172.31.2.51
VPC As A Platform

50. Cloudfront
EC2
Instance
Direct
Connect
Availability Zone “a”
VPC subnet
172.31.0.0/24
EC2
Instance
Availability Zone “b”
VPC subnet
172.31.1.0/24
172.31.0.0/16
Your
Data Center
Your
Users
Edge to Instance

51. Cloudfront
EC2
Instance
Direct
Connect
Availability Zone “a”
VPC subnet
172.31.0.0/24
EC2
Instance
Availability Zone “b”
VPC subnet
172.31.1.0/24
172.31.0.0/16
Your
Data Center
Your
Users
Edge to Instance

52. Thank you!
seymours@amazon.co.ukseymours@amazon.co.uk
@sseymour