Learning how AWS implement AWS VPC CNI

AWS VPC CNI
AWS User Taiwan Group
HungWei Chiu

Bio
• HungWei Chiu(Hwchiu)

• MTS @ Open Networking Foundation (ONF)

• Kubernetes/Container/Linux/Network...etc

• Blog: https://hwchiu.com

• Facebook: 矽⾕⽜的耕⽥筆記

Agenda
• Network Connectivity

• What

• How

• AWS VPC CNI

• What

• Why

• How

Network Connectivity
• How service access outside world

• How service is accessed by other services

• IPAM (IP address management)

• Environments

• Bare metal

• Virtualization

• VM/Contaner

• Orchestrator

• OpenStack/K8s

Bare Metal
Home
https://awei791129.pixnet.net/blog/post/20872246
PPPOE
Laptop
Server
Mobile Phone
Wire
Wireless
Wireless
66.88.99.45
192.168.0.1
192.168.0.12
192.168.0.3
192.168.0.5
Device Router

NAT
• Source NAT (SNAT) and Destination NAT (DNAT)

• SNAT

• Change Source IP

• Internal to external

• DNAT

• Change Destination IP

• External to internal

SNAT
Home
PPPOE
Laptop
Server
Mobile Phone
Wire
Wireless
Wireless
66.88.99.45
192.168.0.1
192.168.0.12
192.168.0.3
192.168.0.5
Device Router
P 8.8.8.8
192.168.0.12
Packet P 8.8.8.8
66.88.99.45
Packet

DNAT(Port Mapping)
Home
PPPOE
Laptop
Server
Mobile Phone
Wire
Wireless
Wireless
66.88.99.45
192.168.0.1
192.168.0.12
192.168.0.3
192.168.0.5
Device Router
P 8.8.8.8
192.168.0.12
Packet P 8.8.8.8
66.88.99.45
Packet

Bare Metal
Home
PPPOE
Laptop
Server
Mobile Phone
Wire
Wireless
Wireless
66.88.99.45
192.168.0.1
10.15.0.12
10.15.0.5
10.15.0.6
Device Router
Wireless AP
Wire
192.168.0.2
10.15.0.2
Router
SNAT/DNAT SNAT/DNAT
Server
192.168.0.3

NAT
• Increase the complexity for debugging

• Decorate the network performance

• Increase the security

• Decrease the accessibility

Container
Home
PPPOE
Server

Wire
66.88.99.45
192.168.0.1
192.168.0.12
Container Router
Container 1

Router
Container 2

Container 3

Linux Bridge

eth0

10.18.0.1
10.18.0.2
10.18.0.12
10.18.0.4

NAT
Home
PPPOE
Server

Wire
66.88.99.45
192.168.0.1
192.168.0.12
Container Router
Container 1

Router
Container 2

Container 3

Linux Bridge

eth0

10.18.0.1
10.18.0.2
10.18.0.12
10.18.0.4
8.8.8.8
10.18.0.2 8.8.8.8
66.88.99.45
8.8.8.8
192.168.0.12

Docker Expose (-p 8080:80)
Home
PPPOE
Server

Wire
66.88.99.45
192.168.0.1
192.168.0.12
Container Router
Container 1

Router
Container 2

Container 3

Linux Bridge

eth0

10.18.0.1
10.18.0.2
10.18.0.12
10.18.0.4
Laptop

Browser

192.168.0.15
10.18.0.4 192.168.0.15
192.168.0.12
192.168.0.15
10.18.0.4
192.168.0.15

Container
• Private subnet by default

• NAT is required

• Docker simplify the DNAY process (-p)

• More and more NAT

Advanced Mode
Home
PPPOE
Server

Wire
66.88.99.45
192.168.0.1
192.168.0.12
Container Router
Container 1

Switch
Container 2

Container 3

eth0

192.168.0.32
192.168.0.28
192.168.0.25
Laptop

Browser

192.168.0.15
192.168.0.28 192.168.0.15
192.168.0.28
192.168.0.15
192.168.0.28
192.168.0.15

Advance Mode
• NAT isn't necessary

• Better performance

• Issue

• How to manage container's IP addresses?

• Con
f
lict?

• Multiple nodes?

Container Clusters
Home
PPPOE
Server

66.88.99.45
192.168.0.1
192.168.0.12
Container Router
Container 1

Router
Container 2

Container 3

Linux Bridge

eth0

10.18.0.1
10.18.0.2
10.18.0.12
10.18.0.4
Server

eth0

Linux Bridge

Container 1

Container 2

192.168.0.15

Same subnet
Home
PPPOE
Server

66.88.99.45
192.168.0.1
192.168.0.12
Container Router
Container 1

Router
Container 2

Container 3

Linux Bridge

eth0

10.18.0.1
10.18.0.2
10.18.0.12
10.18.0.4
Server

eth0

Linux Bridge

Container 1

Container 2

192.168.0.15
10.18.0.1
10.18.0.15 10.18.0.12

Different Subnet
Home
PPPOE
Server

66.88.99.45
192.168.0.1
192.168.0.12
Container Router
Container 1

Router
Container 2

Container 3

Linux Bridge

eth0

10.18.0.1
10.18.0.2
10.18.0.12
10.18.0.4
Server

eth0

Linux Bridge

Container 1

Container 2

192.168.0.15
10.19.0.1
10.19.0.23 10.19.0.15

Across-Node Accessibility
Home
PPPOE
Server

66.88.99.45
192.168.0.1
192.168.0.12
Container Router
Container 1

Router
Container 2

Container 3

Linux Bridge

eth0

10.18.0.1
10.18.0.2
10.18.0.12
10.18.0.4
Server

eth0

Linux Bridge

Container 1

Container 2

192.168.0.15
10.19.0.1
10.19.0.23 10.19.0.15

Across-Node Accessibility
• Have to perform SNAT and DNAT for each
f
low direction.

• Container Cluster(Kubernetes)

• How does K8s solve it?

• CNI (Container Network Interface)

• Tunneling protocol

• VXLAN, IPIP

Tunneling
• IP over IP

• Encapsulate original IP header with additional IP header

• Inner IPv4

• Container to Container

• Outer IPv4

• Node to Node

Example
• NodeA (192.168.0.15)

• Container A (10.19.0.15)

• NodeB (192.168.0.12)

• Container B (10.18.0.2)

• Container A ping Container B

Example
NodeA

tunl

Container

Eth0

NodeB

tunl

Container

Eth0

10.19.0.15 10.18.0.2
192.168.0.12
192.168.0.15

Example
NodeA

tunl

Container

Eth0

NodeB

tunl

Container

Eth0

10.19.0.15 10.18.0.2
192.168.0.12
192.168.0.15
10.19.0.15 10.18.0.2
10.19.0.15 10.18.0.2 192.168.0.15 192.168.0.12
Data
10.19.0.15 10.18.0.2
Data
There are Data now

AWS VPC CNI
• AWS VPC CNI

• AWS VPC

• CNI (Container Network Interface)

• Kubernetes use it to setup the network connectivity

• What are Kubernetes and CNI ?

Kubernetes (container orchestrator)
https://kubernetes.io/blog/2018/07/18/11-ways-not-to-get-hacked/

Kubernetes CNI
Server(K8s Node)

Kubelet
Pod(Sandbox)
CNI(binary)
1
2
3
Server(K8s Node)

Kubelet
Pod(Sandbox)
CNI(binary)
1
2
3
• Executed by Kubelet

• CNI is a standalone binary executable binary

• Help to setup the network connectivity for Sandbox(Pause Container)

AWS VPC CNI
• Goals

• Support high throughput and availability, low latency

• Users must be able to express and enforce network policies and
isolation

• Compare to native EC2 networking and security groups.

AWS VPC CNI
• Goals

• Network operation must be simple and secure.

• Use VPC
f
low logs

• Apply VPC routing polices

• Pod networking should be setup in a matter of seconds

AWS VPC
10.2.0.0/16
Subnet A

10.2.0.0/24
EC2 Instance

10.2.0.5
EC2 Instance

10.2.0.6
EC2 Instance

10.2.0.80
Underlay
Network
Network Tra
ff
ic
Network Tra
ff
ic
Network Tra
ff
ic

AWS VPC and K8S
10.2.0.0/16
Subnet A 10.2.0.0/24
Underlay
Network
Pod
 
Pod
 
Pod
 
Pod
 
Pod
 
Pod
 
EC2 Instance

10.2.0.5
EC2 Instance

10.2.0.6
EC2 Instance

10.2.0.80

Other CNI (IP over IP)
10.2.0.0/16
Subnet A 10.2.0.0/24
Underlay
Network
Pod
 
Pod
 
Pod
 
Pod
 
Pod
 
Pod
 
EC2 Instance

10.2.0.5
EC2 Instance

10.2.0.6
EC2 Instance

10.2.0.80
10.56.2.5 10.56.2.15 10.56.5.5 10.56.5.48 10.56.9.5 10.56.9.25
10.56.9.0/24
10.56.5.0/24
10.56.2.0/24

10.2.0.0/16
Subnet A 10.2.0.0/24
Underlay
Network
Pod
 
Pod
 
Pod
 
Pod
 
Pod
 
Pod
 
EC2 Instance

10.2.0.5
EC2 Instance

10.2.0.6
EC2 Instance

10.2.0.80
10.56.2.5 10.56.2.15 10.56.5.5 10.56.5.48 10.56.9.5 10.56.9.25
10.56.9.0/24
10.56.5.0/24
10.56.2.0/24
10.2.0.5 -> 10.2.0.80

10.2.0.0/16
Subnet A 10.2.0.0/24
Underlay
Network
Pod
 
Pod
 
Pod
 
Pod
 
Pod
 
Pod
 
EC2 Instance

10.2.0.5
EC2 Instance

10.2.0.6
EC2 Instance

10.2.0.80
10.56.2.5 10.56.2.15 10.56.5.5 10.56.5.48 10.56.9.5 10.56.9.25
10.56.9.0/24
10.56.5.0/24
10.56.2.0/24
10.2.0.5 -> 10.2.0.80
Security Group ?

Visibility ?

AWS VPC CNI
10.2.0.0/16
Subnet A 10.2.0.0/24
Underlay
Network
Pod
 
Pod
 
Pod
 
Pod
 
Pod
 
Pod
 
EC2 Instance

10.2.0.5
EC2 Instance

10.2.0.6
EC2 Instance

10.2.0.80
10.2.0.26 10.2.0.16 10.2.0.53 10.2.0.54 10.2.0.82 10.2.0.182
10.2.0.20 -> 10.2.0.82

AWS VPC CNI
10.2.0.0/16
Subnet A 10.2.0.0/24
Underlay
Network
Pod
 
Pod
 
Pod
 
Pod
 
Pod
 
Pod
 
EC2 Instance

10.2.0.5
EC2 Instance

10.2.0.6
EC2 Instance

10.2.0.80
10.2.0.26 10.2.0.16 10.2.0.53 10.2.0.54 10.2.0.82 10.2.0.182
10.2.0.20 -> 10.2.0.82
Security Group

Visibility

AWS VPC CNI
• Requirement

• IPAM (IP addresses management)

• Unique

• Routing rules

Implementation
• Currently

• Each EC2 instance can have multiple elastic network interfaces (ENI)

• ENI can have multiple IPv4/IPv6 addresses.

• EC2-VPC Fabric will deliver the packet to the instance

• The primary ENI IP address is automatically assigned to the interface

• All secondary addresses remain unassigned

• Host owner to con
f
igure them

Components
https://github.com/aws/amazon-vpc-cni-k8s/blob/master/docs/cni-proposal.md

L-IPAMD
• Local IP address Manager (L-IPAM)

• Small and single binary on each host to maintain a warm-pool of
available secondary IP addresses.

L-IPAMD
• Maintaining the warm-pool of available secondary IP addresses

• Number of IP < threshold

• Create a new ENI and attach it to instance

• Allocate all available IP addresses on this new ENI

• Wait for the IP addresses to be ready and then add to warm-pool

• Number of IP > threshold

• Detach a ENI and free it and related IPs

CNI Plugin
• Get a secondary IP address assigned to the instance by L-IPAMD

• Set up the network device

• Host

• Pod(Sandbox)

• Set up the routing rules

• Host

• Pod

AWS VPC CNI
172.31.0.0/16
Subnet A 172.31.0.0/20
Underlay
Network
ENI

IPs
L-IPAMD
IP Pool
VPN CNI
172.31.1.204

AWS VPC CNI
Underlay
Network
ENI

IPs
L-IPAMD
IP Pool
VPN CNI
Pod(Sandbox)

172.31.0.0/16
Subnet A 172.31.0.0/20
172.31.1.204

AWS VPC CNI
Underlay
Network
ENI

IPs
L-IPAMD
IP Pool
VPN CNI
Pod(Sandbox)

veth1
veth2
172.31.0.0/16
Subnet A 172.31.0.0/20
172.31.1.204

AWS VPC CNI
Underlay
Network
ENI

IPs
L-IPAMD
IP Pool
VPN CNI
Pod(Sandbox)

eth0
enixxxx
172.31.15.74/32
172.31.0.0/16
169.254.1.1
Subnet A 172.31.0.0/20
172.31.1.204

Pod ARP/Routing
MAC address of enixxxx
ARP/Routing Table (Pod)

Pod ARP/Routing(Cont.)
ARP/Routing Table (Host)

AWS VPC CNI
Underlay
Network
172.31.0.0/16
Subnet A 172.31.0.0/20
172.31.1.204
ENI

IPs
L-IPAMD
IP Pool
VPN CNI
Pod(Sandbox)

eth0
enixxxx
172.31.10.79/32
172.31.11.162
169.254.1.1
ENI

IPs
L-IPAMD
IP Pool
VPN CNI
Pod(Sandbox)

eth0
enixxxx
172.31.15.74/32
169.254.1.1
2
1
3

Packet
f
low
• 172.31.15.74 ping 172.31.10.79

• Pod (172.31.15.74)

Packet
f
low
• 172.31.15.74 ping 172.31.10.79

• Node (172.31.1.204)

Packet
f
low
• 172.31.15.74 ping 172.31.10.79

• Node (172.31.11.162)

AWS VPC CNI
Underlay
Network
172.31.0.0/16
Subnet A 172.31.0.0/20
172.31.1.204
ENI

IPs
L-IPAMD
IP Pool
VPN CNI
Pod(Sandbox)

eth0
enixxxx
172.31.10.79/32
172.31.11.162
169.254.1.1
ENI

IPs
L-IPAMD
IP Pool
VPN CNI
Pod(Sandbox)

eth0
enixxxx
172.31.15.74/32
169.254.1.1
2
1
3
172.31.15.174 -> 172.31.10.79

Others
• Debugging IPAMD

• Prometheus endpoint

• curl http://localhost:61678/metrics

• Other information (json)

• curl http://localhost:61679/v1/pods

• curl http://localhost:61679/v1/enis

Limitation
• M: Number of ENI

• N: Number of IP address per ENI

• Ignore Primary address

• M*(N-1)

• T3.medium

• M=3, N=6

• 3*(6-1)=15
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html

Limitation
• Final formula is

• M*(N-1) + 2

• Two Pods are deployed before CNI

• L-IPAMD

• kube-proxy

• Both two pods use the hostnetowk
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html

Limitation
• Magic number 2

• Final formula is

• M*(N-1) + 2

• Two Pods are deployed before CNI

• L-IPAMD

• kube-proxy

• Both two pods use the hostnetowk
https://github.com/awslabs/amazon-eks-ami/blob/master/
f
iles/eni-max-pods.txt

Limitation
13 Running Pod
17 = testing(13) + coreDNS (2) + kube-proxy(1) + CNI (1)

Summary
• Have to deploy two binary (L-IPAMD, CNI Binary)

• L-IPAMD is deployed by K8S DaemonSet

• With the help of AWS VPC CNI

• Reduce the number of SNAT/DNAT

• Better performance compared to Tunneling protocol

• User is able to apply existing AWS VPC networking and security best
practices for k8s cluster.

Learning how AWS implement AWS VPC CNI

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Learning how AWS implement AWS VPC CNI

Similar a Learning how AWS implement AWS VPC CNI (20)

Más de HungWei Chiu

Más de HungWei Chiu (20)

Último

Último (20)

Learning how AWS implement AWS VPC CNI