VXLAN with NSX -MH describes VXLAN and how it is implemented with NSX Micro Segmentation. It discusses VXLAN basics like encapsulation and VTEPs. It then covers the NSX control plane and data plane views including logical network view with logical switches/ports and physical transport node view. It provides examples of VXLAN L2 and L3 gateways for inter and intra-subnet communication deployed on NSX managed switches or physical gateways.

1. VXLAN with NSX -MH
VMWARE SDN Solution
Sethuraman Ramanathan
Protocols Sytest Team

2. Agenda
• Virtualization Basics
• VXLAN Basics
• VXLAN - Controller less solution
• VXLAN – Controller based (NSX-MH solution)

3. Virtualization Basics - Physical
Infrastructure
Fibre Channel
storage
Fibre
Channel
Ethernet
NFS
storage
iSCSI
storage
applications
operating system
physical host

4. Virtualization Basics Virtual Infrastructure
hypervisor
VMware ESXi™ host
Fibre
Channel
Fibre Channel
storage
Ethernet
NFS
storage
iSCSI
storage
virtual
machines

5. Servers 10
Utilization 8%
Annual cost per server $4,000
Total Cost $40,000
Servers 3
Utilization 80%
Annual cost per
server
$4,000
Total Cost $12,000
Before Virtualization After Virtualization
More applications per
machine = less machines
$28,000 in cost avoidance
Source: IT Business Edge, “The Business Value of Server Virtualization” – cost for average a 2 x CPU server in three-year amortized hardware purchase, and annual
support and maintenance contract costs 9/07
The CapEx Story: Make better use of existing infrastructure
Virtualization Basics Virtual Infrastructure

7.  Physical Network is tied with virtual Network.
 STP required.Scaling issues is seen as STP is not stable
with scale.
 Network troubleshooting is a challenge in layer 2 Networks
with STP.
 Cannot scale the network beyond 4096 vlans.

8.  Problems being addressed:
 VLAN scale – VXLAN extends the L2 segment ID field to 24-bits,
potentially allowing for up to 16 million unique L2 segments over
the same network
 Layer 2 segment elasticity over Layer 3 boundary – VXLAN
encapsulates L2 frame in IP-UDP header.So No STP is required.
 Removes the need to have additional physical
infrastructure. For example, the forwarding table of the
external switch does not grow with the increase in the
VMs behind the physical port on the server.

9. VXLAN Packet Structure
Original L2 Frame Given a VXLAN Header with VNI
Original L2 FrameVXLAN Header
FCS
Allows for 16M
possible segmentsUDP 4789
Enables better ECMP Load
balancing in the Network.
Src and Dst addresses of
the VTEPs
Src VTEP MAC Address
Next-Hop MAC Address

10. VXLAN Terminology
• VTEP (VXLAN Tunnel
End Point)
– Performs VXLAN encap & decap
– Usually located at the Aggregation Layer
or in the compute devices
• VNI (Virtual Network
Identifier)
– Mapping of VLAN to VXLAN (i.e., VNI
5000 maps to VLAN 20)
– Can have multiple VLANs mapped to the
same VNI
VXLAN Devices
10
VTEP
VTEPVTEP
VTEP

11. VXLAN MAC Learning
 Flood & Learn is used today
 Control-Plane based in future
 Multicast is required
OVS (Open Virtual Switch): Controller based
 Controller learns the mac Data with VTEP information from all OVS devices.
 Controller advertises the mac data with VTEP information to all OVS devices.
PIM-SM or PIM-Bidir : Controller less
11

13. VTEP Discovery (Pim method)
 VTEPs join specified multicast group (*, G)
 PIM-SM or PIM-BiDir
 Can have one multicast group per VNI
 Example: 239.1.1.1 is mapped to vxlan 10
 Can have multiple VNIs per multicast group
 Example: 239.1.1.1 is mapped to vxlan 10 and vxlan11.
How VTEPs find each other in PIM protocol based learning
method ?
13

14. VXLAN Multicast Mode 1mgroup:2 vxlan Mapping
VTEP VTEP VTEP
Pim jointo Multicast Group
239.1.1.1
Pim join Multicast Group
239.1.1.1
Pim join to Multicast Group
239.2.2.2
Pim join to Multicast Group
239.2.2.2
Web
VM
Web
VM
DB
VM
DB
VM
Multicast-enabled
Transport

15. ARP Request
VM 1 VM 3VM 2
VTEP 1
1.1.1.1
VTEP 3
3.3.3.3
VTEP 2
2.2.2.2
IP A  GARP Req
MAC IP Addr
VM 1 VTEP 1
MAC IP Addr
VM 1 VTEP 1
ARP Req
IP A  GARP Req
ARP Req ARP Req
Multicast-enabled
Transport

16. ARP Response
VM 1 VM 3VM 2
VTEP 1
1.1.1.1
VTEP 3
3.3.3.3
VTEP 2
2.2.2.2
ARP Resp
MAC IP Addr
VM 2 VTEP 2
Multicast-enabled
Transport
VTEP 2  VTEP 1ARP Resp
ARP Resp
MAC IP Addr
VM 1 VTEP 1

17. routing-instances {
sw1 {
vtep-source-interface lo0.1;
instance-type virtual-switch;
interface xe-0/0/0.1;
bridge-domains {
vxlan1 {
domain-type bridge;
vlan-id 1;
routing-interface irb.1;
vxlan {
vni 1;
multicast-group 239.1.1.1;
encapsulate-inner-vlan;
decapsulate-accept-inner-vlan;
}
}
}
}
}
Gateway config spine switch– Multicast based
vxlan

18. Gateway config spine switch– Multicast based
vxlanvrf1{
instance-type vrf;
interface xe-0/0/1.1
route-distinguisher 100:1;
vrf-target target:100:1;
protocols {
ospf {
area 0.0.0.0 {
interface all {
bfd-liveness-detection {
minimum-interval 2000;
multiplier 3;
}
}
}
}
pim {
rp {
local {
address 5.0.0.1;
}
}
interface all;
}
}
}

20. NSX Control Plane

21. NSX Control Plane
 The NSX Controller Cluster accepts logical network configuration instructions from
administrators (through the NVP API) or from its clients), calculates the required
network flow entries, and inserts these network flow entries into Open vSwitch
(OVS) instances running on the transport nodes (hypervisor switches and NSX
appliances).
 In each transport node, the flow entries give OVS the routing information it needs
to direct logical Ethernet frames to the right hypervisor or network gateway.

22. NSX Data Plane Views
 Transport Network view
 Hypervisors,Physical gateways/or virtual gateways and service nodes.
 Logical Network View
 Logical switch,Logical switch port,Transport zone

23. NSX Data Plane (Transport Network View)
 The transport network view is the view presented to cloud / data center administrators
(people who deploy hypervisors and their associated network infrastructure).
 This view describes the physical devices that underlie the logical networks.
We refer to these devices as “transport nodes” and they include the
Hypervisors that host VMs and the network hardware that interconnects
hypervisors and connects them to external, physical networks.
 Each transport node runs an instance of Open vSwitch (OVS), so we also refer to
transport nodes as “OVS devices”.
 The cloud/data center administrator works in the transport network view, connecting
hypervisors to the transport network, deploying other NSX transport nodes such as NSX
Gateways, and connecting them to the physical network

24. 8
VLAN
Hardware
Software
L2
L3
Virtual Network
L2
Open vSwitch
NSX Gateway
Physical Network (Arista, Cisco, HP, Juniper, Cumulus,…)
VMVM
NSX vSwitch
ESXi
Open vSwitch
KVM
Open vSwitch
XenServer
Open vSwitch
Hyper-V*
Controller Cluster
Transport Network
NSX Manager
VTEP
HW Partner
VLAN
* Hyper-V plan 2H2014
API

25. NSX Data Plane (Gateway Service)
 An NSX Gateway Service consists of one or more NSX Gateways that attach a
logical network to a physical network not managed by NSX.
 Each Gateway Service can operate as an L2 Gateway Service expanding a logical
L2 segment to include a physical L2 segment, or as an L3 Gateway Service
mapped to a physical router port.
 Each Gateway in the service is a virtual appliance running OVS, or a physical
VTEP-enabled appliance.

27. Destination is in another segment.
Packet is routed to the new segment
VXLANORANGE VXLANBLUE
Ingress VXLAN packet on
Orange segment
VXLAN
Router
 V(X)LAN-to-V(X)LAN Routing (L3 Gateway)
VXLAN on HW Platforms
Supported Functionalities
 VXLAN to VLAN Bridging (L2 Gateway)
VXLANORANGE
Ingress VXLAN packet on
Orange segment
Egress interface chosen (bridge
may .1Q tag the packet)
VXLAN L2
Gateway
Egress interface chosen (bridge
may .1Q tag the packet)

28. VNI 6000
VXLAN-to-VLAN Bridging
Virtual to Physical
VxLAN
VLAN
untagged
VXLAN L2
Gateway
VXLAN L2
Gateway
VNI 5000
VLAN 10
VLAN 20
VXLAN VTEP
HW VXLAN L2 Gateway
Intra-Subnet Communication
L3 Fabric
L3 Cloud
Controller

29. L3 Cloud
VXLAN L3
Gateway
VXLAN L3
Gateway
HW VXLAN Routing
Inter-Subnets Communication
VXLAN-to-VXLAN Routing
VNI 5000 <-> VNI 7000
VXLAN L2
Gateway
VXLAN L2
Gateway
VxLAN
VLAN
untagged
VLAN-to-VXLAN Routing
VNI 6000 <-> L3_Ext_Intf
VNI 5000
VLAN 20VLAN 30
VXLAN-to-VLAN Bridging
VNI 7000 <-> VLAN 30
VXLAN-to-VLAN Bridging
VLAN 20 <-> VNI 6000
L3 Fabric
Controller

31. Inter-VXLAN Routing using SW L3 Gateway
SW
Gwy
VXLAN Routing
VNI 5000 <-> VNI 6000
Virtual to Virtual
VNI 5000 VNI 6000
vMX acting as L3 gateway
L3 Fabric
WAN/Core
VxLAN
untagged
Controller

32. SW L3 Gateway
Communicating with the External L3 Domain
SW
Gwy
VXLAN to VLAN Bridging
VNI 5000 <-> V:LAN 100
Virtual to Physical
VNI 6000
vMX acting as L3 gateway
VLAN
L3 Fabric
WAN/Core
VxLAN
untagged
Controller

33. NSX SW L2 Gateway
SW
Gwy
VXLAN Routing
VNI 5000 <-> VNI 6000
Virtual to Physical
VNI 5000
vMX acting as L2 gateway
L3 Fabric
WAN/Core
VxLAN
untagged
VLAN 10
VXLAN L2
Gateway
Controller

34. NSX Data Plane (Service Node)
 NSX employs NSX Service Nodes, OVS-enabled x86 appliances that are managed
by the Controller Cluster to provide extra packet processing capacity for logical
networks.
 For example, Service Nodes assist with the packet replication required for
logical network broadcast/multicast and unknown unicast flooding in overlay
logical networks.
 In VXLAN Multicast mode packet replication is done by router in the physical
network.

35. Logical Network view
 The logical network view is the set of connectivity and network services a VM sees
in the cloud. The logical view is the view presented to VMs and VM administrators
and is independent of the underlying physical devices of the data center.
 In a multi-tenant cloud, each tenant has his or her own logical network view and
cannot see the logical network views of other tenants. The logical network view
consists of the logical ports, logical switches, and logical routers that interconnect
VMs and connect VMs to the external physical network.
 From the point of view of a VM and its administrators, the logical network is the
network. The VM administrator just connects VMs to logical switches and logical
routers.

36. Logical Network view

37. Logical Network View
Logical Switch:
A logical switch is a layer-2 switching overlay implemented using one of NSX’s
supported encapsulation mechanisms (VXLAN,GRE, IPsecGRE) .
Basic network set-up in NSX is accomplished by connecting VMs to logical switches.
Logical switch port:
Similar to physical switch ports, logical switch ports enable the configuration
of network services.
Attachment:
A Logical wire that can be used to connect virtual interfaces, logical switch ports,.
Common attachment types include L2 gateway attachments, and
L3 gateway attachments.

38. Logical Network View:
Transport zone—Physical network connectivity between transport nodes is modeled in the API
as a transport zone. A transport zone corresponds to a physical network used to send data
traffic between OVS devices. A simple NSX deployment will have a single transport zone that
represents the physical network connectivity within the data center.

39. Logical Network View

40. DEMO

41. L3 Cloud
VXLAN L2
Gateway
VLAN 1
VLAN 1
VXLAN-to-VLAN Bridging
VLAN1<-> VLAN 1
VXLAN L2
Gateway
Topology1
foot Littlefoot
Service node
Controller
Thtys
VxlanL2/L3
gateway

42. L3 Cloud
VXLAN L2
Gateway
VLAN 1
VLAN 1
VXLAN-to-VLAN Bridging
VLAN1<-> VLAN 1
VXLAN L2
Gateway
Topology2
Thtys Littlefoot
Service node
Controller

43. VXLAN L2 GATEWAY CONFIGURATION
{master}
regress@thtys> show configuration interfaces xe-2/3/3.1
family bridge {
interface-mode trunk; << connects to servers
vlan-id-list 1;
}
regress@foot>
{master}
regress@littlefoot> show configuration interfaces xe-1/1/1.1
family bridge {
interface-mode trunk; << connects to servers
vlan-id-list 1;
}
{master}
regress@littlefoot>

44. VXLAN L2 GATEWAY CONFIGURATION
{master}
regress@thtys> show configuration interfaces xe-2/0/0.1
vlan-tagging;
unit 1 {
vlan-id 1;
family inet {
address 1.0.1.1/30; ; << connects to spine
}
}
{master}
regress@littlefoot> show configuration interfaces xe-1/1/0
vlan-tagging;
unit 1 {
vlan-id 1;
family inet {
address 11.0.1.1/30; << connects to spine

45. master}
regress@thtys> show configuration routing-instances sw1
vtep-source-interface lo0.0;
instance-type virtual-switch;
interface xe-2/3/3.1; << Connects to the compute node
86a9576a-3b68-4412-9658-6218b3ac02fa {<< UUID of logical switch
generated by controller
domain-type bridge;
vlan-id 1;
vxlan {
ovsdb-managed;
vni 1; <<<<< VNI 1 is mapped to vlan 1
encapsulate-inner-vlan; ; <<Configure the switch to preserve the
original VLAN tag (in the inner Ethernet packet) when performing VXLAN
encapsulation.
decapsulate-accept-inner-vlan; ; <<<Configure the switch to de-
encapsulate and accept original VLAN tags in VXLAN packets
VXLAN L2 GATEWAY CONFIGURATION:

46. {master}
regress@thtys> show configuration routing-instances vrf1
instance-type vrf;
interface ge-1/0/9.0; <<<< connected to service node;
interface xe-2/0/0.1; <<<< connects to the spine
interface lo0.0;
route-distinguisher 100:1;
vrf-target target:100:1;
protocols {
ospf {
area 0.0.0.0 {
interface all {
bfd-liveness-detection {
minimum-interval 2000;
multiplier 3;
}
}
}
}
}
VXLAN L2 GATEWAY CONFIGURATION:

47. sw1 {
vtep-source-interface lo0.0;
instance-type virtual-switch;
interface xe-1/1/1.1; << Connects to the compute node
bridge-domains {
86a9576a-3b68-4412-9658-6218b3ac02fa {
domain-type bridge;
vlan-id 1;
vxlan {
ovsdb-managed;
vni 1; <<<<< VNI 1 is mapped to vlan 1
encapsulate-inner-vlan; <<Configure the switch to preserve the original VLAN tag
inner Ethernet packet) when performing VXLAN encapsulation.
decapsulate-accept-inner-vlan; <<<Configure the switch to de-encapsulate and a
original VLAN tags in VXLAN packets
}
}
}
VXLAN GATEWAY CONFIGURATION:Littlefoot
< < UUID of logical switch generated
by controller

48. {master}
regress@littlefoot> show configuration routing-instances vrf1
instance-type vrf;
interface xe-1/1/0.1; <<<< connects to the spine
interface lo0.0;
route-distinguisher 100:1;
vrf-target target:100:1;
protocols {
ospf {
area 0.0.0.0 {
interface all {
bfd-liveness-detection {
minimum-interval 2000;
multiplier 3;
}
}
}
}
}
VXLAN GATEWAY CONFIGURATION:Littlefoot

49. {master}
regress@littlefoot> show ovsdb mac logical-switch 86a9576a-3b68-4412-9658-6218b3ac02fa
Logical Switch Name: 86a9576a-3b68-4412-9658-6218b3ac02fa
Mac IP Encapsulation Vtep
Address Address Address
ff:ff:ff:ff:ff:ff 0.0.0.0 Vxlan over Ipv4 10.255.178.140
ff:ff:ff:ff:ff:ff 0.0.0.0 Vxlan over Ipv4 24.24.24.1
{master}
regress@littlefoot>
{master}
regress@littlefoot>
MAC learning:Littlefoot

50. {master}
regress@littlefoot> show ovsdb controller
VTEP controller information:
Controller IP address: 192.168.181.3
Controller protocol: ssl
Controller port: 6632
Controller connection: up
Controller seconds-since-connect: 697109
Controller seconds-since-disconnect: 652143
Controller connection status: backoff
{master}
regress@littlefoot>
Controller status:Littlefoot

51. Logical Network View :Gateway service

52. Logical Network View: Logical port attachment

53. Logical Network view: Gateway service

54. Logical Network view: Logical switch port attachment

55. Logical Network view: Logical switch port status

56. Logical Network view: Logical Switch

57. Logical Network view:Transport Node

58. Logical Network view:Transport Connectors

59. Logical Network view: