Is there something less complicated than connecting two LAN networks? VLAN and VXLAN integration

1. © 2014 VMware Inc. All rights reserved.
Integrating Physical Infrastructure and
Virtual Workloads
How to connect VLAN & VXLAN ?
Emil Gągała
Network & Security Architect
PLNOG
28.09.2015, Kraków

2. Site 3
Site 1
Site2
Site 1
Site2
A1
B1
A3
A2
B2
Ethernet over …
• ATM  LAN Emulation
• MPLS  VPLS, EVPN
• IP  VXLAN, EVPN, OVSDB

3. Ethernet characteristics
• Multipoint to Multipoint connectivity
• No control plane – only data plane MAC learning
• BUM traffic - some form of multicast support required
• Segmentation - VLAN
• No built-in loop prevention mechanism
• No native multi-homing capabilities
• Scalability – scope of broadcast domain and number of MACs
• Lack of Virtual Machines awareness

4. Physical DC Fabric Trends
• From 2- or 3-tier to IP spine/leaf fabrics
• Density & bandwidth jump
• ECMP for layer 3 (and layer 2)
• Reduce network oversubscription
• Wire & configure once
• Uniform configurations
4
WAN/Internet
L3
L2
L3
L2
POD A POD B
WAN/Internet
L3
L2
L2 Ethernet
L3 IP

5. VM5
Virtual Overlay Networking
VM1
Overlay Network
VM2
Logical Switch 5001
VM3
Physical Underlay IP Network
VM4
Logical Switch 5002
Controller
Management Cluster
Control Plane Programming
Data Plane Tunneling
Subnet Red 172.16.10.0/24
Subnet Green 172.16.20.0/24
192.168.150.51 192.168.150.52 192.168.250.51
5
Transport Subnet A 192.168.150.0/24 Transport Subnet B 192.168.250.0/24

6. Traditional VLAN network
VM1 VM2
Server 1
VM3
VM4 VM5
Server 2
VM6
VM7 VM8
Server 3
VM9
Physical switch
Virtual networks:
1 2
3
VLANs

7. OVERLAY Networking
VM1 VM2
Server 1
VM3
VM4 VM5
Server 2
VM6
VM7 VM8
Server 3
VM9
No VM network state
Virtual networks:
S3 VM9 Payload Transport network:

8. Virtual Extensible LAN (VXLAN) Overlay
• VXLAN is an industry standard IP overlay technology - RFC 7348 :
Used to tunnel Layer 2 traffic over an IP infrastructure
8
L2 frame
• Why an IP encapsulation?
– leverage VXLAN in order to decouple its data plane from the physical network:
basic IP connectivity is enough to run SDDC
• Why an additional VXLAN header?
– VXLAN Network Identifier (VNI)
VTEP: Virtual Tunnel End Point
src IP: VTEP1, dst IP:VTEP2 UDP/VXLAN L2 frame VTEP2VTEP1
L2 frame

9. What is VXLAN ? (Overview)
Ethernet in IP overlay network
– Entire L2 frame encapsulated in UDP
– 50+ bytes of overhead
24 bit VXLAN Network Identifier
– 16 M logical networks
VXLAN can cross Layer 3 network boundaries
Technology approved by IETF as standard
– RFC 7348 „Virtual eXtensible Local Area
Network (VXLAN): A Framework for
Overlaying Virtualized Layer 2 Networks over
Layer 3 Networks”
– With Arista, Brocade, Cisco, Cumulus, Dell,
HP, Juniper, Vmware
Overlay between hosts and Gateways
– VMs do NOT see VXLAN ID
VTEP (VXLAN Tunnel End Point)
– Interface which serves as the endpoint for
encapsulation/de-encapsulation of VXLAN
traffic
VTEP acts like a learning bridge
– Missing information relies on ARP generated
from the host or Flood
– Floods ports when encountering an unknown
MAC
– Flooding may happen when
communicating with physical workloads
– Flooding limited to a VXLAN segment
Once destination MAC is known,
communication is direct

10. VXLAN OVERLAY Networking
VM1 VM2
Server 1
VM3
VM4 VM5
Server 2
VM6
VM7 VM8
Server 3
VM9
No VM network state
IP Fabric
Virtual networks:
S3 VM9 Payload Transport network:

11. Software Defined Data Center
VM1 VM2
Server 1
VM3
VM4 VM5
Server 2
VM6
VM7 VM8
Server 3
VM9
Virtual networks:
S3 VM9 Payload Transport network:
Controller

12. Software Defined Data Center
VM1 VM2
Server 1
VM3
VM4 VM5
Server 2
VM6
VM7 VM8
Server 3
VM9
Virtual networks:
Transport network:
Controller
BMS
Server 4

13. Web-Tier
App-Tier
DB-Tier
VMs Connect to
Virtual Networks
Virtual Networks Connect to
non-virtualized Workloads
Physical-Virtual Bridging and Routing
 Requirement: Communication between VMs connected to Logical Networks
(VXLAN) and workloads (virtualized or not) deployed on traditional VLANs
 Different options depending on the connectivity needs:
L2 (Bridging)
SW L2 Bridges
HW VTEP L2 Gateway
L3 (Routing)
 L2/L3 services useful for supporting migration scenarios

14. Overlay to VLAN Bridging

15. Overlay to VLAN Gateway Functionality
• The Overlay to VLAN gateway allows communication between virtual and physical world
Physical Network
VLAN backed network
VM
NSX: Virtual Network,
VXLAN tunnels
VLANVXLAN L2 payload
VXLAN  VLAN
gateway
Physical Workload

16. • L2 as well as L3
• Virtual to virtual, physical to virtual
• Temporary, bandwidth and redundancy not critical
Use Cases: Migration
16
VM
VM
Physical Workload
Virtualized
Workload (VLAN backed)
Physical to Virtual
Virtual to Virtual
VXLAN VLAN

17. • Typically necessary for integrating a non-virtualized appliance
• A gateway takes care of the on ramp/off ramp
Use Cases: Integration of non-Virtualized Workloads
17
VM
Physical Services / Workload
VXLAN VLAN

18. Physical Workload Integration
Physical Workloads
VXLAN VLAN
x86-based bridge
Highest density but requires specific hardware
Leverages any x86 server
Physical Workloads
VXLAN VLAN
HW VTEP
Use-case: Integrate non-virtualized workloads seamlessly with virtual networks

19. x86-based Overlay to VLAN solution

20. P-V Bridging Scale-out
Multiple Bridge instances (VXLAN/VLAN) Pair
 Single bridging instance
(VXLAN/VLAN pair) per Logical Switch
Bandwidth limited by single bridging instance
Bridged VLAN extends to reach physical
devices in multiple racks
VXLAN
VLAN
VLAN extended (!)
SW VTEP
Physical Servers
(VLAN 10) VXLAN 5001
 Scale-out model with multiple bridging instances
active for separate VXLAN/VLAN pairs
 May allow to reduce the spanning of VLANs to a
single rack if physical servers in a VLAN are
contained in that rack
L3-only (VXLAN)
network
VXLAN 5000 Physical Servers
(VLAN 10)
Physical Servers
(VLAN 20)
Bridging Instance 1
(VXLAN 5000 to VLAN 10)
Bridging Instance 2
(VXLAN 5001 to VLAN 20)
20

21. Benefits of x86-based On/off-ramp
Scale-up
• x86 performance curve
• x86 optimized processing (DPDK)
• Encapsulation offloads
• Encryption offloads
Scale-out
• Expand on scale-out
• Active-active services
Flexibility & Operations
• Rich set of stateful services
• Multi-tier logical routing
• Advanced monitoring
• Choice of form-factor
• Scale as you grow
21VMware Confidential
VLAN 10
VLAN 20
VLAN 30

22. HW VTEPs

23. HW VTEPs Motivation
 Integrate non-virtualized workloads seamlessly with virtual networks
Servers with legacy or hard-to-virtualize applications
Physical servers relying on specific hardware not supported by HVs
Physical network & security appliances such as routers, load balancers, firewalls, IPS,
WAN acceleration, etc.
 The following are potential use cases for HW L2 VTEPs:
Low latency traffic
Very large volumes of physical servers (>10G of bandwidth required for P-to-V
communication)
Support of physical hosts connected in different racks  remove the need to extend
VLAN connectivity across the racks
23

24. VM
VXLAN
Distributed Bridging Options with Hardware VTEPs
One bridge instance
• Bandwidth limited by single instance
• VLAN connectivity extended to reach the
physical devices.
Hardware VTEPs deployed where physical
workloads or services exist:
• Bandwidth and physical ports scale-out
• VLANs for Physical workloads only local to a rack
24
VXLAN
VLAN
VM
Single Instance x86 L2 Gateway Multiple Instances 3rd party HW Gateway
Non-virtualized
devices (part of
the same L2
segment)

25. L2 Extension with Hardware VTEPs
Two options:
1. Hardware VTEP with IP multicast in the underlay
2. OVSDB Integration
25
VM
VM
L2 Network
Non-virtualized devices
LS A LS B
VLAN A
VLAN B
VXLAN
VLAN
Hardware VTEP
ToR Switch

26. VXLAN in Multicast Mode

27. VXLAN in Multicast Mode
IGMP to Join VXLANs Assigned Multicast Groups
Web
VM
Web
VM
DB
VM
DB
VM
IGMP Report to Multicast Group
239.1.1.1
VTEP VTEP VTEP
L3 Core
with multicast
IGMP Report to Multicast Group
239.1.1.1
IGMP Report to Multicast Group
239.2.2.2
IGMP Report to Multicast Group
239.2.2.2
Mapping is required between a VXLAN ID and a Multicast Group
VXLAN 5000 VXLAN 6000

28. VTEP 1
1.1.1.1
VTEP 2
2.2.2.2
VTEP 3
3.3.3.3
VXLAN Data Flow Example
VM1 Communicating with VM2 in a VXLAN
VM 1 VM 2 VM 3
MAC1
MAC2
Multicast
Multicast Multicast
ARP Request
ARP Request ARP Request
VM Source MAC Remote VTEP IP
VM1:MAC1 1.1.1.1
MAC Table: VTEP 2 and VTEP 3 (Data Plane Learning)L3 Core
with multicast
VXLAN 5000

29. VTEP 1
1.1.1.1
VTEP 2
2.2.2.2
VTEP 3
3.3.3.3
VXLAN Data Flow Example (2)
VM1 Communicating with VM2 in a VXLAN
VM 1 VM 2 VM 3
MAC1
MAC2
VM Source MAC Remote VTEP IP
VM1:MAC1 1.1.1.1
MAC Table: VTEP 2L3 Core
with multicast
ARP Response
Unicast
VXLAN 5000

30. VTEP 1
1.1.1.1
VTEP 2
2.2.2.2
VTEP 3
3.3.3.3
VXLAN Data Flow Example (3)
VM1 Communicating with VM2 in a VXLAN
VM 1 VM 2 VM 3
MAC1
MAC2
VM Source MAC Remote Host VXLAN
IP
VM1:MAC1 1.1.1.1
MAC Table: VTEP 2L3 Core
with multicast
ARP Response
VM Source MAC Remote Host VXLAN
IP
VM2:MAC2 2.2.2.2
MAC Table: VTEP 1 (Data Plane Learning)
VXLAN 5000

31. VTEP 1
1.1.1.1
VTEP 2
2.2.2.2
VTEP 3
3.3.3.3
VXLAN Data Flow Example (4)
VM1 Communicating with VM2 in a VXLAN
VM 1 VM 2 VM 3
MAC1
MAC2
VM Source MAC Remote Host VXLAN
IP
VM1:MAC1 1.1.1.1
MAC Table: VTEP 2L3 Core
with multicast
VM Source MAC Remote Host VXLAN
IP
VM2:MAC2 2.2.2.2
MAC Table: VTEP 1
Unicast
VXLAN 5000

32. OVSBD Integration

33. • Hardware VTEP enabled physical appliance
• Attach any physical services appliance
• Extensible (schema-based)
• Integration not dependent on Multicast
Overview
• High density of physical ports to connect physical workloads
• Broad ecosystem of partners
• Compatible with HA M-LAG solution
Benefits
OVSDB Integration: Hardware VTEPs
33
Provide connectivity to physical workloads and services
VM1 VM2
LS – VNI
5001
VLAN 100

34. What is OVSDB ?

35. What is OVSDB ?
Open vSwitch Data Base (OVSDB) is a management protocol
 Helps attaching interfaces, gathering statistics, configuring features
 NOT related with and does NOT require Openflow
 RFC 7047 for „The Open vSwitch Database Management Protocol”
3rd Party GW
OVSDB Server
Operational
State
Forwarding
State
IP Fabric
Service Nodes
VM
VMVM
VM VM
VM
Controller
Cluster
CMP

36. • The controller exposes a northbound API - physical ports can be attached to logical switches.
• Virtual ports of VMs are attached to build logical networks that span the physical and virtual
worlds
• The information exchanged by the control plane allows setting up the data plane, i.e. VXLAN
tunnels between VTEPs
• Switch terminates VXLAN tunnels
OVSDB Integration: Hardware VTEPs
VNI
VTEP
VM MAC/IP
port

37. OVS hardware_vtep database schema
Table Purpose Global Top-level configuration.
Manager OVSDB management connection.
Physical_Switch A physical switch.
Physical_Port A port within a physical switch.
Logical_Binding_Stats Statistics for a VLAN on a physical port bound to a logical network.
Logical_Switch A layer−2 domain.
Ucast_Macs_Local Unicast MACs (local)
Ucast_Macs_Remote Unicast MACs (remote)
Mcast_Macs_Local Multicast MACs (local)
Mcast_Macs_Remote Multicast MACs (remote)
Logical_Router A logical L3 router.
Physical_Locator_Set Physical_Locator_Set configuration.
Physical_Locator Physical_Locator configuration.
CONFIDENTIAL 37

38. Summary

39. Integrating Physical Infrastructure and Virtual Workloads
Conclusion
x86-based
 Scale-up & scale-out model
 Pay as you grow
 Rich set of stateful services
 Growing set of routing features
 Software development cycle
 Versatile topology options
HW VTEP
 Highest density & performance
 Broad partner choice
• Limited stateful services (cost)
 Broad set of routing features
• Longer innovation cycle
VMware Confidential 39

40. Thank You