Multi-Site VxLan

2. VXLAN BGP EVPN
based Multi-Pod, Multi-
Fabric, Multi-Site
Max Ardica – Principal Engineer
Lukas Krattiger – Principal Engineer
BRKDCN-2035

3. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cisco Spark
Questions?
Use Cisco Spark to chat with the
speaker after the session
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space
How
cs.co/ciscolivebot#BRKDCN-2035Cisco Spark spaces will be
available until July 3, 2017.

4. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Who Are the Presenters?
Max Ardica
Principal Engineer - INSBU
Lukas Krattiger
Principal Engineer – INSBU
@ccie21921
BRKDCN-2035 4

5. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Session Objectives
At the end of the session, the participants should be able to:
Articulate the different deployment options to interconnect
VXLAN EVPN Networks (Multi-Pod vs. Multi-Fabric vs. Multi-
Site)
Understand the functionalities and specific design
considerations associated to the new VXLAN Multi-Site
architecture
Initial assumption:
The audience already has a good knowledge of the VXLAN
EVPN technology and its use to deploy modern Data Center
Fabrics
BRKDCN-2035 5

6. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Session Reference
• BRKDCN-2304
• L4-L7 Service Integration in Multi-Tenant VXLAN EVPN Data Center Fabrics
• BRKDCN-3378
• Building DataCenter Networks with VXLAN BGP-EVPN
• Wednesday, Jun 28, 1:30 pm
• BRKDCN-2125
• Overlay Management and Visibility with VXLAN
• Thursday, Jun 29, 10:30 am
• BRKDCN-2342
• Programmable Fabric Automation and Management with DCNM 10
• Thursday, Jun 29, 1:00 p.m.
6BRKDCN-2035

7. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Agenda
Introduction
VXLAN EVPN Interconnect Evolution
• Multi-Pod
• Multi-Fabric
• Multi-Site
VXLAN EVPN Multi-Site Deep Dive
• Walkthrough
• Control- and Data-Plane
• Deployment Considerations
Conclusions and Q&A

8. Introduction

9. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Layer 2 Domain Elasticity
Local LAN Fabric
Extended LAN fabric
VN-link
notifications
IP Mobility
Optimal Ingress and Egress
Routing
VM-awareness
VXLAN, DFA, ACI, VN-link
Storage Elasticity
SAN Extensions
Network Service Localization
Any service anywhere
OTV
OTV
OTV
OTV
Fabric Consolidation
Unified Fabric & I/O
Device Virtualization
Segmentation
Data Center Interconnect – DCI Model
Connecting Virtualized Data Centers
Multi-tenancy/Segmentation
Segment-IDs in VXLAN, LISP, FabricPath,
and OTV
Storage Solutions & Partners:
FCIP, I/O Acceleration
EMC, NetApp
BRKDCN-2035 9

10. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Back Then
Yet Another Encapsulation
Flood & Learn (Multicast-based)
Data-Plane only Yesterday
VXLAN for the Data Center – Intra-DC
Control-Plane
Active VTEP Discovery
Multicast and Unicast
Now!
VXLAN for DCI – Inter-DC
DCI Ready
ARP/ND caching/suppress
Multi-Homing
Failure Domain Isolation
Loop Protection
VXLAN Evolves as the Control Plane Evolves!
BRKDCN-2035 10

11. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Back Then
VXLAN for Interconnecting Networks
BRKDCN-2035 11

12. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Inter-X Connectivity
• Single Fabric with End-
to-End Encapsulation
• Build Hierarchy in the
Underlay – Flatten it in
the Overlay
Multi-Pod
Overlay
VTE
P
VTE
P
VTE
P
VTE
P
Bar
em
eta
l
Bar
em
eta
l
Fabric #2
Overlay
VTE
P
VTE
P
Bar
em
eta
l
Bar
em
eta
l
VTE
P
VTE
P
Fabric #1EVPN Control-
Plane Domain 1
EVPN Control-
Plane Domain 2
Single Data-Plane – End-to-End
BGP EVPN
Overlay
VTE
P
VTE
P
VTE
P
VTE
P
Bar
em
etal
Bar
em
etal
Fabric #2
Overlay
VTE
P
VTE
P
Bar
em
etal
Bar
em
etal
VTE
P
VTE
P
Fabric #1EVPN Control-Plane
Domain 1
EVPN Control-Plane
Domain 2
Data-Plane Domain 1 Data-Plane Domain 2
DCI
Data-Plane
• Multiple Fabrics –
Normalized through
Ethernet
• Multiple Fabrics
Interconnect using DCI
(Layer 2 and Layer 3)
Multi-Fabric
Data-Plane
Domain 1
Data-Plane Domain 2
DCI
Data-Plane
Overlay
VTE
P
VTE
P
VTE
P
VTE
P
Bar
em
etal
Bar
em
etal
Fabric #2
Overlay
VTE
P
VTE
P
Bar
em
etal
Bar
em
etal
VTE
P
VTE
P
Fabric #1EVPN Control-Plane
Domain 1
EVPN Control-Plane
Domain 2
BGP EVPN
• Multiple Fabrics with
Integrated DCI
• Integrated DCI –
Scaling within and
between Fabrics
• The Happy Place
Multi-Site
BRKDCN-2035 12

13. VXLAN EVPN Interconnect
Evolution

14. Multi-Pod

15. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
VXLAN EVPN – Single Pod / Single Fabric
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Pod 1
VTEP VTEP
External Network
BRKDCN-2035 15

16. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
VXLAN EVPN – Multi-Pod
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Pod 1
VTEP VTEP
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Pod n
VTEP VTEP
Underlay Extension
BRKDCN-2035 16

17. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Single Overlay Domain – End-to-End Encapsulation
Single Overlay Control-Plane Domain – End-to-End EVPN Updates
Single Underlay Domain End-to-End
Single Replication Domain for BUM
Single VNI Administrative Domain
Multi-Pod Characteristics – ”The Single”
Building Underlay Hierarchies – Non Hierarchical Overlay
BRKDCN-2035 17

18. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multi-Pod – End-to-End Encapsulation
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Pod 1
VTEP VTEP
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Pod n
VTEP VTEP
Underlay Extension
Overlay
Baremetal Baremetal
Unicast
VTEP
10.1.1.1
VTEP
10.2.2.7
BRKDCN-2035 18

19. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multi-Pod – BUM Replication
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Pod 1
VTEP VTEP
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Pod 2
VTEP VTEP
Underlay Extension
Overlay
Baremetal
BUM
BRKDCN-2035 19

20. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Single Overlay Domain – End-to-End Encapsulation
• Scaling the VXLAN EVPN Network
Single Overlay Control-Plane Domain – End-to-End EVPN Updates
• Overlay Control-Plane Update Propagation
Single Underlay Domain End-to-End
• Network must be extended in Underlay (VTEP to VTEP reachability)
Single Replication Domain for BUM
• One BUM flooding domain through out all connected Pods
Multi-Pod Challenges – ”The Single”
BRKDCN-2035 20

21. Multi-Fabric

22. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
VXLAN EVPN – Multi-Fabric
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Fabric 1
VTEP VTEP
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Fabric 2
VTEP VTEP
Underlay No Extension
L2 DCI L2 DCI
L3 DCI L3 DCI
L2 DCI L2 DCI
BRKDCN-2035 22

23. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
• Separate Overlay Domains –Independent L2 and L3 DCI (complexity)
• Separate Overlay Control-Plane Domains – Manual Configuration
• Separate Underlay Domains - Isolated
• Separate Replication Domains for BUM – Independent BUM transport/DCI
• Dedicated Border Leaf – no local End-Point Attachment
Multi-Fabric Characteristics – ”The Separate”
Underlay Isolation – Separate DC Interconnection
BRKDCN-2035 23

24. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Fabric 1
VTEP VTEP
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Fabric n
VTEP VTEP
Underlay No Extension
L2 DCI L2 DCI
L3 DCI L3 DCI
L2 DCI L2 DCI
Multi-Fabric – End-to-End Encapsulation
Overlay Site 1 Overlay Site n
L2 DCI
Baremetal Baremetal
Unicast
VLAN Hand-Off VRF-Lite Hand-Off
BRKDCN-2035 24

25. VXLAN EVPN Multi-Site

26. Multi-Site
Walkthrough

27. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
VXLAN EVPN – Multi-Site
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site 1
VTEP VTEP
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site n
VTEP VTEP
No Underlay Extension
BGW BGW BGW BGW
BRKDCN-2035 27

28. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multiple Overlay Domains – Interconnected & Controlled
Multiple Overlay Control-Plane Domains – Interconnected & Controlled
Multiple Underlay Domains - Isolated
Multiple Replication Domains for BUM – Interconnected & Controlled
Multiple VNI Administrative Domains – Phase 2
Multi-Site Characteristics – ”The Multiple”
Underlay Isolation – Overlay Hierarchies
BRKDCN-2035 28

29. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multi-Site – Hierarchical Overlay Domains
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site 1
VTEP VTEP
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site n
VTEP VTEP
Overlay Site 1 Overlay Site n
Overlay Multi-Site
Baremetal Baremetal
Unicast
BGW BGW BGW BGW
BRKDCN-2035 29

30. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multi-Site – Underlay Isolation
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site 1
VTEP VTEP
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site n
VTEP VTEP
No Underlay Extension
VTEP
10.1.1.1
Border (VIP)
10.1.1.111
Border (VIP)
10.2.2.222
Site 1 Underlay
Routing Table
Leaf:
10.1.1.1
10.1.1.2
10.1.1.3
10.1.1.4
10.1.1.5
10.1.1.6
10.1.1.7
Border:
10.1.1.101
10.1.1.102
10.1.1.111
VTEP
10.2.2.7
Site n Underlay
Routing Table
Leaf:
10.2.2.1
10.2.2.2
10.2.2.3
10.2.2.4
10.2.2.5
10.2.2.6
10.2.2.7
Border:
10.2.2.101
10.2.2.102
10.2.2.222
BGW BGW BGW BGW
Border (PIP)
10.1.1.101
Border (PIP)
10.1.1.102
Border (PIP)
10.2.2.101
Border (PIP)
10.2.2.102
BRKDCN-2035 30

31. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Inter Site Network
Multi-Site – Inter Site Network
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site 1
VTEP VTEP
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site n
VTEP VTEP
VTEP
10.1.1.1
Border (VIP)
10.2.2.222
Border (VIP)
10.1.1.111
Inter-Site Network
Routing Table
Border Site1:
10.1.1.101
10.1.1.102
10.1.1.111
Border Site2:
10.2.2.101
10.2.2.102
10.2.2.222
VTEP
10.2.2.7
BGW BGW BGW BGW
Border (PIP)
10.1.1.101
Border (PIP)
10.1.1.102
Border (PIP)
10.2.2.101
Border (PIP)
10.2.2.102
BRKDCN-2035 31

32. Border Gateways
Deployment
Considerations

33. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Border Gateways Deployment Considerations
Site 1
VTEP
BGW
VTEP
BGW
VTEP
BGW
VTEP
BGW
Site 1
VTEP
BGW
VTEP
BGW
Border Gateways used for two main functions:
1. Interconnecting each site to the Inter-Site network (for
East-West traffic flows)
2. Connecting each site to the external Layer 3 domain
(for North-South traffic flows)
May also be used to connect End-Points and/or
network service nodes (FWs, ADCs)
Two deployment models supported:
1. Anycast Border Gateways
2. VPC Border Gateways
Anycast Border Gateways
VPC Border Gateways
BRKDCN-2035 33

34. Anycast Border
Gateways

35. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Site 1
Anycast Border Gateway (1)
Anycast Border Gateway
Up to 4 Border Gateways
Border Gateway
• Deploying at Leaf – 7.0(3)I7(1)
• Deploying at Spine – 7.0(3)I7(2)
VTEP
BGW
VTEP
BGW
VTEP
BGW
VTEP
BGW
BRKDCN-2035 35

36. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Site 1
Anycast Border Gateway (2)
Anycast Border Gateway
Common Virtual IP (VIP) across BGW
• VIP is used for Intra- and Inter-Site
Communication
• VIP for communication between the Border
Gateways in different Sites
• VIP for communication between Border
Gateway and Leaf within a Site
Individual Primary IP (PIP) per BGW
• Used for Broadcast, Unknown Unicast and
Multicast (BUM) replication
• PIP for communication with Single-Homed
End-Points (routed only), intra- and inter-Site
VTEP
BGW
VTEP
BGW
VTEP
BGW
VTEP
BGW
Border VIP
10.1.1.111
Border VIP
10.1.1.111
PIP-BGW1
10.1.1.101
PIP-BGW2
10.1.1.102
PIP-BGW3
10.1.1.103
PIP-BGW4
10.1.1.104
BRKDCN-2035 36

37. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Site 1
Anycast Border Gateway (3)
Anycast Border Gateway
Per-VNI Designated Forwarder (DF) election
• Each BGW can serve as DF for a single or a
set of Layer-2 VNI
• DF election and assignment is automatic
Using BGP EVPN Route Type 4 for DF election
• Operator Managed Assignment (Type: 00)
• Six Octet Site Identifier (System MAC:
00:00:00:00:00:01)
• Multi-Site Discriminator (Ethernet-Segment:
00:00:07)
• Originators IP Address (PIP): 10.1.1.101
• Layer-2 VNI: 30010
VTEP
BGW
VTEP
BGW
VTEP
BGW
VTEP
BGW
Spine
RR
Spine
RR
BGP EVPN
Type: 00
System MAC: 00:00:00:00:00:01
Ethernet Segment: 00:00:074
IP: 10.1.1.101
VNI: 30010
DF
30010
DF
30099
DF
30012
DF
30011
BRKDCN-2035 37

38. VPC Border
Gateways

39. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Site 1
VPC Border Gateway (1)
VPC Border Gateway
2 Border Gateways
Border Gateway
• Using a Leaf – 7.0(3)I7(2)
VTEP
BGW
VTEP
BGW
BRKDCN-2035 39

40. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Site 1
VPC Border Gateway (2)
VPC Border Gateway
Common Virtual IP (VIP) across BGW
• VIP is used for Intra- and Inter-Site
Communication
• VIP for communication between the Border
Gateways in different Sites
• VIP for communication between Border
Gateway and Leaf within a Site
Individual Primary IP (PIP) per BGW
• Used for Broadcast, Unknown Unicast and
Multicast (BUM) replication
• PIP for communication with Single-Homed
End-Points, intra- and inter-Site
VTEP
BGW
VTEP
BGW
Border VIP
10.1.1.111
Border VIP
10.1.1.111
PIP-BGW1
10.1.1.101
PIP-BGW2
10.1.1.102
BRKDCN-2035 40

41. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Site 1
VPC Border Gateway (3)
VPC Border Gateway
VPC-based Designated Forwarder Election
Per-Site Designated Forwarder (DF) election
• Using same approach as in VPC
• Best Path to Rendezvous-Point or VPC
Primary Node
VTEP
BGW
VTEP
BGW
DF
BRKDCN-2035 41

42. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Site 1
VPC Border Gateway (4)
VPC Border Gateway
Single- or Dual-Homed End-Points
• Services Appliance (i.e. Firewall, ADC etc.)
• Physical or Virtual Servers
Advertised and Reachable through Virtual IP
Address (VIP)
• Intra-Site: Leaf nodes use VIP to reach End-
Points connected to Border Gateways
• Inter-Site: Remote Border Gateways use VIP
to reach End-Points connected to Border
• Traffic potentially traverses VPC Peer-Link
VTEP
BGW
VTEP
BGW
VTEP
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.
2 0000.3010.1101/48 30010, 65599:30010 192.168.10.101/32 50001, 65599:50001 10.1.1.111
2 0000.3010.1102/48 30010, 65599:30010 192.168.10.102/32 50001, 65599:50001 10.1.1.111
Border VIP
10.1.1.111
Border VIP
10.1.1.111
ADC
0000.3010.1102
192.168.10.102
ADC
ADC
0000.3010.1101
192.168.10.101
ADC
BRKDCN-2035 42

43. Multi-Site Control
Plane

44. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Control Plane Deployment Considerations
Both MP-eBGP or MP-iBGP peering supported intra-Site between leaf nodes
Only MP-eBGP EVPN sessions supported inter-Sites mandates that each
site is part of a separate AS
Full mesh of MP-eBGP EVPN adjacencies only currently supported across sites
• Recommended to deploy a couple of Route-Servers in the Inter-Site network when 3 or
more sites are deployed
• Route-Servers only perform control plane functions (“eBGP Route-Reflectors”)
• Need to ensure that Route-Servers offer support for Route Type 4 EVPN routes,
required for DF election
BRKDCN-2035 44

45. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
DCI
Multi-Site – Overlay Control-Plane (L3Core)
Spine Spine
VXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP
….
Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP
….BGW BGW BGW BGW
RR RR
iBGP-EVPN iBGP-EVPN
DC Core
(Layer-3 Unicast)
BRKDCN-2035 45

46. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
DCI
Multi-Site – Overlay Control-Plane (L3Core)
Spine Spine
VXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP
….
Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP
….BGW BGW BGW BGW
RR RR
iBGP-EVPN iBGP-EVPN
DC Core
(Layer-3 Unicast)
RS Route Server (eBGP ”Route Reflector”)
RS
BRKDCN-2035 46

47. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
DCI
Multi-Site – Overlay Control-Plane (L3Core)
DC Core
(Layer-3 Unicast)
RS
Spine Spine
VXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP
….
Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP
….BGW BGW BGW BGW
RR RR
iBGP-EVPN iBGP-EVPN
RS – Route Server (eBGP ”Route Reflector”) BRKDCN-2035 47

48. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
DCI
Multi-Site – Overlay Control-Plane (L3Core, no RS)
DC Core
(Layer-3 Unicast)
Spine Spine
VXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP
….
Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP
….BGW BGW BGW BGW
RR RR
iBGP-EVPN iBGP-EVPN
RS – Route Server (eBGP ”Route Reflector”)
eBGP-EVPN
BRKDCN-2035 48

49. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
DCI
Multi-Site – Overlay Control-Plane
DC Core
(Layer-3 Unicast)
RS
Spine Spine
VXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP
….
Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP
….BGW BGW BGW BGW
RR RR
iBGP-EVPN
Host1
0000.3010.1101
192.168.10.101
Host3
0000.3010.1102
192.168.10.102
Host2
0000.3020.2101
192.168.20.101
VRF
Tenant1
L3VNI: 50001
Route-Target: 65501:50001
VRF
Tenant1
L3VNI: 50001
Route-Target: 65502:50001
L2VNI: 30010 (VLAN 10)
L3VNI: 50001 (Tenant1)
L2VNI: 30020 (VLAN 20)
L3VNI: 50001 (Tenant1)
L2VNI: 30010 (VLAN 10)
L3VNI: 50001 (Tenant1)
iBGP-EVPN
VIP1
10.1.1.111
VIP2
10.2.2.222
RS – Route Server (eBGP ”Route Reflector”) BRKDCN-2035 49

50. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
DCI
Multi-Site – Overlay Control-Plane (Site1)
DC Core
(Layer-3 Unicast)
RS
Spine Spine
VXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP
….
Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP
….BGW BGW BGW BGW
RR RR
Host1
0000.3010.1101
192.168.10.101
Host3
0000.3010.1102
192.168.10.102
Host2
0000.3020.2101
192.168.20.101
VRF
Tenant1
L3VNI: 50001
Route-Target: 65502:50001
L2VNI: 30010 (VLAN 10)
L3VNI: 50001 (Tenant1)
L2VNI: 30020 (VLAN 20)
L3VNI: 50001 (Tenant1)
L2VNI: 30010 (VLAN 10)
L3VNI: 50001 (Tenant1)
VRF
Tenant1
L3VNI: 50001
Route-Target: 65501:50001
VIP1
10.1.1.111
VIP2
10.2.2.222
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.
2 0000.3010.1101/48 30010, 65501:30010 192.168.10.101/32 50001, 65501:50001 10.1.1.1
2 0000.3020.2101/48 30020, 65501:30020 192.168.20.101/32 50001, 65501:50001 10.1.1.111
2 0000.3010.1102/48 30010, 65501:30010 192.168.10.102/32 50001, 65501:50001 10.1.1.111
BRKDCN-2035 50

51. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
DCI
Multi-Site – Overlay Control-Plane (Site2)
DC Core
(Layer-3 Unicast)
RS
Spine Spine
VXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP
….
Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP
….BGW BGW BGW BGW
RR RR
Host1
0000.3010.1101
192.168.10.101
Host3
0000.3010.1102
192.168.10.102
Host2
0000.3020.2101
192.168.20.101
VRF
Tenant1
L3VNI: 50001
Route-Target: 65502:50001
L2VNI: 30010 (VLAN 10)
L3VNI: 50001 (Tenant1)
L2VNI: 30020 (VLAN 20)
L3VNI: 50001 (Tenant1)
L2VNI: 30010 (VLAN 10)
L3VNI: 50001 (Tenant1)
VRF
Tenant1
L3VNI: 50001
Route-Target: 65502:50001
VIP1
10.1.1.111
VIP2
10.2.2.222
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.
2 0000.3010.1101/48 30010, 65502:30010 192.168.10.101/32 50001, 65502:50001 10.2.2.222
2 0000.3020.2101/48 30020, 65502:30020 192.168.20.101/32 50001, 65502:50001 10.2.2.1
2 0000.3010.1102/48 30010, 65502:30010 192.168.10.102/32 50001, 65502:50001 10.2.2.3
BRKDCN-2035 51

52. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
DCI
Multi-Site – Overlay Control-Plane (DCI)
DC Core
(Layer-3 Unicast)
RS
RS – Route Server (eBGP ”Route Reflector”)
Spine Spine
VXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP
….
Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP
….BGW BGW BGW BGW
RR RR
VRF
Tenant1
L3VNI: 50001
Route-Target: 65501:50001
VRF
Tenant1
L3VNI: 50001
Route-Target: 65502:50001
VIP1
10.1.1.111
VIP2
10.2.2.222
L2VNI: 30010 (VLAN 10)
L3VNI: 50001 (Tenant1)
L2VNI: 30020 (VLAN 20)
L3VNI: 50001 (Tenant1)
L2VNI: 30010 (VLAN 10)
L3VNI: 50001 (Tenant1)
Host1
0000.3010.1101
192.168.10.101
Host3
0000.3010.1102
192.168.10.102
Host2
0000.3020.2101
192.168.20.101
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.
2 0000.3010.1101/48 30010, 65599:30010 192.168.10.101/32 50001, 65599:50001 10.1.1.111
2 0000.3020.2101/48 30020, 65599:30020 192.168.20.101/32 50001, 65599:50001 10.2.2.222
2 0000.3010.1102/48 30010, 65599:30010 192.168.10.102/32 50001, 65599:50001 10.2.2.222
BRKDCN-2035 52

53. Multi-Site –
Selective
Advertisements

54. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multi-Site – Selective Advertisements
The Multi-Site architecture provides granular control on how Layer-2
and Layer-3 communication is extended across sites
Layer-2 and/or Layer-3 VNIs configured on the Border Gateways
(BGW) control the Control-Plane advertisement towards DCI
Enhances the overall scalability of the solution
• Scale up the total number of End-Points supported across sites
BRKDCN-2035 54

55. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
DCI
Multi-Site – Selective Advertisements (DCI)
DC Core
(Layer-3 Unicast)
RS
Spine Spine
VXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP
….
Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP
….BGW BGW BGW BGW
RR RR
RS – Route Server (eBGP ”Route Reflector”)
Only prefixes of VRF ”Tenant1” and L2VNI 30010 are
advertised from Site1 towards DCI. In this example this is
Host1.
All prefixes of VRF ”Tenant2” and L2VNI 30020 are not
advertised from Site2 towards DCI. These prefixes are not
seen within the DCI
VRF
Tenant1
L3VNI: 50001
Route-Target: 65501:50001
VIP1
10.1.1.111
VIP2
10.2.2.222Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.
2 0000.3010.1101/48 30010, 65599:30010 192.168.10.101/32 50001, 65599:50001 10.1.1.111
Host1
0000.3010.1101
192.168.10.101
L2VNI: 30010 (VLAN 10)
L3VNI: 50001 (Tenant1)
L2VNI: 30020 (VLAN 20)
L3VNI: 50002 (Tenant2)
Host2
0000.3020.2101
192.168.20.101
BRKDCN-2035 55

56. Multi-Site Data Plane

57. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
DCI
Multi-Site – Overlay Data Plane
DC Core
(Layer-3 Unicast)
Spine Spine
VXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP
….
Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP
….
VIP1
10.1.1.111
VIP2
10.2.2.222
BGW BGW BGW BGW
Host1
0000.3010.1101
192.168.10.101
Host3
0000.3010.1102
192.168.10.102
Host2
0000.3020.2101
192.168.20.101
Intra-site VXLAN
Data Plane
Inter-site VXLAN
Data Plane
De-capsulation and
Re-encapsulation on
BGW
De-capsulation and
Re-encapsulation on
BGW
BRKDCN-2035 57

58. Multi-Site Packet
Walk (BUM)

59. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet Walk – Layer-2 (BUM) – Site1
VXLAN EVPN
Site2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPN
Site1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPN
DCI
Baremetal
Host 1
0000.3010.1101
192.168.10.101
Baremetal
Host 2
0000.3010.1102
192.168.10.102
SIP DIP VXLAN SMAC DMAC SIP DIP
Payload
L10 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255
Bridge
DF
30010
DF
30010
Host 1 sends a L2
BUM frame
1
2
Leaf10 replicates
traffic intra-Site
BRKDCN-2035 59

60. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet Walk – Layer-2 (DF & Split Horizon) – Site1
VXLAN EVPN
Site2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPN
Site1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPN
DCI
Baremetal
Host 1
0000.3010.1101
192.168.10.101
Baremetal
Host 2
0000.3010.1102
192.168.10.102
SIP DIP VXLAN SMAC DMAC SIP DIP
Payload
L10 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255
Bridge
DF
30010
DF
30010
BUM Forward
Drop due to Split-Horizon rule
Drop due to Designated Forwarder (DF) rule
BRKDCN-2035 60

61. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet Walk – Layer-2 (BUM) – DCI
VXLAN EVPN
Site2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPN
Site1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPN
DCI
Baremetal
Host 1
0000.3010.1101
192.168.10.101
Baremetal
Host 2
0000.3010.1102
192.168.10.102
Bridge
DF
30010
DF
30010
SIP DIP VXLAN SMAC DMAC SIP DIP
Payload
BGW-VIP1 BGW21 30010 H1-MAC ALL-F H1-IP ALL-255
BGW-VIP1 BGW22 30010 H1-MAC ALL-F H1-IP ALL-255
BGW-VIP1 BGW12 30010 H1-MAC ALL-F H1-IP ALL-255
3
BGW11 replicates traffic inter-
Sites toward BGW nodes
BUM Forward
BRKDCN-2035 61

62. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet Walk – Layer-2 (DF & Split Horizon) – DCI
VXLAN EVPN
Site2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPN
Site1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPN
DCI
Baremetal
Host 1
0000.3010.1101
192.168.10.101
Baremetal
Host 2
0000.3010.1102
192.168.10.102
Bridge
DF
30010
DF
30010
SIP DIP VXLAN SMAC DMAC SIP DIP
Payload
BGW-VIP1 BGW21 30010 H1-MAC ALL-F H1-IP ALL-255
BGW-VIP1 BGW22 30010 H1-MAC ALL-F H1-IP ALL-255
BGW-VIP1 BGW12 30010 H1-MAC ALL-F H1-IP ALL-255
BUM Forward
BUM Forward
Drop due to Split-Horizon rule
Drop due to Designated Forwarder (DF) rule
BRKDCN-2035 62

63. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet Walk – Layer-2 (BUM) – Site2
VXLAN EVPN
Site2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPN
Site1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPN
DCI
Baremetal
Host 1
0000.3010.1101
192.168.10.101
Baremetal
Host 2
0000.3010.1102
192.168.10.102
Bridge
DF
30010
DF
30010
SIP DIP VXLAN SMAC DMAC SIP DIP
Payload
BGW-VIP2 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255
4
BGW22 replicates traffic
intra-Site
BUM Forward
BRKDCN-2035 63

64. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet Walk – Layer-2 (DF & Split Horizon) – Site2
VXLAN EVPN
Site2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPN
Site1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPN
DCI
Baremetal
Host 1
0000.3010.1101
192.168.10.101
Baremetal
Host 2
0000.3010.1102
192.168.10.102
Bridge
DF
30010
DF
30010
SIP DIP VXLAN SMAC DMAC SIP DIP
Payload
BGW-VIP2 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255
BUM Forward
Drop due to Split-Horizon rule
Drop due to Designated Forwarder (DF) rule
BRKDCN-2035 64

65. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet Walk – Layer-2 (BUM) – Site2
VXLAN EVPN
Site2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPN
Site1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPN
DCI
Baremetal
Host 1
0000.3010.1101
192.168.10.101
Baremetal
Host 2
0000.3010.1102
192.168.10.102
Bridge
DF
30010
DF
30010
Leaf20 sends traffic to
local Host 2
5
BRKDCN-2035 65

66. Multi-Site Packet
Walk (Bridging)

67. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet Walk – Layer-2 (Host 1 to Host 2) – Site1
VXLAN EVPN
Site2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPN
Site1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPN
DCI
Baremetal
Host 1
0000.3010.1101
192.168.10.101
Baremetal
Host 2
0000.3010.1102
192.168.10.102
SIP DIP VXLAN SMAC DMAC SIP DIP
Payload
L10 BGW-VIP1 30010 H1-MAC H2-MAC H1-IP H2-IP
Bridge
Host 1 sends traffic
destined to remote Host 2
1
2
Leaf10 performs L2 lookup
and encapsulates toward
local BGW VIP1 address
VIP2VIP1
BRKDCN-2035 67

68. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet Walk – Layer-2 (Host 1 to Host 2) – DCI
VXLAN EVPN
Site2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPN
Site1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPN
DCI
Baremetal
Host 1
0000.3010.1101
192.168.10.101
Baremetal
Host 2
0000.3010.1102
192.168.10.102
SIP DIP VXLAN SMAC DMAC SIP DIP
Payload
BGW-VIP1 BGW-VIP2 30010 H1-MAC H2-MAC H1-IP H2-IP
Bridge
3
BGW11 performs L2 lookup
and encapsulates toward
remote BGW VIP2 address
VIP2VIP1
BRKDCN-2035 68

69. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet Walk – Layer-2 (Host 1 to Host 2) – Site2
VXLAN EVPN
Site2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPN
Site1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPN
DCI
Baremetal
Host 1
0000.3010.1101
192.168.10.101
Baremetal
Host 2
0000.3010.1102
192.168.10.102
Bridge
SIP DIP VXLAN SMAC DMAC SIP DIP
Payload
BGW-VIP2 L20 30010 H1-MAC H2-MAC H1-IP H2-IP
4
BGW22 performs L2 lookup
and encapsulates toward
destination L20 node
Leaf20 bridges traffic to
local Host 2
5
VIP2VIP1
BRKDCN-2035 69

70. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet Walk – Layer-2 (Host 2 to Host 1) – Site2
VXLAN EVPN
Site2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPN
Site1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPN
DCI
Baremetal
Host 1
0000.3010.1101
192.168.10.101
Baremetal
Host 2
0000.3010.1102
192.168.10.102
Bridge
SIP DIP VXLAN SMAC DMAC SIP DIP
Payload
L20 BGW-VIP2 30010 H2-MAC H1-MAC H2-IP H1-IP
Host 2 replies to remote
Host 1
6
7
Leaf20 performs L2 lookup
and encapsulates toward
local BGW VIP2 address
VIP2VIP1
BRKDCN-2035 70

71. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet Walk – Layer-2 (Host 2 to Host 1) – DCI
VXLAN EVPN
Site2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPN
Site1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPN
DCI
Baremetal
Host 1
0000.3010.1101
192.168.10.101
Baremetal
Host 2
0000.3010.1102
192.168.10.102
Bridge
SIP DIP VXLAN SMAC DMAC SIP DIP
Payload
BGW-VIP2 BGW-VIP1 30010 H2-MAC H1-MAC H2-IP H1-IP
8
BGW21 performs L2 lookup
and encapsulates toward
remote BGW VIP1 address
VIP2VIP1
BRKDCN-2035 71

72. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet Walk – Layer-2 (Host 2 to Host 1) – Site1
VXLAN EVPN
Site2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPN
Site1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPN
DCI
Baremetal
Host 1
0000.3010.1101
192.168.10.101
Baremetal
Host 2
0000.3010.1102
192.168.10.102
Bridge
SIP DIP VXLAN SMAC DMAC SIP DIP
Payload
BGW-VIP1 L10 30010 H2-MAC H1-MAC H2-IP H1-IP
9
BGW12 performs L2 lookup
and encapsulates toward
destination L10 node
Leaf10 bridges traffic
toward Host 1
10
VIP2VIP1
BRKDCN-2035 72

73. Multi-Site Packet
Walk (Routing)

74. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet Walk – Layer-3 (Host 1 to Host 3) – Site1
VXLAN EVPN
Site2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPN
Site1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPN
DCI
Baremetal
Host 1
0000.3010.1101
192.168.10.101
Baremetal
Host 3
0000.3010.1102
192.168.20.102
SIP DIP VXLAN SMAC DMAC SIP DIP
Payload
L10 BGW-VIP1 50001 L10-MAC BGW-VMAC1 H1-IP H3-IP
Route
Host 1 sends a data
packet to the remote
Host 3
1
2
Leaf10 performs a L3 lookup
and encapsulates toward
local BGW VIP1 address
VIP2
VMAC2
VIP1
VMAC1
BRKDCN-2035 74

75. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet Walk – Layer-3 (Host 1 to Host 3) – DCI
VXLAN EVPN
Site2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPN
Site1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPN
DCI
Baremetal
Host 1
0000.3010.1101
192.168.10.101
Baremetal
Host 3
0000.3010.1102
192.168.20.102
Route
SIP DIP VXLAN SMAC DMAC SIP DIP
Payload
BGW-VIP1 BGW-VIP2 50001 BGW-VMAC1 BGW-VMAC2 H1-IP H3-IP
3
BGW11 performs a L3 lookup
and encapsulates toward
remote BGW VIP2 address
VIP2
VMAC2
VIP1
VMAC1
BRKDCN-2035 75

76. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet Walk – Layer-3 (Host 1 to Host 3) – Site2
VXLAN EVPN
Site2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPN
Site1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPN
DCI
Baremetal
Host 1
0000.3010.1101
192.168.10.101
Baremetal
Host 3
0000.3010.1102
192.168.20.102
Route
SIP DIP VXLAN SMAC DMAC SIP DIP
Payload
BGW-VIP2 L20 50001 BGW-VMAC1 L20-MAC H1-IP H3-IP
4
BGW21 performs a L3
lookup and encapsulates
toward destination L20 node
Leaf20 routes traffic to
local Host 3
5
VIP2
VMAC2
VIP1
VMAC1
BRKDCN-2035 76

77. Multi-Site and
Failure Detection
on BGW

78. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Steady State Traffic – Site1
VXLAN EVPN
Site2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPN
Site1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPN
DCI
Baremetal
Host 1
0000.3010.1101
192.168.10.101
Baremetal
Host 2
0000.3010.1102
192.168.10.102
SIP DIP VXLAN SMAC DMAC SIP DIP
Payload
L10 BGW-VIP1 30010 H1-MAC H2-MAC H1-IP H2-IP
VIP2VIP1
BRKDCN-2035 78

79. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
DCI Link Failure BGW12 – Site1
VXLAN EVPN
Site2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPN
Site1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPN
DCI
Baremetal
Host 1
0000.3010.1101
192.168.10.101
Baremetal
Host 2
0000.3010.1102
192.168.10.102
SIP DIP VXLAN SMAC DMAC SIP DIP
Payload
L10 BGW-VIP1 30010 H1-MAC H2-MAC H1-IP H2-IP
On DCI Link Failure (i.e. BGW12)
• Virtual IP (VIP) on BGW is disabled
• BGW will stop participating in DF election
• BGW acts like a Leaf (Layer-3 only)
• Traffic towards others Sites is served by remaining
BGWs (i.e. BGW11)
BGW12 gets isolated from
the DCI Core Network
Intra-site VXLAN traffic re-
routing
VIP2VIP1
BRKDCN-2035 79

80. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric Link Failure BGW12 – Site1
VXLAN EVPN
Site2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPN
Site1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPN
DCI
Baremetal
Host 1
0000.3010.1101
192.168.10.101
Baremetal
Host 2
0000.3010.1102
192.168.10.102
SIP DIP VXLAN SMAC DMAC SIP DIP
Payload
L10 BGW-VIP1 30010 H1-MAC H2-MAC H1-IP H2-IP
VIP2VIP1
BRKDCN-2035 80

81. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric Link Failure BGW12 – Site1
VXLAN EVPN
Site2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPN
Site1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPN
DCI
Baremetal
Host 1
0000.3010.1101
192.168.10.101
Baremetal
Host 2
0000.3010.1102
192.168.10.102
SIP DIP VXLAN SMAC DMAC SIP DIP
Payload
L10 BGW-VIP1 30010 H1-MAC H2-MAC H1-IP H2-IP
VIP2VIP1
BGW12 gets isolated from
the Spine nodes
Intra-site VXLAN traffic re-
routing
On Fabric Link Failure (i.e. BGW12)
• Virtual IP (VIP) on BGW is disabled
• Primary IP (PIP) on BGW is disabled
• BGW will stop participating in the Overlay
BRKDCN-2035 81

82. Multi-Site Setup
Walkthrough

83. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
Site 1 Setup – Enable Border Gateway
Spine Spine
VTEP VTEP VTEP VTEP
VTEP VTEP
….BGW1 BGW2
Multi-Site Commands are marked in red
Various options do exist but the recommended design
choices are:
• Fabric Internal
IGP Underlay, iBGP Overlay
• DCI (primary choice)
eBGP Underlay, eBGP Overlay
Route Server for DCI Overlay peerings
DC Core for reachability across n Sites
• DCI (alternative option)
Any Routing Protocol Underlay, eBGP Overlay
Full-Mesh for DCI Overlay peerings
Back-to-Back Site Reachability (physical, full-
mesh)
BRKDCN-2035 83

84. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
Site 1 Setup – Enable Border Gateway
Spine Spine
VTEP VTEP VTEP VTEP
VTEP VTEP
….BGW1 BGW2
feature nv overlay
nv overlay evpn
feature bgp
feature interface-vlan
feature vn-segment-vlan-based
evpn multisite border-gateway
BGW2BGW1
BRKDCN-2035 84

85. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
Site 1 Setup – BGW 1 Loopback & VTEP
Spine Spine
VTEP VTEP VTEP VTEP
VTEP VTEP
….BGW1
interface loopback1
description PIP VTEP
ip address 10.1.1.101/32 tag 12345
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
interface loopback100
description VIP Multi-Site 1
ip address 10.1.1.111/32 tag 12345
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
interface loopback0
description RID
ip address 10.10.10.101/32 tag 12345
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
BGW1
BRKDCN-2035 85

86. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
Site 1 Setup – BGW 2 Loopback & VTEP
Spine Spine
VTEP VTEP VTEP VTEP
VTEP VTEP
…. BGW2
interface loopback1
description PIP VTEP
ip address 10.1.1.102/32 tag 12345
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
interface loopback100
description VIP Multi-Site 1
ip address 10.1.1.111/32 tag 12345
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
interface loopback0
description RID
ip address 10.10.10.102/32 tag 12345
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
BGW2
BRKDCN-2035 86

87. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
Site 1 Setup – Fabric Link Tracking BGW 1
Spine Spine
VTEP VTEP VTEP VTEP
VTEP VTEP
….BGW1
interface Ethernet1/53
description TO-SPINE1
ip address 10.0.1.1/30
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
evpn multisite fabric-tracking
interface Ethernet1/54
description TO-SPINE2
ip address 10.0.2.1/30
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
evpn multisite fabric-tracking
BGW1
Allows to bring down the PIP/VIP
loopback interfaces when the
BGW is isolated from the spines
BRKDCN-2035 87

88. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
Site 1 Setup – Fabric Link Tracking BGW 2
Spine Spine
VTEP VTEP VTEP VTEP
VTEP VTEP
…. BGW2
interface Ethernet1/53
description TO-SPINE1
ip address 10.0.1.5/30
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
evpn multisite fabric-tracking
interface Ethernet1/54
description TO-SPINE2
ip address 10.0.2.5/30
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
evpn multisite fabric-tracking
BGW2
BRKDCN-2035 88

89. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Core
(Layer-3 Unicast)
Fabric
Site 1 Setup – Multi-Site Underlay Interface
Spine Spine
VTEP VTEP VTEP VTEP
VTEP VTEP
….BGW1 BGW2
interface Ethernet1/1
description TO-DC-CORE1
ip address 10.111.111.1/30 tag 12345
evpn multisite dci-tracking
interface Ethernet1/2
description TO-DC-CORE2
ip address 10.111.222.1/30 tag 12345
evpn multisite dci-tracking
BGW1
DCI
interface Ethernet1/1
description TO-DC-CORE1
ip address 10.222.111.1/30 tag 12345
evpn multisite dci-tracking
interface Ethernet1/2
description TO-DC-CORE2
ip address 10.222.222.1/30 tag 12345
evpn multisite dci-tracking
BGW2
Allows to bring down the PIP/VIP loopback interfaces
when the BGW is isolated from the DC core BRKDCN-2035 89

90. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Core
(Layer-3 Unicast)
Fabric
Site 1 BGW 1 Setup – Multi-Site Overlay Peering
Spine Spine
VTEP VTEP VTEP VTEP
VTEP VTEP
….BGW1
DCI
RS router bgp 65501
router-id 10.10.10.101
address-family ipv4 unicast
redistribute direct route-map REDIST-LOCAL
neighbor 10.111.111.2
remote-as 65599
update-source ethernet1/1
address-family ipv4 unicast
neighbor 10.111.222.2
remote-as 65599
update-source ethernet1/2
address-family ipv4 unicast
neighbor 10.99.99.201
remote-as 65599
update-source loopback0
ebgp-multihop 5
peer-type fabric-external
address-family l2vpn evpn
rewrite-evpn-rt-asn
send-community
send-community both
BGW1
RS – Route Server (eBGP ”Route Reflector”) BRKDCN-2035 90

91. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Core
(Layer-3 Unicast)
Fabric
Site 1 BGW 2 Setup – Multi-Site Overlay Peering
Spine Spine
VTEP VTEP VTEP VTEP
VTEP VTEP
….
DCI
RS router bgp 65501
router-id 10.10.10.102
address-family ipv4 unicast
redistribute direct route-map REDIST-LOCAL
neighbor 10.222.111.2
remote-as 65599
update-source ethernet1/1
address-family ipv4 unicast
neighbor 10.222.222.2
remote-as 65599
update-source ethernet1/2
address-family ipv4 unicast
neighbor 10.99.99.201
remote-as 65599
update-source loopback0
ebgp-multihop 5
peer-type fabric-external
address-family l2vpn evpn
rewrite-evpn-rt-asn
send-community
send-community both
BGW1
RS – Route Server (eBGP ”Route Reflector”)
BGW2
BRKDCN-2035 91

92. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Core
(Layer-3 Unicast)
Fabric
Site 1 Setup – Multi-Site Overlay Peering
Spine Spine
VTEP VTEP VTEP VTEP
VTEP VTEP
….BGW1
DCI
RS
RS – Route Server (eBGP ”Route Reflector”)
BGW2
peer-type fabric-external
• Enables Next-Hop Rewrite for Multi-Site
• Defines Site External BGP neighbors for EVPN
exchange
rewrite-evpn-rt-asn
• Rewrites Route-Target Auto information to simplify
MAC-VRF and IP-VRF configuration
• Normalizes outgoing Route-Targets AS number to
match remote AS number
• Uses BGP configured Neighbors Remote AS
BRKDCN-2035 92

93. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
DCI
Site 1 Setup – Multi-Site Overlay Peering
DC Core
(Layer-3 Unicast)
Spine Spine
VXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP
….
Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP
….
VIP1
10.1.1.111
VIP2
10.2.2.222
BGW BGW BGW BGW
Host1
0000.3010.1101
192.168.10.101
peer-type fabric-external peer-type fabric-external
BGP Update:
MAC: 0000.3010.1101 (L2VNI 30001)
IP: 192.168.20.101 (L3VNI 50001)
NH: 10.2.2.222
RMAC: BGW-VMAC2
BGP Update:
MAC: 0000.3010.1101 (L2VNI 30001)
IP: 192.168.20.101 (L3VNI 50001)
NH: 10.1.1.111
RMAC: BGW-VMAC1
Rewrite Next-Hop IP and Next-
Hop MAC (RMAC) based on
Neighbor Site BGW
BGP Update:
MAC: 0000.3010.1101 (L2VNI 30001)
IP: 192.168.20.101 (L3VNI 50001)
NH: 10.1.1.1
RMAC: Leaf1
Rewrite Next-Hop IP and Next-
Hop MAC (RMAC) based on
Neighbor Site BGW
BRKDCN-2035 93

94. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
DCI
Site 1 Setup – Multi-Site Overlay Peering
DC Core
(Layer-3 Unicast)
Spine Spine
VXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP
….
Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP
….
VIP1
10.1.1.111
VIP2
10.2.2.222
BGW BGW BGW BGW
Host1
0000.3010.1101
192.168.10.101
Host2
0000.3020.2101
192.168.20.101
rewrite-evpn-rt-asn rewrite-evpn-rt-asn
BGP Update:
Remote AS: 65502
VNI: 50001
Route-Target: 65502:50001
Rewrite Route-Target based on
BGP Neighbors Remote ASN
BGP Update:
Remote AS : 65501
VNI: 50001
Route-Target: 65501:50001
BGP Update:
Remote AS: 65502
VNI: 50001
Route-Target: 65502:50001
BRKDCN-2035 94

95. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Core
(Layer-3 Unicast)
Fabric
Site 1 Setup – Anycast BGW VTEP Configuration
Spine Spine
VTEP VTEP VTEP VTEP
VTEP VTEP
….BGW1
DCI
RS – Route Server (eBGP ”Route Reflector”)
interface nve1
no shutdown
host-reachability protocol bgp
multisite ethernet-segment 7
system-mac 0000.0000.0001
source-interface loopback1
multisite border-gateway interface loopback100
member vni 30010
multisite ingress-replication
mcast-group 239.1.1.1
member vni 30011-30020
mcast-group 239.1.1.2
member vni 50001 associate-vrf
BGW1
BGW2
BGW2
BRKDCN-2035 95

96. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Core
(Layer-3 Unicast)
Fabric
Site 1 Setup – Anycast BGW VTEP Configuration
Spine Spine
VTEP VTEP VTEP VTEP
VTEP VTEP
….BGW1
DCI
BGW2
multisite ethernet-segment
• Defines the discriminator for Sites in a common Domain
system-mac
• Defines the Multi-Site Site-Id (6 octets hex)
multisite border-gateway interface loopback#
• Defines the Loopback Interface used for the Border
Gateway Virtual IP Address (VIP)
multisite ingress-replication
• Per-VNI knob for extending Layer-2 VNI
• Defines the Multi-Site BUM Replication method
BRKDCN-2035 96

97. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Core
(Layer-3 Unicast)
Fabric
Site 1 Setup – Multi-Site Overlay Traffic Policy
Spine Spine
VTEP VTEP VTEP VTEP
VTEP VTEP
….BGW1
DCI
BGW2
• BUM Traffic Policing
• Limits Broadcast, Unknown Unicast and Layer-2
Multicast Traffic across Multi-Site
• Level 0 = No B/U/M Forwarding
• Level 100 = All B/U/M Forwarding Forwarding
• Enforced on Encapsulation towards remote Sites
evpn storm-control broadcast level 10
evpn storm-control unicast level 10
evpn storm-control multicast level 10
BGW1 BGW2
BRKDCN-2035 97

98. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Site 1 Setup – Multi-Site Overlay Traffic Policy
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site 1
VTEP VTEP
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site n
VTEP VTEP
Overlay Multi-Site
Overlay Site 1 Overlay Site n
Baremetal
BUM
BGW BGW BGW BGW
BRKDCN-2035 98

99. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Site 1 Setup – Multi-Site Overlay Traffic Policy
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site 1
VTEP VTEP
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site n
VTEP VTEP
Overlay Multi-Site
Overlay Site 1 Overlay Site n
Baremetal
BUM
Storm Control
Broadcast 0-100%
Unknown Unicast 0-100%
Multicast 0-100%
Storm Control
Broadcast 0-100%
Unknown Unicast 0-100%
Multicast 0-100%
BGW BGW BGW BGW
BRKDCN-2035 99

100. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Site 1 Setup – Multi-Site BUM Replication Modes
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site 1
VTEP VTEP
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site n
VTEP VTEP
Overlay Multi-Site
Overlay Site 1 Overlay Site n
Multicast Multicast
Ingress Replication
BGW BGW BGW BGW
BRKDCN-2035 100

101. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Site 1 Setup – Multi-Site BUM Replication Modes
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site 1
VTEP VTEP
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site n
VTEP VTEP
Overlay Multi-Site
Overlay Site 1 Overlay Site n
Ingress Replication Ingress Replication
Ingress Replication
BGW BGW BGW BGW
BRKDCN-2035 101

102. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Site 1 Setup – Multi-Site BUM Replication Modes
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site 1
VTEP VTEP
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site n
VTEP VTEP
Overlay Multi-Site
Overlay Site 1 Overlay Site n
Ingress Replication Multicast
Ingress Replication
BGW BGW BGW BGW
BRKDCN-2035 102

103. Connectivity to the
External Layer 3
Domain

104. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Connectivity to the External Layer 3 Domain
The BGW nodes can be used to provide Layer-3 external
connectivity to each site
Different connectivity models are supported
• VRF-Lite peering with an external pair of WAN Edge routers
• MP-BGP EVPN peering with the external WAN Edge routers (GOLF)
• Dedicated or shared pair of WAN Edge routers across sites
External Layer-3 network may be different from the DCI network
used for inter-site communication
BRKDCN-2035 104

105. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
DCI
Multi-Site – Border Gateway and VRF-Lite
Spine Spine
VXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP
….
Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP
….
VIP1
10.1.1.111
VIP2
10.2.2.222
BGW BGW BGW BGW
Host1
0000.3010.1101
192.168.10.101
Host3
0000.3010.1102
192.168.10.102
Host2
0000.3020.2101
192.168.20.101
VRF-CVRF-BVRF-ASeparate routing
peering for each VRF
(IGP or eBGP)
Dedicated interface
(logical or physical) for
each VRF
BRKDCN-2035 105

106. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
DCI
Multi-Site – Border Gateway and GOLF
Spine Spine
VXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP
….
Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP
….
VIP1
10.1.1.111
VIP2
10.2.2.222
BGW BGW BGW BGW
Host1
0000.3010.1101
192.168.10.101
Host3
0000.3010.1102
192.168.10.102
Host2
0000.3020.2101
192.168.20.101
VRF-CVRF-BVRF-A
Single MP-BGP EVPN
instance to exchange routes
for all VRFs
VXLAN Data Plane
between BGW and WAN
Edge Router
BRKDCN-2035 106

107. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Core
(Layer-3 Unicast)
MPLS
L3VPN
Fabric
DCI
Spine Spine
VXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP
….
Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP
….
VIP1
10.1.1.111
VIP2
10.2.2.222
BGW BGW BGW BGW
Host1
0000.3010.1101
192.168.10.101
Host2
0000.3020.2101
192.168.20.101
Host3
0000.3010.1102
192.168.10.102
Multi-Site – Shared Internet/WAN Gateways
Internet/WAN
BorderPE BorderPE
Inter-Site VXLAN
Communication between
Border Gateways
BRKDCN-2035 107

108. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multi-Site – Per Site Internet/WAN Gateway
DC Core
(Layer-3 Unicast)
MPLS
L3VPN
Fabric
DCI
Spine Spine
VXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP
….
Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP
….
VIP1
10.1.1.111
VIP2
10.2.2.222
BGW BGW BGW BGW
Host1
0000.3010.1101
192.168.10.101
Host2
0000.3020.2101
192.168.20.101
Host3
0000.3010.1102
192.168.10.102
Internet/WAN
BorderPE BorderPE BorderPE BorderPE
Inter-Site VXLAN
Communication between
Border Gateways
BRKDCN-2035 108

109. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
MPLS
L3VPN
Fabric
DCI
Spine Spine
VXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP
….
Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP
….
VIP1
10.1.1.111
VIP2
10.2.2.222
BGW BGW BGW BGW
Host1
0000.3010.1101
192.168.10.101
Host2
0000.3020.2101
192.168.20.101
Host3
0000.3010.1102
192.168.10.102
Internet/WAN BorderPE BorderPE BorderPE BorderPE
Multi-Site – Consolidated WAN and DCI NetworkPerform simple routing for
inter-site flows, VXLAN (or
VRF-Lite) to MPLS VPN
hand-off for north-south
communication
Inter-Site VXLAN
Communication between
Border Gateways
BRKDCN-2035 109

110. Ingress and Egress
Traffic Optimization

111. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Spine Spine
VXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP
Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP
BGW
WAN
Active FWActive FW
The stretching of Layer-2 domains
across separate sites may lead to
the creation of asymmetric traffic
paths
Deploying independent stateful
services (like FWs) across sites
would result in traffic drops
In this case it is required to ensure
the symmetry of ingress and egress
communication paths
Ingress and Egress Traffic Optimization
The Issue of Extending Layer 2 Domains
BGW BGW BGW
DC Core
(Layer-3 Unicast)
BRKDCN-2035 111

112. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Core
(Layer-3 Unicast)
Guarantee routing symmetry with the
outside of the Data Center
• Egress Always prefer the local BGW
• Ingress Steer traffic to the specific destination
End-Point’s location
Maintain optimal routing over the
dedicated DCI network (if existing) for
Server-to-Server traffic
• The DC fabric must discriminate between DC
and WAN destinations
If required provide a fallback path via
DCI for WAN isolation situations
Spine Spine
VXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP
Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP
BGW BGW BGW BGW
WAN
Ingress and Egress Traffic Optimization
Maintaining Traffic Symmetry over Optimal Paths
BRKDCN-2035 112

113. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Core
(Layer-3 Unicast)
MPLS
L3VPN
Spine Spine
VXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP
Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEPVIP2
10.2.2.222
BGW BGW BGW BGW
Host1
0000.3010.1101
192.168.10.101
Host3
0000.3010.1102
192.168.10.102
BorderPE BorderPE BorderPE BorderPE
VIP1
10.1.1.111
Multi-Site – Egress Path Optimization
172.16.1.10
172.16.1.0/24 Border-PEs 1-2 172.16.1.0/24 Border-PEs 3-4
172.16.1.0/24 VIP1 172.16.1.0/24 VIP2
eBGP-EVPN
Less preferred
advertisement of
172.16.1.0 because
of longer AS-Path
BRKDCN-2035 113

114. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Core
(Layer-3 Unicast)
MPLS
L3VPN
Spine Spine
VXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP
Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEPVIP2
10.2.2.222
BGW BGW BGW BGW
Host1
0000.3010.1101
192.168.10.101
Host3
0000.3010.1102
192.168.10.102
BorderPE BorderPE BorderPE BorderPE
VIP1
10.1.1.111
Multi-Site – Egress Path Optimization
172.16.1.10
eBGP-EVPN
Optimized Egress
Traffic Path
Optimized Egress
Traffic Path
BRKDCN-2035 114

115. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Core
(Layer-3 Unicast)
MPLS
L3VPN
Spine Spine
VXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP
Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEPVIP2
10.2.2.222
BGW BGW BGW BGW
Host1
0000.3010.1101
192.168.10.101
Host3
0000.3010.1102
192.168.10.102
BorderPE BorderPE BorderPE BorderPE
VIP1
10.1.1.111
Multi-Site – Egress Path Optimization
WAN Isolation Scenario
172.16.1.10
172.16.1.0/24 Border-PEs 3-4
172.16.1.0/24 VIP1 172.16.1.0/24 VIP2
eBGP-EVPN
WAN Isolation
Scenario
172.16.1.0/24 VIP2
BRKDCN-2035 115

116. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Core
(Layer-3 Unicast)
MPLS
L3VPN
Spine Spine
VXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP
Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEPVIP2
10.2.2.222
BGW BGW BGW BGW
Host1
0000.3010.1101
192.168.10.101
Host3
0000.3010.1102
192.168.10.102
BorderPE BorderPE BorderPE BorderPE
VIP1
10.1.1.111
Multi-Site – Ingress Path Optimization
192.168.10.0/24 BGW 1-2
192.168.10.101/32 BGW-1-2
192.168.10.0/24 BGW 3-4
192.168.10.102/32 BGW 3-4
192.168.10.101/32 Leaf1 192.168.10.102/32 -> Leaf3
eBGP-EVPN
Host routes
advertised across
sites but NOT re-
advertised toward the
local Border-PEs
192.168.10.0/24 Border-PE 1-4
192.168.10.101/32 Border-PE 1-2
192.168.10.102/32 Border-PE 3-4Host routes
advertisement in
the WAN
Deploying LISP on the
Border-PEs is a viable
alternative to host routes
advertisement
Filter out host routes
received from remote
sites. Only announce
local host route
information
BRKDCN-2035 116

117. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Core
(Layer-3 Unicast)
MPLS
L3VPN
Spine Spine
VXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP
Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEPVIP2
10.2.2.222
BGW BGW BGW BGW
Host1
0000.3010.1101
192.168.10.101
Host3
0000.3010.1102
192.168.10.102
BorderPE BorderPE BorderPE BorderPE
VIP1
10.1.1.111
Multi-Site – Ingress Path Optimization
eBGP-EVPN
192.168.10.0/24 Border-PE 1-4
192.168.10.101/32 Border-PE 1-2
192.168.10.102/32 Border-PE 3-4
Optimized Ingress
Traffic Path
Optimized Ingress
Traffic Path
BRKDCN-2035 117

118. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Core
(Layer-3 Unicast)
MPLS
L3VPN
Spine Spine
VXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP
Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEPVIP2
10.2.2.222
BGW BGW BGW BGW
Host1
0000.3010.1101
192.168.10.101
Host3
0000.3010.1102
192.168.10.102
BorderPE BorderPE BorderPE BorderPE
VIP1
10.1.1.111
eBGP-EVPN
192.168.10.0/24 Border-PE 3-4
192.168.10.101/32 Border-PE 1-2
192.168.10.102/32 Border-PE 3-4
Multi-Site – Ingress Path Optimization
WAN Isolation Scenario
WAN Isolation
Scenario
BRKDCN-2035 118

119. Network Services
Integration

120. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Network Services Integration
Couple of different options where to connect network services:
1. Service Leaf nodes: recommended to connect devices used for east-
west communication
2. Border Gateway Nodes: used to connect network services for north-
south traffic flows
Depending on the specifics of the Multi-Site deployment, the
following deployment models would be possible:
• Active/Standby Service Nodes pair connected to different sites
• Active/Active cluster of Service Nodes deployed across sites
• Independent Active/Standby Service nodes pairs deployed in separate
sites
BRKDCN-2035 120

121. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Network Services Integration
Active/Standby Pair Deployed across Sites
Spine Spine
VXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP
Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP
BGW
WAN
Standby FWActive FW
BGW BGW BGW
DC Core
(Layer-3 Unicast)
Requirement to extend Layer 2 communication
between Active/Standby nodes for keep-alives
and state information exchange
Perimeter service nodes connected to VPC Border
Gateways
Ingress and egress traffic always traversing the
Active node in Site 1
No issues related to the creation of asymmetric
traffic paths
East-West flows must be hair-pinned to the
active FW connected to the Service leaf nodes
in Site 1
• Need to properly dimension bandwidth in the DC
Core to accommodate for this extra traffic
Active FW Standby FW
Baremetal BaremetalBaremetal
North-South traffic
flows
North-South traffic
flows
East-West traffic
flows
BRKDCN-2035 121

122. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Network Services Integration
Active/Active Cluster of Service Nodes Deployed across Sites
Spine Spine
VXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP
Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP
BGW
WAN
Active/Active
FW Cluster
BGW BGW BGW
DC Core
(Layer-3 Unicast)
Requirement to extend Layer 2 communication
between Active/Active nodes for intra-cluster
communication and traffic redirection
Perimeter service nodes connected to VPC Border
Gateways
Asymmetric traffic issues taken care by native
intra-cluster traffic redirection
Option to deploy ingress/egress optimization
technique to avoid inter-site traffic hair-pinning
Service node cluster integration not supported
at FCS and planned for a future SW release
Active/Active
FW Cluster
Logical Intra-
Cluster Link (ICL)
Baremetal
North-South traffic
flows
BRKDCN-2035 122

123. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Network Services Integration
Independent Active/Standby Pair Deployed in Separate Sites
Spine Spine
VXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP
Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP
BGW
WAN
Active/Standby
FW
Active/Standby
FW
BGW BGW BGW
DC Core
(Layer-3 Unicast)
Mandates the deployment of Ingress/Egress
traffic optimization to avoid creation of
asymmetric traffic path for north-south
communication
Active/Standby nodes can use direct links to
sync state
Perimeter service nodes can connected to Anycast
Border Gateways
Active/Standby pair (or cluster) still required for
service nodes used for east-west traffic flows
Baremetal Baremetal
Active FW Standby FW
North-South traffic
flows
North-South traffic
flows
BRKDCN-2035 123

124. Legacy Site
Integration

125. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multi-Site and Legacy Site Integration
Extend Layer-2 and Layer-3 connectivity between sites
• Coexistence and/or application migration use cases
Proposed approach is to deploy a pair of ‘remote’ VPC Border Gateways
in the legacy site
• Offers native Multi-Site functionalities (BUM containment, etc) to the legacy site
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Greenfield Site
VTEP VTEP
Legacy Site
VTEP VTEP
BGW BGW BGW BGW
Pair of VPC
Border Gateways
BRKDCN-2035 125

126. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multi-Site and Legacy Site Integration
Layer-2 Connectivity with the ‘Remote’ BGW
Legacy Site
VTEP VTEP
BGW BGW
Legacy Aggregation Layer
devices support MLAG
Single logical link to extend
VLANs toward the Greenfield
VXLAN EVPN site
VLANs mapped to L2VNIs
on the Border Gateways
Legacy Site
VTEP VTEP
BGW BGW
Legacy Aggregation Layer
devices do not support MLAG
Single port-channel
from each aggregation
layer device
VLANs mapped to L2VNIs
on the Border Gateways
Recommended to
move the STP root to
the BGW devices
BRKDCN-2035 126

127. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multi-Site and Legacy Site Integration
Layer-2 Control Plane Exchange across Sites
Baremetal
Host 1
0000.3010.1101
192.168.10.101
Baremetal
Host 2
0000.3010.1102
192.168.10.102
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Greenfield Site
VTEP VTEP
Legacy Site
VTEP VTEP
BGW BGW BGW BGW
VIP1
10.1.1.111
VIP2
10.2.2.222
MAC NH
0000.3010.1101 Leaf1
0000.3010.1102 VIP2
MAC NH
0000.3010.1101 VIP1
0000.3010.1102 Po1
Po1
All End-Points in the legacy
site are learned as directly
connected to the BGW
eBGP-EVPN
BRKDCN-2035 127

128. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Integration between Anycast Gateway and legacy default gateway
(HSRP, VRRP, etc.) not initially supported with VXLAN Multi-Site
First option is to keep on the legacy network the active default gateway
for the stretched IP subnets
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Greenfield Site
VTEP VTEP
Legacy Site
VTEP VTEP
BGW BGW BGW BGW
Default Gateway
deployed on the
legacy aggregation
devices
Multi-Site and Legacy Site Integration
Default Gateway Deployment – Option 1
L3
L2
Greenfield VXLAN
EVPN Fabric only offers
L2 services for the
stretched IP subnets
BRKDCN-2035 128

129. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Recommended approach is to migrate the default gateway from the
legacy aggregation devices to the Border Gateways (VXLAN EVPN
Anycast Gateway)
Optimize routing between End-Points deployed across sites
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Greenfield Site
VTEP VTEP
Legacy Site
VTEP VTEP
BGW BGW BGW BGW
Default Gateway
migrated to the Border
Gateways (VXLAN
EVPN Anycast
Gateway)
Multi-Site and Legacy Site Integration
Default Gateway Deployment – Option 2
L3
L2
Greenfield VXLAN
EVPN Fabric offers L2
and L3 services for the
stretched IP subnets
Legacy infrastructure
offers only L2
services
Distributed Anycast
Gateway function
L3
L2
BRKDCN-2035 129

130. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multi-Site and Legacy Site Integration
Layer-3 Control Plane Exchange across Sites
Baremetal
Host 1
0000.3010.1101
192.168.10.101
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Greenfield Site
VTEP VTEP
Legacy Site
VTEP VTEP
BGW BGW BGW BGW
VIP1
10.1.1.111
VIP2
10.2.2.222
Po1
All End-Points in the legacy
site are learned as directly
connected to the BGW
eBGP-EVPN
L3
L2
Baremetal
Host 3
0000.3010.1102
192.168.20.101
IP NH
192.168.10.101 Leaf1
192.168.20.101 VIP1
IP L3VNI
192.168.10.101 VIP1
192.168.20.101 Po1
BRKDCN-2035 130

131. Migration to Multi-
Site

132. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Migration to Multi-Site
Use Cases
1. Site addition: need to connect a Greenfield VXLAN EVPN
Fabric to an existing VXLAN EVPN Fabric built with 1st
generation Nexus 9000
2. Migrating a VXLAN Multi-Pod Fabric to Multi-Site
3. Migrating a VXLAN Multi-Fabric design to Multi-Site
BRKDCN-2035 132

133. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Migration to Multi-Site
Site Addition
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Existing VXLAN
EVPN Fabric
VTEP VTEP
BGW BGW
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Greenfield
Fabric
VTEP VTEP
BGW BGW
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Existing VXLAN
EVPN Fabric
Step 1: add a pair of Border Gateways to the existing VXLAN EVPN Fabric, running the
proper SW release supporting Multi-Site
Note: no requirement to change the HW/SW version on existing leaf nodes
Step 2: connect the BGW to the inter-site network and establish control plane peering with
the BGW in the Greenfield Fabric
Step 3: configure on the BGW the L2VNIs and L3VNIs to be extended
eBGP-EVPN
BRKDCN-2035 133

134. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Migration to Multi-Site
Multi-Fabric to Multi-Site
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site 1
VTEP VTEP
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site 2
VTEP VTEP
No Underlay Extension
BGW BGW BGW BGW
Step 1: add a pair of Border Gateways to
each Pod (if needed) and connect them
to the spines and to the inter-site
network
Step 2: upgrade the SW on both Fabrics
BGW to be able to support Multi-Site
Step 3: establish control plane
adjacencies across sites
Step 4: disconnect the previously used
DCI and extend Layer-2 and Layer-3
across Multi-Site
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Fabric 1
VTEP VTEP
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Fabric 2
VTEP VTEP
BRKDCN-2035 134

135. Conclusion

136. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multiple Overlay Domains – Interconnected & Controlled
• Scaling and Segregating VXLAN EVPN Networks
Multiple Overlay Control-Plane Domains – Interconnected & Controlled
• Limited Overlay Control-Plane Update Propagation
Multiple Underlay Domains - Isolated
• Isolated Underlay Domains – No need for Extension
Multiple Replication Domains for BUM – Interconnected & Controlled
• Individual BUM flooding domain with Traffic control
Multi-Site Advantages – ”The Multiple”
BRKDCN-2035 136

137. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
• New IETF Draft for Multi-Site Design
• Multi-site EVPN based VXLAN using Border Gateways
• https://tools.ietf.org/html/draft-sharma-multi-site-evpn
VXLAN EVPN – Multi-Site
BRKDCN-2035 137

138. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
• Give us your feedback to be
entered into a Daily Survey
Drawing. A daily winner will
receive a $750 gift card.
• Complete your session surveys
through the Cisco Live mobile
app or on www.CiscoLive.com/us.
Complete Your Online
Session Evaluation
Don’t forget: Cisco Live sessions will be
available for viewing on demand after the
event at www.CiscoLive.com/Online.

139. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions
BRKDCN-2035 139

140. Thank you