Openstack Summit Vancouver 2018 - Multicloud Networking

Multicloud Networking –
Connecting OpenStack Private
Clouds to Public Clouds
Shannon McFarland – CCIE #5245
Distinguished Consulting Engineer
@eyepv6

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Agenda
• Multicloud Networking Overview
• Extending On-Prem Private Clouds to a Public Cloud
• Adding More Public Cloud Providers to the Mix
• DMVPN
• Amazon Web Services
• Google Cloud Platform
• Microsoft Azure
• Automation
• Conclusion
2

Disclaimer
• There are a gazillion ways to accomplish the same thing for ALL of this
• You can build multicloud connections using software, hardware, commercial and open source gadgets
• You or someone you work with needs to know IPsec/IKE, BGP, OSPF, EIGRP and FHRP stuff
• Dead Peer Detection
• IPsec SA lifetimes
• IPsec SA replay window-size
• Perfect Forward Secrecy (PFS)
• BGP timers, Local Preference, MED, inbound soft reset (check if cloud provider supports dynamic inbound soft reset)
• BGP graceful restart - Note: Each cloud provider uses BGP graceful restart with default timers (120 sec) – My configs do not
show that due to slide space but know that it is enabled on each on-prem router
• IGP timers, tuning
• FHRP (HSRP, GLBP, VRRP) timers, tracking
3
router bgp 65002
bgp log-neighbor-changes
bgp graceful-restart restart-time 120
bgp graceful-restart stalepath-time 360
bgp graceful-restart

Multicloud
Networking
Overview

Hybrid vs Multicloud Networking
• Hybrid Cloud Networking = Network transport from on-premises (on-prem) to a single public cloud provider
• Multicloud Networking = Network transport from on-prem to multiple public cloud providers and/or between multiple public
cloud providers
• The technologies used can be identical for every connection or they can be per-provider, per-region, per-project, etc..
• Common network transport ingredients for hybrid and multicloud:
• Encryption (IPsec/IKEv2/IKEv2, SSL, PKI)
• Routing (Static, BGP and with supported public cloud-hosted routers: OSPF, EIGRP)
• Tunneling (IPsec tunnel mode, GRE, mGRE, MPLS, segment routing, etc..)
• Common network endpoint options:
• Native VPN (IPsec over Internet) using public cloud provider services that connect to on-prem router/firewall
• Commercial/Open Source VPN platform hosted on the public cloud provider connecting to an on-prem router/firewall
• Colocation/Direct Peering: Service from public cloud provider to on-prem via a 3rd party colo facility
• Google Cloud Platform Dedicated Internconnect/Direct Peering/Carrier Peering: https://cloud.google.com/interconnect/
• Amazon Web Services Direct Connect/PrivateLink: https://aws.amazon.com/directconnect/
• Microsoft Azure ExpressRoute: https://azure.microsoft.com/en-us/services/expressroute/
5

Why Would You Use Multiple Cloud Providers?
• Cloud provider high availability
• M&A may dictate public cloud provider preference (for a time)
• Regional cloud provider access
• Feature disparity between providers, regions and/or services
• Per-project service requirements
6

Extending On-Prem
Private Cloud to a
Public Cloud

Options – IPsec-over-the-Internet or Dedicated Connections
8
VPC Network
10.138.0.0/20
IPsec/IKEv2
BGP/OSPF/EIGRP
On-Prem Private Cloud
Google
Cloud
VPN
Google Cloud Router
VPC Network
10.138.0.0/20
BGP/OSPF/EIGRP
Google
Cloud
VPN
VPC Network
10.138.0.0/20 Google
Cloud
Router
Cloud Partner
Interconnect
Colocation Facility
IPsec VPN + Internet
Colocation
Commercial/Open Source
&
Native OpenStack VPNaaS
Private Network
192.168.200.0/24
Private Network
192.168.200.0/24

Multicloud Topologies With OpenStack
9
OpenStack
VM
Neutron
Router
+
VPNaaS
VPNaaS Based
Multicloud Networking
Data Center Infra.
TOR(s)
Internet Edge
Infra.
VPN/CoLo
Virtual Router Based
Data Center Infra.
OpenStack
TOR(s)
Internet Edge
Infra.
VPN/CoLo
VM
Virtual
Router
Neutron
Router
OpenStack
VM
Neutron
Router
Hardware Based
Data Center Infra.
TOR(s)
Internet Edge
Infra.
VPN/CoLo
*Also, provider networks

Public Cloud Provider - Native VPN Services
• Google Cloud Platform (GCP):
• VPN: https://cloud.google.com/compute/docs/vpn/overview
• Dedicated Interconnect: https://cloud.google.com/interconnect/
• Amazon Web Services (AWS):
• VPN: http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpn-connections.html
• Direct Connect: https://aws.amazon.com/directconnect/
• Microsoft Azure:
• VPN: https://docs.microsoft.com/en-us/azure/vpn-gateway/
• ExpressRoute: https://azure.microsoft.com/en-us/services/expressroute/
• OpenStack public cloud goodness: https://www.openstack.org/passport
The Big Three
10
Reference

Starting Simple
Public Cloud Provider Native IPsec VPN Service
11
VPC Network
10.138.0.0/20
IPsec/IKEv2
BGP/OSPF/EIGRP
eBGP<>IGP Redistribution
Google
Cloud
VPN
Google Cloud Router
BGP AS65000
BGP AS65003
Private Network
192.168.200.0/24Cisco
ASR1000

Add More On-Prem Stuff
12
VPC Network
10.138.0.0/20
BGP AS65000 BGP AS65003
Routes this side should see:
10.138.0.0/20
Private Network
192.168.100.0/24
BGP AS65002
On-Prem Tenant 1
On-Prem Tenant 2Routes this side should see:
192.168.100.0/24
192.168.200.0/24
Google
Cloud
VPN
Google Cloud Router
BGP/OSPF/EIGRP
BGP/OSPF/EIGRP
Private Network
192.168.200.0/24
Cisco
ASR1000
Cisco
ASR1000

Physical/Virtual
13
VPC Network
10.138.0.0/20
Virtual Router
Physical Firewall
Google
Cloud
VPN
Google Cloud Router
CSR 1000v
ASA Firewall
Private Network
192.168.100.0/24
Private Network
192.168.200.0/24

Add More Public
Cloud Providers to
the Mix

Stepping into Multicloud Networking
Multiple Native IPsec VPN Services
15
Private Network
192.168.200.0/24
VPC Network
10.138.0.0/20
BGP/OSPF/EIGRP
Google
Cloud VPN
Google Cloud Router
VPC Network
172.31.0.0/16
VPN
Gateway
VPC
Router

Stepping into Multicloud Networking
Multiple Native IPsec VPN Services
16
VPC Network
10.138.0.0/20
BGP/OSPF/EIGRP
Google
Cloud VPN
Google Cloud Router
VPC Network
172.31.0.0/16
VPN
Gateway
VPC
Router
As the number of these
connections increase and/or
change frequently... You can see
where this is going
Private Network
192.168.200.0/24

Site-to-Site + Manual Configuration per Site =
Unpleasant Times
17

Example – OpenStack VPNaaS
18
• Lots of sites and lots
of variation in policies
can lead to lots of
human errors
• Per-Cloud provider
IKE/IPsec compatibility
polices required
• Automation helps but
only with the
configuration
challenge
[root@mc-os-q-aio-sm ~]# openstack vpn ike policy create ikepolicy
[root@mc-os-q-aio-sm ~]# openstack vpn ipsec policy create ipsecpolicy
[root@mc-os-q-aio-sm ~]# openstack vpn service create vpn
> --router a6c58be0-7e32-4a14-b648-8b8178f8de8c
[root@mc-os-q-aio-sm ~]# openstack vpn endpoint group create ep_subnet
> --type subnet
> --value 7fe62bea-49ee-42a0-8c6a-5ec982983e98
[root@mc-os-q-aio-sm ~]# openstack vpn endpoint group create ep_cidr
> --type cidr
> --value 10.0.1.0/24
[root@mc-os-q-aio-sm ~]# openstack vpn ipsec site connection create GCP-Conn
--vpnservice vpn
--ikepolicy ikepolicy
--ipsecpolicy ipsecpolicy
--peer-address 35.xx.xx.xx
--peer-id 35.xx.xx.xx
--psk demo-secret
--local-endpoint-group ep_subnet
--peer-endpoint-group ep_cidr
Reference

Moving Away From Native VPN Services
• Large Site-to-Site designs suck due to configuration complexity (even with
Heat or other automation)
• If on-prem routers/firewalls are behind NAT – Check for provider support
of NAT-T
• You need to extend your on-prem IGP (OSPF/EIGRP) into the public cloud
• Operational consistency
• You need SSL-based VPNs
• You need MPLS VPN
• QoS, specific network monitoring (IP SLA, NetFlow), Enterprise toolsets for
configuration and monitoring
What Conditions Cause a Change in Design?
19

DMVPN – Dynamic
Multipoint VPN

DMVPN – Enable Dynamic Multicloud Networking
21
VNet Network
10.50.0.0/16
DMVPN
BGP/OSPF/EIGRP
On-Prem Private CloudVPC Network
172.31.0.0/16 Cisco
CSR1000v
Cisco
CSR1000v
DMVPN:
https://www.cisco.com/c/en/us/products/security/
dynamic-multipoint-vpn-dmvpn/index.html
Hub
Spoke
Spoke
Cisco
ASR1000
Private Network
192.168.200.0/24

22
VNet Network
10.50.0.0/16
DMVPN
FHRP
VPC Network
172.31.0.0/16 Cisco
CSR1000v
Cisco
CSR1000v
IGP Support: OSPF, EIGRP, iBGP
QoS Policies
IP SLA, NetFlow
NAT-T (Transparency)
MPLS
etc...
Hub
Spoke
Spoke
Cisco
ASR1000
Private Network
192.168.200.0/24

DMVPN (Dynamic Multipoint VPN)
• DMVPN is a Cisco innovation for building GRE/mGRE + IPsec VPN connections in a
dynamic and scalable manner
• Cisco DMVPN
• https://www.cisco.com/c/en/us/products/security/dynamic-multipoint-vpn-dmvpn/index.html
• Cisco IWAN CVD
• https://www.cisco.com/c/en/us/solutions/design-zone/networking-design-guides/branch-
wan-edge.html
• OpenNHRP:
• https://sourceforge.net/projects/opennhrp/
• https://wiki.alpinelinux.org/wiki/Dynamic_Multipoint_VPN_(DMVPN)
23

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
Terminology and Features
192.168.102.0/24
Hub1
Spoke 1
Hub 2
Spoke 2
192.168.101.0/24
192.168.1.0/24 192.168.2.0/24
Tunnel: 10.0.0.101
Physical: 172.16.101.1
Tunnel: 10.0.0.1
Physical: 172.16.1.1
Tunnel: 10.0.0.2
Physical: 172.16.2.1
Overlay Addresses
NBMA Address
Core Network
192.168.128.0/17
On Demand
Spoke Tunnels
Tunnel Address
Tunnel: 10.0.0.102
Physical: 172.16.102.1
GRE/IPsec
Tunnels

DMVPN Components
• Next Hop Resolution Protocol (NHRP)
• Creates a distributed (NHRP) mapping database of all the spoke’s tunnel to real (public
interface) addresses
• Multipoint GRE Tunnel Interface (mGRE)
• Single GRE interface to support multiple GRE/IPsec tunnels
• Simplifies size and complexity of configuration
• IPsec tunnel protection
• Dynamically creates and applies encryption policies
• Routing
• Dynamic advertisement of branch networks; almost all routing protocols (EIGRP, RIP, OSPF,
BGP, ODR) are supported
25
Reference

DMVPN Implementation
26
.
Hub and spoke
(Phase 1)
Spoke-to-spoke
(Phase 2)
Server Load Balancing
Hierarchical (Phase 3)
VRF-lite
2547oDMVPN
Spoke-to-hub tunnels
Spoke-to-spoke tunnels
2547oDMVPN tunnels

Amazon Web
Services – Cisco
CSR & DMVPN

AWS with Cisco CSR 1000v Support
• Amazon Web Services Marketplace + Cisco CSR:
• https://aws.amazon.com/marketplace/search/results?x=0&y=0&searchTerms=csr&page=1&ref_=nav_search_
box
• Cisco CSR for AWS Deployment
• DMVPN
https://www.cisco.com/c/en/us/td/docs/solutions/Hybrid_Cloud/Intercloud/CSR/AWS/CSRAWS/CSRAWS_3.
html
• Deployment https://www.cisco.com/c/en/us/td/docs/routers/csr1000/software/aws/b_csraws.html
• Cisco Live Session for AWS with Cisco CSR:
• https://www.ciscolive.com/global/on-demand-library/?search=brkarc-
2023#/session/1486155288098001AhER
• Transit VPC with CSR: http://d2zmdbbm9feqrf.cloudfront.net/2017/usa/pdf/BRKARC-2749.pdf
28

AWS CSR to On-Prem ASR – DMVPN
29
VPC Network
172.16.2.0/24
VPC
Router
192.xxx.xxx.x52.xxx.xxx.x
192.168.200.0/24
172.16.2.0/16
Public-side Network
172.16.1.0/24
Cisco
CSR1000v
DMVPN
Hub
Tunnel:
10.1.0.2
Spoke
Tunnel:
10.1.0.4
OSPF
Private Network
192.168.200.0/24
OSPF 10 Area 0
OpenStack
VM
.30
DataCenter
Infra.
Provider Networks with VLANs Example
Cisco
ASR1000

AWS CLI: Create VPC, Subnets and Internet GW
Create a new AWS VPC (vpc)
# aws ec2 create-vpc --cidr-block 172.16.0.0/16
Create a new subnet in the VPC (this one will be used for the CSR’s ’outside’ interface)
# aws ec2 create-subnet --vpc-id vpc-66a0a102 --cidr-block 172.16.1.0/24
Create another new subnet in the VPC (this one will be used for the CSR’s ‘inside’ interface)
# aws ec2 create-subnet --vpc-id vpc-66a0a102 --cidr-block 172.16.2.0/24
Create a new AWS Internet Gateway (igw)
# aws ec2 create-internet-gateway
Attach the Internet gateway to the VPC
# aws ec2 attach-internet-gateway --vpc-id vpc-66a0a102 --internet-gateway-id igw-591fba3d
Reference

AWS CLI: Create Route Tables
Create a new route table in the VPC (rtb) that will be used by the CSR’s ‘outside’ subnet
# aws ec2 create-route-table --vpc-id vpc-66a0a102
Create a new default route in the route table and point it to the Internet gateway
# aws ec2 create-route --route-table-id rtb-aaa37dcd --destination-cidr-block 0.0.0.0/0 --gateway-id igw-591fba3d
Associate the new routable with the ‘outside’ VPC subnet
# aws ec2 associate-route-table --subnet-id subnet-0c15b86b --route-table-id rtb-aaa37dcd
Create a new route table in the VPC (rtb) that will be used by the CSR’s ‘inside’ subnet
# aws ec2 create-route-table --vpc-id vpc-66a0a102
Create a new default route in the route table for the ‘inside’ subnet and point it to the Internet gateway
# aws ec2 create-route --route-table-id rtb-3741e750 --destination-cidr-block 0.0.0.0/0 --gateway-id igw-591fba3d
Create a new default route in the route table and point it to the Internet gateway
# aws ec2 create-route --route-table-id rtb-3741e750 --destination-cidr-block 192.168.200.0/24 --network-interface-id eni-af67db80
Associate the new route table with the ‘inside’ VPC subnet
# aws ec2 associate-route-table --subnet-id subnet-c617baa1 --route-table-id rtb-3741e750
Reference

AWS CLI: Create a Security Group/Rules
32
Create a new security group for the ‘outside’ facing interface (Optional: You can just use an existing group)
# aws ec2 create-security-group --group-name csr --description csr-rules --vpc-id vpc-66a0a102
Create a new security group rule for SSH to the CSR
# aws ec2 authorize-security-group-ingress --group-id sg-65c39b03 --protocol tcp --port 22 --cidr 0.0.0.0/0
Create a new security group rule for ICMP from the other CSRs (On-Prem and GCP CSR [optional: Just showing the format for your use])
# aws ec2 authorize-security-group-ingress --group-id sg-65c39b03
--ip-permissions '[{"IpProtocol": "icmp", "FromPort": -1, "ToPort": -1, "IpRanges": [{"CidrIp": "192.x.x.x/32"}, {"CidrIp": "35.x.x.x/32"}]}]
Create a new security group rule for ESP (IP 50) from the other CSRs
--ip-permissions '[{"IpProtocol": "50", "IpRanges": [{"CidrIp": ”192.x.x.x/32"}, {"CidrIp": "35.x.x.x/32"}]}]'
Create a new security group rule for IKE from the other CSRs
--ip-permissions '[{"IpProtocol": "17", "FromPort": 500, "ToPort": 500, "IpRanges": [{"CidrIp": "192.x.x.x/32"}, {"CidrIp": "35.x.x.x/32"}]}]'
Create a new security group rule for IKE/NAT-T from the other CSRs
--ip-permissions '[{"IpProtocol": "17", "FromPort": 4500, "ToPort": 4500, "IpRanges": [{"CidrIp": "192.x.x.x/32"}, {"CidrIp": "35.x.x.x/32"}]}]’
Optional: You may want to create a security group just for the ’inside’ subnet that has
different rules than the one for the ‘outside’ subnet
# aws ec2 authorize-security-group-ingress --group-id sg-84aef7e2 --protocol all --cidr 192.168.200.0/24
# aws ec2 authorize-security-group-ingress --group-id sg-84aef7e2 --protocol all --cidr 172.16.2.0/24
Reference

AWS CLI: Run a new CSR Instance Using Previous Parameters
33
{
"ImageId": "ami-99e5d0f9",
"InstanceType": "t2.medium",
"KeyName": "mc-aws-key",
"NetworkInterfaces": [
{
"DeviceIndex": 0,
"Description": "Primary network interface",
"Groups": [
"sg-65c39b03"
],
"PrivateIpAddresses": [
{
"Primary": true,
"PrivateIpAddress": "172.16.1.10"
}
],
"SubnetId": "subnet-0c15b86b"
},
{
"DeviceIndex": 1,
"PrivateIpAddresses": [
{
"Primary": true,
"PrivateIpAddress": "172.16.2.10"
}
],
"SubnetId": "subnet-c617baa1"
}
]
}
csr-create.json
Create a CSR instance using the JSON file shown to the left
# aws ec2 run-instances --cli-input-json file://csr-create.json
Create a tag/name and associate it with the CSR (Optional)
# aws ec2 create-tags --resources i-0f2a0ee857e9c2540
--tags Key=Name,Value=csr-aws-01
Create a new External IP (EIP) allocation (or use an existing one)
# aws ec2 allocate-address
eipalloc-ab35cb96 vpc 52.xxx.xxx.x
Associate the EIP with the ’outside’ interface of the CSR (GigabitEthernet 1)
# aws ec2 associate-address --allocation-id eipalloc-ab35cb96
--network-interface-id eni-dd5bd6f2
Modify the ’inside’ subnet to disable source/destination checking
# aws ec2 modify-network-interface-attribute
--network-interface-id eni-af67db80
--source-dest-check "{"Value": false}"
A note about NAT: If you plan to use the CSR for NAT
operation, you must disable source/destination checking
on the outside CSR interface/subnet
http://docs.aws.amazon.com/AmazonVPC/latest/UserG
uide/VPC_NAT_Instance.html#EIP_Disable_SrcDestCh
eck
Reference

Connect to the AWS CSR – Enable Interfaces
# ssh -i "mc-aws-key.pem" ec2-user@52.xxx.xxx.x
csr-aws-01#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
csr-aws-01(config)#interface gigabitEthernet 2
csr-aws-01(config-if)#ip address dhcp
csr-aws-01(config-if)#no shutdown
Connect to the new AWS-hosted CSR and enable the GigabitEthernet 2 interface for DHCP
csr-aws-01#show ip interface brief
Interface IP-Address OK? Method Status Protocol
GigabitEthernet1 172.16.1.10 YES DHCP up up
VirtualPortGroup0 192.168.35.1 YES TFTP up up
Wait a few seconds and check to make sure that both interfaces on the CSR are up and have the correct
IP addresses:
Note: This can all be automated (along with the DMVPN configs) by creating AWS CloudFormation templates
Reference

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
AWS Cisco CSR DMVPN Config
Spoke
35BRKCLD-3440
crypto ikev2 proposal AES/GCM/256
encryption aes-gcm-256
prf sha512
group 19
!
crypto ikev2 policy AES/GCM/256
match fvrf any
proposal AES/GCM/256
!
crypto ikev2 keyring DMVPN-KEYRING
peer ANY
address 0.0.0.0 0.0.0.0
pre-shared-key <PSK_PASSWORD_GOES_HERE>
!
crypto ikev2 profile DMVPN-IKEv2-PROFILE
description PSK Profile
match identity remote address 0.0.0.0
identity local address 52.xxx.xxx.x
authentication remote pre-share
authentication local pre-share
keyring local DMVPN-KEYRING
dpd 40 5 on-demand
!
crypto ipsec security-association replay window-size 1024
!
crypto ipsec transform-set AES256/GCM/TRANSFORM esp-gcm 256
mode transport
!
crypto ipsec profile DMVPN-IPSEC-PROFILE
set transform-set AES256/GCM/TRANSFORM
set ikev2-profile DMVPN-IKEv2-PROFILE
... Output summarized
interface Tunnel0
description DMVPN
ip address 10.1.0.4 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp authentication <NHRP_PASSWORD>
ip nhrp network-id 100
ip nhrp nhs 10.1.0.2 nbma 192.xxx.xxx.x multicast
ip tcp adjust-mss 1360
ip ospf authentication-key 7 <OSPF_PASSWORD>
ip ospf network point-to-multipoint
ip ospf hello-interval 10
tunnel source GigabitEthernet1
tunnel mode gre multipoint
tunnel key 100
tunnel protection ipsec profile DMVPN-IPSEC-PROFILE
!
interface GigabitEthernet1
description Internet
ip address dhcp
no ip redirects
no ip unreachables
no ip proxy-arp
negotiation auto
!
router ospf 10
router-id 10.1.0.4
network 172.16.2.0 0.0.0.255 area 2
network 10.1.0.0 0.0.0.255 area 0
!
ip route 0.0.0.0 0.0.0.0 172.16.1.1
Reference

On-Prem Cisco ASR DMVPN Config
Hub – Nothing ever changes on the hub for each example
36BRKCLD-3440
prf sha512
group 19
!
match fvrf any
!
peer ANY
address 0.0.0.0 0.0.0.0
!
dpd 40 5 on-demand
!
!
mode transport
!
interface Tunnel0
description DMVPN
ip address 10.1.0.2 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp map multicast dynamic
ip nhrp redirect
tunnel source GigabitEthernet0/0/0
tunnel key 100
!
interface GigabitEthernet0/0/0
ip address 192.xxx.xxx.x 255.255.255.0
no ip redirects
no ip unreachables
no ip proxy-arp
negotiation auto
!
router ospf 10
router-id 10.1.0.2
network 10.1.0.0 0.0.0.255 area 0
network 192.168.200.0 0.0.0.255 area 0
!
ip route 0.0.0.0 0.0.0.0 192.xxx.xxx.x
Reference

Verify Routing and Reachability
[ec2-user@ip-172-16-2-192 ~]$ ping 192.168.200.30
PING 192.168.200.30 (192.168.200.30) 56(84) bytes of data.
64 bytes from 192.168.200.30: icmp_seq=1 ttl=62 time=2.75 ms
Connect to an AWS instances and ping to the on-prem private network
asr-mc-01#show ip route | i 172.16.2.0
O IA 172.16.2.0 [110/1001] via 10.1.0.4, 00:11:41, Tunnel0
On the on-prem ASR check the route for the AWS VPC network 172.16.2.0/24
csr-aws-01#show ip route | i 192.168.200.0
O 192.168.200.0/24 [110/1001] via 10.1.0.2, 6d17h, Tunnel0
On AWS check for the route for the on-prem network (192.168.200.0/24)
VPC Network
172.16.2.0/24
.10.192
OSPF
Hub
Tunnel:
10.1.0.2
Spoke
Tunnel:
10.1.0.4
Cisco
CSR1000v
Reference
Private Network
192.168.200.0/24
OSPF 10 Area 0
VM
.30
DataCenter
Infra.
Cisco
ASR1000

Google Cloud
Platform– Cisco
CSR & DMVPN

Private Network
192.168.200.0/24inside-network
10.0.1.0/24
OSPF 10 Area 0
10.0.1.0/24
192.168.200.0/24
Cisco
CSR1000v
OpenStack
Default Network
10.138.0.0/20
.100.3 .2
.1
VM
.30192.xxx.xxx.x35.xxx.xxx.x
DMVPN
Hub
Tunnel:
10.1.0.2
Spoke
Tunnel:
10.1.0.1
GCP CSR to On-Prem ASR – DMVPN
Coming in 16.9.1 Release
OSPF
Compute
Engine
2 1
DataCenter
Infra.
Cisco
ASR1000

gcloud – Create the GCP External IP, Inside VPC
Network & Route
Create a new external IP reservation that will be used for the GCP CSR NATed connection (or use an existing one)
# gcloud compute addresses create csr-to-csr-ext-ip --region us-west1
Capture the external IP address
# gcloud compute addresses list --filter="csr-to-csr-ext-ip"
NAME REGION ADDRESS STATUS
csr-to-csr-ext-ip us-west1 35.xxx.xxx.x RESERVED
Create a new GCP inside network that will be attached to the ‘inside’ interface of the CSR
# gcloud compute networks create inside-network --subnet-mode=custom
Create a new GCP inside subnet - Associate it with the inside network
# gcloud compute networks subnets create inside-subnet
--network=inside-network
--range=10.0.1.0/24
Create a new GCP route from the CSR inside network to the on-prem private network which routes through the IPsec VPN
# gcloud compute routes create inside-to-csr-private
--destination-range=192.168.200.0/24
--next-hop-address=10.0.1.2
Reference

gcloud – Create GCP Firewall Rules
Create a new GCP firewall rule to allow traffic into the inside CSR network from the default network
# gcloud compute firewall-rules create allow-default-to-csr-inside
--direction=INGRESS
--action=ALLOW
--rules=all
--source-ranges=0.0.0.0/0
Create a new GCP firewall rule to allow traffic between the default network and the on-prem ASR public IP for IKE, IPsec
# gcloud compute firewall-rules create csr-csr-vpn
--direction=INGRESS
--network=default
--action=ALLOW
--rules=udp:500,udp:4500,esp
--source-ranges=192.xxx.xxx.x
Reference

gcloud – Create CSR and Test Instances
Create a new GCE CSR instance and set fixed IPv4 addresses to each of the two interfaces
# gcloud compute instances create "csr-gcp-01"
--zone "us-west1-a"
--machine-type "n1-standard-4"
--network-interface subnet="default",private-network-ip="10.138.0.100",address="35.xxx.xxx.x"
--can-ip-forward
--network-interface subnet="inside-subnet",private-network-ip="10.0.1.2",no-address
--image ”name_of_csr_image"
--boot-disk-size "10"
--boot-disk-type "pd-standard"
--boot-disk-device-name "csr-gcp-01"
Create a new GCE test instance that will be used to validate the VPN and routing
# gcloud compute instances create "csr-inside-vm"
--zone "us-west1-a"
--machine-type "g1-small"
--subnet "inside-subnet"
--private-network-ip "10.0.1.3"
--image "debian-9-stretch-v20170918"
--image-project "debian-cloud"
--boot-disk-size "10"
--boot-disk-type "pd-standard"
--boot-disk-device-name "csr-inside-vm"
Reference

Connect to the GCP CSR – Enable Interfaces
43
# gcloud compute ssh cisco-user@csr-gcp-01
csr1kv-gcp#configure terminal
csr1kv-gcp(config)#interface gigabitEthernet 2
csr1kv-gcp(config-if)#ip address dhcp
csr1kv-gcp(config-if)#no shutdown
Connect to the new GCP-hosted CSR and enable the GigabitEthernet 2 interface for DHCP
csr1kv-gcp#show ip interface brief
GigabitEthernet1 10.138.0.100 YES TFTP up up
Wait a few seconds and check to make sure that both interfaces on the CSR are up and with the correct IP
addresses:
Reference

GCP Cisco CSR DMVPN Config
Spoke
44
prf sha512
group 19
!
match fvrf any
!
peer ANY
address 0.0.0.0 0.0.0.0
!
dpd 40 5 on-demand
!
!
mode transport
!
interface Tunnel0
description DMVPN
ip address 10.1.0.1 255.255.255.0
no ip redirects
ip mtu 1400
tunnel key 100
!
ip address 10.138.0.100 255.255.255.0
no ip redirects
no ip unreachables
no ip proxy-arp
negotiation auto
!
router ospf 10
router-id 10.1.0.1
network 10.0.1.0 0.0.0.255 area 1
network 10.1.0.0 0.0.0.255 area 0
!
ip route 0.0.0.0 0.0.0.0 138.0.0.1
Reference

Hub
45
prf sha512
group 19
!
match fvrf any
!
peer ANY
address 0.0.0.0 0.0.0.0
!
dpd 40 5 on-demand
!
!
mode transport
!
interface Tunnel0
description DMVPN
ip address 10.1.0.2 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp redirect
tunnel key 100
!
no ip redirects
no ip unreachables
no ip proxy-arp
negotiation auto
!
router ospf 10
router-id 10.1.0.2
network 10.1.0.0 0.0.0.255 area 0
network 192.168.200.0 0.0.0.255 area 0
!
ip route 0.0.0.0 0.0.0.0 192.xxx.xxx.x
Reference

46
# gcloud compute ssh "csr-inside-vm“
shmcfarl@csr-inside-vm:~$ ping 192.168.200.30
Connect to the GCP test instance that was created earlier and ping to the on-prem private network
csr1kv-gcp#show ip route | i 192.168.200.0
. . .
O 192.168.200.0/24 [110/1001] via 10.1.0.2, 00:09:51, Tunnel0
On the GCP CSR, check for the private network route from the on-prem side(192.168.200.0/24)
. . .
O IA 10.0.1.0/24 [110/1001] via 10.1.0.1, 00:40:08, Tunnel0
On the on-prem ASR, check for the VPC inside network route (10.1.0/24)
csr1kv-gcp#show ip nhrp
10.1.0.2/32 via 10.1.0.2
Tunnel0 created 5d14h, never expire
Type: static, Flags:
NBMA address: 192.xxx.xxx.x
Check the DMVPN Next-Hop Resolution Protocol (NHRP) Status
asr-mc-01#show ip nhrp
10.1.0.1/32 via 10.1.0.1
Tunnel0 created 00:40:25, expire 00:08:20
Type: dynamic, Flags: registered used nhop
(Claimed NBMA address: 10.138.0.100)
Reference

Microsoft Azure –
Cisco CSR and
DMVPN

Azure CSR to On-Prem ASR – DMVPN
192.xxx.xxx.x40.xxx.xxx.x
192.168.200.0/24
10.10.1.0/24
Cisco
CSR1000v
DMVPN
Hub
Tunnel:
10.1.0.2
Spoke
Tunnel:
10.1.0.6
OSPF
Inside Subnet
10.10.1.0/24
Outside Subnet
10.10.0.0/24
Cisco
ASR1000
Private Network
192.168.200.0/24
OSPF 10 Area 0
OpenStack
VM
.30
DataCenter
Infra.

Microsoft Azure with Cisco CSR 1000v
• Microsoft Azure Marketplace
• https://azuremarketplace.microsoft.com/en-
us/marketplace/apps/cisco.cisco-csr-basic-template
• https://github.com/Azure/azure-quickstart-templates/tree/master/cisco-
csr-1000v
• Cisco CSR 1000v with Azure Deployment
• https://www.cisco.com/c/en/us/td/docs/routers/csr1000/software/azu/b_c
sr1000config-azure.html
49

Azure CLI: Create Resource Group, Networks,
Subnets
Create a new Azure Resource Group (rg)
# az group create --name multicloud-rg --location westus
Create a new public (external IP) IPv4 address to be used for the CSR’s ‘outside’ interface
# az network public-ip create --resource-group multicloud-rg --name csr-azure-01-eip --allocation-method static
Create a new virtual network (vnet) and a subnet to be used for the CSR’s ‘outside interface
# az network vnet create
--resource-group multicloud-rg
--name mc-csr-vnet
--address-prefix 10.10.0.0/16
--subnet-name csr-outside
--subnet-prefix 10.10.0.0/24
Create a new subnet for the CSR’s ‘inside’ interface and associate it with the vnet created above
# az network vnet subnet create
--vnet-name mc-csr-vnet
--name csr-inside
--address-prefix 10.10.1.0/24
Reference

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 51BRKCLD-3440
Azure CLI: Create Route Tables
Create a new route table (rt) that will be used for the CSR’s ’outside’ subnet
# az network route-table create
--name csr-outside-rt
Create a new route table that will used for the CSR’s ‘inside’ subnet
# az network route-table create
--name csr-inside-rt
Create a new route table entry for the ‘inside’ subnet to reach the on-prem network (192.168.200.0) via the CSR’s IP (10.10.1.4)
# az network route-table route create
--name csr-to-on-prem-route
--route-table-name csr-inside-rt
--address-prefix 192.168.200.0/24
--next-hop-type VirtualAppliance
--next-hop-ip-address 10.10.1.4
Associate the ‘outside’ route table with the ‘outside’ subnet
# az network vnet subnet update
--name csr-outside
--route-table csr-outside-rt
Associate the ‘inside’ route table with the ‘inside’ subnet
# az network vnet subnet update
--name csr-inside
--route-table csr-inside-rt
Reference

Azure CLI: Create Network Security Group (NSG)
Create a new Network Security Group (NSG) to be used for the ‘outside’ CSR interface
# az network nsg create
--name csr-nsg-outside
Create a new NSG rule to allow inbound SSH access to the CSR (can make it specific to an IP/Prefix)
# az network nsg rule create
--nsg-name csr-nsg-outside
--name SSHRule
--priority 100
--source-address-prefixes 'Internet'
--source-port-ranges '*'
--destination-address-prefixes '*'
--destination-port-ranges 22
--access Allow
--protocol Tcp
--direction inbound
Create a new NSG rule to allow inbound UDP 500 (IKE) traffic to the CSR (can make it specific to an IP/Prefix)
--name UDP-500
--priority 101
--access Allow
--protocol Udp
--direction inbound
Reference

Azure CLI: Create NSG Rule & NICs
Create a new Network Security Group (NSG) to be used for the ‘outside’ CSR interface
--name UDP-4500
--priority 102
--access Allow
--protocol Udp
--direction inbound
Create a new NIC to be used by the CSR’s ‘outside’ interface. Associate the NIC with the NSG, Subnet, Public IP & enable forwarding
# az network nic create
--name csr-nic-g1
--subnet csr-outside
--network-security-group csr-nsg-outside
--ip-forwarding true
--public-ip-address csr-azure-01-eip
Create a new NIC to be used by the CSR’s ‘inside’ interface. Associate the NIC with the NSG, Subnet and enable forwarding
# az network nic create
--name csr-nic-g2
--subnet csr-inside
--ip-forwarding true
Reference

Azure CLI: Run a new CSR Instance Using
Previous Parameters
Create a CSR VM using a Azure Marketplace image. Associate the VM with two the NICs created earlier.
# Note: The VM can be created with a large number of options to include SSH keys, image (BYOL, # of NICs), and size
# az vm create
--name csr-azure-01
--admin-username csr-azure
--admin-password <PASSWORD>
--authentication-type password
--image cisco:cisco-csr-1000v:16_6:16.6.120170804 <<< Change image based on current release
--nics csr-nic-g1 csr-nic-g2
--size Standard_D2_v2

Connect to the Azure CSR – Enable Interfaces
# ssh csr-azure@40.xxx.xxx.x
csr-azure-01#configure terminal
csr-azure-01(config)#interface gigabitEthernet 2
csr-azure-01(config-if)#ip address dhcp
csr-azure-01(config-if)#no shutdown
Connect to the new Azure-hosted CSR and enable the GigabitEthernet 2 interface for DHCP
csr-azure-01#show ip interface brief
VirtualPortGroup0 192.168.35.1 YES TFTP up up
Wait a few seconds and check to make sure that both interfaces on the CSR are up and have the correct
IP addresses:
Note: This can all be automated (along with the DMVPN configs) by creating Azure Automation/Resource Manager
Reference

Azure Cisco CSR DMVPN Config
Spoke
prf sha512
group 19
!
match fvrf any
!
peer ANY
address 0.0.0.0 0.0.0.0
!
dpd 40 5 on-demand
!
!
mode transport
!
interface Tunnel0
description DMVPN
ip address 10.1.0.6 255.255.255.0
no ip redirects
ip mtu 1400
tunnel key 100
!
ip address dhcp
no ip redirects
no ip unreachables
no ip proxy-arp
negotiation auto
!
router ospf 10
router-id 10.1.0.6
network 10.1.0.0 0.0.0.255 area 0
network 10.10.1.0 0.0.0.255 area 3
!
ip route 0.0.0.0 0.0.0.0 10.10.0.1

Hub - Nothing ever changes on the hub for each example
prf sha512
group 19
!
match fvrf any
!
peer ANY
address 0.0.0.0 0.0.0.0
!
dpd 40 5 on-demand
!
!
mode transport
!
interface Tunnel0
description DMVPN
ip address 10.1.0.2 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp redirect
tunnel key 100
!
no ip redirects
no ip unreachables
no ip proxy-arp
negotiation auto
!
router ospf 10
router-id 10.1.0.2
network 10.1.0.0 0.0.0.255 area 0
network 192.168.200.0 0.0.0.255 area 0
!
ip route 0.0.0.0 0.0.0.0 192.xxx.xxx.x

shmcfarl@AzTestVm:~$ping 192.168.200.30
Connect to an AWS instances and ping to the on-prem private network
O IA 10.10.1.0/24 [110/1001] via 10.1.0.6, 00:19:15, Tunnel0
On the on-prem CSR check the route for the AWS VPC network 172.16.2.0/24
csr-azure-01#show ip route | i 192.168.200.0
O 192.168.200.0/24 [110/1001] via 10.1.0.2, 00:17:57, Tunnel0
On AWS check for the route for the on-prem network (192.168.200.0/24)
Inside Subnet
10.10.1.0/24
.4.5
OSPF
Hub
Tunnel:
10.1.0.2
Spoke
Tunnel:
10.1.0.6
Cisco
CSR1000v
Reference
Cisco
ASR1000
Private Network
192.168.200.0/24
OSPF 10 Area 0
VM
.30
DataCenter
Infra.

60
VNet Network
10.10.1.0/24
DMVPN
VPC Network
172.16.2.0/24
Cisco
CSR1000v
Cisco
CSR1000v
VPC Network
10.0.1.0/24
Cisco
CSR1000v
Cisco
ASR1000
Hub
Tunnel:
10.1.0.2
Spoke
Tunnel:
10.1.0.1
Spoke
Tunnel:
10.1.0.4
Spoke
Tunnel:
10.1.0.6
Private Network
192.168.200.0/24
OSPF 10 Area 0
OpenStack
VM
.30
DataCenter
Infra.

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS BRKCLD-3440
General Guidelines for DMVPN Between Clouds
• Set the VPC routes for each site
• Set the firewall/security groups/network security groups for each site/protocol
gcloud compute routes create inside-to-aws
--network=csr-inside-network
gcloud compute routes create inside-to-azure
--network=csr-inside-network
Create a very specific per-site route (AWS example allowing UDP500 from each cloud provider public IP)
aws ec2 authorize-security-group-ingress --group-id sg-65c39b03
--ip-permissions '[{"IpProtocol": "17", "FromPort": 500, "ToPort": 500, "IpRanges": [{"CidrIp": "192.x.x.x/32"},
{"CidrIp": "35.x.x.x/32"}, {"CidrIp": "40.x.x.x/32"}]}]’
Alternatively, you can open it up (Azure example)
az network nsg rule create
--name UDP-4500
--priority 102
--access Allow
--protocol Udp
--direction inbound

Routing Example – All Sites
62
• For spoke-to-spoke direct routing with DMVPN/NHRP:
• ‘ip nhrp redirect’ on the hubs
• ‘ip nhrp shortcut’ on the spokes
asr-mc-01#show ip route ospf
10.0.0.0/8 is variably subnetted, 7 subnets, 2 masks
O IA 10.0.1.0/24 [110/1001] via 10.1.0.1, 02:40:45, Tunnel0
O 10.1.0.1/32 [110/1000] via 10.1.0.1, 02:40:45, Tunnel0
O 10.1.0.4/32 [110/1000] via 10.1.0.4, 01:18:49, Tunnel0
O 10.1.0.6/32 [110/1000] via 10.1.0.6, 00:56:19, Tunnel0
O IA 10.10.1.0/24 [110/1001] via 10.1.0.6, 00:55:34, Tunnel0
172.16.0.0/24 is subnetted, 1 subnets
O IA 172.16.2.0 [110/1001] via 10.1.0.4, 01:18:49, Tunnel0
Hub On-Prem CSR
csr1kv-gcp#show ip route ospf
O 10.1.0.2/32 [110/1000] via 10.1.0.2, 02:43:14, Tunnel0
O % 10.1.0.4/32 [110/2000] via 10.1.0.2, 01:21:14, Tunnel0
O % 10.1.0.6/32 [110/2000] via 10.1.0.2, 00:58:47, Tunnel0
O IA% 10.10.1.0/24 [110/2001] via 10.1.0.2, 00:58:00, Tunnel0
O IA% 172.16.2.0 [110/2001] via 10.1.0.2, 01:21:14, Tunnel0
O 192.168.200.0/24 [110/1001] via 10.1.0.2, 02:43:14, Tunnel0
Spoke – Google Cloud Platform CSR
csr-aws-01#show ip route ospf
O IA% 10.0.1.0/24 [110/2001] via 10.1.0.2, 01:21:32, Tunnel0
O % 10.1.0.1/32 [110/2000] via 10.1.0.2, 01:21:32, Tunnel0
O 10.1.0.2/32 [110/1000] via 10.1.0.2, 01:21:32, Tunnel0
O % 10.1.0.6/32 [110/2000] via 10.1.0.2, 00:59:01, Tunnel0
O IA% 10.10.1.0/24 [110/2001] via 10.1.0.2, 00:58:14, Tunnel0
O 192.168.200.0/24 [110/1001] via 10.1.0.2, 01:21:32, Tunnel0
Spoke – Amazon Web Services CSR
csr-azure-01#show ip route ospf
O IA% 10.0.1.0/24 [110/2001] via 10.1.0.2, 00:58:44, Tunnel0
O % 10.1.0.1/32 [110/2000] via 10.1.0.2, 00:58:44, Tunnel0
O 10.1.0.2/32 [110/1000] via 10.1.0.2, 00:58:44, Tunnel0
O % 10.1.0.4/32 [110/2000] via 10.1.0.2, 00:58:44, Tunnel0
O IA% 172.16.2.0 [110/2001] via 10.1.0.2, 00:58:44, Tunnel0
O 192.168.200.0/24 [110/1001] via 10.1.0.2, 00:58:44, Tunnel0
Spoke – Azure CSR
IA - OSPF inter area
% - next hop override

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS BRKCLD-3440
NHRP Example – Hub/Spoke
asr-mc-01#show ip nhrp
10.1.0.1/32 via 10.1.0.1
10.1.0.4/32 via 10.1.0.4
10.1.0.6/32 via 10.1.0.6
asr-mc-01#show ip nhrp multicast
I/F NBMA address
Tunnel0 35.xxx.xxx.x Flags: dynamic (Enabled)
Hub On-Prem ASR
csr-azure-01#show ip nhrp
10.0.1.0/24 via 10.1.0.1
Type: dynamic, Flags: router rib nho
10.1.0.1/32 via 10.1.0.1
Type: dynamic, Flags: router nhop rib nho
10.1.0.2/32 via 10.1.0.2
Tunnel0 created 00:21:28, never expire
Type: static, Flags:
10.1.0.4/32 via 10.1.0.4
Type: dynamic, Flags: router nhop rib nho
10.10.1.0/24 via 10.1.0.6
Type: dynamic, Flags: router unique local
NBMA address: 10.10.0.4
(no-socket)
172.16.2.0/24 via 10.1.0.4
Type: dynamic, Flags: router rib nho
csr-azure-01#show ip nhrp multicast
I/F NBMA address
Tunnel0 192.xxx.xxx.x Flags: nhs (Enabled)
Spoke – Azure CSR
shmcfarl@AzureTestVm:~$ traceroute 10.0.1.3
traceroute to 10.0.1.3 (10.0.1.3), 30 hops max, 60 byte packets
1 10.10.1.4 (10.10.1.4) 1.220 ms 1.192 ms 1.328 ms
2 10.0.1.3 (10.0.1.3) 25.794 ms * 25.782 ms
Spoke – Azure VM

65
VNet Network
10.10.1.0/24
DMVPN
OpenStack Private Cloud
VPC Network
172.16.2.0/24
Cisco
CSR1000v
Cisco
CSR1000v
VPC Network
10.0.1.0/24
Cisco
CSR1000v
Cisco
ASR1000
Hub
Tunnel:
10.1.0.2
Spoke
Tunnel:
10.1.0.1
Spoke
Tunnel:
10.1.0.4
Spoke
Tunnel:
10.1.0.6
192.168.200.0/24
OSPF 10 Area 0
VM.110
DataCenter
Infra.
10.40.0.0/24
.6
Neutron
Router

Split-
Tunneling/Routing
Options

Split-Tunnel/Routing Options
67
• All three public cloud providers allow for either split-tunneling or forced/direct routing
• Split-tunneling:
• Public cloud resources (instances/VMs, container clusters) will use the default VPC gateway for
non-on-prem routes
• Public cloud resources will use the on-prem-specific routes advertised by the CSR
• Forced/Direct routing – All public cloud resources will use the VPN connection as their
default route for ALL traffic (forces traffic through the on-prem site)
BGP
Google Cloud VPN
Google Cloud Router
Cisco
ASR1000
10.0.0.1
VPC Subnetwork
GW
External/NAT
Routing
192.xxx.xxx.x
35.xxx.xxx.x10.0.0.5
Compute
Engine
2 1

Dealing with Split Routes
• OpenStack with two possible routes:
• Typically the Neutron L3 agent is the default
route for VMs on the Private-Network
(172.16.0.1)
• Adding a CSR for GCP-facing connections
requires route changes:
• Static definition or dynamically learned via Neutron
BGP service
68
[centos@c7-os-vm1 ~]$ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 172.16.0.1 0.0.0.0 UG 0 0 0 eth0
10.138.0.0 172.16.0.11 255.255.240.0 UG 0 0 0 eth0
169.254.169.254 172.16.0.1 255.255.255.255 UGH 0 0 0 eth0
172.16.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
openstack subnet set --host-route destination=10.138.0.0/20,gateway=172.16.0.11 Private-Subnet
Default
10.138.0.0/20

Summary
• Public cloud native IPsec VPN support is good, but it is always point-to-point, does not have
consistent support for NAT and lacks network-rich features
• DMVPN can greatly improve the deployment, HA, scalability and operations of the VPN
connections
• Multicloud between multiple public cloud providers and on-prem look like distinctly separate
hybrid cloud deployments but..
• You have to take into consideration:
• Team knowledge of public cloud operations, tools, automation
• Cross cloud tools and automation
• Diversity of network designs, protocols, security
• Multi-region designs
• Availability zones within and across providers
69

A Note On MTU
• All three providers recommend a different size interface MTU for the IPsec tunnel
interface:
• Google recommends 1460 on the tunnel:
https://cloud.google.com/vpn/docs/concepts/advanced#mtu
• AWS recommends 1399 on the tunnel:
https://docs.aws.amazon.com/AmazonVPC/latest/NetworkAdminGuide/Introduction.html
• Azure recommends 1400 on the tunnel: https://docs.microsoft.com/en-us/azure/vpn-
gateway/vpn-gateway-about-vpn-devices
• In addition to MTU, you need to set and test your TCP MSS values
• In my testing, an IP MTU of 1400 and TCP MSS of 1360 worked for all sites but this
may need to change based on your applications and if you are adding other encaps
like MPLS
72

Automating the Multicloud Network
• Challenges:
• Different toolsets for different jobs (Ansible, Python, Bash scripts, Terraform, etc..)
• Different toolsets for different clouds (Heat for OpenStack, CloudFormation for AWS, Deployment Manager for GCP,
Azure Automation)
• Different toolsets for different vendor products (Cisco NSO, CloudCenter, Prime, YANG development kit, etc..)
• There is no silver bullet - Start simple:
• Use what your team knows – Perform a gap analysis on what you have against what you need
• Initially, automate the things that hurt a lot to do by hand and that change frequently – I use free tools but that doesn’t
mean the process is free J
• I use public cloud clients (gcloud, aws cli, azure cli) for services that don’t change frequently or that need very unique/non-
repeatable configurations
• I use public cloud provider automation tools (GCP Deployment Manager) for in-project work (new instances with new
networks for a GCP-only project)
• I use REST for things that change a lot
• When you want to stop pulling your hair out, move to something that can front-end each API that you need to talk to and
treat the environment as a whole – Cisco CloudCenter: https://www.cisco.com/c/en/us/products/cloud-systems-
management/cloudcenter/index.html
74

Amazon CloudFormation
• https://aws.amazon.com/cloudformation/
• Template-based (JSON/YAML) – Build a stack(s) from a template file
• Sometimes you need to run more than one stack (in order) to get what you need
• Race conditions: Understand ‘DependsOn’ and the use of the wait condition
• If you need to use more than one stack, use “Outputs” to export values that the next stack will
need to build the next set of resources
• Example template: https://github.com/shmcfarl/multicloud/tree/master/aws/cloudformation
75

Google Cloud Platform – Deployment Manager
• https://cloud.google.com/deployment-manager/
• Configuration files (YAML), Templates (Python/Jinja2), Schema files (JSON)
• Sometimes you need to run more than one stack (in order) to get what you need
• Race conditions
• Use “Outputs” to export values that the next stack will need to build the next set of resources
• Example templates: https://github.com/shmcfarl/multicloud/tree/master/gcp/deployment-
manager
• Make your own changes to the files: <ZONE>, <PROJECT>, <IMAGE>, etc.
• Deploy the main stack:
• Deploy any custom routes that may be needed for other sites
76
gcloud deployment-manager deployments create gcp-stack
--config gcp_main_stack.yaml
--automatic-rollback-on-error
gcloud deployment-manager deployments create gcp-stack-route
--config inside-private-routes.yaml
--automatic-rollback-on-error

Microsoft Azure Automation/Resource Manager
• https://azure.microsoft.com/en-us/services/automation/
• Runbooks (create graphically, PowerShell, Python)
• Read and select these carefully: https://docs.microsoft.com/en-
us/azure/automation/automation-runbook-types
• Resource Manager: https://docs.microsoft.com/en-us/azure/azure-resource-
manager/resource-group-overview
• https://github.com/Azure/azure-quickstart-templates/tree/master/cisco-csr-1000v
• Example template:
https://github.com/shmcfarl/multicloud/blob/master/azure/resource-manager/az-
arm-csr-cleaned.json
77

Call APIs Directly
• Google Cloud Platform:
https://cloud.google.com/compute/docs/reference/latest/
• Amazon Web Services:
https://docs.aws.amazon.com/AWSEC2/latest/APIReference/Welc
ome.html
• Microsoft Azure: https://docs.microsoft.com/en-us/rest/api/
78

Google VPN –
Creating Google VPN,
Router, IPsec, BGP via
REST APIs

Google Cloud API – Creating GCP Cloud
VPN/Routers
• Assumptions/environment:
• Understand how to authenticate to GCP APIs:
https://cloud.google.com/docs/authentication/
• In this example, the Paw application was used to craft GET, POST and PATCH calls
• Some configurations have been sanitized for security purposes
• Have on-prem Cloud infrastructure deployed and a CSR/ASR configured (can be
done after GCP side is deployed)
• In this example, the configuration will be deployed against the OpenStack use case
discussed in the earlier slides
• In this example, the default network created by GCP will be used
• Note: gcloud has VERY long delays in commands if you have IPv6 enabled on your
local machine – set to “link-local” mode on your Mac
80

Reference Topology for GCP API Example
Private Network
172.16.0.0/24
.11Default Network
10.138.0.0/20
IPsec/IKEv2
Tunnel Mode
OSPF 10 Area 0
OSPF<>BGP Redistribution
192.yyy.yyy.y
35.yyy.yyy.y
10.138.0.0/20
172.16.0.0/24
On-Prem Cloud
Google
Cloud
VPN
Google Cloud Router
BGP AS65000 BGP AS65003
169.254.0.5
169.254.0.6

GCP API (1) – Create VPN GW and External IP
POST /compute/v1/projects/<gcp_project_number>/regions/us-west1/targetVpnGateways HTTP/1.1
Authorization: Bearer XXXX
Content-Type: application/json; charset=utf-8
Host: www.googleapis.com
Connection: close
Content-Length: 138
{
"name": "csr-gcp-os-aio-gw",
"network": "projects/<gcp_project_number>/global/networks/default",
"region": "projects/<gcp_project_number>/regions/us-west1"
}
POST /compute/v1/projects/<gcp_project_number>/regions/us-west1/addresses HTTP/1.1
Connection: close
Content-Length: 29
{
"name": "gcp-to-os-dmz"
}
GET /compute/v1/projects/<gcp_project_number>/regions/us-west1/addresses/gcp-to-os-dmz HTTP/1.1
Connection: close
RESPONSE - SUMMARIZED:
"name": "gcp-to-os-dmz",
"description": "",
"address": ”35.yyy.yyy.y",
"status": "RESERVED",
POST: Create VPN
Gateway
POST: Create External IP
Address
GET: Get the External IP
Address

GCP API (2) – Create Forwarding Rules
83
POST /compute/v1/projects/<gcp_project_number>/regions/us-west1/forwardingRules HTTP/1.1
Connection: close
Content-Length: 257
{
"name": "csr-gcp-os-aio-rule-esp",
"IPProtocol": "ESP",
"IPAddress": "35.yyy.yyy.y",
"region": "projects/<gcp_project_number>/regions/us-west1",
"target": "https://www.googleapis.com/compute/v1/projects/<gcp_project_number>/regions/us-west1/targetVpnGateways/csr-gcp-os-aio-gw"
}
Connection: close
Content-Length: 278
{
"name": "csr-gcp-os-aio-rule-udp500",
"IPProtocol": "UDP",
"target": "https://www.googleapis.com/compute/v1/projects/<gcp_project_number>/regions/us-west1/targetVpnGateways/csr-gcp-os-aio-gw",
"portRange": "500"
}
Connection: close
Content-Length: 280
{
"name": "csr-gcp-os-aio-rule-udp4500",
"IPProtocol": "UDP",
"target": "https://www.googleapis.com/compute/v1/projects/<gcp_project_number>/regions/us-west1/targetVpnGateways/csr-gcp-os-aio-gw",
"portRange": "4500"
}
POST: Create
Forwarding rule for ESP
POST: Create
Forwarding rule for UDP
500
POST: Create
Forwarding rule for UDP
4500

GCP API (3) – Create Cloud Router & BGP Session
84
POST /compute/v1/projects/<gcp_project_number>/regions/us-west1/routers HTTP/1.1
Connection: close
Content-Length: 574
{
"name": "csr-gcp-os-bgp-rtr",
"bgp": {
"asn": "65000"
},
"interfaces": [
{
"name": "if-csr-gcp-os-bgp-rtr-02",
"linkedVpnTunnel": "https://www.googleapis.com/compute/v1/projects/<gcp_project_number>/regions/us-west1/vpnTunnels/csr-gcp-os-aio-gw-tunnel-1",
"ipRange": "169.254.0.5/30"
}
],
"bgpPeers": [
{
"name": "csr-gcp-os-bgp-peer",
"interfaceName": "if-csr-gcp-os-bgp-rtr-02",
"ipAddress": "169.254.0.5",
"peerIpAddress": "169.254.0.6",
"peerAsn": "65003"
}
],
"network": "https://www.googleapis.com/compute/v1/projects/<gcp_project_number>/global/networks/default"
}
POST: Create Cloud Router, BGP session and link to the Cloud VPN tunnel

GCP API (5) – Create Cloud VPN Tunnel
85
POST /compute/v1/projects/<gcp_project_number>/regions/us-west1/vpnTunnels HTTP/1.1
Connection: close
Content-Length: 417
{
"name": "csr-gcp-os-aio-gw-tunnel-1",
"sharedSecret": " <pre-shared-password-goes-here> ",
"router": "https://www.googleapis.com/compute/v1/projects/<gcp_project_number>/regions/us-west1/routers/csr-gcp-os-bgp-rtr",
"peerIp": "192.yyy.yyy.y",
"ikeVersion": "2",
"targetVpnGateway": "https://www.googleapis.com/compute/v1/projects/<gcp_project_number>/regions/us-west1/targetVpnGateways/csr-gcp-os-aio-gw"
}
POST: Create a Cloud VPN tunnel and associated it with the Cloud router

Openstack Summit Vancouver 2018 - Multicloud Networking

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Openstack Summit Vancouver 2018 - Multicloud Networking

Similar a Openstack Summit Vancouver 2018 - Multicloud Networking (20)

Último

Último (20)

Openstack Summit Vancouver 2018 - Multicloud Networking