Tungsten Fabric SDN Controller overview, Microservices Architecture, and Multi-Cloud feature overview

1. Tungsten Fabric
Overview & Deepdive
Qasim Arham
July 19, 2018 (OSNUG Dallas Meetup)

2. MISSION
Build the world’s most ubiquitous, easy-to-use, scalable, secure, and cloud-grade SDN
stack, providing a network fabric connecting all environments, all clouds, all people.

3. CODE & COMMUNITY

4. CODE
• 2013-Today: >300 years of work
• 200-300 developer contributions
• ~100 active developers
• Languages: C++, Python, Node, Go
• Apache 2.0 license
• Part of the Linux Foundation Networking
• GitHub repositories
• Gerrit review processes
• Launchpad bug tracking and blueprints
• Other OSS used: Cassandra, Kafka, HAproxy,
Docker, Keystone

5. COMMUNITY
Principles:
• Open and inclusive
• Provide strong technical and
architectural oversight
• Competitive ideas welcome
• Rough consensus and running code
will always win
• Iterate and evolve

6. COMMUNITY
• Online:
• Downloads and trial sandbox
• Talk with 900+ people: Slack, Mailing lists
• Follow: Blog, YouTube, Facebook, Twitter
• GitHub: Presentations, Tutorials
• Live (see calendar) :
• Conferences: OpenStack, KubeCon, ONS,
Re:invent and GC Next
• Meetups: host your own or join some
• User Group events: often at conferences
• Governance summits
• Groups: Governance, Technical, Infrastructure
• Community manager: Greg Elkinbard
JOIN
• tungsten.io/slack
• tungsten.io/community

7. COMMUNITY MEMBERS
your logo here

8. PAST, PRESENT & FUTURE
• OpenStack networking at scale
• NFV service chaining
• Analytics collection/querying
• REST API and GUI
• Docker & ESXi runtime support
• VMware vSphere support
• DPDK vRouter
• Prototype with Kubernetes v1.1
• Node-port service chaining
• Improve analytics with Kafka
• LBaaS
• ToR switch as OVSDB gateway
• Kubernetes and CNI support
• OpenShift and Mesos support
• Containerize project
• New install w/ Ansible or Helm
• Security focus
• Multicloud deployability
• Switching fabric focus
• Declarative network as code
v1 v2 v4v3 v5+

9. FEATURES

10. Tungsten Fabric Overview
DC
Interconnect
controller
VM VM VM VM VM VM VM VM
VN VN VN
Network
Appliances
(eg SRX)
Bare Metal
(e.g. S/PGW )
e.g. OpenClos, VC, VCF or QF fabric
Virtual Compute Server Infrastructure
VLAN
BGP
control plane
vRouter
Hypervisor
vRouter
Hypervisor
vRouter
Hypervisor
VNF
XMPP control plane
Orchestrators
REST APIs
Configuration
Nodes
Control
Nodes
Analytics
Nodes
IF-MAP
REST
XMPP
BGP
BGP,
Netconf
vRouters Gateways
BGP
REST
Container Container
vRouter
EVPN/VXLAN
Containerize Control Plane
(MicroServices)
vRouter AgentvRouter AgentvRouter Agent vRouter Agent
Tungsten Fabric

11. ARCHITECTURE OVERVIEW
Ethernet / IP
underlay network
TF CONTROLLER, API & GUI
scale-out control and
management container
micro-services
REST
XMPP
ORCHESTRATION NODES
XMPP
virtual overlay networks
TF
Orchestration plug-ins
Control
COMPUTE NODE 2…
TF
vRouter
COMPUTE NODE 1
TF
vRouter
Compute Runtime Compute Runtime
Control
Networks isolated unless
connected with policy

12. USER EXPERIENCE
• REST API
• HTTPS authentication and
role-based authorization
• Used for GUI
• Used for declarative
configurations as code
• Generated from data model
NORTH-BOUND
API
GUI

13. VIRTUAL
NETWORK
GREEN
Host + Hypervisor
Host + Hypervisor
Visualizing Tungsten Fabric’s Operational Effects
VIRTUAL
NETWORK
BLUE
VIRTUAL
NETWORK
YELLOW
TF Security Policy
(e.g. allow only HTTP traffic)
Service Chain
Policy with a
Firewall VNF
IP fabric
(switch underlay)
G1 G2 G3
B3
B1
B2
G1
G3
G2
Y1 Y2 Y3B1 B2 B3
Y2Y3
Y1
VM and virtualized Network
function pool
Intra-network traffic Inter-network traffic traversing a service
… …
LOGICAL
(PolicyDefinition)
PHYSICAL
(PolicyEnforcement)
Non-HTTP
traffic
Security
Groups

14. Seamless Multi-Cloud Overlay SDN
Telco POPs Private Cloud DC Public Cloud VPCUsers
Multicloud SDN
Virtual Networking: Overlay Virtual Networking provides connectivity for VM’s and Containers
Distributed Compute Platforms: Leverage the right balance of edge compute, private cloud
compute, and public cloud compute to deploy services
Ubiquitous Security – Centralized security policy orchestration with distributed enforcement across multiple clouds
Performance and Scale: Manage remote compute resources, high performance virtual network
functions, and containers using the same tools
Overlay SDN

15. Tungsten Fabric vRouter Architecture & Overview
Host Compute
Kernel space
User
space
vRouter Kernel
vRouter Agent
Config
VRFs
Policy
Table
vhost0
pkt0 tap-xyztap-abc
Netlink
Virtual
Machine
(Tenant A)
Routing
Instance
Routing
Instance
Virtual
Machine
(Tenant B)
Control Node
XMPP ethX OR bondX
vRouter Agent
• Exchanging control state such as routes with the Control nodes
using XMPP.
• Receiving low-level configuration state such as routing instances
and forwarding policy from the Control nodes using XMPP
• Reporting analytics state such as logs, statistics, and events to the
analytics nodes.
• Installing forwarding state into the forwarding plane
• Discovering the existence and attributes of VMs in cooperation
with the Nova agent.
• Applying forwarding policy for the first packet of each new flow
and installing a flow entry in the flow table of the forwarding
plane.
• Proxying DHCP, ARP, DNS
vRouter Kernel/DPDK
• Encapsulating packets sent from the overlay network and
de-capsulating packets received for the overlay network.
• Packets received from the overlay network are assigned to a
routing instance based on the MPLS label or Virtual Network
Identifier (VNI).
• Doing a lookup of the destination address of the in the Forwarding
Information Base (FIB) and forwarding the packet to the correct
destination. The routes may be layer-3 IP prefixes or layer-2 MAC
addresses.
• Doing RPF check before sending Virtual machine traffic to
destination. This is configurable.

16. TF vRouter Architecture & Overview (cont.)
Overlay tunnels
MPLS over GRE or VXLAN
Compute Node
vRouter Forwarding Plane
Virtual
Machine
(VN-IP1)
Routing
Instance
Flow Table
FIB
Eth1 (Phy-IP1)
Tap Interfaces (vif)
Compute Node
vRouter Forwarding Plane
Virtual
Machine
(VN-IP2)
Routing
Instance
Flow Table
FIB
Eth1 (Phy-IP2)
Tap Interfaces (vif)
Virtual-IP2
Payload
Virtual-IP2
Payload
MPLS
Phy-IP2
Virtual-IP2
Payload
Virtual-IP2
Payload
MPLS
Phy-IP2
1. Guest OS ARPs for destination within subnet or
default GW
2. vRouter receives the ARP and responds with VRRP
MAC
3. Guest OS sends traffic to the VRRP MAC, vrouter
encapsulates the packet with appropriate MPLS tag
and GRE header
4. Physical fabric routes on physical IP address
5. Returning packets get forwarded to appropriate
routing instance by the MPLS tag
6. vrouter decapsulates the packet, and forwards it to
the guest OS

17. VROUTER DEPLOYMENT MODELS
KERNEL VROUTER DPDK VROUTER
SRIOV/ VROUTER COEXISTENCE SMARTNIC VROUTER
…
VM 1vRouter
Agent
VNF 2
…VM 1vRouter
Agent
VM 2
…VM 1vRouter
Agent
VM 2
…VM 1vRouter
Agent
VM 2
▪ vRouter runs as a user
space process and
uses DPDK for fast
path Packet I/O.
▪ Full set of SDN
Capabilities Supported
▪ Requires the VMs to
have DPDK enabled
for performance
benefits
▪ vRouter fwding plane
runs within the NIC
▪ Workloads are
SRIOV-connected to the
NIC
▪ Some workloads can directly
SRIOV into the NIC, while
others go through the vRouter
▪ Sometimes a VNF can have
multiple interfaces some of
which are SRIOV-ed to the NIC
▪ Interfaces that are SRIOV-ed
into NIC don’t get the benefits /
features of vRouter
▪ This the normal operation where
fwding plane of vRouter runs in
the kernel and are connected to
VMs using TAP interface (or
veth pair for containers)
▪ vRouter itself is enhanced using
other performance related
features:
o TSO / LRO
o Multi-Q Virtio

18. The Latest from Tungsten Fabric
➢ Microservices
architecture
➢ Better cloud
native
deployment
options
➢ Comprehensive
support for
Network objects
➢ Ingress/Egress
Network Policy
➢ High
performance
load balancing
➢ Improved flow
performance and
management
➢ SDN for Edge
Compute – Beta
Quality
Housekeeping Container SDN VM’s and NFV

19. Tungsten Fabric
Kubernetes Support (helm)

20. What is Tungsten Fabric Helm charts?
Tungsten Fabric Helm

21. OpenStack and TF Helm Overview
OpenStack Helm
Tungsten Fabric Helm
OpenStack &
TF Helm
Solution

22. Tungsten Fabric Integration with k8s
Compute Node-01
CNI Plugin
Kubelet
POD 1 POD 2
CNI Plugin
Kubelet
POD 3 POD 4
vRouter
(replaces kube-proxy)
vRouter
(replaces kube-proxy)
Contrail-kube-mgr
Contrail Controller
API Server
Scheduler
Controller/Replication Manager
kubectl
(user commands)
etcd
Discovery Dashboard Contrail Analytics
Compute Node-02
Namespace: kube-system
* Contrail-Kube-manager listens to K8s API Server and
conveys the API request to Contrail Controller
…

23. Tungsten Fabric Evolution to Microservices
VM VMVM
Contrail Controller: 2n+1
Contrail 1.X/2.X/3.X
BMS or VMs base
(SDN Controller)
OR
BMS
Contrail 4.X (Containers)
BMS or VMs base
(SDN Controller)
Contrail 5.X (Containers)
Microservices
(SDN Controller)
Analytics
DB
Config +
Control
Analytics
Kube
MGR
HA
Proxy
vRouter
Agent
Multiple Process running in one
Container (FAT Containers)
● Contrail-Control (5 daemons)
● Contrail-Config ( 8 daemons)
● Contrail-Analytics (5 daemons)
● Contrail-WebUI ( 4 daemons)
● Contrail-DB (3 daemons)
● Contrail-vRouter (3 D) + Kernel/DPDK (FP)
DaemonSet, Ingress Services with Host
Networking
with choice of run single or multiple
containers per PODs
27-30 Containers Images

24. Tungsten Fabric Helm Charts
Tungsten Fabric Helm Charts
Contrail Contrail-3rd-Party Contrail-Controller Contrail-Analytics Contrail-vRouter
● Contrail: Parent helm chart for all contrail networking
● Contrail-Third Party: Helm chart for Contrail third-party components
● Contrail-Controller: Helm chart for contrail controller
● Contrail-Analytics: Helm chart for contrail analytics
● Contrail-vrouter: Helm Chart for contrail vRouter (DPDK & Kernel)
● Helm-Toolkit-Contrail: Chart where we define common templates/method used by all
other contrail charts
Helm-Contrail-Toolkit

25. TF Helm Microservices Architecture (Helm Charts)
Kubernetes Cluster
contrail-analytics-api
Container (1/7)
Host-Networking
Contrail-Analytics
contrail-control
contrail-config
contrail-webui
contrail-analytics
Contrail-Controller
contrail-vrouter
Contrail-
vRouter
Contrail-Third-Party
Contrail Helm
Toolkitcontrail-analytics-nodemgr
Container (2/7)
Host-Networking
contrail-collector
Container (3/7)
Host-Networking
contrail-snmp-collector
Container (4/7)
Host-Networking
contrail-query-engine
Container (5/7)
Host-Networking
Contrail-topology
Container (6/7)
Host-Networking
Contrail-alarm-gen
Container (7/7)
Host-Networking
POD – DaemonSet (7/7)
analytics-api
Service
Cluster-SVC-Networking
analytics-ingress
Service
Cluster-SVC-Networking
analytics-api
Ingress
Cluster-SVC-Networking
contrail-svc-monitor
Container (3/5)
Host-Networking
contrail-config-nodemgr
Container (2/5)
Host-Networking
contrail-config-api
Container (1/5)
Host-Networking
contrail-schema-transf
Container (4/5)
Host-Networking
contrail-device-mgr
Container (5/5)
Host-Networking
POD – DaemonSet (5/5)
contrail-control
Container (1/4)
Host-Networking
contrail-dns
Container (2/4)
Host-Networking
contrail-named
Container (3/4)
Host-Networking
contrail-webui
Container (1/2)
Host-Networking
contrail-webui-middleware
Container (2/2)
Host-Networking
analyticsdb
POD - DaemonSet
Host-Networking
analytics-zookeeper
POD - DaemonSet
Host-Networking
configdb
POD - DaemonSet
Host-Networking
config-zookeeper
POD - DaemonSet
Host-Networking
kafka
POD - DaemonSet
Host-Networking
redis
POD - DaemonSet
Host-Networking
configdb-nodemgr
POD - DaemonSet
Host-Networking
analyticsdb-nodemgr
POD - DaemonSet
Host-Networking
POD – DaemonSet (3/3)
POD – DaemonSet (2/2)
config-api
Service
Cluster-SVC-Networking
config-ingress
Service
Cluster-SVC-Networking
web-controller
Service
Cluster-SVC-Networking
web-ingress
Service
Cluster-SVC-Networking
config-api
Ingress
Cluster-SVC-Networking
webui
Ingress
Cluster-SVC-Networking
contrail-control-nodemgr
Container (1/4)
Host-Networking
contrail-vrouter-nodemgr
Container (2/2)
Host-Networking
contrail-vrouter-agent
Container (1/2)
Host-Networking
POD
DaemonSet
(2/2)
contrail-vrouter-agent-dpdk
Container (1/3)
Host-Networking
contrail-vrouter-dpdk
Container (2/3)
Host-Networking
contrail-vrouter-nodemgr
Container (3/3)
Host-Networking
POD DaemonSet (2/2)
Other Containers:
• Contrail-status
• node-init
• vrouter-init-kernel
• vrouter-init-dpdk

26. DIFFERENT LEVELS OF ISOLATION
N a m e s p a c e - B
S3 S4
POD 9
…
POD 13
…
…
N a m e s p a c e - A
S1 S2
POD 1
…
POD 5
…
…
N a m e s p a c e - D
S7 S8
POD 25
…
POD 29
…
…
N a m e s p a c e - C
S5 S6
POD 17
…
POD 21
…
…
N a m e s p a c e - F
S11 S12
POD 41
…
POD 45
…
…
N a m e s p a c e - E
S9 S10
POD 33
…
POD 37
…
…
DEFAULT CLUSTER MODE NAMESPACE ISOLATION POD / SERVICE ISOLATION
▪ This is how Kubernetes networking works
today
▪ Flat subnet where -- Any workload can talk to
any other workload
▪ In addition to default cluster, operator can add
isolation to different namespaces transparent to
the developer
▪ In this mode, each POD is isolated from
one another
▪ Note that all three modes can co-exist

27. INSTALLATION
• Ansible playbook to flexibly deploy Tungsten Fabric binaries
• Helm charts to easily operate Tungsten Fabric components on Kubernetes
• Install-time option with OpenShift to deploy with Tungsten Fabric
• Tungsten Fabric binaries available on DockerHub and we’re improving
CI/CD
• Commercial integrations into lifecycle tools like RH OpenStack Director

28. VERSATILE SDN SOLUTION
L4 Policy
Tungsten Fabric network and security policies
provide fine grain traffic control, while
abstracting away the underlay topology.
1
Svc Chain Policy2
Containers
App Tier DB Tier
BMS
VMs VMsFWLB
Web Tier
VMs
1
2
1
Consistent security and network functionality between VMs, containers, or bare metal.
…
VM
Compute Node
Nested Container
Compute Node
Tungsten Fabric
Username
Passwor
d
…
NFV
Compute Node

29. SOFTWARE DEFINED SECURE NETWORKING
…
Web App db
App1, Deployment = Dev
Web App db
App1, Deployment = Staging
Web App db
App1, Deployment = Prod
Tungsten fabric provides a rich, consistent set of security policy capabilities across multiple platforms.
Web App db
App1, Deployment = Dev-K8s
Web App db
App1, Deployment =
Dev-Mesos
vRouter Security Groups
Web App db
App1, Deployment = Staging-BMS
B a r e M e t a l S e r v e r
s
Network Policy
Device
Manager
1. Simplified Manageability (change control, etc. is much easier)
2. Improved Scalability
3. Define / Review / Approve Once → Use Everywhere

30. Tungsten Fabric Multi Cloud
Multi-Cloud Networking for Converged Operators

31. TF SDN Controller for VM & Containers
Neutron/CNI
SDN Controller
Kubernetes
CNI
Neutron
ML2 Plugin
Edge Site
Edge Site
Edge Site
Basic Networking:
L2/L3 or L2/L3 Network
IPAM/DHCP, DNS, Multi-Tenancy
Advance Networking:
VLAN-ID, VRRP, VIP, LB, Routes
Advertisement,
GW Function, Service Chaining, Traffic
Steering, Flow awareness,
QoS, SR-IOV/DPDK, BGP-VPN,
Inter Site Federation, Health Checks, FW,
IPSec/TLS Support

32. 5G Edge Computing and Encryption
Cell Sites
RRU
DU
(Distributed Unit)
Edge Site (Data Center)
IPSec or SSL Tunnel
Centralized
Data Center
UPF
CCF
Core Network VNFs
RPF
CU
(Central Unit)
APP
UPFPPF
RCF
Secure RAN to CN
● Use Contrail Encryption to secure Remote Edge and Central DC connection.
● Secure Overlay site to site communication via Contrail encryption support
● Policy based encryption model
VNFs
VNFs
VNFs
APP Application
CCF Core Control Function (Core Network)
UPF User Plane Function (Core Network)
RCF Radio Control Function (RAN)
PPF Packet Processing Function (RAN)
RPF Radio Processing Function (RAN)
RRU Remote Radio Unit (RAN)

33. Try Tungsten Fabric
https://tungstenfabric.github.io/website/Tungsten-Fabric-15-minute-d
eployment-with-k8s-on-AWS.html

34. Carbide Sandbox Environment
Tungsten Fabric + Kubernetes on AWS
https://tungsten.io/start/

35. 0-60 in 15 Minutes Flat w/Carbide (TF+k8s on
AWS)

36. 0-60 in 15 Minutes Flat w/Carbide (TF+k8s on
AWS)

37. 0-60 in 15 Minutes Flat w/Carbide (TF+k8s on AWS)

38. 0-60 in 15 Minutes Flat w/Carbide (TF+k8s on AWS)