6. Native Container Networking
6
Without NSX
Challenge
• Microservices are
connected to Private Container network that only
spans the PaaS platform
• Requires ramp nodes and NAT for integrating
physical services – e.g. Firewall, Load Balancer
Benefits
• A single network fabric that connects VMs, network services and
containers across on premise and public cloud
• Container Network integrates with rest of Data Center network with
BGP
• Layer 3 reachability between LB, FW and Containers
simplifies integration of network services
CaaS / PaaS platformWith NSX
Ramp
Node (NAT)
CaaS / PaaS platform
Container Network
7. Microsegmentation for Microservices
77
Without NSX
Challenge
• No means for a devops and security admin to
define, implement & monitor security policy
for microservices
• Not possible to apply policy for
Microservice → database traffic due to NAT
Benefits
• NSX enables both the devops admin and the security admin to
define & monitor policy for microservices
• Prioritizes security admin policy
• Enables users to define policy for
1. Microservice ←→ Microservice traffic
2. Microservice →Database traffic
With NSX
Ramp
Node (NAT)
CaaS / PaaS platform
Container Network
CaaS / PaaS platform
1
2
8. NSX-T & PaaS / CaaS integration
PaaS Control Plane
etcd
API-Server
Scheduler
NCM
Infra
K8s / OS
Adapter
CloudFoundry
Adapter
Libnetwork
Adapter
NSX Container Plugin
More…
NSX
Manager
API Client
Proj: foo Proj: bar
NSX topology for K8s / CF
• NSX integration with K8s/PCF and NSX Container Plugin
(NCP) for integrating with Caas/PaaS with NSX Manager
• Native Container Networking
• IP address per container / POD
• Container Network integration with DC network via routing & BGP
• Microsegmentation – inter project and intra project isolation
• Operations – Same operational tools likes Traceflow and Port Connectivity are available for visibility.
9. NSX Operational Tools for Enterprise CaaS and PaaS
NSX-T Traceflow
NSX-T Operational
Tools
• Traceflow
• Port Mirroring
• Port Connection
Tool
• Spoofguard
• Syslog
• Port Counters
• IPFIX
11. Cloud Foundry NSX-T Topology
Org: foo Org: bar
NSX/ CF topology
• Orgs: We are dynamically building a separate network
topology per CF Org, every CF Org gets one Tier-1 router
• Spaces: We are creating one or more Logical Switches
per Space, and are attaching them to the Org T1 router
• Cells: Are not doing NAT, every AI (container) has its
own logical port on a NSX logical switch. Every Cell can
have AIs from different Orgs & Spaces, and with this from
different IP Subnets / Topologies
• North/South: High performant North/South routing using
NSX’s routing infrastructure, including dynamic routing to
physical network. Direct Gorouter to Container routing (no
NAT through Cell VM), NAT or No-NAT selectable at
install time
• East/West: Direct C-to-C traffic – No Gorouter hairpin
• Firewall: Every AI (container) has DFW rules applied on
its Interface, with policies defined in the new
cf-networking policy server. ASGs are also mapped to Fw
• Visibility and troubleshooting: Every AI (container) has
a logical port on the logical switch with:
• Counters, SPAN / Remote mirroring, IPFIX export,
Traceflow & Port-Connection tool, Spoofguard
• IPAM: NSX is used to provide IP Address Management
by supplying Subnets from IP Block to Namespaces, and
Individual IPs and MAC to AI (container)
Cloud Foundry NSX Topology
10.12.0.0/24 10.12.1.0/24 10.12.3.0/24
12. • NSX Container Plugin: NCP is a software
component provided by VMware in form of a
BOSH add-on release. It is deployed as a pair of
HA VMs as part of the ERT (using a Ops
Manager Tile)
• Adapter layer: NCP is build in a modular way, so
that individual adapters can be added for different
CaaS and PaaS systems
• NSX Infra layer: Implements the logic that
creates topologies, attaches logical ports, etc.
based on triggers from the Adapter layer
• NSX API Client: Implements a standardized
interface to the NSX API
Network Container Plugin (NCP)
CF / NSX-T Components
Network Container Plugin (NCP)
NCP
Infra
K8s / OS
Adapter
CloudFoundry
Adapter
More ...
NSX Container Plugin
More…
NSX
Manager
API Client
NSX
Manager
Org: foo Org: bar
NSX/ CF topology
BBS
mysql Brain
Cloud Controller
CAPI
Policy Server
Policy API
Space: Prod Space: Prod
15. Namespace creation workflow
PKS / NSX Workflows
Namespace / Topology creation
NCM
Infra
K8s / OS
Adapter
CloudFoundry
Adapter
Libnetwork
Adapter
NSX Container Plugin
More…
NSX
Manager
API Client
NSX
Manager
NS: foo
NSX/ K8s topology
NS: bar
K8s master
etcd
API-Serve
r
Scheduler
1)
2)
3)
4)
1. NCP creates a ‚watch‘ on K8s API for
any Namespace events
2. A user creates a new K8s Namespace
3. The K8s API Server notifies NCP of the
change (addition) of Namespaces
4. NCP creates the network topology for the
Namespace :
a) Requests a new subnet from the
pre-configured IP block in NSX
b) Creates a logical switch
c) Creates a T1 router and attaches it to
the pre-configured global T0 router
d) Creates a router port on the T1 router,
attaches it to the LS, and assigns an
IP from the new subnet