More Related Content Similar to Architecture of Cisco Container Platform: A new Enterprise Multi-Cloud Kubernetes Platform (20) Architecture of Cisco Container Platform: A new Enterprise Multi-Cloud Kubernetes Platform3. Agenda
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
• Product Introduction
• Architecture
• Operations
• Infrastructure
• network, storage, load balancing, add-ons
• Multi-cloud
• Recent features, upgrades
• Demos
BRKCLD-2676 3
My contact info:
Email: srampal@cisco.com
Twitter: @sr2357
4. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Cisco Webex Teams
Questions?
Use Cisco Webex Teams (formerly Cisco Spark)
to chat with the speaker after the session
Find this session in the Cisco Events App
Click “Join the Discussion”
Install Webex Teams or go directly to the team space
Enter messages/questions in the team space
How
Webex Teams will be moderated
by the speaker until Dec 15, 2018.
cs.co/ciscolivebot#BRKCLD-2676
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
1
2
3
4
BRKCLD-2676 4
5. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Containerization Challenges and Trends
in a Multicloud World
5BRKCLD-2676
Multiple Open Source Solutions Hybrid Environments
Container Complexity Networking, Security and Storage
Source: CNCF Survey, June’2017
Container Trends
1. Kubernetes is emerging as the leading container orchestration platform
2. Containers are being adopted heavily in on-premise data centers
Source: Jan 17, 2017 cncf.io blog
6. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Automates deploying, running, scaling, and
operating containers on physical or virtual
machines. Incl. Scheduling, Load balancing,
Rolling updates
Kubernetes Goals
• API and implementation 100% open
• Modular and replaceable
• Don’t force apps to know about
concepts that are
• Cloud Provider Specific
• Kubernetes Specific
Enable Users To
• Write once, run anywhere
• Avoid vendor lock-in
• Avoid coupling app to infrastructure
What Kubernetes provides
6BRKCLD-2676
7. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
• Inside: Container Runtime and Workloads
• Above: Management, Services and Tools
• Logging + Monitoring
• Kubernetes Stack Lifecycle Management, Patches, Upgrades
• CI/CD
• PaaS
• Workflow Orchestration
• Data processing
• OTS applications:
• Middleware + Storage + Databases + …
• Below: Diverse Infra Environments
• Container Storage, Container Network
• Image registry
• Cloud provider
• Cluster + host lifecycle management
• Identity and secret management
What Kubernetes does not provide
7BRKCLD-2676
Kubernetes
Docker
Infra Environments
Services and Management
8. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Cisco Container Platform
Hybrid Cloud Optimized
E.g: Google, istio, external secure registry, …
Flexible Deployment Model
VM | Bare metal ßà HX, ACI | Public cloud
Integrated
Networking | Management | Security | Analytics
Native Kubernetes (100% Upstream)
Direct updates and best practices from open source community
Turnkey Solution
For Production-Grade Optimized
Container Environments
Easy to acquire, deploy & manage | Open & consistent | Extensible platform | World-class advisory & support
9. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
• Deploy Kubernetes clusters on
HyperFlex and vSphere
• Container Networking – CNI and
service mesh (Istio)
• Persistent storage (Flex Driver)
• L3 / L7 Load Balancing (Nginx)
• Container Registry (Harbor)
• AD Authentication / RBAC
• Communication between containers
and VMs / BMs
• Resource based node pools
• UI – Kubernetes, API
• Security (policies, encryption)
• Add / remove Kubernetes nodes
• Lifecycle management (OS updates,
Kubernetes upgrades)
• Monitoring (Prometheus)
• Logging (EFK)
• High Availability
Cisco Container Platform Feature Set
Kubernetes-as-a-Service
Setup ManageConsume
Cisco is the single point of contact for support !
10. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
20182017
Nov’17
(Demo Release)
August’18
(VMware on UCS)
Google Hybrid cloud
Jan’18
(Early Access)
May ’18
Baseline CCP v1.0
(HyperFlex 3.0)
20182018 2019
Cisco Container Platform Timeline*
*Roadmap dates subject to change
More releases /
Functionality
Nov’18
AWS EKS,
Load Balancing
Istio, Harbor, Node Pools
11. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Cisco Container Platform for HyperFlex
11BRKCLD-2676
IaaS
HyperFlex
Compute/Storage
HyperFlex
Network ACI
Nexus 9k standalone
On prem Kubernetes
Cisco Container Platform
Container Networking
Contiv / ACI CNI / Calico
Container Storage
HyperFlex Flex driver
Turnkey Kubernetes
• Simple & Seamless Day0 &
DayN K8S operations
integrated into HyperFlex
• HyperFlex IaaS
Enterprise Storage
• Scale-out, HA Filesystem
• Data protection, efficiency
and resiliency
Enterprise Networking
and Security
• Multi-tenant architecture,
Micro-segmentation,
Security policies
Common Platform for
Legacy and Modern Apps
• Co-existence of VMs and
containers on same
platform
DevOps Ready IT
• Enable developer agility
with IT & security policies
• Avoid Shadow IT
Turnkey Appliance for
Enterprise Kubernetes
Cisco Container Platform
Single Vendor Support
• Fully supported by Cisco
Global TAC
• Single throat to choke for
entire stack
13. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Cisco Container Platform Stack
13BRKCLD-2676
Control Plane Data Plane
VM VM VM
Control Plane Kubernetes
Automation
Orchestration
Operations
HX Connect
Cluster/
Machine
Controllers
VM VM VM
Cluster 1 Kubernetes
Cluster1
Workloads
Cluster1
Ops
Pod
Pod
Pod
VM VM VM
Cluster 2 Kubernetes
Cluster2
Workloads
Cluster2
Ops
Pod
Pod
Pod
Kubernetes Fluentd Prometheus Kibana Hyperflex Contiv
Storage (Hyperflex)
Networking (e.g. Nexus 9K or other)
Compute Hardware (UCS)
Hypervisor Layer (HyperFlex/VMW)
VM
14. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
• Releases v1.0-1.2
• May - June 2018
• Baseline On-premises Container platform
• API driven cluster life-cycle management
• Core platform foundation: immutable images,
disconnected installs, Kubeadm, Helm
• Hardware: Hyperflex only
• Kubernetes 1.10
• Networking: Calico, ACI CNI, Contiv*
• Initial or tech preview for add-on services (e.g.
EFK, L7 LB)
Cisco Container Platform release content
14Presentation ID
• Releases v1.4 – v2.2
• August – December 2018
• Hardware: Hyperflex or non-Hyperflex UCS, vSphere
storage, dynamic Flexvolume provisioning
• New features & services e.g Harbor registry, Node
Pools, Istio
• Multi-cloud GA: AWS EKS support with IAM and
cluster mgmt., Google Hybrid
• Kubernetes 1.11, web based installer etc
• Readiness of 1.0 baseline services (EFK, L3/ L4/ L7 LB)
15. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Competitive: Technical differentiators & Benefits
15Presentation ID
API driven cluster management (no Ansible, Puppet ..)
Multiple clusters, single management & control endpoint
100% upstream Kubernetes experience (no proprietary lock-in)
Unified full stack management of hosts/ node OS, Kubernetes & add-on services
Single point of support from Cisco (hardware, open source software, integrations, proprietary software options)
Multi-cloud and platform integrations (AWS, Google, vSphere, Bare Metal*, Openstack*)
Rich roadmap with value adds in networking, multi-cloud, AI/ ML, security, analytics
16. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Pre-requisites for v1.0 release
• HyperFlex 3.0.1b, 3.5
• VMware vCenter server 6.0, 6.5
• DRS and HA enabled
• Shared datastore
• ACI fabric (optional)
• DHCP for VMs
• IPs reserved for VIPs
Pre-requisites & Packaging
16BRKCLD-2676
Software Release Packaging
A CCP release currently consists of two artifacts:
• CCP Tenant Image OVA
• Supports both Ubuntu 16.04 and Ubuntu 18.04
•e.g. ccp-tenant-image-1.10.1-1.0.0.ova
• Control Plane Installer OVA
•e.g. kcp-vm-1.0.1.ova
These are available for download from cisco.com
Note: Disconnected deploys supported
17. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
CCP – CP1
K8S-Red
K8S-Blue
vCenter
PG10
PG20
PG30
100.1.1.0/28
100.1.2.0/24
100.1.3.0/24
HX vSphere Cluster
ASR1K or any L3 GW
Leaf e.g. N93xx
Spine e.g. N95xx 100.1.x.x
17BRKCLD-2676
DHCP server
18. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
IP subnets for cluster nodes
18BRKCLD-2676
• Flexible model: Can use shared port-groups or separate port-groups per tenant cluster
• DHCP pool: Used to allocate node/VM IPs
• VIP pool: Used to allocate Virtual IPs (for Kubernetes master IP & ingress load balancer VIP)
M
N“Tenant K8S”
Port Group 30
10.1.1.0/24
10.1.1.2-200 for VM IPs DHCP
10.1.1.201-254 for VIPs
DHCP
Server
pool
VIP pool
(managed by CCP)
20. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
CCP 2.0 (web based installer added)
Tenant cluster 1
Devops admin/ Dev
K8s api, RBAC
K8s data plane
Tenant cluster 2
Devops Admin/ Dev
K8s api, RBAC
K8s data plane
CCP Admin (IT Ops)
CCP api, RBAC
(Transient)
Installer VM
Full cluster & services
life-cycle mgmt
“Immutable” infra
Ubuntu
K8s
Add-ons
Ubuntu
K8s
Add-onsUbuntu
K8s
CCP app
CCP admin
Web based
Installer VM
21. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Increased agility for IT to support App/ Dev Teams
21BRKCLD-2676
(while retaining enterprise-wide consistency, hardware integrations, security, compliance …)
22. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Side note: “Immutable software” model
22BRKCLD-2676
• Sometimes also called “Golden image” model
• What does this mean ?
• Cisco does not officially support users installing software on these nodes
• e.g. “sudo apt-get install mysql-server” à Not supported by Cisco
• Currently these operations are not blocked to allow installing any urgent or
minor packages that may be needed say for troubleshooting
• If customer does install additional packages, Cisco support is “best effort” but
not guaranteed
23. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Interacting with Cisco Container Platform
23BRKCLD-2676
Kubernetes
Kubernetes Lifecycle
IT Admin
UI
+
API
CLI
+
UI
+
API
Monitoring / Logging
Storage / Network
Developers
KubernetesCisco CP
25. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Cisco Container Platform CNI Options
ACI CNI Contiv (Tech Preview) Calico
Network Policy • K8s network policy
• ACI policy (EPGs + Contracts)
for K8s network policy
• K8s network policy • K8s network policy
Underlay Network Integration • Underlay integration with ACI
fabric
• Policy extends beyond single
K8s cluster across VMs, Bare
Metal, Multi-clusters
Load Balancer Integration • Hardware L3 Load Balancer
integrated with ACI CNI to
provide optimal data path
• Software metalLB L3 Load
Balancer
• Contiv-metalLB optimization
(roadmap)
• Software metalLB L3 Load
Balancer
Istio Integration • Istio integration • Istio integration
• Contiv-Envoy data path
acceleration (roadmap)
• Istio Integration
26. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Single tenant: Calico/ Contiv VxLan mode on vSphere
30BRKCLD-2676
K8S master
nodes/ VMs 1..3
K8S compute
nodes/ VMs 1..M
VMWare VM Port group 100
Physical L3 gateways
Contiv
VXLAN overlays Non-contiv
VLAN traffic
K8S compute
nodes/ VMs 1..M
27. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Contiv CNI
31BRKCLD-2676
Optimized for Performance and Scale
Uses https://fd.io/technology/ Vector Path Processing (VPP)
CNI for Cisco Container Platform
CNI for Production Grade Container Environments
Supports Any Networking Underlay
For ACI fabric use ACI CNI
100% Open Source
https://github.com/contiv/vpp
K8s Container Network Interface (CNI)
Plugin for Network Connectivity and
Security
Easy installation | User space; No kernel tax | Provides container traffic operational visibility / monitoring /
debugging | World-class advisory and support
28. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Kubelet
CNI
CRI
tapv2
Contiv vswitch
Agent
Pod
Pod
Pod
VPP
…
K8s Master
IPv4/IPv6/SRv6 Network
• High performance user-space networking
• Agile feature development without dependency on Linux kernel
• Integration with Envoy side car for high performance service mesh (future)
• Data path optimizations for NFV
App
Kernel Host stack
Legacy Apps
Contiv
Netmaster
Contiv
Etcd
Kubelet
CNI
CRI
tapv2
Contiv vswitch
Agent
Pod
Pod
Pod
VPP
App
Kernel Host stack
High Performance
Apps
Pod
Pod
Pod
Istio Envoy App
VPP
TCP
Stack
Pod
Pod
Pod
High Performance
Apps
Istio EnvoyApp
VPP
TCP
Stack
memif
Legacy Apps
Pod
Pod
Pod
VNF
memif
Cloud-Native
VNFs
Pod
Pod
Pod
VNF
Cloud-Native
VNFs
K8s policy & state
distribution
Contiv CNI Architecture
32BRKCLD-2676
29. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA 33BRKCLD-2676
Contiv CNI node internal data path architecture
Pod1 Pod2 PodN
PodM
tap-1 tap-2 tap-n
tap-0 vpp1
lo0
Gige0/8/0
NIC NIC
BD1
BVI
VPP
enp0s9
Host stack
30. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
ACI Physical topology
Integrated k8s container networking + BM/ VM networking fabric
34BRKCLD-2676
Leaf: N9k
L2 from Contiv
OVS to fabric leaf
switch
via ethernet VPC/
link bond
Spine Layer: N9k
DC Core
ACI/ Nexus CLOS fabric
.…
Host-n
V M V MV M V M
.…
V M V MV M V M
.…
Host-2
V M V MV M V M
Contiv Host Plug-Ins
External IP
network
L3 out
Nx K8S tenant cluster nodes
31. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Technical Description
• Network policies of Kubernetes supported using standard
upstream format but enforced through OpFlex / OVS using APIC
Host Protection Profiles
• Kubernetes app configurations can be moved without
modification to/from ACI and non-ACI environments
• Embedded fabric and virtual switch load balancing
• PBR in fabric for external service load balancing
• OVS used for internal service load balancing
• VMM Domain for Kubernetes
• Statistics per namespace, deployment, service, pod
• Physical to container correlation
35BRKCLD-2676
ACI CNI Solution Overview
Node
OpFlex OVS
Kubernetes
ACI Policies
Network Policy
Node
OpFlex OVS
32. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Visibility: Live statistics in APIC per
container and health metrics
Hardware-accelerated:
Integrated load balancing
Enhanced Multitenancy and
unified networking for
containers, VMs, bare metal
Flexible policy: Native
platform policy API and
ACI policies
Fast, easy, secure
and scalable
networking for
your Application
Container Platform
Turnkey solution for
node and container
connectivity
Why Use ACI CNI
36BRKCLD-2676
33. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
2
3 Deploy and scale clusters
Build service definitions and
define network policy
ACI Fabric
Create Kubernetes system
resources in ACI
Fabric bring up
2
1
(Optional) Create EPGs and
contracts for use in Kubernetes
3
(Optional) Create EPGs and
contracts
4
Container Team Network Administrator
Node
OpFlex OVS
(Optional) Annotate
deployments to move
between EPGs
5
Monitor and observe network
telemetry
4
1 Install Kubernetes and ACI
plugin
Deploy and
scale clusters
ACI CNI Plugin for Kubernetes
37BRKCLD-2676
Native Security Policy Support
34. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Using Network Policy and EPGs
38BRKCLD-2676
Cluster Isolation Namespace Isolation Deployment Isolation
Single EPG for entire cluster.
(Default behavior)
No need for any internal contracts.
Each namespace is mapped to
its own EPG.
Contracts for inter-namespace traffic.
Each deployment mapped to an EPG
Contracts tightly control service traffic
EPG Network PolicyKey Map Contract
35. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Fabric Administrator has inventory of Kubernetes
objects – simplify operations
39BRKCLD-2676
APIC keeps inventory of pods and their
metadata (labels, annotations),
deployments, replicasets, etc.
View pods per node, map to
encapsulation, physical point
in the fabric.
Fabric admin can search APIC for k8s
nodes, masters, pods, services …
36. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
• In production environments certain services like
high performance databases will be running as
VMs or Bare Metal Servers
• This calls for the ability to easily provide
communication between Kubernetes PODs and
VMs/Bare Metal endpoints
• Simply deploy a contract between your EPGs, ACI
will do the rest!
• This works for any VMM domain and Physical
Domains, for example you can have a Container
Domain using VXLAN speaking with a Microsoft
SCVMM Domain using VLAN.
Container to Non-Container Communications
40BRKCLD-2676
38. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
• Integration with Kubernetes FlexVolume
Plugin framework
• Developed by HX team as part of HX 3.0
release
• New HX 3.5 (Dynamic Flexvolume)
• Enables developers to leverage HyperFlex
storage for state-full container storage
• HyperFlex Data Performance and Resiliency
• Note: BRKCLD-2016 "HyperFlex FlexVolume Driver
for Kubernetes Persistent Volumes” M.
Zimmerman
HyperFlex 3.0 & 3.5 FlexVolume Driver
42BRKCLD-2676
K8s Node VM
Kubelet
HX FlexVolume
Driver
SW iSCSI Initiator
private host-only vswitch
ESXi vmkernel interface
iSCSI
LUN
File
HX iSCSI Proxy
HX Controller VM
vswitch-hx-storage-data
NFS Datastore
HX ESXi Node
API
39. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
HyperFlex Storage for Kubernetes Node VMs
44BRKCLD-2676
DATASTORE
The “vmdk” blocks are synchronously replicated within the cluster based on the HyperFlex “Replication Factor”
RF3 = three copies of data (recommended)
Worker 1 VM
Based on cluster-wide Replication Factor
HYPERVISOR CONTROLLER
VM
IOVISOR
A
HYPERVISOR CONTROLLER
VM
IOVISOR
HYPERVISOR CONTROLLER
VM
IOVISOR
Master VM Worker 2 VM
BC
Kubernetes Cluster
VMDK FileVMDK File VMDK File
A B CB C A
C3 A2B2A3B2
B3C2 C3 A2
C2
B3
C2
A3B3
A2
B2
A3
C3
40. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
HyperFlex Persistent Storage for K8s Pods
45BRKCLD-2676
DATASTORE
If the “Worker 1 VM” node is moved to another physical host or if the Pod is restarted on a Kubernetes node
on a different physical host, the Pod retains access to the persistent volume
Worker 1 VM
Pod
HYPERVISOR CONTROLLER
VM
IOVISOR
Persistent
Volume
HYPERVISOR CONTROLLER
VM
IOVISOR
HYPERVISOR CONTROLLER
VM
IOVISOR
Master VM Worker 2 VM
PodPersistent
Volume
Pod Restarted Here
vMotion Node VM
or
A BC
B2C2
B3 A3
A2
C3
42. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Cisco Container Platform – Load Balancers
Features ACI CNI Contiv Calico
L7 Load Balancer • Nginx (Default)
• Istio (tech preview)
• Nginx (Default)
• Istio (tech preview)
• Nginx (Default)
• Istio (tech preview)
L3 / L4 Load Balancer • Hardware based implementation
on ACI (L3 / L4)
• Nginx ( L4 / SSL Termination)
• metalLB - (L3 / L4 )
• Nginx -- ( L4/ SSL Termination)
• metalLB (L3/L4)
• Nginx ( L4 / SSL
Termination)
Egress Traffic • Istio (tech preview) • Istio (tech preview) • Istio (tech preview)
(Tech Preview)
43. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Container Load Balancing As a Service
- K8s + NGINX Ingress Controller
http://guestbook.com https://cafe.test.com/tea
https://cafe.test.com/coffee
Container
networking
Guestbook app / service
Guest
Redis Master Redis Slave
Persistent storage
tea-svc
Tea Coffee CoffeeTea
coffee-svc
Virtual IP Address (VIP / External IP)
Guest Guest
Cisco Container Platform L7 Load Balancer
44. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Kubernetes Ingress with HA (metalLB)
Internal container SDN (Calico, ACI, Contiv)
External routable IPs
https://café.example.com/tea
VIP: 125.1.1.10
Ports: 80, 443, 8080
NGINX
NGINX
controller
Tea pod1
Coffee pod2
Coffee pod3
Tea pod2
Coffee pod1
Tea pod3
https://café.example.com/coffee
NGINX
NGINX
controller
metalLB metalLB
45. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Role Based Access Control
50BRKCLD-2676
46. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Monitoring with Prometheus / Grafana
51BRKCLD-2676
47. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Logging with EFK
52BRKCLD-2676
48. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
High Availability and Disaster Recovery
Control Plane
Machines
Four instances
configured in an anti-
affinity role. Failure
and restarting of
instances is done via
VMware DRS and
vMotion
Control Plane
Data
Persistent volumes
can be backed up,
and CCP instance can
be restored in case of
failure
Tenant Cluster
Nodes
Node failures are
monitored and
managed by CCP
control plane where
new nodes are
provisioned when
needed
50. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Hybrid complexity
90% 14%
Have taken steps
toward hybrid1
Have an optimized cloud
strategy2
1. Source: IDC CloudView, May, 2018, n=5,740 worldwide respondents, unweighted
2. Source: IDC CloudView, May, 2018, n=5,740 worldwide heavy cloud-using respondents, unweighted
51. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
On-premises
environment
Cisco Nexus9K / ACI
Cisco HyperFlex / UCS
VPC
EC2 / EBS
ElasticContainerRegistry
Identity and Access
Management (IAM)
Cisco CloudCenter
Stealthwatch Cloud
AppDynamics
Cisco Hybrid Solution for Kubernetes on AWS
Optional Mandatory
Cisco CSR1000v
Cisco Container
Platform
Amazon EKS
Legend:
Production-grade
consistent environment
52. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
On-premises
environment
Solution differentiation
First Hybrid Solution for Kubernetes on AWS
Consistent Identity and Authentication
Production Grade environment
Cisco Enterprise-class support
53. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Multi-cloud architecture: Cisco CP + AWS EKS
54. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
vSphere IAM Authentication
55. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
On-premises
environment
Google
Kubernetes Engine
Existing
Services
Apps | Data
Cisco Hybrid Cloud Platform for Google Cloud
Cisco HyperFlex
Cisco Nexus9K / ACI
Cisco CSR1000v
Cisco Stealthwatch Cloud
Cisco Container
Platform
Consistent Environment
Google Apigee
Cisco CloudCenter
Istio
BigQuery
Cloud SQL
Pub/Sub
Big Table
Cloud Storage
Cloud Spanner
Open Service Broker
56. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Open hybrid cloud solution use cases
61BRKCLD-2676
Developers Use the latest cloud services to differentiate their application
IT Admin Production-ready Kubernetes solution installed and maintained
Security Team Extend visibility, threat detection and control
An application running on premises
consumes leading edge cloud services
2
Developers Optimize my development lifecycle wherever it makes sense, not
location dependent
IT Admin Ensure services can reach other services between on-prem and
cloud
Security Team Insights into network traffic between on-prem and cloud
Seamless CICD workflow for containerized applications
across both cloud and on-premises3
Cloud application consumes data
from a legacy application running on-premises
Developers Legacy applications can participate in a cloud native architecture
IT Admin Support developer’s current and future container needs
Security Team Maintain and enhance control in containers, across multiple
environments
1
58. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Harbor Registry
63
59. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
CCP and Harbor Registry
64
• Harbor registry as one of core
components of CCP with limited feature
set (no notary, no image scanning yet)
• Two registry models – 1) Central
Registry 2) Environment / org specific
registries
• Dedicated registry cluster is
recommended with initial size of
registry volumes
• Harbor registry is lifecycle managed
during version upgrades
• Customers can use other registries as
well e.g. Docker Trusted Registry, Quay
etc.
60. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Not all workloads created equal!
65
CPU Intensive
• Financial
Modelling
work
• Apache Spark
• Encoders /
decoders
Memory
Intensive
• High paging
applications
• In-memory
databases
GPU Intensive
• 3D Rendering
applications
• AI / ML
Applications
with
Tensorflow
Kubernetes can manage different types of workloads through tag based node pools
61. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Node pools in CCP
66
• Machines sizes can be different
between pools (high CPU or high
memory)
• Individual pool can be separately
managed (change size, delete)
• Planning to add node pool for
Kubernetes masters with multi-master
support
• Planning to add GPU based node pool
support in future releases
62. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Web Installer
67
• VMware OVA based installation via
web installer
• Web installer allows validation of user
inputs and data population
• Ubuntu 18.04 based OS image
included as part of CCP
• Web installer takes about 20 minutes
(environment dependent) to install and
configure CCP
• CCP control plane is 4 VMs footprint
(2 vCPU, 8 Gb memory, 40GB disk per
VM)
64. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Upgrades, Updates and Scalability
Release Support Timeline
Within a major release family (e.g. 1.y.z) No change in major K8s versions supported N/A
Major Releases Add a new K8s version and
deprecate/remove the oldest K8 version
Quarterly
Minor Releases New features and fixes Monthly
Patch Releases Critical bug fixes only As Required
Cisco container platform can support Kubernetes clusters up to size of 256 nodes, and one
Cisco container platform can support up to 100 Kubernetes clusters
65. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Platform Components and Pre-requisites (v2.x)
Function Component Version
Container Runtime Docker CE 17.03.2
Operating System Ubuntu 16.04, 18.04
Orchestration Kubernetes 1.10.1, 1.11.3
IaaS (pre-req) vSphere 6.0 U3, 6.5
Infrastructure (pre-req) Hyperflex 3.0.1b+, 3.5.1a
CNI ACI, Calico 1.9r32, 3.1.3
SDN ACI 3.2(2o)
Container Storage Flex Driver 1.0
L7 Load Balancing Nginx (community) Ingress 0.24.0
Monitoring Prometheus, Grafana 2.3.1, 5.2.1
Logging EFK 6.4.2, 2.0.2, 6.4.2
L3 Load Balancing MetalLB 0.6.2
Service Mesh Istio / Envoy 1.0 / 1.6
Registry Harbor 1.6.0
66. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Platform Components Support (v2.x)
Supported Integrated Tech Preview
Docker CE (container runtime) HyperFlex Flex Driver Contiv
Kubernetes ACI CNI Istio
Kubernetes Host OS (Ubuntu)
Calico
Nginx / MetalLB
Prometheus / Grafana
EFK
Harbor registry
Supported: Solution Support via TAC and CCP team
Integrated: Partner component supported by partner or different Cisco product
Tech preview: Not supported by Cisco TAC or partner
67. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
In conclusion: IT (& Ops) gets to be Oprah
72BRKCLD-2676
My contact info:
Email: srampal@cisco.com
Twitter: @sr2357
69. Complete your online session evaluation
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Give us your feedback to be entered into a
Daily Survey Drawing.
Complete your session surveys through the
Cisco Live mobile app
Don’t forget: Cisco Live sessions will be available for viewing on
demand after the event at www.CiscoLive.com/Online.
BRKCLD-2676 74
70. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Demos in the
Cisco campus
Walk-in
self-paced labs
Meet the
engineer 1:1
meetings
Related
sessions
Continue
your
education
75BRKCLD-2676
71. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA 76BRKCLD-2676
Continue your education
Demos in the Cisco
campus
Walk-in
self-paced labs
Meet the engineer 1:1
meetings
Related
sessions
74. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Cloud Cisco education offerings
79BRKCLD-2676
Course Description Cisco Certification
Understanding Cloud Fundamentals (CLDFND)
Introducing Cloud Administration (CLDADM)
Learn how to perform foundational tasks related to
Cloud computing, and the essentials of Cloud
infrastructure, administration and operations
CCNA® Cloud
Implementing and Troubleshooting the Cisco Cloud Infrastructure (CLDINF)
Designing the Cisco Cloud (CLDDES)
Automating the Cisco Enterprise Cloud (CLDAUT)
Building the Cisco Cloud with Application Centric Infrastructure (CLDACI)
Obtain professional level skills to design, automate,
secure, provision and manage private and hybrid Clouds
CCNP®
Cloud
Product Training Portfolio:
CloudCenter: CLDCTR*
UCS Director: UCSDF, UCSDACI
Prime Service Catalog: PSCF, PSCI, PSCD
MetaPod: MPODF20
Gain in-depth hands-on skills using Cisco solutions to
configure, deploy, manage and troubleshoot Cloud
deployments
For more details, please visit: http://learningnetwork.cisco.com
Questions? Visit the Learning@Cisco Booth
*Available Q3FY18
75. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Data Center / Virtualization Cisco education offerings
80BRKCLD-2676
Course Description Cisco Certification
Introducing Cisco Data Center Networking (DCICN)
Introducing Cisco Data Center Technologies (DCICT)
Get job-ready foundational-level certification and skills
in installing, configuring, and maintaining next
generation data centers.
CCNA® Data Center
Implementing Cisco Data Center Unified Computing (DCUCI)
Implementing Cisco Data Center Infrastructure (DCII)
Implementing Cisco Data Center Virtualization and Automation (DCVAI)
Designing Cisco Data Center Infrastructure (DCID)
Troubleshooting Cisco Data Center Infrastructure (DCIT)
Obtain professional level skills to design, configure,
implement, troubleshoot next generation data center
infrastructure.
CCNP® Data Center
Product Training Portfolio:DCAC9K, DCINX9K, DCMDS, DCUCS, DCNX1K,
DCNX5K, DCNX7K, CACND, DSACI, HFLEX
UCSDF, UCSDACI, DCUCCEN
Gain hands-on skills using Cisco solutions to configure,
deploy, manage and troubleshoot unified computing,
policy-driven and virtualized data center infrastructure.
Designing the FlexPod® Solution (FPDESIGN)
Implementing and Administering the FlexPod®
Solution (FPIMPADM)
Learn how to design, implement and administer
FlexPod®
solutions
Cisco and NetApp Certified FlexPod®
Specialist
Designing the VersaStack Solution (VSDESIGN)
Implementing and Administering the VersaStack Solution (VSIMP)
Learn how to design, implement and administer
VersaStack solutions
For more details, please visit: http://learningnetwork.cisco.com
Questions? Visit the Learning@Cisco Booth
77. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Ubuntu
K8s
CP app
Ubuntu
K8s
Add-ons
End-user apps
Add-ons
CCP GUI
K8s
dashboard
Add-on
GUIs
e.g. Grafana
Add-on
GUIs
e.g. Grafana
CP REST api client
Kubectl client
Ssh to nodes
Ssh to nodes
Control
Plane cluster
Tenant
Plane cluster
Interacting with Control & Tenant Clusters
82BRKCLD-2676
78. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Cisco Container Platform Deployment through vCenter
83BRKCLD-2676
79. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
Kubernetes Tenant Cluster Creation Wizard
84BRKCLD-2676
80. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
M
N
Port Group 20
100.1.2.0/24
M + N M
N
Ctrl Plane “Tenant K8S” ‘Blue’ “Tenant K8S” ‘Red’
Port Group 10
100.1.1.0/26
L3 physical gateways
Port Group 30
100.1.3.0/24
Cisco Container Platform example with 2 tenant k8s
clusters
86BRKCLD-2676
100.1.0.0/16
81. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
GbE
TAP TAP
Client
10.1.1.3
Replica
10.1.1.4:8080
NAT plugin
REQ: LB+DNAT
RESP: SNAT
src: 10.1.1.3:40000
dst:
10.103.233.222:80
GbE
TAP TAP
Replica
10.1.1.6:8080
NAT plugin
REQ: FW
RESP: FW
Replica
10.1.1.5:8080
Service: 10.103.233.222:80
Contiv Contiv
src: 10.1.1.3:40000
dst: 10.1.1.5:8080
src:
10.103.233.222:80
dst: 10.1.1.3:40000
network
src: 10.1.1.3:40000
dst: 10.1.1.5:8080
src: 10.1.1.5:8080
dst: 10.1.1.3:40000
src:
10.103.233.222:80
dst: 10.1.1.3:40000
87BRKCLD-2676
Service load balancing data path
82. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA
GbE
TAP TAP
Replica
10.1.1.4:8080
NAT plugin
REQ: FW
RESP: FW
GbE
TAP TAP
Replica
10.1.1.6:8080
NAT plugin
Replica
10.1.1.5:8080
Service: 10.103.233.222:80
VPP VPP
src:
172.30.1.2:40000
dst: 10.1.1.4:8080
src: 10.1.1.4:8080
dst:
172.30.1.2:40000
src: 10.1.1.4:8080
dst:
172.30.1.2:40000
network
src:
172.30.1.2:40000
dst:
10.103.233.222:80
src:
172.30.1.2:40000
dst: 10.1.1.4:8080
Host
172.30.1.2
Kube-proxy
REQ: LB+DNAT
RESP: SNAT
TAP
src:
10.103.233.222:80
dst: 172.30.1.2:40000
88BRKCLD-2676
Host-service, LB to the same node
83. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CiscoLiveLA 89BRKCLD-2676
Contiv CNI internal addressing