SpringOne Platform 2017
Thomas Kraus, VMware; Merlin Glynn, VMware
Today's developer needs to rapidly build and deploy code in a consistent, predictable, and declarative manner. This session will illustrate how companies can leverage PKS, Kubernetes, Harbor, Clair, and Concourse to achieve these goals. The session will provide a solution overview for developing, building, and deploying applications using Container technologies from VMware and Pivotal. A brief review of each of the technologies being discussed will be provided. The session will include a proposed end to end solution leveraging all of these technologies to provide a better developer experience. The session will conclude with a demonstration illustrating a development workflow leveraging these technologies to initially develop and then update an Application running on PKS and Kubernetes.
2. • This presentation may contain product features that are currently under development.
• This overview of new technology represents no commitment from VMware to deliver these
features in any generally available product.
• Features are subject to change, and must not be included in contracts, purchase orders, or
sales agreements of any kind.
• Technical feasibility and market demand will affect final delivery.
• Pricing and packaging for any new technologies or features discussed or presented have not
been determined.
Disclaimer
2
6. PKS
BOSH
K8S-1
Work
er
Worker
K8S-2
BOSH
Agent
BOSH
Agent
K8s-api
Team A
K8s-api
KUBO
BOSH
Release
(tgz)
DAY 2 Ops
- Auto/Manual Rebuild
- Auto/Manual Repair
- Manual Scale
- Patch & Upgrade
- Control & Audit OPS Events
NAMESPACE_1: TEAM A
NAMESPACE_2: TEAM B
Team C
Team B
NAMESPACE_1: DEFAULT
DAY 1 Ops
DEPLOY
OperateK8s+RunApps/Containers
UI
&
API
Worker
Application
Dev/Ops Owner
Application
Dev/Ops Owner
Application
Dev/Ops Owner
Work
er
MASTER
WorkerMASTER
ETCD
WorkerMASTER
ETCD
MASTER
MASTER
ETCD
Platform
Reliability Engineer
Self Service K8s
PKS Technical Overview
1.7 -> 1.8
1.7 -> 1.8
PKS Controller
6
Ops
Manager
(OVA)
14. Reference
CONFIDENTIAL 14
• Hugo – website framework template and instructions
• https://github.com/tkrausjr/my-conference
• Example Conference Go Application
• https://github.com/tkrausjr/go-http
• Concourse Pipeline
• https://github.com/virtmerlin/spring1-goapp-ci
• Kubernetes Resource Type for Concourse
• https://github.com/virtmerlin/concourse-kubernetes-resource
15. Lessons Learned / Enhancements
CONFIDENTIAL 15
• Enhance Security - Notary Image Signing right now has signing keys in the pipeline
for updating an existing image which has initially been signed and pushed.
• Create a Harbor and Notary Resource Type for Concourse instead of using clairctl
and Docker Cli to increase reusability.
• Consider using separate Docker Images for Build and Run and add in artifact
repository like Nexus or Artifactory to store build artifacts. Right now the same Base
Docker Image is used for both the Application Build and the Application run via
Kubernetes.
• Concourse or any other CI / Automation tool is almost required for a solutions like this
and removes the human element and creates a version controlled and tracked
location for everything from your Application and
Goals of PKS
GOAL – Bring the release engineering, deployment, and lifecycle management qualities of BOSH to Kubernetes (K8s) clusters.
Provide Day 1 and Day 2 Deployment, and Operations of Enterprise Grade Prod ready K8s
Fully Supported Kubernetes
Better Operational experience running K8s in Production especially Day 2(upgrades, HA, deployment, Scaling, self healing, life-cycle management)
Deep Integration with NSX
Hardened, Production-grade
Runs on vSphere and VMC
HA, Security, Multi-tenancy, Tools
Bosh – Open Source software that delivers the capability to deploy complex distributed software (Day1) and keep it healthy (Day2)
1 – CPI Is how BOSH can deploy software across many clouds. PKS will support GCP & vSphere at GA
2 – BOSH Stemcells allow BOSH to abstract the OS image from the software being deployed. Stemcells for PKS 1.0 will be Ubuntu based VMDKs with a minimal OS and a BOSH agent
3 – BOSH Releases are collections of software and the metadata require to tell BOSH how to deploy that software. KUBO (Now Called CFCR) is literally a BOSH release to deploy Kubernetes
4 – BOSH Manifest is a set of YAML (which also uses another set of global YAML called a cloud-config, which is not pictured above). The Manifest will define which Stemcellls, releases, configuration options for the releases, and number of instances should be deployed. The manifest is then given to BOSH
5 – BOSH deployments is a given instantiation of a BOSH Deployment manifest
PKS
3 Key components of PKS
PKS Controller : The API that will allow self service
KUBO : CFCR BOSH Releases
BOSH
HARBOR
role-based access control w/ AD/LDAP integration
Security vulnerability scanning (Clair)
content trust - image signing
policy based image replication
audit and logs
open-source under Apache 2 license
This image is the first ‘introduction’ to to what PKS actually provides.
At it heart PKS provides a self service API for development teams to consume container services, while still allowing the PRE (or Operator) the ability to control the environment for the enterprise.
The animations will play thru to demonstrate key Day 1 & Day 2 capabilities;
3 Key components of PKS
PKS Controller : The API that will allow self service
KUBO : CFCR BOSH Releases
BOSH
*** Manual scale is intended to denote that BOSH scale ops of each K8s cluster is not automated. While various K8s mechanisms can be used to auto scale pod workloads, K8s itself is not auto scaled in 1.0
I have had a lot of experience w/ large scale deployment of Container Orchestration systems like Mesos or Kubernetes and on of the often overlooked aspects of these projects is the Container Registry and the Enterprise Control over both Container Images and SW artifacts used in the build process. Harbor addresses this problem.
Harbor – Open Source Docker v2 Registry fronted by a Reverse Proxy and automatically included Clair & Notary. *** Harbor provides
Authentication & Authorization with AD/LDAP integration
Security vulnerability scanning (Clair) and content trust - image signing
policy based image replication
audit and logs
Restful API
open-source under Apache 2 license
Clair CVE scanning. Clair is a French word which translates to clear, bright, transparent.
Clair API, Scanning Engine for scanning Docker Images Layer by Layer for CVE vulnerabilities. Returns a list of all files and packages in use by a particular LAYER and compares those files against the files in the CVE database that is stored in Postgres.
Clair API Allows for Creation, Put and Deletes of existing vulnerabilities BUT if you are using a FETCHER to obtain vulnerabilities (CVE) the vulnerabilities may be re-inserted when the next scheduled fetch occurs.
Image signing
Curated Images is kind of a big thing.
Picture this -
- Dev on laptop pulls git repo modifies a Dockerfile and copies binaries or packages (vulnerable) into the Docker container and builds an image.
- Dev has permissions to push to your registry REPO or REPO is PUBLIC.
- You now have vulnerable images in your registry which can be deployed 100's of times and exploited. UNLESS
1) You enable Notary (DTR) , so you cant pull or push without correctly signing the image first.
2) You si
From the Developer point of view:
I check my code in just like if I were pushing to PCF
But in addition to application artifacts, the pipeline is going to build an image for me …
In this visual we have a K8S cluster already running docker as the backend container engine, so our CI/CD pipeline will build a docker image for us and post it to a registry, in this case VMware Harbor
Afterwhich, the pipeline will instantiate a K8S deployment to run our docker image based application as a set of pods in a replica set in case a worker note goes offline.
The developer can than create a ‘service’ that gives worker nodes (or any external node) running the kube-proxy service the ability to route to where those pods are and access the apps/microservices running in them.
Ingress routing from external is similar to that of CF with an external DNS map being required to forward requests to 1 or more worker nodes running kube-proxy
One of the key differences is that Kubernetes isn’t opinionated on how the container image should be built, this give more flex to the developers but in some cases can make things more difficult for operators as we’ll see later on in the presentation
Agility is why developers want it
Harbor Core Features avail to PKS deployed clusters
*** Harbor will ship (as in will be packaged with and entitled with PKS)