Overview of kubernetes and its use as a DevOps cluster management framework. Problems with deployment via kube-up.sh and improving kubernetes on AWS via custom cloud formation template.

1. DevOps with Kubernetes

2. Agenda
• Kubernetes overview
• Usage and demo
• Architecture
• Kubernetes on AWS with Cloud Formation

3. Kubernetes Overview

4. What is Kubernetes?
Quick facts
• System for managing and orchestrating containerized applications in
clusters, a.k.a. cluster management software
• Open source, MIT licensed, developed by Google
• Used in GCE, OpenShift, other projects

5. Kubernetes is
• portable: public, private, hybrid, multi-cloud, written in Go
• extensible: modular, pluggable, hookable, composable
• self-healing: auto-placement, auto-restart, auto-replication, auto-scaling
• scalable and reliable: all components are scalable and clear setup path
exists to setup scalable and reliable cluster
• documented: a lot of documentation, training materials, community
support
• open source: MIT license, large and active community

6. With Kubernetes you can
• Orchestrate complex application deployments quickly and predictably
• Scale your applications on the fly
• Seamlessly roll out new features
• Easily setup complex operations scenarios, e.g. rolling update, canary
deployments etc
• Optimize use of your hardware by using only the resources you need
• Manage persistent storage
• Automate

7. Kubernetes solves
• application composition: co-
locating helper processes
preserving the “one-application-
per-container” model,
• mounting storage systems,
• distributing configuration and
secrets,
• application health checking,
• replicating application instances,
• horizontal (auto-)scaling,
• naming and discovery,
• load balancing,
• rolling updates,
• resource monitoring,
• log access and ingestion,
• support for introspection and
debugging, and
• identity and authorization.

8. Kubernetes Usage

9. Kubernetes management
• Kubectl CLI
• Independent binaries for multiple platforms (Go)
• put config file to $HOME/.kube or set $KUBECONFIG
• Automation friendly with multiple output formats: text, json, yaml, jsonpath
• Supports proxy into cluster network, container attachment and log retrieval
• REST API
• Available at https://<master-ip>
• Self-documented, swagger documentation
• Supports proxy into cluster network
• Basic Web dashboard
• Available at https://<master-ip>/ui
• Only some objects are displayed

10. Kubernetes objects
Primitives
• Namespace
• Node
• Pod
• Service
• Config Map
• Secret
• Volume
• Persistent Volume
Controllers
• Replication Controller
• Deployment
• Job
• Daemon Set
• Ingress
• ...

11. Kubernetes objects: common
• All objects include metadata with
• Name – unique
• Labels – searchable, selectable
• Annotations – arbitrary additional information
• Spec – object specific description/specification of the object
• Status – object status within the cluster
• Object information may be received in different formats

12. Kubernetes objects: names and namespaces
• Namespaces are used to separate groups of objects, e.g. by user, team,
project etc
• Namespaces are scopes for names; names are unique per type within
namespace
• Namespaces may also be a basis for access control separation
• Resource quotas may be associated with namespaces
kubectl get namespaces [ <ns> ... ]
kubectl describe namespaces [ <ns> ... ]

13. Kubernetes objects: nodes
• Nodes represent a physical or virtual worker machine where kubelet,
kube proxy, and docker run
• Kubelet registers a node on the master and maintains keep-alive check
• Nodes may be annotated and labeled to specify workload affinity and
constraints
kubectl get nodes [ <nd> ... ]
kubectl describe nodes [ <nd> ... ]

14. Kubernetes objects: pods
• Pod is a group of containers
• Run on the same node – co-located and co-scheduled
• Shared storage
• Shared localhost network and port space
• Unique IP within a cluster
• Example: app server and log shipper

15. Kubernetes objects: nodes and pods
Node 1 Node 2
Pod A-1
10.0.0.3
Cnt1
Cnt2
Pod A-2
10.0.0.5
Cnt1
Cnt2
Pod B-1
10.0.0.8
Cnt3

16. Kubernetes objects: volumes and persistent volumes
• Used to manage persistent storage
• Multiple types supported:
• AWS EBS
• Azure block store
• Git
• NFS
• GlusterFS
• Ceph
• ...

17. Kubernetes objects: pods and volumes
Pod
Container 1 Container 2
Persistent
Volume
Volume
Volume
Claim
Volume
Mount
Volume
Mount

18. Kubernetes objects: config maps and secrets
• Config maps and secrets are used for distribution of configuration
information including secrets like password, certificates, keys etc
• Kubelet registers a node on the master and maintains keep-alive check
• Nodes may be annotated and labeled to specify workload affinity and
constraints
kubectl get configmaps [ <cm> ... ]
kubectl describe configmaps [ <cm> ... ]
kubectl get secrets [ <sc> ... ]
kubectl describe secrets [ <sc> ... ]

19. Kubernetes objects: services
• Service is an abstraction that defines a set of pods a policy to access
them
• Service is a distributed L3 load balancer
• Single unique IP within a cluster
• Used to expose pods to the world:
• Default
• NodePort
• LoadBalancer

20. external
port
Kubernetes objects: pods and services abstraction
Cluster
Pod A-1
10.0.0.3
Pod A-2
10.0.0.5
Pod B-1
10.0.0.8
SrvA
10.7.0.1
SrvB
10.7.0.3

21. Kubernetes objects: pods and services
Node 1 Node 2
Pod A-1
10.0.0.3
Pod A-2
10.0.0.5
Pod B-1
10.0.0.8
SrvA
10.7.0.1
SrvB
10.7.0.3
SrvA
10.7.0.1
SrvB
10.7.0.3
external
port
external
port
iptables iptables

22. Service Discovery: DNS
DNS
• <service-name>.<namespace-name>.svc.cluster.local
• <service-name>.<namespace-name>
• <service-name> - in the same namespace
• DNS SRV _<port>._<proto>.<service-name> - for port number
e.g. “SRV _http._tcp.nginx”

23. Kubernetes Object: Controllers
• Deployment
• Daemon Set
• Job
• Ingress
• Replication Controller
• Replication Set

24. Kubernetes Controller: Job
• Create one or more pods and ensure that specified number of them
successfully terminates
• Jobs may be used for operations automation

25. Kubernetes Controller: Deployment
• Deployment provides declarative updates for Pods and Replica Sets
• Orchestrate updates and rollbacks
• Scale up or down

26. Kubernetes Addons
• DNS
• UI
• Logging
• Monitoring

27. Kubernetes Architecture

28. Kubernetes architecture

29. Kubernetes architecture: node
• kubelet manages pods, their
containers, images, volumes,
network etc
• kube-proxy is a simple network
proxy and load balancer
responsible for reflecting
services on the nodes

30. Kubernetes architecture: node
• kubelet manages pods, their
containers, images, volumes,
network etc
• kube-proxy is a simple network
proxy and load balancer
responsible for reflecting
services on the nodes.
Userspace (legacy) or iptables
(modern) modes are supported.

31. Kubernetes architecture: control plane
• etcd is a reliable watchable
storage for all persistent master
state
• API Server is a CRUD-y REST
server with most/all logic
implemented in plugins that
serves Kubernetes API.
It mainly processes REST
operations, validates them, and
updates the corresponding
objects in etcd.

32. Kubernetes architecture: control plane
• etcd is a reliable watchable
storage for all persistent master
state
• API Server is a CRUD-y REST
server with most/all logic
implemented in plugins that
serves Kubernetes API.
It mainly processes REST
operations, validates them, and
updates the corresponding
objects in etcd.

33. Kubernetes architecture: control plane
• Scheduler binds unscheduled
pods to nodes
• Control manager performs all
other cluster-level functions,
e.g. deployments rollout, job
control, pod replication control
etc

34. Kubernetes architecture: control plane
• Scheduler binds unscheduled
pods to nodes
• Control manager performs all
other cluster-level functions,
e.g. deployments rollout, job
control, pod replication control
etc

35. Kubernetes architecture: control plane
• Kubectl client is CLI to manage
K8S cluster

36. Kubernetes architecture: security
• Authentication and
authorization are pluggable. By
default – file based, but may be
easily switched to external
resources (OAuth, authorization
service)
• Transport security is based on
TLS, key distribution is
deployment specific

37. Kubernetes architecture: security
• Authentication and
authorization are pluggable: file
based by default, but may be
easily switched to external
resources (OAuth, authorization
service)
• Transport security is based on
TLS, key distribution is
deployment specific

38. Example orchestration scenario
1. User creates a new Deployment
object via REST
2. Controller Manager sees a
Deployment object with no Pods and
creates Pod objects based on the
Deployment object specification
3. Scheduler sees Pod objects not
assigned to Nodes and allocates
them according to the Nodes load
and the Pods specifications
4. Kubelets running on Nodes see Pod
objects allocated to their
corresponding Nodes and start Pods’
containers based on the Pods’
specifications
1
23
4
4

39. Kubernetes Deployment on AWS

40. Deployment options
• kube-up.sh script
Available in k8s distro and supported by the developers
• Other methods as described in kubernetes documentation
• Other projects and systems based on kubernetes, such as GCE
• EBT AWS CloudFormation template

41. AWS Cloud Formation K8S Cluster Improvements
• Master is in auto scaling group for auto recovery
• Nodes are in multi-zone auto scaling group for high availability
• Multiple auto scaling groups are supported for nodes
• Simple no-client cluster rollout and teardown
• Support for node EIP auto-assignment

42. Summary

43. The good, the bad, and the ugly
Pros
• Multi-platform
• Rich OTB abstractions and
functionality
• Extensibility
Cons and problems
• Complex architecture and setup
(AWS CF Template solves the
problem for AWS)
• Manifest parameterization is
outside K8S

44. Future work
• Simple deployment to other clouds
(Azure) and on-prem
• Multi-zone master
• Single-node deployment (reusable
master)
• Multi-region, multi-cloud and
federated deployment
• Persistent volume management and
backup in prod
• Monitoring and log collection in prod
• Integration with Jenkins
• ...
• Use Vault for key and secret
storage
• Packaged components: HAProxy
ingress, glusterfs, elasticsearch,
mongo DB, MySql Cluster(?),
Galena Cluster(?), WildFly,
ActiveMQ, RabbitMQ (?),
HippoCMS, Keycloak, OpenAM,
Hadoop (?), Rstudio Server,
Jupyter, etc
• Web UI
• ...

45. THANK YOU
Oleg Chunikhin
Chief Architect
ochunikhin@eastbanctech.com
202-295-3000
eastbanctech.com