3. Goal
• Be familiar with the basics of how to use Kubernetes
• Not in scope
• Advanced Usages of Kubernetes
• Administering Kubernetes
3
4. Structure
• Lecture followed by Watch/Follow
• I'll show it, and you can follow along
• Ask questions as we go along
• We'll take a poll with each section - if 50% are good continuing, we will continue
• Given time and size, unable to do individual attention
• But happy to follow up afterwards
4
5. Biases
• Focused on Linux based containers
• Expect some (though not much) familiarity with Docker
• Exercises written on MacOS with /bin/sh
• But should work with minor tweaks
5
11. Container Info Dump
• Lightweight isolation
• More like process than like VM but properties of both
• Same Kernel
• Separate File System (with some mappings)
• Average lifetime of minutes-days
• VMs: Average lifetime of hours-weeks
• Machines: Average lifetime of weeks-years
11
12. Container Info Dump
• Container : Process with a specific namespace/capability/cgroups configuration
• Namespaces: Create process visibility separation (e.g. what `ps -ef` show, `ls /` looks different)
• Capabilities: Allow/Disallow privileged functions (e.g. raw sockets for packet captures)
• CGroups: Allocate resources for processes (e.g. max cpu, memory, I/O)
• Container Image
• Packaging which becomes the basis for the root file system of a container
• Different ways to maintain it (tarball, layers, etc) depending on runtime which did packaging
12
Key
13. Container Info Dump
• Multiple Container Runtimes
• Docker : Most common
• Rocket
• CRI-O
• Focus on image management and running individual containers
• Get Image/Start/Stop
13
14. Packaging -> 12 Factor Model
• Containers can be as simple as a packaging format
• Or can follow https://12factor.net/
• Many usages follow the 12 Factor Model, but some support other apps
14
17. Interface
• API Driven (provided by Kubernetes' apiserver)
• Ubiquitous client - comment line `kubectl`
• Have multiple resources: pod, replicaset, deployment, job, volume, etc
• Defined by specification
• Put/Get specifications to/from API server
17
18. Desired State
• Focus on the *what* (specification passed to the API server) not the *how*
• "I want 5 small compute units doing my web app front end, and 2 large compute units doing my web
app database"
• Orchestrator takes definition of "what" and figures out "how" to reconcile it
18
19. Controllers
• Automation Providers
• Examine the current state of the cluster (via the API server), what the desired state says it should be
and takes the necessary steps
• All kinds of controllers:
• Scheduler, Manager
• Cloud Provider(s)
• Ingress
19
21. Processes
• State Storage (etcd) - 1 cluster : Live DB of cluster
• API Server - enough to cover load : How everything interacts
• Scheduler - 3 for redundancy, 1 active : Decides what should be running where
• Controller Manager - 3 for redundancy, 1 active : Spawns internal controllers which handle all heavy
lifting functions
• kubelet - 1 per workload node : Runs wherever a workload runs.
• Kube Proxy - 1 per workload node : Handles traffic forwarding into/out of cluster and to known
endpoints
• Container Networking - TBD : Handles routing of workloads to each other
21
22. Master vs Minion
• Master : Core of Control Plane runs here
• Minion : Where the actual workloads run
• kubelet and kube-proxy run on both (depending on control plane installation)
• Lots of variations
• Can have internal/external etcd
• Can run control plane in the cluster even
22
23. Add ons
• Pluggable cluster features
• network overlay : Provides host<->host connectivity which makes the cluster network look connected
• kube-dns : Provides a DNS domain which provides naming for cluster resources
• das
23
26. Pod
• Basic unit of compute
• The "What" of the workload
• Meant for portions of a workload that are tightly coupled
• 1 or more containers scheduled together
• Typically expected to run indefinitely
• Containers sharing a network namespace
• I.e. "host" from a network perspective
26
27. Configuration Resources
• Under the 12 Factor model, containers end up being immutable, so you need to be able to get
configuration parameters in.
• ConfigMap : key/value collections which can be made available to Containers in Pods
• Secrets : key/value collections which can be made available to Containers in Pods but should be
handled carefully
27
28. Pod Collections
• ReplicaSet : Multiple identically configured pods (differ by IP)
• Deployment : Mutable collection of Pods performing a workload
• Job : Pods meant to run ones
• CronJob : Pods meant to run on a regular but not constant basis
• StatefulSet : Pods meant to have some consistency over their lifetimes (name, Pod IP, etc)
• DaemonSet : Pods meant to running on all nodes (or subset of nodes with a label)
28
Two Most Common
29. Service
• Abstraction of how to get to a workload
• The "Way" to the workload
• Uses labels to decide membership (which Pods are "behind" the Service)
• For Kubernetes base, this mapping is a L4 load balancer
• But add-ons provide this via DNS and other methods
29
30. (Slight aside) Labels
• Not a resource, but is metadata on resources
• Every resource can have labels added to it
• Labels: key/value pairs that tag resource
• Used to select subsets of those resources
• "Gimme all Pods with the label `app=web`
• Services say "connect this frontend with all pods with the label `app=web`"
• Metadata also includes name, annotations (structured data), timestamps, status, etc
30
33. Unified Network Space
• All Pods are reachable from all other Pods from an IP perspective
• Policy may not allow, but common, non-conflicting, and known IP space
• No need to port map ports for Pod<->Pod inside of the cluster
• May need to map Outside->Pod
• Need to map from Container->Pod (container runtime behavior)
• Typically
• Implemented as an overlay but can be done with direct routing
• Must NAT to outside cluster (IPv6 is coming not full support yet)
33
34. IPs
• Cluster IP Space : Used to provide Unified Network Space. Pool of Pod IPs
• Service IP Space : Used to provide area to map services into
• Node IPs : The IPs assigned to master/minions
• Typically used when referring to getting into/out of the cluster
34
35. IPTables
• Any mapping in the cluster is done via IPTables
• Handles Service IP to backend Pod IPs
• Handles inside->outside IP NAT
• Handles outside->inside IP NAT
• Controlled by kube-proxy (not really a proxy anymore)
35
37. Common things to remember
• Pods : multiple containers working closely together
• Pod IP : Ultimately how traffic gets into a workload. One CIDR over a cluster.
• IPTables : How cube proxy maps everything into/out of the cluster (and how it maps services)
• Controllers : Entities that do a piece of automation
• Labels and Selectors : Ways to classify resources
• IPTables, IPTables, IPTables
37
40. EX00: Starting minikube
$ minikube start
Starting local Kubernetes v1.7.0 cluster...
Starting VM...
Downloading Minikube ISO
97.80 MB / 97.80 MB [=====================================] 100.00% 0s
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Starting cluster components...
Connecting to cluster...
Setting up kubeconfig...
Kubectl is now configured to use the cluster.
40
41. EX00: Starting minikube
$ minikube status
minikube: Running
localkube: Running
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.100
$
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.5",
GitCommit:"17d7182a7ccbb167074be7a87f0a68bd00d58d97", GitTreeState:"clean",
BuildDate:"2017-08-31T09:14:02Z", GoVersion:"go1.8.3", Compiler:"gc",
Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.0",
GitCommit:"d3ada0119e776222f11ec7945e6d860061339aad", GitTreeState:"clean",
BuildDate:"2017-07-26T00:12:31Z", GoVersion:"go1.8.3", Compiler:"gc",
Platform:"linux/amd64"}
41
Is everything running ok?
42. EX00: Starting minikube
$ kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health": "true"}
$
$
$ kubectl cluster-info
Kubernetes master is running at https://192.168.99.100:8443
To further debug and diagnose cluster problems, use 'kubectl cluster-info
dump'.
42
43. EX01: helloshell
• Purpose
• Show how to run a simple command
• What's going on when you run the command
• Demonstrate pod, replicaset, deployment
43
44. EX01: helloshell
$ kubectl run -it --image=busybox bb1 /bin/sh
If you don't see a command prompt, try pressing enter.
/ # ps
PID USER TIME COMMAND
1 root 0:00 /bin/sh
7 root 0:00 ps
/ #
44
Is everything running ok?
45. EX01: helloshell
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
bb1-1176220718-z09mj 1/1 Running 1 46s
$
$
$
$
$ kubectl get rs
NAME DESIRED CURRENT READY AGE
bb1-1176220718 1 1 1 19s
$
$
$ kubectl get deploy
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
bb1 1 1 1 1 1m
45
What's this actually running?
*-##########-XXXXX format tends
to look like a deployment/replicaset
Why 1?
46. EX01: helloshell
$ kubectl attach bb1-1176220718-z09mj -c bb1 -i -t
If you don't see a command prompt, try pressing enter.
/ # Session ended, resume using 'kubectl attach bb1-1176220718-z09mj -c bb1
-i -t' command when the pod is running
$
$
$
$ kubectl attach bb1-1176220718-z09mj -c bb1 -i -t
If you don't see a command prompt, try pressing enter.
error: unable to
upgrade connection: container bb1 not found in pod bb1-1176220718-
z09mj_default
$
$ kubectl get pod bb1-1176220718-z09mj
NAME READY STATUS RESTARTS AGE
bb1-1176220718-z09mj 0/1 Completed 2 10m
46
Let's try exiting and entering quickly
What's this error?
What does it mean when the pod is running?
0/1 means it's in the middle of (re-)starting
47. EX01: helloshell
$ kubectl get pod bb1-1176220718-z09mj -o yaml
apiVersion: v1
kind: Pod
...
spec:
containers:
...
restartPolicy: Always
47
Pod description says it's going to try to restart
49. EX02: Official Introduction
• Purpose
• Connect docker and kubernetes
• Build artifacts that can go into kubernetes
• Reinforce pod, replicaset, deployment
• Demonstrate services
• From: https://kubernetes.io/docs/tutorials/stateless-application/hello-minikube/
49
50. EX02: Official Introduction
$ eval $(minikube docker-env)
$ docker ps
CONTAINER ID IMAGE COMMAND
6d5fac7595bd gcr.io/google_containers/pause-amd64:3.0 "/pause"
...
50
Connect to minikube's docker daemon
Now we can interact with it just as if it was a local docker daemon
51. EX02: Official Introduction
$ cd ex02
$ ls
Dockerfile server.js
$
$ docker build -t intro:0.0.1 .
Sending build context to Docker daemon 3.072kB
Step 1 : FROM node:6.9.2
6.9.2: Pulling from library/node
75a822cd7888: Pull complete
57de64c72267: Pull complete
...
Step 4 : CMD node server.js
---> Running in 22b57d427b1c
---> 280abd363feb
Removing intermediate container 22b57d427b1c
Successfully built 280abd363feb
51
Build a container image from the official example
52. EX02: Official Introduction
$ kubectl run intro --image=intro:0.0.1 --port=8080
$
$ kubectl get deployments
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
intro 1 1 1 1 7s
$
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
intro-1197849725-75kk9 1/1 Running 0 10s
$
52
Start the image
So, it's running - now what?
53. EX02: Official Introduction
$ kubectl expose deploy/intro --type=NodePort
service "intro" exposed
$
$
$ kubectl get svc
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
intro 10.0.0.238 <nodes> 8080:30561/TCP 6s
kubernetes 10.0.0.1 <none> 443/TCP 9h
$
$
$ minikube service intro
Opening kubernetes service default/intro in default browser...
$
$
$ minikube service intro --url
http://192.168.99.100:30561
53
Make it available over the network
Tells to map it to a port on all (one) of the nodes
Automatically opens browser
Or get the URL yourself
We're going to leave this running, for the next exercise...
54. EX03: The Dashboard Add-on
• Purpose
• Demonstrate the dashboard add-on
• Demonstrate minikube dashboard shortcuts
54
55. EX03: The Dashboard Add-on
$ minikube dashboard
Opening kubernetes dashboard in default browser...
$
$
$ minikube dashboard --url
http://192.168.99.100:30000
$
55
Start up the dashboard
Automatically opens browser
As before, can get it yourself
But before, we did `minikube service $NAME`...
56. EX03: The Dashboard Add-on
$ kubectl get service -n kube-system
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns 10.0.0.10 <none> 53/UDP,53/TCP 9h
kubernetes-dashboard 10.0.0.114 <nodes> 80:30000/TCP 9h
$
$
$ minikube service -n kube-system kubernetes-dashboard
Opening kubernetes service kube-system/kubernetes-dashboard in default
browser...
$ minikube service -n kube-system kubernetes-dashboard --url
http://192.168.99.100:30000
56
There is a `kubernetes-dashboard` service running; it's just in the kibe-system namespace (we'll come back to that...)
`minikube dashboard` is a shortcut for `minikube service...` for the dashboard
58. EX04: Add-ons
• Purpose
• Explore the add-ons and add-ons manager
• Explore the kibe-system namespace
58
59. EX04: Add-ons
$ kubectl get -n kube-system pods
NAME READY STATUS RESTARTS AGE
kube-addon-manager-minikube 1/1 Running 0 21h
kube-dns-910330662-rnwgp 3/3 Running 0 21h
kubernetes-dashboard-tlh94 1/1 Running 0 21h
$
$
$ minikube addons list
- ingress: disabled
- dashboard: enabled
- heapster: disabled
- kube-dns: enabled
- registry: disabled
- registry-creds: disabled
- addon-manager: enabled
- default-storageclass: enabled
59
Already saw kibe-system services - what about pods?
This are background processes which are managed by the qinikube add-ons manager
60. EX04: Add-ons
$ minikube addons enable heapster
heapster was successfully enabled
$
$
$ kubectl -n kube-system get pods
NAME READY STATUS RESTARTS AGE
heapster-t00zx 1/1 Running 0 2s
influxdb-grafana-ll71w 2/2 Running 0 2s
kube-addon-manager-minikube 1/1 Running 0 22h
kube-dns-910330662-rnwgp 3/3 Running 0 22h
kubernetes-dashboard-tlh94 1/1 Running 0 22h
60
Let's turn on something else
Check on the pods again
61. EX04: Add-ons
• addon-manager : Controller which provides these add-ons
• dashboard : Web interface for cluster information and status
• kube-dns : Provides cluster DNS mapping (we'll come back to this)
• heapster : Gathers container and node statistics
• registry : Can run a container image registry
• default-storageclass : Provides a simply host path persistent volume
• ingress : Provides a Layer 7 load balancer as Kubernetes primitive
• registry-creds : Simplified way to provide container registry user/password for image pulls
61
63. EX05: Working with pods
• Purpose
• Explore multiple ways of seeing pod information
• Explore the pod spec
63
64. EX05: Working with pods
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
intro-1197849725-g22tx 1/1 Running 0 15m
$
$
$ kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE
intro-1197849725-g22tx 1/1 Running 0 15m 172.17.0.4 minikube
$
$
$ kubectl get pods -o name
pods/intro-1197849725-g22tx
$
64
Get pods - normal
Get pods - show Pod IP and Node
Get pods - show just name (good for shell loops)
65. EX05: Working with pods
$ kubectl describe pod/intro-1197849725-g22tx
Name: intro-1197849725-g22tx
Namespace: default
Node: minikube/192.168.99.100
Start Time: Sat, 09 Sep 2017 13:26:55 -0700
Labels: pod-template-hash=3978227742
run=intro
Annotations: ...
Status: Running
IP: 172.17.0.10
Created By: ReplicaSet/intro-1197849725
Controlled By: ReplicaSet/intro-1197849725
Containers:
intro:
Container ID: docker://...
Image: intro:0.0.2
Image ID: docker://sha256:...
Port: 8080/TCP
State: Running
...
65
This is `describe pod`. It gives you some
human readable information about the pod.
66. EX05: Working with pods
$ kubectl get pods/intro-1197849725-g22tx -o yaml
apiVersion: v1
kind: Pod
metadata:
name: intro-1197849725-g22tx
namespace: default
...
spec:
containers:
- image: intro:0.0.1
...
status:
hostIP: 192.168.99.100
podIP: 172.17.0.4
...
66
This is what a pod spec looks like.
This can be used for specific search/display
or to configure the system.
67. EX05: Working with pods
$ kubectl get pod -o=custom-columns=NAME:.metadata.name,IP:.status.podIP
NAME IP
intro-1197849725-g22tx 172.17.0.4
67
Show just the name and podIP
68. EX05: Working with pods
$ cat redis.yaml
apiVersion: v1
kind: Pod
metadata:
name: redis-manual
spec:
containers:
- image: redis:4.0.1
name: redis
$
$ kubectl apply -f redis.yaml
pod "redis-manual" created
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
redis-manual 1/1 Running 0 2s
...
68
Write a spec file manually
Apply it to the cluster
69. EX05: Working with pods
$ kubectl apply -f redis.yaml
pod "redis-manual" created
$
$
$
$
$ kubectl delete -f redis.yaml
pod "redis-manual" deleted
$
$
$
$
$
$ kubectl apply -f redis.yaml
pod "redis-manual" created
$
69
$ kubectl get pods -w | grep redis
redis-manual 0/1 Pending 0 0s
redis-manual 0/1 Pending 0 0s
redis-manual 0/1 ContainerCreating 0 0s
redis-manual 1/1 Running 0 1s
redis-manual 1/1 Terminating 0 7s
redis-manual 0/1 Terminating 0 8s
redis-manual 0/1 Terminating 0 9s
redis-manual 0/1 Terminating 0 9s
redis-manual 0/1 Pending 0 15s
redis-manual 0/1 Pending 0 15s
redis-manual 0/1 ContainerCreating 0 15s
redis-manual 1/1 Running 0 16s
We can also -w(atch) the pod changes
70. EX05: Working with pods
$ kubectl get events | grep redis-manual
3m 3m 1 redis-manual Pod Normal
Scheduled default-scheduler Successfully assigned redis-manual to minikube
3m 3m 1 redis-manual Pod Normal
SuccessfulMountVolume kubelet, minikube MountVolume.SetUp succeeded for volume "default-
token-3j744"
3m 3m 1 redis-manual Pod spec.containers{redis} Normal
Pulled kubelet, minikube Container image "redis:4.0.1" already present on
machine
3m 3m 1 redis-manual Pod spec.containers{redis} Normal
Created kubelet, minikube Created container
3m 3m 1 redis-manual Pod spec.containers{redis} Normal
Started kubelet, minikube Started container
3m 3m 1 redis-manual Pod spec.containers{redis} Normal
Killing kubelet, minikube Killing container with id docker://redis:Need to kill
Pod
...
70
Can see the same a -w(atch) and more in the events
71. EX06: Working in a container
• Purpose
• Explore starting points for debugging
• Explore how to get logs
• Explore how to get inside a container
71
72. EX06: Working in a container
$ kubectl logs redis-manual
...:32.344 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
...:32.344 # Redis version=4.0.1, bits=64, commit=00000000, modified=0,
pid=1, just started
...:32.344 # Warning: no config file specified, using the default config. In
order to specify a config file use redis-server /path/to/redis.conf
...:32.345 * Running mode=standalone, port=6379.
...:32.345 # WARNING: The TCP backlog setting of 511 cannot be enforced
because /proc/sys/net/core/somaxconn is set to the lower value of 128.
...:32.345 # Server initialized
...:32.345 * Ready to accept connections
72
Getting "logs" == stdout and stderr
Convention of 12 Factor logging approach.
https://12factor.net/logs
73. EX06: Working in a container
$ kubectl exec -it redis-manual /usr/local/bin/redis-cli
127.0.0.1:6379> set foo 10
OK
127.0.0.1:6379> get foo
"10"
127.0.0.1:6379>
$
$
$ kubectl exec -it redis-manual /usr/local/bin/redis-cli
127.0.0.1:6379> get foo
"10"
73
How to work on the redis container?
It persists across invocations of the client command
74. EX06: Working in a container
$ kubectl delete pod/redis-manual
pod "redis-manual" deleted
$ kubectl apply -f ./redis.yaml
pod "redis-manual" created
$ kubectl exec -it redis-manual /usr/local/bin/redis-cli
127.0.0.1:6379> get foo
(nil)
74
But does not persist across invocations of the pod itself
75. EX07: Deployment replicas
• Purpose:
• Explore deployment keeping replicas running
• Explore adding/removing replicas from a deployment
75
76. EX07: Deployment replicas
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
intro-1197849725-g22tx 1/1 Running 0 30m
redis-manual 1/1 Running 0 10m
$
$
$ kubectl get pods -o name | xargs kubectl delete
pod "intro-1197849725-g22tx" deleted
pod "redis-manual" deleted
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
intro-1197849725-g22tx 1/1 Terminating 0 30m
intro-1197849725-v78kg 1/1 Running 0 17s
76
Let's do a little cleanup
A new intro pod is already around
77. EX07: Deployment replicas
$ kubectl get deploy intro
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
intro 1 1 1 1 30m
$
$
$
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
intro-1197849725-v78kg 1/1 Terminating 0 2m
$
77
Deployment tries to keep CURRENT equal to DESIRED
Delete the deployment to make the pod go away
78. EX07: Deployment replicas
$ kubectl run intro --image=intro:0.0.1 --port=8080 --replicas=3
deployment "intro" created
$ kubectl get deploy
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
intro 3 3 3 3 13s
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
intro-1197849725-bl6qs 1/1 Running 0 17s
intro-1197849725-gdr6f 1/1 Running 0 17s
intro-1197849725-qm7zd 1/1 Running 0 17s
78
Let's start it with more instances
replicas == pod count
79. EX07: Deployment replicas
$ kubectl delete po/intro-1197849725-qm7zd
pod "intro-1197849725-qm7zd" deleted
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
intro-1197849725-bl6qs 1/1 Running 0 1m
intro-1197849725-gdr6f 1/1 Running 0 1m
intro-1197849725-l50k3 1/1 Running 0 3s
intro-1197849725-qm7zd 1/1 Terminating 0 1m
$ kubectl get deploy intro
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
intro 3 3 3 3 1m
79
Delete a pod again
The deployment will do what it needs to to get the count back to 3
80. EX07: Deployment replicas
$ kubectl scale deploy/intro --replicas=1
deployment "intro" scaled
$ kubectl get deploy/intro
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
intro 1 1 1 1 2m
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
intro-1197849725-bl6qs 1/1 Terminating 0 3m
intro-1197849725-gdr6f 1/1 Running 0 3m
intro-1197849725-l50k3 1/1 Terminating 0 1m
$
$
80
Scale it down to 1
83. EX08: Deployment updates
$ grep Hello server.js
response.end('Hello Kubernetes Tutorial!');
$ docker build -t intro:0.0.2 .
Sending build context to Docker daemon 3.072kB
Step 1 : FROM node:6.9.2
---> faaadb4aaf9b
Step 2 : EXPOSE 8080
---> Using cache
---> 20e3088f6122
Step 3 : COPY server.js .
---> 9aa0164faa7e
Removing intermediate container abe7416c8707
Step 4 : CMD node server.js
---> Running in 4d672a4e6fac
---> 1df0203ce037
Removing intermediate container 4d672a4e6fac
Successfully built 1df0203ce037
83
First, we need a new image to update to.
84. EX08: Deployment updates
$ kubectl scale deploy/intro --replicas=3
deployment "intro" scaled
$
$ # kubectl get pods -w ### watch the deployment as it happens
$
$
$ kubectl set image deploy/intro intro=intro:0.0.2
deployment "intro" image updated
$ kubectl rollout status deploy/intro
deployment "intro" successfully rolled out
$
$ minikube service intro
84
Next, let's make sure we have some additional copies for resilience
86. EX08: Deployment updates
intro-3978227742-hllw8 0/1 Pending 0 0s
intro-1197849725-j3l1c 1/1 Terminating 0 1m
intro-3978227742-hllw8 0/1 Pending 0 0s
intro-3978227742-hllw8 0/1 ContainerCreating 0 0s
intro-3978227742-6kb68 0/1 Pending 0 0s
intro-3978227742-6kb68 0/1 Pending 0 0s
intro-3978227742-6kb68 0/1 ContainerCreating 0 0s
intro-3978227742-hllw8 1/1 Running 0 0s
intro-1197849725-lt7fs 1/1 Terminating 0 1m
intro-3978227742-5d5w9 0/1 Pending 0 0s
intro-3978227742-5d5w9 0/1 Pending 0 0s
intro-3978227742-5d5w9 0/1 ContainerCreating 0 0s
intro-3978227742-5d5w9 1/1 Running 0 1s
intro-1197849725-2nzz4 1/1 Terminating 0 2m
intro-3978227742-6kb68 1/1 Running 0 1s
86
Our deployment strategy (default rollingUpdate) will create new Pods before deleting the old ones,
and it will roll over some of the pods "slowly". (In this exercise, the pods come up too quickly so not much waiting.)
87. EX09 Pod information inside
• Purpose
• Show how to expose information to pod
• Explore the deployment specification
• Explore the `edit` command
87
88. EX09 Pod information inside
88
Since I have 3 pods, how do I know which one I'm hitting?
Let's add the pod IP to our response.
89. EX09 Pod information inside
$ grep Hello server.js
response.end('Hello Kubernetes Tutorial from ' + process.env.PODIP + '!n');
$ docker build -t intro:0.0.3 .
Sending build context to Docker daemon 3.072kB
Step 1 : FROM node:6.9.2
---> faaadb4aaf9b
Step 2 : EXPOSE 8080
---> Using cache
---> 20e3088f6122
Step 3 : COPY server.js .
---> 83e6090ec153
Removing intermediate container 76bf52dc48dc
Step 4 : CMD node server.js
---> Running in c08880cc596d
---> e2c588c47a0a
Removing intermediate container c08880cc596d
Successfully built e2c588c47a0a
$
$ kubectl set image deploy/intro intro=intro:0.0.3
deployment "intro" image updated
89
Start by adding a new image (0.0.3) which pulls an environment variable called PODIP
And roll this out
90. EX09 Pod information inside
90
It's updated, but we haven't defined the environment variable in it yet.
91. EX09 Pod information inside
$ kubectl get deploy/intro -o yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: intro
...
spec:
replicas: 3
strategy:
rollingUpdate:
...
template:
spec:
containers:
- image: intro:0.0.3
name: intro
ports:
- containerPort: 8080
protocol: TCP
...
91
Deployments have specs just like pods do
The pod spec is nested inside of the deployment spec
92. EX09 Pod information inside
$ kubectl edit deploy/intro
spec:
...
template:
spec:
containers:
- image: intro:0.0.3
name: intro
env:
- name: PODIP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
...
92
Opens up the EDITOR
Once it's written and EDITOR is exited, it'll save and cycle the pods
93. EX09 Pod information inside
...
deployment "intro" edited
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
intro-1725170555-306cj 1/1 Terminating 0 28m
intro-1725170555-pw8zd 1/1 Terminating 0 28m
intro-1725170555-qbctx 1/1 Terminating 0 28m
intro-3265745252-07l0b 1/1 Running 0 3s
intro-3265745252-0qw03 1/1 Running 0 4s
intro-3265745252-sxz2s 1/1 Running 0 4s
$
$ minikube service intro
...
93
The pods cycle...
95. EX09 Pod information inside
$ kubectl get pod -o=custom-columns=NAME:.metadata.name,IP:.status.podIP
NAME IP
intro-3265745252-07l0b 172.17.0.11
intro-3265745252-0qw03 172.17.0.9
intro-3265745252-sxz2s 172.17.0.10
95
And confirm the Pod IPs
96. EX09 Pod information inside
$ curl http://192.168.99.100:30561/
Hello Kubernetes Tutorial from 172.17.0.11!
$ curl http://192.168.99.100:30561/
Hello Kubernetes Tutorial from 172.17.0.9!
$ curl http://192.168.99.100:30561/
Hello Kubernetes Tutorial from 172.17.0.10!
$ curl http://192.168.99.100:30561/
Hello Kubernetes Tutorial from 172.17.0.9!
$ curl http://192.168.99.100:30561/
Hello Kubernetes Tutorial from 172.17.0.10!
$ curl http://192.168.99.100:30561/
Hello Kubernetes Tutorial from 172.17.0.11!
96
Run it from the command line since browsers will pool the connection...
98. EX10 Configmaps
$ grep -A 2 Hello server.js
response.end('Hello Kubernetes Tutorial from ' + process.env.PODIP + '!n' +
'The configuration variable is ' + process.env.CONFIGVAR + 'n'
);
$ docker build -t intro:0.0.4 .
Sending build context to Docker daemon 3.072kB
Step 1 : FROM node:6.9.2
---> faaadb4aaf9b
Step 2 : EXPOSE 8080
---> Using cache
---> 20e3088f6122
Step 3 : COPY server.js .
---> 5f8dad93c9b3
Removing intermediate container 29b6ad3411b5
Step 4 : CMD node server.js
---> Running in c6911d08376d
---> f72a2166a111
Removing intermediate container c6911d08376d
Successfully built f72a2166a111
98
Update our server to output something with more environment variables in it
99. EX10 Configmaps
$ kubectl set image deploy/intro intro=intro:0.0.4
deployment "intro" image updated
$
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
intro-3265745252-0qw03 1/1 Terminating 0 24m
intro-3265745252-7wg78 1/1 Terminating 0 29s
intro-3265745252-sxz2s 1/1 Terminating 0 24m
intro-4010436465-3sbgd 1/1 Running 0 4s
intro-4010436465-lr2qh 1/1 Running 0 2s
intro-4010436465-nchsj 1/1 Running 0 4s
$
$ curl http://192.168.99.100:30561/
Hello Kubernetes Tutorial from 172.17.0.8!
The configuration variable is undefined
99
See that it's using it, but like with the Pod IP, it's not being set yet
100. EX10 Configmaps
$ kubectl create configmap --from-literal=configvar=valuea intro
configmap "intro" created
$
$ kubectl edit deploy/intro
...
spec:
template:
spec:
containers:
- env:
- name: CONFIGVAR
valueFrom:
configMapKeyRef:
name: intro
key: configvar
...
deployment "intro" edited
100
Create a configmap with one key/value in it
Map that configmap's key in as an environment variable
101. EX10 Configmaps
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
intro-2942694475-948fj 1/1 Running 0 6s
intro-2942694475-9tf89 1/1 Running 0 6s
intro-2942694475-vh007 1/1 Running 0 6s
intro-3265745252-07l0b 1/1 Terminating 0 3s
intro-3265745252-0qw03 1/1 Terminating 0 4s
intro-3265745252-sxz2s 1/1 Terminating 0 4s
$
$
$ curl http://192.168.99.100:30561/
Hello Kubernetes Tutorial from 172.17.0.10!
The configuration variable is valuea
101
Deployment changed, so pods roll
And now it's using the CONFIGVAR environment variable
to output value
102. EX10 Configmaps
$ kubectl get cm/intro -o yaml
apiVersion: v1
data:
configvar: valuea
kind: ConfigMap
metadata:
creationTimestamp: 2017-09-10T05:36:09Z
name: intro
namespace: default
resourceVersion: "170767"
selfLink: /api/v1/namespaces/default/configmaps/intro
uid: f2f0527d-95e9-11e7-b635-080027358e48
$ kubectl get cm/intro -o yaml > intro-cm.yaml
102
Another way is to use the configmap spec like pod and deployment.
Can get that by looking at what's in there already.
Save that out to a file
103. EX10 Configmaps
$ vi intro.yaml
...
apiVersion: v1
data:
configvar: values
kind: ConfigMap
metadata:
name: intro
...
$
$ kubectl delete cm/info
configmap "intro" deleted
$
$ kubectl apply -f intro-cm.yaml
configmap "intro" created
103
Reduce it down to take out the Kuberentes server decoration
Delete the old info
`apply` tries to create/update the resource in sync with the file
In this case, it creates
104. EX10 Configmaps
$ vi intro.yaml
...
apiVersion: v1
data:
configvar: valueb
kind: ConfigMap
metadata:
name: intro
...
$
$ kubectl apply -f intro-cm.yaml
configmap "intro" created
$
$ curl http://192.168.99.100:30561/
Hello Kubernetes Tutorial from 172.17.0.11!
The configuration variable is valuea
104
Let's update the configvar
And testing... we see that it hasn't updated
105. EX10 Configmaps
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
intro-2942694475-t0dk8 1/1 Running 0 9m
intro-2942694475-tlpvm 1/1 Running 0 9m
intro-2942694475-vfwwf 1/1 Running 0 9m
$
$
$ kubectl get pods -o name | xargs kubectl delete
pod "intro-2942694475-t0dk8" deleted
pod "intro-2942694475-tlpvm" deleted
pod "intro-2942694475-vfwwf" deleted
$
$
$ curl http://192.168.99.100:30561/
Hello Kubernetes Tutorial from 172.17.0.10!
The configuration variable is valuec
105
A configmap change isn't readily identified as causing the deployment to change,
these pods don't get restarted automatically.
Delete the pods manually
Test again, and we see that it has changed
106. EX10 Configmaps
• Can also define config maps from files
• Include the whole file verbatim: kubectl create cm test --from-file=configs=/path/to/file
• Include the file as a list of key/value pairs: kubectl create cm test --from-env-file=/path/to/file
106
108. EX11: Secrets
$ kubectl create secret generic intro --from-literal=password=reallysecret
secret "intro" created
$
$ kubectl get secret intro -o yaml
apiVersion: v1
data:
password: cmVhbGx5c2VjcmV0
kind: Secret
metadata:
name: intro
namespace: default
...
type: Opaque
108
Secert is very similar to the configmap, but it's meant to have some meaning behind it (and handling is in progress)
Stores as base64 encoded values available from the API
109. EX11: Secrets
$ grep -A 2 Hello server.js
response.end('Hello Kubernetes Tutorial from ' + process.env.PODIP + '!n' +
'The secret password is "' + process.env.PW + '"n'
);
$ docker build -t intro:0.0.5 .
Sending build context to Docker daemon 3.072kB
Step 1 : FROM node:6.9.2
---> faaadb4aaf9b
Step 2 : EXPOSE 8080
---> Using cache
---> 20e3088f6122
Step 3 : COPY server.js .
---> cb2fb7acc119
Removing intermediate container 409c93df3ec7
Step 4 : CMD node server.js
---> Running in 35465f243ef9
---> 566294badefd
Removing intermediate container 35465f243ef9
Successfully built 566294badefd
109
Can use it the same way - set up the secret as an environment variable
110. EX11: Secrets
$ cat intro-deploy.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
...
spec:
...
template:
..
spec:
containers:
- env:
...
- name: PW
valueFrom:
secretKeyRef:
name: intro
key: password
image: intro:0.0.5
110
Updated intro deployment specification
Make the password available to the app as part of the environment
Update to our latest build
111. EX11: Secrets
$ kubectl apply -f intro-deploy.yaml
Warning: kubectl apply should be used on resource created by either kubectl
create --save-config or kubectl apply
deployment "intro" configured
$
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
intro-2942694475-fpnfh 1/1 Terminating 0 30m
intro-2942694475-gqdvf 1/1 Terminating 0 30m
intro-2942694475-rxn92 1/1 Terminating 0 30m
intro-3353884051-kdz2c 1/1 Running 0 2m
intro-3353884051-nnkln 1/1 Running 0 2m
intro-3353884051-rln6s 1/1 Running 0 2m
$
$ minikube service intro
Opening kubernetes service default/intro in default browser...
111
Update the new deploy spec with `apply`
Updated deployment causes pods to roll
And see if it worked...
113. EX11: Secrets
• You can change access to secrets separate from access to configmaps (see RBAC)
• Exposing via the environment may leak it (env available in other ways) --- we'll look at that next
• There is work to protect the secrets more
• Not allow any node access to the secret -- only ones where the secret is scheduled
• Sealing it all the way to the process
• Can use external secret stores - Vault, CyberArk, KMS, but mileage may vary
113
114. EX12: Volumes
• Purpose
• Explore the volumes, volumeMounts fields in the spec
• Explore secrets, configmaps as mounts
114
115. EX12a: Volumes
$ cat server.js
var http = require('http');
var fs = require('fs');
var password = fs.readFileSync('/data/password', 'UTF8');
var handleRequest = function(request, response) {
console.log('Received request for URL: ' + request.url);
response.writeHead(200);
response.end('Hello Kubernetes Tutorial from ' + process.env.PODIP + '!n' +
'The secret password from env is "' + process.env.PW + '"n' +
'The secret password from fs is "' + password +'"n'
);
};
var www = http.createServer(handleRequest);
www.listen(8080);
115
Update to read a secret from a file system path `/data/password`
116. EX12a: Volumes
$ cat intro-deploy.yaml
kind: Deployment
...
spec:
template:
...
spec:
volumes:
- name: intro
secret:
secretName: intro
containers:
- volumeMounts:
- name: intro
readOnly: true
mountPath: /data
...
image: intro:0.0.6
116
Update the deployment spec to map the intro secret to `/data`. This puts the `password` key at `/data/password`
117. EX12a: Volumes
$ docker build -t intro:0.0.6 .
Sending build context to Docker daemon 5.632kB
...
$ kubectl apply -f intro-deploy.yaml
deployment "intro" configured
deployment "intro" configured
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
intro-2431729973-nn0t0 1/1 Running 0 4s
...
intro-538233644-8tj16 1/1 Terminating 0 4m
...
$ minikube service intro
117
Deploy it all - build image, apply deployment, check for changed pods, and open the browser
119. EX12b: Volumes
$ cat server.js
var http = require('http');
var fs = require('fs');
var configvar = fs.readFileSync('/cm/configvar', 'UTF8');
var handleRequest = function(request, response) {
console.log('Received request for URL: ' + request.url);
response.writeHead(200);
response.end('Hello Kubernetes Tutorial from ' + process.env.PODIP + '!n' +
'The configvar from fs is "' + configvar + '"n'
);
};
var www = http.createServer(handleRequest);
www.listen(8080);
119
Same can be done for config map
121. EX12b: Volumes
$ docker build -t intro:0.0.7 .
Sending build context to Docker daemon 5.12kB
...
$ kubectl apply -f intro-deploy.yaml
deployment "intro" configured
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
...
intro-1784550256-3qvbn 0/1 ContainerCreating 0 3s
...
intro-2431729973-m7fxw 1/1 Terminating 0 3m
...
121
Redeploy
122. EX12b: Volumes
$ curl http://192.168.99.100:30561/
Hello Kubernetes Tutorial from 172.17.0.3!
The configvar from fs is "valuec"
122
And test...
123. EX12: Volumes
• Additional Volume types, but depend on environment
• HostPath volume
• Local volume
• NFS and NAS volumes
• Ceph, Gluster, ScaleIO, etc volumes
• Cloud volumes - AWS EBS/EFS, GCP Persistent Disk, Azure Disk/File
123
124. EX13: Stateful Sets
• Purpose
• Explore support for applications expecting consistent IPs
124
125. EX13: Stateful Sets
$ cat redis-statefulset.yaml
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: redis
spec:
serviceName: redis
replicas: 1
template:
metadata:
labels:
app: redis
spec:
containers:
- name: redis
image: redis:4.0.1
volumeMounts:
- mountPath: /data
name: redis-data
volumes:
- name: redis-data
hostPath:
path: /data
125
StatefulSet spec is similar to Deployment where it has a nested Pod spec inside of it
hostPath volume creates a place to preserve data
(separate from the Name/IP preservation)
126. EX13: Stateful Sets
$ kubectl apply -f redis-statefulset.yaml
statefulset "redis" created
$ kubectl get statefulset
NAME DESIRED CURRENT AGE
redis 1 1 24s
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
intro-1784550256-300fz 1/1 Running 0 1m 172.17.0.10 minikube
intro-1784550256-cxq4v 1/1 Running 0 1m 172.17.0.9 minikube
intro-1784550256-lc4l8 1/1 Running 0 1m 172.17.0.11 minikube
redis-0 1/1 Running 0 1m 172.17.0.3 minikube
126
Apply just like any of the others
Pod IP is allocated
Create Pod names based on statefulset
name with an identifier after
127. EX13: Stateful Sets
$ kubectl run -it --image=redis:4.0.1 shell /bin/sh
If you don't see a command prompt, try pressing enter.
#
# redis-cli -h 172.17.0.3
172.17.0.3:6379>
172.17.0.3:6379> set foo bar
OK
172.17.0.3:6379> get foo
"bar"
172.17.0.3:6379> save
OK
172.17.0.3:6379>
#
Session ended, resume using 'kubectl attach shell-2621852270-816gf -c shell
-i -t' command when the pod is running
127
Let's operate inside of the pod a little bit
Connect to the server pod based on IP
Set some data and to check back later
Make sure the data is saved to disk
128. EX13: Stateful Sets
$ kubectl delete pod redis-0
pod "redis-0" deleted
$ kubectl get pods/redis-0 -o wide -w
NAME READY STATUS RESTARTS AGE IP NODE
redis-0 1/1 Terminating 0 13s 172.17.0.3 minikube
redis-0 0/1 Terminating 0 14s <none> minikube
redis-0 0/1 Terminating 0 23s <none> minikube
redis-0 0/1 Terminating 0 23s <none> minikube
redis-0 0/1 Pending 0 4s <none> <none>
redis-0 0/1 Pending 0 4s <none> minikube
redis-0 0/1 ContainerCreating 0 4s <none> minikube
redis-0 1/1 Running 0 5s 172.17.0.3 minikube
128
Delete the pod, and it's recreated automatically
with the same name/IP
129. EX13: Stateful Sets
$ kubectl attach shell-2621852270-816gf -c shell -it
If you don't see a command prompt, try pressing enter.
# redis-cli -h 172.17.0.3
172.17.0.3:6379> get foo
"bar"
172.17.0.3:6379>
#
129
Check back in the new pod and see if the connection IP and data is preserved
130. EX13: Stateful Sets
• Tied together with volumes and storage classes, StatefulSets can help with non-12 Factor Apps
• Downsides
• Can't pick IP ahead of time
• Affects pod scheduling (has to map to existing node)
130
132. EX14: Services
$ kubectl apply -f redis.yaml
pod "redis" created
$
$ kubectl get pods/redis -o wide
NAME READY STATUS RESTARTS AGE IP NODE
redis 1/1 Running 0 21s 172.17.0.3 minikube
$ kubectl attach shell-2621852270-816gf -c shell -it
If you don't see a command prompt, try pressing enter.
# redis-cli -h 172.17.0.3
172.17.0.3:6379> GET foo
"bar"
172.17.0.3:6379>
#
Session ended, resume using 'kubectl attach shell-2621852270-816gf -c shell
-i -t' command when the pod is running
132
133. EX14: Services
$ kubectl delete pod/redis
pod "redis" deleted
$
$ kubectl scale deploy/intro --replicas=5
deployment "intro" scaled
$
$ kubectl apply -f redis.yaml
pod "redis" created
$ kubectl get pod/redis -o wide
NAME READY STATUS RESTARTS AGE IP NODE
redis 1/1 Running 0 9s 172.17.0.12 minikube
$
133
New pod has new IP
Delete to free up the IP
Make something else take up the existing IP (.3)
Recreate
134. EX14: Services
$ kubectl attach shell-2621852270-816gf -c shell -it
If you don't see a command prompt, try pressing enter.
#
# redis-cli -h 172.17.0.3
Could not connect to Redis at 172.17.0.3:6379: Connection refused
Could not connect to Redis at 172.17.0.3:6379: Connection refused
not connected> exit
#
# redis-cli -h 172.17.0.12
172.17.0.12:6379> GET foo
"bar"
172.17.0.12:6379>
#
Session ended, resume using 'kubectl attach shell-2621852270-816gf -c shell
-i -t' command when the pod is running
134
Let's try to get the data again
Using the old IP address will fail
Try the new iP
Data is still there
135. EX14: Services
$ kubectl get service
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
intro 10.0.0.238 <nodes> 8080:30561/TCP 1h
kubernetes 10.0.0.1 <none> 443/TCP 1h
$ kubectl get service/intro -o yaml
apiVersion: v1
kind: Service
metadata:
labels:
run: intro
name: intro
...
spec:
clusterIP: 10.0.0.238
ports:
- nodePort: 30561
port: 8080
protocol: TCP
targetPort: 8080
selector:
run: intro
type: NodePort
...
135
Look at the existing services
This came from the `expose` in EX02
136. EX14: Services
$ kubectl expose pod redis --port 6379
service "redis" exposed
$ kubectl get service/redis
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
redis 10.0.0.184 <none> 6379/TCP 6s
$ kubectl get service/redis -o yaml
apiVersion: v1
kind: Service
metadata:
labels:
app: redis
name: redis
...
spec:
clusterIP: 10.0.0.184
- port: 6379
protocol: TCP
targetPort: 6379
selector:
app: redis
...
136
This is the IP to use to connect to
137. EX14: Services
$ kubectl get service/redis -o yaml
apiVersion: v1
kind: Service
spec:
selector:
app: redis
...
$ kubectl get pods/redis -o yaml
apiVersion: v1
kind: Pod
metadata:
labels:
app: redis
name: redis
...
137
How the service maps to the pods
138. EX14: Services
• Ensure that the `kube-dns` add-on is running
• This provides a mapping from $SERVICENAME to $IP
• FQDN: $SERVICE_NAME.$NAMESPACE.svc.$CLUSTER_DOMAIN
• So can use DNS instead of L4 mappings
138
$ minikube addons list | grep kube-dns
- kube-dns: enabled
139. EX14: Services
$ kubectl attach shell-2621852270-816gf -c shell -it
If you don't see a command prompt, try pressing enter.
#
# redis-cli -h 10.0.0.184
10.0.0.184:6379> GET foo
"bar"
10.0.0.184:6379>
#
# redis-cli -h redis
redis:6379> GET foo
"bar"
redis:6379>
#
Session ended, resume using 'kubectl attach shell-2621852270-816gf -c shell
-i -t' command when the pod is running
139
Try to access it via the IP address of the service
Try to access it via the DNS name
141. Where to go from here?
• Topics
• Running the Kubernetes cluster itself
• Persistent Volumes
• Ingresses
• Access Control
• Operators
• Helm
• Multicontainer Pods, Sidecars
141
142. October 29–November 3, 2017 | San Francisco, CA
www.usenix.org/lisa17 #lisa17
Remember to fill in
your tutorial evaluations!
Thank You!
F2 - Kubernetes : Hit the Ground Running
Chris "mac" McEniry