This talk provides a 101 introdution to Kubernetes from a user point of view. Aimed at service providers, it was presented at the GPN Annual Meeting 2019. https://conferences.k-state.edu/gpn/

1. An overview of the
Kubernetes architecture
Presented by Igor Sfiligoi, UCSD
Workshop at the Great Plains Network Annual Meeting 2019
GPN Annual Meeting 2019 - Kubernetes Architecture 1

2. Outline
• Kubernetes history
• Basic building blocks
• Provided bells and whistles
• Scheduling
• User interface
GPN Annual Meeting 2019 - Kubernetes Architecture 2

3. Kubernetes
• Now maintained by
Cloud Native Computing Foundation
https://kubernetes.io
Originally created by Google
• With very large and active
development community
Open source
• But also available out-of-the-box on
all major Clouds (GCP, AWS and Azure)
Can be deployed on-prem
GPN Annual Meeting 2019 - Kubernetes Architecture 3

4. Container based
• Typically Docker based
Containers are the
basic building block
• Creating custom ones almost trivial
Standard images for
many applications exist
• If state needed, must be held outside
Just remember
containers are stateless
GPN Annual Meeting 2019 - Kubernetes Architecture 4

5. Container Orchestration
• Once you have many containers on many nodes, you need something to manage the whole
• This is usually referred to as Orchestration
Attribution: https://kubernetes.io
GPN Annual Meeting 2019 - Kubernetes Architecture 5

6. Packing containers into pods
The smallest concept is actually the Pod
A Pod is a set of containers
• Having a single Container in a Pod OK
Containers within a Pod are
guaranteed to run alongside
• And can share (ephemeral) state
Pod
Container
Container
https://kubernetes.io/docs/concepts/workloads/pods/pod/
GPN Annual Meeting 2019 - Kubernetes Architecture 6

7. Packing Pods into Deployments
• If it terminates for whatever reason, it is gone
A Pod is ephemeral
• Initially launches a single Pod (no obvious benefit)
• If a Pod is removed, a new Pod is automatically re-submitted
A Deployment is persistent
• E.g. for load balancing and horizontal scaling
A Deployment can also manage multiple replicas
Great
for service
applications
https://kubernetes.io/docs/concepts/workloads/controllers/deployment/
GPN Annual Meeting 2019 - Kubernetes Architecture 7

8. Configuration
management
• Kubernetes provides an easy mechanism to inject
information into the Container images at runtime
Most applications need to be configured
Three types of information
Environment variables Whole files Secrets
https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/
https://kubernetes.io/docs/concepts/configuration/secret/
GPN Annual Meeting 2019 - Kubernetes Architecture 8

9. Linking to external storage
• Most applications will need it!
External storage essential for persistency
• Local storage
• Distributed storage, e.g. CEPH, NFS, etc.
• Custom filesystems via CSI – e.g. CVMFS
Kubernetes provides the necessary hooks at Pod launch time
https://kubernetes.io/docs/concepts/storage/volumes/
https://kubernetes-csi.github.io/docs/
GPN Annual Meeting 2019 - Kubernetes Architecture 9

10. Networking
Each container get its own private IP address
A Deployment can be registered as a Service
• Gets its own IP address and DNS entry
• Traffic routes to the Pods in Deployment based on selected policy (e.g. RR)
Service can also serve as a NAT
• Routing traffic from WAN using the Kubernetes public IPs
https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
GPN Annual Meeting 2019 - Kubernetes Architecture 10

11. Networking
Each container get its own private IP address
A Deployment can be registered as a Service
• Gets its own IP address and DNS entry
• Traffic routes to the Pods in Deployment based on selected policy (e.g. RR)
Service can also serve as a NAT
• Routing traffic from WAN using the Kubernetes public IPs
https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
• Unprivileged Pods better for regular users to minimize risk
Privileged Pods can get access to the host/public IP
• E.g. due to the use of X.509
Useful for Network Servers tied to a specific node
GPN Annual Meeting 2019 - Kubernetes Architecture 11

12. Pod scheduling
Kubernetes comes with a pretty decent scheduler
Will match Pods to available resources (CPU, Memory, GPU, etc.)
• Nodes advertise what is available
• Pods specify what they require, may also limit itself to a subset of Nodes
• A Pod will start on a Node only if a match can be made
There is also a notion of Priorities
• If a match for a higher priority Pod cannot be made,
the scheduler will kill one or more lower priority Pods to make space for it (if at all possible)
https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/
https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/
GPN Annual Meeting 2019 - Kubernetes Architecture 12

13. The DaemonSet
• E.g. a Monitoring probe
Sometimes an application must run on all the nodes
• Like a Deployment, but with fixed all-nodes scheduling
The DaemonSet automates this
https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/
GPN Annual Meeting 2019 - Kubernetes Architecture 13

14. Users and Permissions
Kubernetes does not really have a concept of a “User”
Permissions are set as part of the Namespace concept
• Anyone having access to a Namespace can operate on the objects inside that Namespace
• Including creating, monitoring and modifying them
Namespace conceptually provides virtual-private Kubernetes clusters
• But very little additional restrictions within
• And relatively hard coordinating Pods in separate Namespaces
https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
GPN Annual Meeting 2019 - Kubernetes Architecture 14

15. Users and Permissions
Kubernetes does not really have a concept of a “User”
Permissions are set as part of the Namespace concept
• Anyone having access to a Namespace can operate on the objects inside that Namespace
• Including creating, monitoring and modifying them
Namespace conceptually provides virtual-private Kubernetes clusters
• But very little additional restrictions within
• And relatively hard coordinating Pods in separate Namespaces
PRP Nautilus provides
user management as a
side concept.
https://nautilus.optiputer.net
GPN Annual Meeting 2019 - Kubernetes Architecture 15

16. Driving
Kubernetes
GPN Annual Meeting 2019 - Kubernetes Architecture 16

17. YAML
Everywhere
• Both for creating/configuring
Pods/Deployments/Services
• And for querying their (detailed) status
Most interactions with Kubernetes
will involve YAML documents
• Describes itself as
“a human friendly markup language”
• Uses Python-indentation
to indicate nesting
YAML is actually quite easy to use
https://en.wikipedia.org/wiki/YAML
GPN Annual Meeting 2019 - Kubernetes Architecture 17

18. An example YAML file
kind: Deployment
metadata:
name: osg-collector-prp-sdsc
namespace: osg
labels:
k8s-app: osg-collector-prp
spec:
template:
metadata:
labels:
k8s-app: osg-collector-prp
spec:
containers:
- name: osg-collector-prp
image: sfiligoi/prp-osg-pool:collector
ports:
- containerPort: 9618
volumeMounts:
- name: condordata
mountPath: /var/lib/condor
- name: configpasswd
mountPath: /var/lock/condor/pool_password
subPath: pool_password
readOnly: true
volumes:
- name: condordata
persistentVolumeClaim:
claimName: pvc-xcache11-t2-ucsd-edu-persistent-1
- name: configpasswd
secret:
secretName: osg-pool-sdsc-config
items:
- key: pool_password
path: pool_password
defaultMode: 256
GPN Annual Meeting 2019 - Kubernetes Architecture 18

19. kubectl
• kubectl create -f <filename> - Create new object
• kubectl get <type> -n <namespace> - Query existing objects
• kubectl edit <type> -n <namespace> <id> - Update existing object
• kubectl delete -f <filename> - Delete existing object
Most often used cmdline tool
https://kubernetes.io/docs/reference/kubectl/
GPN Annual Meeting 2019 - Kubernetes Architecture 19

20. Installing kubectl
• Just a static binary
• Available for all major platforms
(Linux, MacOS, Windows)
• Detailed download instructions at
https://kubernetes.io/docs/tasks/tools/install-kubectl/
• Can be used over WAN
• Just put the config file in
~/.kube/config
Get yours from
PRP’s Nautilus
GPN Annual Meeting 2019 - Kubernetes Architecture 20

21. The end
GPN Annual Meeting 2019 - Kubernetes Architecture 21

22. Acknowledgents
This work was partially funded by
US National Science Foundation (NSF) awards
CNS-1456638, CNS-1730158,
ACI-1540112, ACI-1541349,
OAC-1826967, OAC 1450871,
OAC-1659169 and OAC-1841530.
GPN Annual Meeting 2019 - Kubernetes Architecture 22