Kubernetes Pod manifests, port redirection, liveness probes, and other features make a great single-node container manager. Learn how to use the kubelet in this manner for edge deployments and other single machine use-cases. We'll cover the tradeoffs, what you get, what won't work, and more.

1. CONFIDENTIAL designator
Kubelet with no Kubernetes
Control Plane
Rob Szumski
OpenShift Product Manager
1

2. CONFIDENTIAL designator
Who am I?
● Now: OpenShift Product Manager
● Early employee at CoreOS
● Playing with containers and
Kubernetes since they existed
● Planning out an “edge” deployment
for my shop

3. CONFIDENTIAL designator
Where did we start from?
1. etcd 2. docker 3. systemd
coordination containment lifecycle/logs

4. CONFIDENTIAL designator
Where did we start from?
1. etcd 2. docker 3. systemd
coordination containment lifecycle/logs
GitHub src

5. CONFIDENTIAL designator
Where did we start from?
1. etcd 2. docker 3. systemd
coordination containment lifecycle/logs
etcd via
Kubernetes
CRI via
Kubernetes
Kubernetes

6. CONFIDENTIAL designator
Kubernetes Architecture
Kubernetes
Workload
YAML
Controller
Manager
etcd
Scheduler
Deployment
StatefulSet
...etc...
Node
P
P
P
P
P
P
P

7. CONFIDENTIAL designator
Detailed look at a Node
Node
P
Kubelet
Operating
System
Container
Runtime
P
apiVersion: v1
kind: Pod
metadata:
name: nginx
spec:
containers:
- name: nginx
image: quay.io/robszumski/nginx
securityContext:
privileged: true
ports:
- name: tls
containerPort: 443
hostPort: 443
protocol: TCP
resources:
limits:
cpu: "100m"
memory: "100Mi"
volumeMounts:
- name: letsencrypt
mountPath: /etc/letsencrypt
mountPropagation: Bidirectional
- name: proxy3
image: quay.io/pusher/oauth2_proxy
volumes:
- name: letsencrypt
hostPath:
path: /etc/letsencrypt
Pod logs
Pod status
Node status
Resource usage

8. CONFIDENTIAL designator
Minimal deployment for my shop
● Single machine, consumer hardware
● Control software for industrial equipment (CNC)
● Security cameras
● Camera recording/DVR
● Camera transcoding/timelapse
● Nginx web server
● OAuth proxy to secure access

9. STRICTLY INTERNAL ONLY
End-User Premises Edge Provider Edge Provider/Enterprise Core
Edge
Server/Gateway
Regional
Data Center
Infrastructure
Edge
Provider
Far
Edge
Provider
Access
Edge
Provider
Aggregation
Edge
Core
Data Center
Device or
Sensor
9
“last mile”
“Edge”
deployments

10. CONFIDENTIAL designator
Other edge examples

11. CONFIDENTIAL designator
Do you need a control plane?
Kubernetes
Workload
YAML
Controller
Manager
etcd
Scheduler
Deployment
StatefulSet
...etc...
Node
P
P
P
P
P
P
P

12. CONFIDENTIAL designator
Do you need a control plane?
Kubernetes
Pod
YAML
Controller
Manager
etcd
Scheduler
Node
P
P
P
P
P
P
P
??

13. CONFIDENTIAL designator
The Standalone Kubelet
● Does not talk to a control plane
● Can be fully disconnected if desired
● Uses the kubelet’s static manifest feature
● Compatible with fully automatic provisioning
○ Golden image
○ Ignition/Cloud-Init
● More powerful with a self-managing operating
system
Node
Kubelet
Operating
System
Container
Runtime
P

14. CONFIDENTIAL designator
The Standalone Kubelet
$ systemctl cat kubelet
# /etc/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet Server
After=crio.service
Requires=crio.service
[Service]
WorkingDirectory=/var/lib/kubelet
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/kubelet
ExecStart=/usr/bin/kubelet
$KUBE_LOGTOSTDERR
$KUBE_LOG_LEVEL
$KUBELET_API_SERVER
$KUBELET_ADDRESS
$KUBELET_PORT
$KUBELET_HOSTNAME
$KUBE_ALLOW_PRIV
$KUBELET_ARGS
Restart=on-failure
KillMode=process
[Install]
WantedBy=multi-user.target
$ cat /etc/kubernetes/kubelet
###
# kubernetes kubelet (minion) config
# The address for the info server to serve on (set to
0.0.0.0 or "" for all interfaces)
KUBELET_ADDRESS="--address=127.0.0.1"
# The port for the info server to serve on
# KUBELET_PORT="--port=10250"
# You may leave this blank to use the actual hostname
KUBELET_HOSTNAME="--hostname-override=127.0.0.1"
# Add your own!
KUBELET_ARGS="--cgroup-driver=systemd --fail-swap-on=false
--pod-manifest-path=/etc/kubernetes/manifests
--container-runtime=remote
--container-runtime-endpoint=unix:///var/run/crio/crio.sock
--runtime-request-timeout=10m"

15. CONFIDENTIAL designator
Pros/Cons of No Control Plane
What works:
● Pods with multiple containers
● Volume mounts, security contexts, resource limits
● Port redirection, NodePorts
● Log streaming
● Attaching to containers, exec-ing commands
● Restarting Pods on failure
What doesn’t work:
● DaemonSets, ReplicaSets, Deployments
● Services, Ingress, NetworkPolicy
● Events, RBAC, authentication
● Persistent Volumes and Persistent Volume Claims
● Operators, custom controllers, admission controllers
Changes to our workflow
Services & Ingress:
To ingest traffic, we will need to run nginx on a
NodePort, similar to running an Ingress
controller on a full cluster.
Volumes:
Instead of persisting data to a PV, you will
need to write directly to the host’s storage.
CLI tool:
Directly interact with the container runtime,
crictl logs -f instead of kubectl logs -f

16. CONFIDENTIAL designator
Running a Container
$ ssh root@fedora
$ crictl ps
CONTAINER IMAGE STATE
proxy3 quay.io/pusher/oauth2_proxy@sha256:b5c44a0aba0e146a776a6a2a07353a3dde3ee78230ebfc56bc973e37ec68e425 Running
nginx7 quay.io/robszumski/nginx-for-drone@sha256:aee669959c886caaf7fa0c4d11ff35f645b68e0b3eceea1280ff1221d88aac36 Running
cncjs9 quay.io/robszumski/cncjs@sha256:3d11bc247c023035f2f2c22ba4fa13c5c43d7c28d8f87588c0f7bdfd3b82121c Running
transcode15 quay.io/robszumski/rtsp-to-mjpg@sha256:52dd81db58e5e7c9433da0eedb1c02074114459d4802addc08c7fe8f418aead5 Running
$ scp pod.yaml root@fedora:/etc/kubernetes/manifests
Deployment:
List running workloads:
$ crictl logs 86fadc1aee09c
[2020/07/05 20:15:46] [oauthproxy.go:252] mapping path "/" => upstream "http://192.168.7.62:8080/"
[2020/07/05 20:15:46] [http.go:57] HTTP: listening on :4180
[2020/07/05 20:18:45] [google.go:270] refreshed access token Session{email:xxxx@robszumski.com user:168129640651108868061
PreferredUsername: token:true id_token:true created:2020-07-05 16:49:23.445238687 +0000 UTC expires:2020-07-05 21:18:44 +0000 UTC
refresh_token:true} (expired on 2020-07-05 20:08:08 +0000 UTC)
173.53.xx.xxx - xxxx@robszumski.com [2020/07/05 20:18:44] xxxx.robszumski.com GET - "/oauth2/auth" HTTP/1.0 "Mozilla/5.0 (Macintosh;
Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36" 202 0 1.221
Stream logs:

17. CONFIDENTIAL designator
Kubernetes Pod is the API
Deconflict names
apiVersion: v1
kind: Pod
metadata:
name: nginx
spec:
containers:
- name: nginx
image: quay.io/robszumski/nginx
securityContext:
privileged: true
ports:
- name: tls
containerPort: 443
hostPort: 443
protocol: TCP
resources:
limits:
cpu: "100m"
memory: "100Mi"
volumeMounts:
- name: letsencrypt
mountPath: /etc/letsencrypt
mountPropagation: Bidirectional
- name: proxy3
image: quay.io/pusher/oauth2_proxy
volumes:
- name: letsencrypt
hostPath:
path: /etc/letsencrypt
Standardized format
that is well understood
Multiple containers
Port mapping
Resource limits,
hard and soft
Volume mapping, including
shared mounts
Security contexts
Liveness/readiness probes
Restart policy

18. CONFIDENTIAL designator
Node
Scaling this deployment method
Node
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
manual steps
golden image
scp
Machine
Configuration
Pod
YAMLs
systemd timer + curlremote ignition
config file
cloud storage buckets
Machine
Configuration
Pod
YAMLs
Small scale, manual Hundreds or thousands, fully automated

19. CONFIDENTIAL designator
Try it out!
● Tweet at me if you try it out: @robszumski
● Bleeding edge, so just a gist
○ https://gist.github.com/dmesser/ffa556788660a7d23999427be4797d38
● Kelsey Hightower has an older tutorial based on CoreOS Container Linux (EOLd)
○ https://github.com/kelseyhightower/standalone-kubelet-tutorial

20. CONFIDENTIAL designator
Future: the real solution
{
"ignition": {
"version": "2.2.0",
"config": {
"replace": {
"source": "http://ignition-server-public.xxx.robszumski.com/ignition.json",
"verification": { "hash":
"sha512-a4d77e4915a74c0828bdddb952d965f0aa7d2f7f80b315f7cbf475cc2e442b72d9ca8bc48269c0
9d2b14c05720ffb57662fc10f564d871ab8f13160cdfe20115" }
}
}
}
}
Take this Ignition stub that references the remote one:
Pass it to your cloud provider
$ aws ec2 run-instances --image-id ami-abcd1234 --count 1 --instance-type m3.medium
--key-name my-key-pair --subnet-id subnet-abcd1234 --security-group-ids sg-abcd1234
--user-data file://remote-ignition.json
Pass it to your bare metal
$ sudo coreos-installer install /dev/sda --ignition-file remote-ignition.json