At Giant Swarm, we manage Kubernetes clusters for customers 24/7, both on-premises and in the cloud. That means we do not just set something up and hand it over, but we actually take care that it’s operational and up-to-date at all times. In this talk Timo explains how Giant Swarm are using Operators to codify all operational tasks of managing Kubernetes cluster and distributed applications on top. The operators manage PKI infrastructures, networks, VMs and storage both on-premises and in the cloud. There have been a lots of challenges and learnings in the past year and Timo would like to share them with you.

1. Automating Kubernetes Cluster
Operations with Operators
by Timo Derstappen

2. Running Kubernetes in Kubernetes

3. What this talk is about
Provisioning Kubernetes Clusters
Advanced usage of Kubernetes

4. What this talk is not about
Intro to Kubernetes
How to build cloud native applications

5. Overview
1. Background / Motivation
2. Architecture
3. Operators
4. Outlook

6. Background
We manage Kubernetes clusters for customers 24/7.
Both on bare metal and in the cloud.

7. Kubernetes as a Service (KaaS)
Own DC in Frankfurt, Germany
AWS (Azure and Google next)
Full access to clusters

8. On-premises KaaS
We also manage Kubernetes in the DCs or AWS accounts of
the customers itself.

9. Open Source
Our tooling is open-source and we are working with the
community to improve kubernetes operations.

10. Multiple Clusters
"Soft" multi-tenancy not enough in enterprise context
Different service classifications
Different environments: separation of dev, test, prod
Test tooling on new K8s versions

11. People must come to things in their
own time, in their own way, for their
own reasons, or they never truly come
at all. - Dee Hock

12. RBAC, NetworkPolicies FTW
We encourage to use these features and help with
integration.
Small clusters make little sense
Support while processes need to be adapted
Kubernetes matures - trust over time

13. Introducing:
Giantnetes

14. Motivation
Rather obvious. If you provide others with Kubernetes you
would like to use the same concepts within your product as
well.

15. De nition
G8s: Giantnetes
K8s: Kubernetes == Guest cluster

17. Overview
1. Background / Motivation
2. Architecture
3. Operators
4. Outlook

21. Why are we doing this again?
Fully automate a powerful container platform
Take away complexity of distributed applications from the
developers

22. Immutability
CoreOS (Hosts and Guests)
Continuous pipeline (Container build, registry, kubernetes)

23. Reproducable builds and
deployments at any point in time

24. Every manual change is ephemeral

25. Mean time to recovery!

26. Networking
Flannel/VPC network between guest cluster nodes
Calico between containers
Network policy controller

31. Certi cates
Each cluster has its own root ca
Using PKI backends in Vault
Certificates are rotated every day

32. Certi cates #2
Kubernetes components use mutual TLS
Client certificates are used for RBAC

33. Overview
1. Background / Motivation
2. Architecture
3. Operators
4. Outlook

34. Iterations of the platform
1. K8s clusters via systemd units over fleet
2. K8s manifests to create K8s clusters used as templates
3. Writing operators

35. Operators
Codify all operational tasks
Manage desired state

36. Third-Party Resources
Easily extend the Kubernetes API

37. KVM operator
Create k8s clusters on bare-metal via KVM based on a TPR

38. AWS operator
Create k8s clusters on AWS based on a TPR

45. The Cluster TPR de nes
Kubernetes configuration/version
Ingress configuration/version
Network configuration
Master/Node configurations

46. The Cert TPR de nes
TTL
Common Name
SANs
Cluster / Component

47. Overview
1. Background / Motivation
2. Architecture
3. Operators
4. Outlook

48. Micro Operators
Operators should have single responsibilities.
Keep them simple and maintainable

49. Operatorkit
Our services are based upon microkit. We would like to
reduce boilerplate in our operators and collect them in a
library called operatorkit.

50. Self-hosted
Having Giantnetes and all guest clusters running self-hosted
will further ease the lifecycle management of the clusters.

51. Thank you.
Timo Derstappen
@teemow