TODO

1. 1@Horgix@ContainerDayFRKIND: Next level inception
Alexis “Horgix” Chotard
4th
June 2019
Kubernetes in Docker
Next level inception

2. 2@Horgix@ContainerDayFRKIND: Next level inception
SRE & Technical Officer
▼ Automation Addict
▼ Craftsman
▼ ❤ CNCF & Open Source
▼ 30% Development
▼ 40% Operations
▼ 30% Training, coaching, ...
@Horgix
HorgixAlexis “Horgix” Chotard

3. 3@Horgix@ContainerDayFRKIND: Next level inception
Let’s talk about testing!
Kubernetes in Docker?
I lied. It was a bait.

4. 4@Horgix@ContainerDayFRKIND: Next level inception
Craftsmanship applied to complex systems
Testing & Kubernetes

5. 5@Horgix@ContainerDayFRKIND: Next level inception
Reminder: tests pyramid

6. 6@Horgix@ContainerDayFRKIND: Next level inception
Reminder: tests pyramid
What does it
mean for
Kubernetes
applications?

7. 7@Horgix@ContainerDayFRKIND: Next level inception
Kubernetes applications?
▼ Let’s not talk about your typical application. You write simple manifests to
create Deployments, Services, Ingresses, etc. and apply them. Fine.
▼ What about applications deeply interfacing with Kubernetes?
▽ Operators are popping everywhere and for everything
Reminder: Operator = Custom Resource Definition (CRD) + Controller
▽ Controllers & CRDs have to be tested
▼ … and what about Kubernetes itself?

8. 8@Horgix@ContainerDayFRKIND: Next level inception
The overly simplified CRDs & Controllers 101

9. 9@Horgix@ContainerDayFRKIND: Next level inception
The overly simplified CRDs & Controllers 101

10. 10@Horgix@ContainerDayFRKIND: Next level inception
The overly simplified CRDs & Controllers 101

11. 11@Horgix@ContainerDayFRKIND: Next level inception
The overly simplified CRDs & Controllers 101

12. 12@Horgix@ContainerDayFRKIND: Next level inception
Unit tests
▼ client-go fake client [1][2]
▼ In-memory implementation of an apiserver/client
▼ Great for simple things but lots of caveats:
▽ Majority of the functionalities of apiserver’s does not exist
▽ Race & timing issues not surfaced
▽ No other controllers running
(i.e. creating a ReplicaSet does not mean Pods)
[1] https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/kubernetes/fake/clientset_generated.go
[2] Example usage: https://github.com/diazjf/fakeclient

13. 13@Horgix@ContainerDayFRKIND: Next level inception
Integration tests
▼ Kubebuilder [1] & controller-runtime [2] use this approach a lot
▼ Run etcd + apiserver (+ optionnally controller-manager)
▼ Why?
▽ Admission control
▽ Timing issues
[1] https://github.com/kubernetes-sigs/kubebuilder
[2] https://github.com/kubernetes-sigs/controller-runtime

14. 14@Horgix@ContainerDayFRKIND: Next level inception
End-to-end tests
▼ Start a full Kubernetes cluster
▼ Give us the ultimate flexibility
▼ Black box testing - we don’t assume implementation
▼ But this can be slow and/or expensive
▼ Why End-to-End (E2E) tests?
▽ Certain edge cases only picked up in a real environment
▽ Kubernetes has a lot of controllers
The way these inter-operate is really important
▽ “Fighting” can cause massive issues for a level based system

15. 15@Horgix@ContainerDayFRKIND: Next level inception
End-to-end tests - What they usually run on
▼ Remote, managed cluster (GKE, EKS, …)
▼ Minikube
▼ Your own cluster
▼ … all with their own pitfalls:
▽ Often used as “persistent” clusters
▽ Lead to behaviors related to a previous state
▽ Might be slow and/or costly
▽ What about testing different Kubernetes versions easily?
▼ “When you’re developing tools against the Kubernetes APIs it’s often best to
be throwing things away fairly regularly” [1]
[1] https://garethr.dev/2019/05/ephemeral-kubernetes-clusters-with-kind-and-make/#167

16. 16@Horgix@ContainerDayFRKIND: Next level inception
Let’s run temporary instances inside containers!
What if we did just like our other apps?

17. 17@Horgix@ContainerDayFRKIND: Next level inception
https://kind.sigs.k8s.io/
https://github.com/kubernetes-sigs/kind

18. 18@Horgix@ContainerDayFRKIND: Next level inception
What’s kind?
▼ Kubernetes IN Docker
▼ Uses containers to simulate nodes
▼ Multi-node
▼ Multi-cluster
▼ Works offline
▼ Boot a cluster in ~30sec
(well, a bit more since a recent kubelet systemd unit change [1]
... but soon down to 20sec [2])
[1] https://github.com/kubernetes-sigs/kind/issues/576
[2] https://github.com/kubernetes-sigs/kind/pull/585

19. 19@Horgix@ContainerDayFRKIND: Next level inception
How kind works
▼ Leverage Existing Tooling [1] :
▽ kubeadm handles node configuration, certificates, etc.
▽ kustomize handles merging user-provided config patches
with generated kubeadm configs
▼ Node image
▽ You can build it yourself - useful when developing on K8s
▽ Embed everything that’s need
▽ Docker in Docker pattern
[1] https://kind.sigs.k8s.io/docs/design/principles/

20. 20@Horgix@ContainerDayFRKIND: Next level inception
Architecture summary - what is spawned by kind

21. 21@Horgix@ContainerDayFRKIND: Next level inception
Let’s spawn some clusters
Demo time ⚡

22. 22@Horgix@ContainerDayFRKIND: Next level inception
Demo #1 - What did we just do?
▼ We created a cluster using kind create
▼ We learned that kind doesn’t persist any state and
instead use Docker labels in a smart way
▼ We saw a full Kubernetes cluster with real components
… running inside Docker
▼ In a really fast and efficient way!

23. 23@Horgix@ContainerDayFRKIND: Next level inception
With end-to-end (conformance) tests
Using kind clusters for testing

24. 24@Horgix@ContainerDayFRKIND: Next level inception
Conformance tests?
▼ Requirements:
▽ Ginkgo
▽ e2e.test
▽ kubectl
▼ Official test suite used to validate releases

25. 25@Horgix@ContainerDayFRKIND: Next level inception
Kubernetes conformance tests
Demo time 🔧

26. 26@Horgix@ContainerDayFRKIND: Next level inception
Demo #2 - What did we just do?
▼ We built the end-to-end test tooling of Kubernetes
▼ We used it against our test cluster from the outside as we may do with
any local cluster
▼ We found that kubetest is already integrated with kind

27. 27@Horgix@ContainerDayFRKIND: Next level inception
Not everyone is developing on Kubernetes itself
What about controllers?

28. 28@Horgix@ContainerDayFRKIND: Next level inception
Sample controller deployment/injection
Demo time 🚀

29. 29@Horgix@ContainerDayFRKIND: Next level inception
Demo #3 - What did we just do?
▼ We built a sample controller
▼ We created CRD and a custom object
… and saw that the controller reacted to it
▼ Given more time for the demo, we also could have:
▽ Added an end-to-end test for this behavior and ran it against our cluster
▽ Loaded and deployed the controller into our cluster

30. 30@Horgix@ContainerDayFRKIND: Next level inception
kind in a CI pipeline
Demo time ♻

31. 31@Horgix@ContainerDayFRKIND: Next level inception
… Not enough time for it today.
See the repo at the end of these slides :)
kind in a CI pipeline

32. 32@Horgix@ContainerDayFRKIND: Next level inception
...
Wrapping up 🎁

33. 33@Horgix@ContainerDayFRKIND: Next level inception
kind vs The world
▼ https://github.com/kubernetes/minikube
▼ https://github.com/ubuntu/microk8s
▼ https://github.com/bsycorp/kind
▼ https://github.com/kinvolk/kube-spawn
▼ https://github.com/danderson/virtuakube
▼ https://github.com/kubernetes-sigs/kubeadm-dind-cluster

34. 34@Horgix@ContainerDayFRKIND: Next level inception
So… Why kind?
▼ “kind is actually literally my favorite thing right now” -- Bryan Liles @ KCCNC
▼ It’s simple, yet complete
▽ Multi-node (including HA) clusters
▽ Supports building Kubernetes release builds from source
▽ Can be used as a library [1]
▽ Supports Windows in addition to MacOS and Linux
▼ kind is a CNCF certified conformant Kubernetes installer
▼ Developed by cool & smart people
▼ Integrated with official Kubernetes testing tools
▼ Being integrated with more and more stuff every day
[1] https://github.com/kubernetes/kubeadm/tree/master/kinder

35. 35@Horgix@ContainerDayFRKIND: Next level inception
Take away
▼ kind allows you to spawn a real & lightweight Kubernetes clusterS
▼ This is awesome for debugging and testing:
▽ Kubernetes itself
▽ Applications interacting with Kubernetes directly such as controllers
▼ It’s really fast to create clusters: ~30sec
▼ Integrates well in a CI pipeline and other use-cases
Kubernetes is all about community: go contribute to kind :)

36. 36@Horgix@ContainerDayFRKIND: Next level inception
Alexis “Horgix” Chotard
4th
June 2019
Kubernetes in Docker
Next level inception
Thank you!
Demo : https://github.com/Horgix/kind-demo
Slides : https://www.slideshare.net/Horgix
Thanks @BenTheElder