Five Great Ways to Lose Data on Kubernetes - KubeCon EU 2020

1. Five Great Ways to Lose Data on Kubernetes (And How to Avoid Them) Robert Hodges - KubeCon Europe 2020 1

2. Presenter and Company Bio www.altinity.com Enterprise provider for ClickHouse, a popular, open source data warehouse. Implementors of ClickHouse Kubernetes operator. Robert Hodges - Altinity CEO 30+ years on DBMS plus virtualization and security. Using Kubernetes since 2018. 2

3. Two boring deﬁnitions 3

4. What do we mean by “data”? Rep SKU Date City Customer Units Price ... 25 #556 25 Nov SFO Wells Fargo 378 $25.00 36 #556 25 Nov SEA Boeing 259 $25.50 55 #558 28 Nov BOS Au Bon Pain 100 $29.33 ... ... ... ... ... ... ... Which products have the best gross margins over time? Do some SKUs sell better on different days of the week? Which kinds of companies are most likely to buy SKU 556? 4

5. And what do we mean by “lose”? The data loss “arrow of evil” Temporarily unavailable All of it gone, forever Can’t process transactions Unable to see market conditions Business stops functioning 5

6. The single copy catastrophe 6

7. Starting a database is easy in Kubernetes (Install helm) kubectl create ns mysql helm install --namespace mysql --name mysql-server stable/mysql 7

8. Kubernetes Node A delicate ﬂower is our database Container “mysql” Storage Delete pod Delete volume Delete node 8

9. Kubernetes Node Traditional database solution: replicas Storage Kubernetes Node Storage Replication Container “mysql” Backup DBMS Copy DBMS Copy DBMS Backup Static copies Live replica Primary Replica Container “mysql” 9

10. Simplest K8s path to replicas: use an operator Kubernetes Operator Single speciﬁcation Best practice deployment Custom Resource Definition Kubernetes API Native Controller Native Controller Native Controllers Etcd Pod Service StatefulSet 10

11. Complex systems made simple[r] apiVersion: "clickhouse.altinity.com/v1" kind: "ClickHouseInstallation" metadata: name: "ch01" spec: configuration: clusters: - name: replicated layout: shardsCount: 2 replicasCount: 2 zookeeper: nodes: - host: zookeeper.zk Name to identify resource Deﬁnition of cluster Name of service we depend on 11

12. Blast-radius blues 12

13. What is “blast radius?” HostKubernetes Data Center Region 13

14. Host 172.20.49.116 Affinity vs. anti-affinity zookeeper-0 Host 172.20.71.114 zookeeper-1zookeeper-1 Anti- Affinity Affinity 14

15. Pod anti-aﬃnity covers host failures apiVersion: v1 kind: Pod spec: affinity: "podAntiAffinity": { "requiredDuringSchedulingIgnoredDuringExecution": [ { "labelSelector": { "matchLabels": { "clickhouse.altinity.com/chi": "zdist1", } }, "topologyKey": "kubernetes.io/hostname" . . . 15

16. Node aﬃnity + failure domain covers AZ failure apiVersion: v1 kind: Pod spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: failure-domain.beta.kubernetes.io/zone operator: In values: - us-west-2a 16

17. Protect replicas using distance CH PodCH Pod Zookeeper Pod CH PodCH Pod Zookeeper Pod us-west-2a us-west-2b CH PodCH Pod Zookeeper Pod us-west-2c DBMS Copy DBMS CopyBackup Host Object Storage 17

18. Blast protection across regions or K8s (It’s complicated) 18

19. Aﬃnity aﬄictions 19

20. Where are my pods running? kubectl get pod -o=custom-columns=NAME:.metadata.name,STATUS:.status.phase,NODE:.spec.nod eName -n zk NAME STATUS NODE zookeeper-0 Running ip-172-20-49-116.us-west-2.compute.internal zookeeper-1 Running ip-172-20-71-114.us-west-2.compute.internal zookeeper-2 Running ip-172-20-49-116.us-west-2.compute.internal 20

21. Which hosts are in which AZs? kubectl get node -o=custom-columns=NODE:.metadata.name,ZONE:.metadata.labels.'failure-doma in.beta.kubernetes.io/zone' NODE ZONE ip-172-20-47-4.us-west-2.compute.internal us-west-2a ip-172-20-49-116.us-west-2.compute.internal us-west-2a ip-172-20-52-196.us-west-2.compute.internal us-west-2a ip-172-20-70-184.us-west-2.compute.internal us-west-2b ip-172-20-71-114.us-west-2.compute.internal us-west-2b 21

22. 172.20.49.116172.20.47.4 Aﬃnity is useless if you don’t actually use it zookeeper-0 zookeeper-2 172.20.70.184 172.20.71.114 zookeeper-1 us-west-2a us-west-2b 22

23. Searching quickly for missing aﬃnity rules kubectl get pods -o json -n zk | jq -r "[.items[] | {name: .metadata.name, affinity: .spec.affinity}]" [ { "name": "zookeeper-0", "affinity": null }, . . . ] 23

24. The persistent volume that wasn’t 24

25. Kubernetes Node Ephemeral storage is a feature, not a bug! “bad” pod Kubernetes Node “good” pod Network Storage Application Application 25

26. Things to look for in database storage kubectl get pvc -o=custom-columns=NAME:.metadata.name ,SIZE:.spec.resources.request.storage, CLASS:.spec.storageClassName,VOLUME:.spec.volumeName ... NAME SIZE CLASS VOLUME storage...0-0-0 10Gi kops-ssd-1-17 pvc-8f86...7a8 kubectl get storageclass/kops-ssd-1-17 ... NAME PROVISIONER RECLAIMPOLICY.. kops-ssd-1-17 (default) kubernetes.io/aws-ebs Delete 26

27. PVs are not enough if you don’t use them! Kubernetes Node EBS Storage Application “zookeeper” pod /data /datalog /var/lib/zk/data Ephemeral Storage 27

28. Testing to the point of abuse is the solution Check Kubernetes resource deﬁnitions Inspect ﬁle system mounts Kill pods Kill nodes Kill and restart all pods in replicated databases Delete volumes Test with large amounts of data 28

29. Fat ﬁngers of fate 29

30. The best way to lose data: do it yourself helm delete mysql-server --purge 30

31. Fix PV reclaim policy to prevent vaporization kubectl get pv -o=custom-columns=PV:.metadata.name,NAME:.spec.claimRef.name, POLICY:.spec.persistentVolumeReclaimPolicy PV NAME POLICY pvc-3969c6e7-1e79-424e-afdb-be050ba47f08 mysql-server Delete kubectl patch pv pvc-3969c6e7-1e79-424e-afdb-be050ba47f08 -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}' 31

32. Or use a different storage provider... apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: kops-ssd-1-17-retain parameters: encrypted: "true" type: gp2 provisioner: kubernetes.io/aws-ebs reclaimPolicy: Retain volumeBindingMode: WaitForFirstConsumer 32

33. “Orphan” PV can now be reclaimed apiVersion: v1 kind: PersistentVolumeClaim metadata: name: mysql-server namespace: mysql spec: accessModes: - ReadWriteOnce resources: requests: storage: 8Gi storageClassName: "kops-ssd-1-17" volumeMode: Filesystem volumeName: pvc-3969c6e7-1e79-424e-afdb-be050ba47f08 PVC deﬁnition must exactly match volume! 33

34. Steps to reclaim -- Automation anyone? Remove claimRef tag from PV using ‘kubectl edit’ kubectl apply -f reclaim-pvc.yaml helm install --namespace mysql --name mysql-server --set persistence.existingClaim=mysql-server,mysqlRootPassword=secret stable/mysql 34

35. Wrap-up 35

36. How to avoid losing data on Kubernetes Replicas! Indulg i Paranoi ! Distance! Testing! Aﬃnity Rules! Reclaim Policies! Use operators if available 36

37. Thank you! We’re hiring Email: rhodges@altinity.com Code: https://github.com/Altinity/click house-operator Company: https://www.altinity.com 37

Five Great Ways to Lose Data on Kubernetes - KubeCon EU 2020

Altinity Ltd

Five Great Ways to Lose Data on Kubernetes - KubeCon EU 2020