Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Five Great Ways to Lose Data on Kubernetes - KubeCon EU 2020

670 visualizaciones

Publicado el

Slides from Altinity CEO, Robert Hodges, presentation during Kubecon + CloudNativeCon EU Virtual 2020. August 18th, 2020.

Publicado en: Software
  • Inicia sesión para ver los comentarios

Five Great Ways to Lose Data on Kubernetes - KubeCon EU 2020

  1. 1. Five Great Ways to Lose Data on Kubernetes (And How to Avoid Them) Robert Hodges - KubeCon Europe 2020 1
  2. 2. Presenter and Company Bio www.altinity.com Enterprise provider for ClickHouse, a popular, open source data warehouse. Implementors of ClickHouse Kubernetes operator. Robert Hodges - Altinity CEO 30+ years on DBMS plus virtualization and security. Using Kubernetes since 2018. 2
  3. 3. Two boring definitions 3
  4. 4. What do we mean by “data”? Rep SKU Date City Customer Units Price ... 25 #556 25 Nov SFO Wells Fargo 378 $25.00 36 #556 25 Nov SEA Boeing 259 $25.50 55 #558 28 Nov BOS Au Bon Pain 100 $29.33 ... ... ... ... ... ... ... Which products have the best gross margins over time? Do some SKUs sell better on different days of the week? Which kinds of companies are most likely to buy SKU 556? 4
  5. 5. And what do we mean by “lose”? The data loss “arrow of evil” Temporarily unavailable All of it gone, forever Can’t process transactions Unable to see market conditions Business stops functioning 5
  6. 6. The single copy catastrophe 6
  7. 7. Starting a database is easy in Kubernetes (Install helm) kubectl create ns mysql helm install --namespace mysql --name mysql-server stable/mysql 7
  8. 8. Kubernetes Node A delicate flower is our database Container “mysql” Storage Delete pod Delete volume Delete node 8
  9. 9. Kubernetes Node Traditional database solution: replicas Storage Kubernetes Node Storage Replication Container “mysql” Backup DBMS Copy DBMS Copy DBMS Backup Static copies Live replica Primary Replica Container “mysql” 9
  10. 10. Simplest K8s path to replicas: use an operator Kubernetes Operator Single specification Best practice deployment Custom Resource Definition Kubernetes API Native Controller Native Controller Native Controllers Etcd Pod Service StatefulSet 10
  11. 11. Complex systems made simple[r] apiVersion: "clickhouse.altinity.com/v1" kind: "ClickHouseInstallation" metadata: name: "ch01" spec: configuration: clusters: - name: replicated layout: shardsCount: 2 replicasCount: 2 zookeeper: nodes: - host: zookeeper.zk Name to identify resource Definition of cluster Name of service we depend on 11
  12. 12. Blast-radius blues 12
  13. 13. What is “blast radius?” HostKubernetes Data Center Region 13
  14. 14. Host 172.20.49.116 Affinity vs. anti-affinity zookeeper-0 Host 172.20.71.114 zookeeper-1zookeeper-1 Anti- Affinity Affinity 14
  15. 15. Pod anti-affinity covers host failures apiVersion: v1 kind: Pod spec: affinity: "podAntiAffinity": { "requiredDuringSchedulingIgnoredDuringExecution": [ { "labelSelector": { "matchLabels": { "clickhouse.altinity.com/chi": "zdist1", } }, "topologyKey": "kubernetes.io/hostname" . . . 15
  16. 16. Node affinity + failure domain covers AZ failure apiVersion: v1 kind: Pod spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: failure-domain.beta.kubernetes.io/zone operator: In values: - us-west-2a 16
  17. 17. Protect replicas using distance CH PodCH Pod Zookeeper Pod CH PodCH Pod Zookeeper Pod us-west-2a us-west-2b CH PodCH Pod Zookeeper Pod us-west-2c DBMS Copy DBMS CopyBackup Host Object Storage 17
  18. 18. Blast protection across regions or K8s (It’s complicated) 18
  19. 19. Affinity afflictions 19
  20. 20. Where are my pods running? kubectl get pod -o=custom-columns=NAME:.metadata.name,STATUS:.status.phase,NODE:.spec.nod eName -n zk NAME STATUS NODE zookeeper-0 Running ip-172-20-49-116.us-west-2.compute.internal zookeeper-1 Running ip-172-20-71-114.us-west-2.compute.internal zookeeper-2 Running ip-172-20-49-116.us-west-2.compute.internal 20
  21. 21. Which hosts are in which AZs? kubectl get node -o=custom-columns=NODE:.metadata.name,ZONE:.metadata.labels.'failure-doma in.beta.kubernetes.io/zone' NODE ZONE ip-172-20-47-4.us-west-2.compute.internal us-west-2a ip-172-20-49-116.us-west-2.compute.internal us-west-2a ip-172-20-52-196.us-west-2.compute.internal us-west-2a ip-172-20-70-184.us-west-2.compute.internal us-west-2b ip-172-20-71-114.us-west-2.compute.internal us-west-2b 21
  22. 22. 172.20.49.116172.20.47.4 Affinity is useless if you don’t actually use it zookeeper-0 zookeeper-2 172.20.70.184 172.20.71.114 zookeeper-1 us-west-2a us-west-2b 22
  23. 23. Searching quickly for missing affinity rules kubectl get pods -o json -n zk | jq -r "[.items[] | {name: .metadata.name, affinity: .spec.affinity}]" [ { "name": "zookeeper-0", "affinity": null }, . . . ] 23
  24. 24. The persistent volume that wasn’t 24
  25. 25. Kubernetes Node Ephemeral storage is a feature, not a bug! “bad” pod Kubernetes Node “good” pod Network Storage Application Application 25
  26. 26. Things to look for in database storage kubectl get pvc -o=custom-columns=NAME:.metadata.name ,SIZE:.spec.resources.request.storage, CLASS:.spec.storageClassName,VOLUME:.spec.volumeName ... NAME SIZE CLASS VOLUME storage...0-0-0 10Gi kops-ssd-1-17 pvc-8f86...7a8 kubectl get storageclass/kops-ssd-1-17 ... NAME PROVISIONER RECLAIMPOLICY.. kops-ssd-1-17 (default) kubernetes.io/aws-ebs Delete 26
  27. 27. PVs are not enough if you don’t use them! Kubernetes Node EBS Storage Application “zookeeper” pod /data /datalog /var/lib/zk/data Ephemeral Storage 27
  28. 28. Testing to the point of abuse is the solution Check Kubernetes resource definitions Inspect file system mounts Kill pods Kill nodes Kill and restart all pods in replicated databases Delete volumes Test with large amounts of data 28
  29. 29. Fat fingers of fate 29
  30. 30. The best way to lose data: do it yourself helm delete mysql-server --purge 30
  31. 31. Fix PV reclaim policy to prevent vaporization kubectl get pv -o=custom-columns=PV:.metadata.name,NAME:.spec.claimRef.name, POLICY:.spec.persistentVolumeReclaimPolicy PV NAME POLICY pvc-3969c6e7-1e79-424e-afdb-be050ba47f08 mysql-server Delete kubectl patch pv pvc-3969c6e7-1e79-424e-afdb-be050ba47f08 -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}' 31
  32. 32. Or use a different storage provider... apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: kops-ssd-1-17-retain parameters: encrypted: "true" type: gp2 provisioner: kubernetes.io/aws-ebs reclaimPolicy: Retain volumeBindingMode: WaitForFirstConsumer 32
  33. 33. “Orphan” PV can now be reclaimed apiVersion: v1 kind: PersistentVolumeClaim metadata: name: mysql-server namespace: mysql spec: accessModes: - ReadWriteOnce resources: requests: storage: 8Gi storageClassName: "kops-ssd-1-17" volumeMode: Filesystem volumeName: pvc-3969c6e7-1e79-424e-afdb-be050ba47f08 PVC definition must exactly match volume! 33
  34. 34. Steps to reclaim -- Automation anyone? Remove claimRef tag from PV using ‘kubectl edit’ kubectl apply -f reclaim-pvc.yaml helm install --namespace mysql --name mysql-server --set persistence.existingClaim=mysql-server,mysqlRootPassword=secret stable/mysql 34
  35. 35. Wrap-up 35
  36. 36. How to avoid losing data on Kubernetes Replicas! Indulg i Paranoi ! Distance! Testing! Affinity Rules! Reclaim Policies! Use operators if available 36
  37. 37. Thank you! We’re hiring Email: rhodges@altinity.com Code: https://github.com/Altinity/click house-operator Company: https://www.altinity.com 37

×