Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Automatically Scaling Your Kubernetes Workloads - SVC209-S - Anaheim AWS Summit

840 visualizaciones

Publicado el

As our need for more computing resources has accelerated, so too have the ways in which computing has evolved. The cloud has enabled us to easily scale to suit our needs. To keep pace, we need more automated way to scale our infrastructure. In this session, we discuss automatic scaling with Kubernetes, how to set it up, and—most importantly—what to monitor in order to drive your automatic scaling. This session is brought to you by AWS partner, Datadog.

  • Sé el primero en comentar

Automatically Scaling Your Kubernetes Workloads - SVC209-S - Anaheim AWS Summit

  1. 1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Automatically Scaling Your Kubernetes Workloads H. “Waldo” Grunenwald Tech Evangelist Datadog S V C 2 0 9 - S
  2. 2. Automatically Scaling Your Kubernetes Workloads
  3. 3. Automatically Scaling Your Kubernetes Workloads H. “Waldo” Grunenwald Technical Evangelist Datadog
  4. 4. @gwaldo Tech Evangelist & Writer “Docs & Talks” Recovering Ops Engineer
  5. 5. @datadoghq SaaS-based monitoring 250+ integrations Trillions of data points per day https://datadoghq.com
  6. 6. “I think there is a world market for maybe five computers.” – Thomas Watson, IBM
  7. 7. “I think there is a world market for maybe five computers.” – Thomas Watson, IBM, Wildly misquoted
  8. 8. Why the HPA is a Big Deal
  9. 9. Why the HPA is a Big Deal
  10. 10. History of Kubernetes HPA ● K8S v1.2: autoscaling based off basic metrics within cluster (e.g. CPU util.) ● K8S v1.6: autoscaling on custom metrics within cluster ● K8S v1.10: autoscale based on ANY metric, including external
  11. 11. History of Kubernetes HPA ● K8S v1.2: autoscaling based off basic metrics within cluster (e.g. CPU util.) ● K8S v1.6: autoscaling on custom metrics within cluster ● K8S v1.10: autoscale based on ANY metric, including external
  12. 12. History of Kubernetes HPA ● K8S v1.2: autoscaling based off basic metrics within cluster (e.g. CPU util.) ● K8S v1.6: autoscaling on custom metrics within cluster ● K8S v1.10: autoscale based on ANY metric, including external
  13. 13. Choosing the Autoscaling Metric
  14. 14. Recurse until you find the cause
  15. 15. Recurse until you find the cause
  16. 16. Implementing Autoscaling with a Custom Metric
  17. 17. Implementing Autoscaling with a Custom Metric
  18. 18. Prerequisites ❏ Kubernetes running v1.10 or higher ❏ Enable the aggregation layer: https://kubernetes.io/docs/tasks/access- kubernetes-api/configure-aggregation-layer/ ❏ Datadog account (free trials available) ❏ K8S nodes running Datadog Agent (ideally as a DaemonSet), with Autodiscovery enabled ❏ Agents are configured to securely communicate with Cluster Agent (not mandatory, but has more enriched data). https://github.com/DataDog/datadog-agent/blob/master/Dockerfiles/cluster- agent/README.md#security-premise
  19. 19. Spin up the Datadog Cluster Agent (RBAC) kubectl apply -f manifests/cluster-agent/rbac/rbac-cluster- agent.yaml clusterrole.rbac.authorization.k8s.io "dca" created clusterrolebinding.rbac.authorization.k8s.io "dca" created serviceaccount "dca" created
  20. 20. Spin up the Datadog Cluster Agent (RBAC) output kubectl apply -f manifests/cluster-agent/rbac/rbac-cluster- agent.yaml clusterrole.rbac.authorization.k8s.io "dca" created clusterrolebinding.rbac.authorization.k8s.io "dca" created serviceaccount "dca" created
  21. 21. Create Datadog Cluster Agent & Services In the Cluster Agent’s deployment manifest (cluster-agent.yaml) add Datadog <API_KEY> and <APP_KEY>, and set DD_EXTERNAL_METRICS_PROVIDER_ENABLED to true. kubectl apply -f manifests/cluster-agent/cluster-agent.yaml kubectl apply -f manifests/cluster-agent/datadog-cluster- agent_service.yaml kubectl apply -f manifests/cluster-agent/hpa-example/cluster- agent-hpa-svc.yaml
  22. 22. Create Datadog Cluster Agent & Services In the Cluster Agent’s deployment manifest (cluster-agent.yaml) add Datadog <API_KEY> and <APP_KEY>, and set DD_EXTERNAL_METRICS_PROVIDER_ENABLED to true. kubectl apply -f manifests/cluster-agent/cluster-agent.yaml kubectl apply -f manifests/cluster-agent/datadog-cluster- agent_service.yaml kubectl apply -f manifests/cluster-agent/hpa-example/cluster- agent-hpa-svc.yaml
  23. 23. Create Datadog Cluster Agent & Services In the Cluster Agent’s deployment manifest (cluster-agent.yaml) add Datadog <API_KEY> and <APP_KEY>, and set DD_EXTERNAL_METRICS_PROVIDER_ENABLED to true. kubectl apply -f manifests/cluster-agent/cluster-agent.yaml kubectl apply -f manifests/cluster-agent/datadog-cluster- agent_service.yaml kubectl apply -f manifests/cluster-agent/hpa-example/cluster- agent-hpa-svc.yaml
  24. 24. Services in Kubernetes Services == Load Balancer Service == VIP
  25. 25. Verifying Cluster Agent kubectl get pods, svc -l app=datadog-cluster-agent PODS: NAMESPACE NAME READY STATUS RESTARTS AGE default datadog-cluster-agent-7b7f6d5547-cmdtc 1/1 Running 0 28m SVCS: NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default datadog-custom-metrics-server ClusterIP 192.168.254.87 <none> 443/TCP 28m default datadog-cluster-agent ClusterIP 192.168.254.197 <none> 5005/TCP 28m
  26. 26. Verifying Cluster Agent Output kubectl get pods, svc -l app=datadog-cluster-agent PODS: NAMESPACE NAME READY STATUS RESTARTS AGE default datadog-cluster-agent-7b7f6d5547-cmdtc 1/1 Running 0 28m SVCS: NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default datadog-custom-metrics-server ClusterIP 192.168.254.87 <none> 443/TCP 28m default datadog-cluster-agent ClusterIP 192.168.254.197 <none> 5005/TCP 28m
  27. 27. Register the External Metrics Provider kubectl apply -f manifests/cluster-agent/hpa-example/rbac-hpa.yaml clusterrolebinding.rbac.authorization.k8s.io "system:auth-delegator" created rolebinding.rbac.authorization.k8s.io "dca" created apiservice.apiregistration.k8s.io "v1beta1.external.metrics.k8s.io" created clusterrole.rbac.authorization.k8s.io "external-metrics-reader" created clusterrolebinding.rbac.authorization.k8s.io "external-metrics-reader" created
  28. 28. Register the External Metrics Provider Output kubectl apply -f manifests/cluster-agent/hpa-example/rbac-hpa.yaml clusterrolebinding.rbac.authorization.k8s.io "system:auth-delegator" created rolebinding.rbac.authorization.k8s.io "dca" created apiservice.apiregistration.k8s.io "v1beta1.external.metrics.k8s.io" created clusterrole.rbac.authorization.k8s.io "external-metrics-reader" created clusterrolebinding.rbac.authorization.k8s.io "external-metrics-reader" created
  29. 29. Datadog Cluster Agent running kubectl get pods,svc PODS NAMESPACE NAME READY STATUS RESTARTS AGE default datadog-agent-4c5pp 1/1 Running 0 14m default datadog-agent-ww2da 1/1 Running 0 14m default datadog-agent-2qqd3 1/1 Running 0 14m [...] default datadog-cluster-agent-7b7f6d5547-cmdtc 1/1 Running 0 16m
  30. 30. Example HPA Manifest ● is configured to scale a deployment called 'nginx' ● set a maximum number of replicas to create to 3, with a minimum of 1 ● HPA triggers autoscaling off of the Datadog metric 'nginx.net.request_per_sec' over the scope 'kube_container_name: nginx'
  31. 31. Example HPA Manifest ● is configured to scale a deployment called 'nginx' ● set a maximum number of replicas to create to 3, with a minimum of 1 ● HPA triggers autoscaling off of the Datadog metric 'nginx.net.request_per_sec' over the scope 'kube_container_name: nginx'
  32. 32. Creating an Autoscaling Deployment # Nginx deployment kubectl apply -f manifests/cluster-agent/hpa- example/nginx.yaml # HPA manifest kubectl apply -f manifests/cluster-agent/hpa-example/hpa- manifest.yaml # Verify kubectl get pods,svc
  33. 33. Creating an Autoscaling Deployment # Nginx deployment kubectl apply -f manifests/cluster-agent/hpa- example/nginx.yaml # HPA manifest kubectl apply -f manifests/cluster-agent/hpa-example/hpa- manifest.yaml # Verify kubectl get pods,svc
  34. 34. Creating an Autoscaling Deployment # Nginx deployment kubectl apply -f manifests/cluster-agent/hpa- example/nginx.yaml # HPA manifest kubectl apply -f manifests/cluster-agent/hpa-example/hpa- manifest.yaml # Verify kubectl get pods,svc
  35. 35. POD: default nginx-6757dd8769-5xzp2 1/1 Running 0 3m SVC: NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default nginx ClusterIP 192.168.251.36 none 8090/TCP 3m HPAS: NAMESPACE NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE default nginxext Deployment/nginx 0/9 (avg) 1 3 1 3m
  36. 36. curl <NGINX_SVC>:8090/nginx_status while true; do curl <NGINX_SVC>:8090/nginx_status; sleep 0.1; done Let’s Stress it out
  37. 37. +
  38. 38. +
  39. 39. kubectl get pods,svc PODS: NAMESPACE NAME READY STATUS RESTARTS AGE default datadog-cluster-agent-7b7f6d5547-cmdtc 1/1 Running 0 9m default nginx-6757dd8769-5xzp2 1/1 Running 0 2m default nginx-6757dd8769-k6h6x 1/1 Running 0 2m default nginx-6757dd8769-vzd5b 1/1 Running 0 29m HPAS: NAMESPACE NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE default nginxext Deployment/nginx 30/9 (avg) 1 3 3 29m
  40. 40. function yamllint() { for i in $(find . -name '*.yml' -o -name '*.yaml'); do echo $i; ruby -e "require 'yaml';YAML.load_file("$i")"; done }
  41. 41. In Conclusion ● Walked through a brief history of scaling and deploying software ● Described some of the key components and history of the HPA ● How to choose a metric on which to base autoscaling ● Think of autoscaling as an output for your monitoring ● Walk-through implementing Autoscaling with a custom provider ● How a simple linting script can save you a lot of time, effort, and embarrassment.
  42. 42. In Conclusion ● Walked through a brief history of scaling and deploying software ● Described some of the key components and history of the HPA ● How to choose a metric on which to base autoscaling ● Think of autoscaling as an output for your monitoring ● Walk-through implementing Autoscaling with a custom provider ● How a simple linting script can save you a lot of time, effort, and embarrassment.
  43. 43. Datadog Cluster Agent: https://github.com/DataDog/datadog-agent/tree/master/Dockerfiles/manifests/cluster-agent (includes an example) HPA blog: https://www.datadoghq.com/blog/autoscale-kubernetes-datadog/ DCA blog: https://www.datadoghq.com/blog/datadog-cluster-agent/ HPA Docs: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/ Enable the Kubernetes Aggregation Layer: https://kubernetes.io/docs/tasks/access-kubernetes-api/configure-aggregation-layer/ Configure Agents to communicate securely with Cluster Agent: https://github.com/DataDog/datadog- agent/blob/master/Dockerfiles/cluster-agent/README.md#security-premise Datadog Cluster Agent Deployment Manifest: https://github.com/DataDog/datadog-agent/blob/master/Dockerfiles/manifests/cluster- agent/cluster-agent.yaml Datadog Example HPA: https://github.com/DataDog/datadog-agent/blob/master/Dockerfiles/manifests/cluster-agent/hpa-example/hpa- manifest.yaml Resources
  44. 44. Automatically Scaling Your Kubernetes Workloads
  45. 45. Visit Our Booth
  46. 46. Thank you! S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. H. “Waldo” Grunenwald waldo@datadoghq.com @gwaldo

×