Speaker: Spencer Brown, Solutions Engineer, Datadog

1. Autoscaling Your
Kubernetes Workloads

2. Autoscaling Your
Kubernetes Workloads
Spencer Brown
Solutions Engineer
Datadog

3. Solutions Engineer for
Cloud Integrations

4. @datadoghq
SaaS-based monitoring
250+ integrations
Trillions of data points per day
https://datadoghq.com

5. “I think there is a world market for maybe
five computers.”
– Thomas Watson, IBM

6. “I think there is a world market for maybe
five computers.”
– Thomas Watson, IBM, Wildly
misquoted

17. Why the HPA is a Big Deal

18. History of Kubernetes HPA
● K8S v1.2: autoscaling based off basic metrics within
cluster (e.g. CPU util.)
● K8S v1.6: autoscaling on custom metrics within
cluster
● K8S v1.10: autoscale based on ANY metric, including
external

19. History of Kubernetes HPA
● K8S v1.2: autoscaling based off basic metrics within
cluster (e.g. CPU util.)
● K8S v1.6: autoscaling on custom metrics within
cluster
● K8S v1.10: autoscale based on ANY metric, including
external

20. History of Kubernetes HPA
● K8S v1.2: autoscaling based off basic metrics within
cluster (e.g. CPU util.)
● K8S v1.6: autoscaling on custom metrics within
cluster
● K8S v1.10: autoscale based on ANY metric, including
external

21. Choosing the Autoscaling Metric

23. Recurse until you find the cause

27. Implementing Scaling with a Custom
Metric

29. Prerequisites
❏ Kubernetes running v1.10 or higher
❏ Enable the aggregation layer: https://kubernetes.io/docs/tasks/access-
kubernetes-api/configure-aggregation-layer/
❏ Datadog account (free trials available)
❏ K8S nodes running Datadog Agent (ideally as a DaemonSet), with
Autodiscovery enabled
❏ Agents are configured to securely communicate with Cluster Agent (not
mandatory, but has more enriched data).
https://github.com/DataDog/datadog-agent/blob/master/Dockerfiles/cluster-
agent/README.md#security-premise

30. Spin up the Datadog Cluster Agent (RBAC)
kubectl apply -f manifests/cluster-agent/rbac/rbac-cluster-
agent.yaml
clusterrole.rbac.authorization.k8s.io "dca" created
clusterrolebinding.rbac.authorization.k8s.io "dca" created
serviceaccount "dca" created

31. Spin up the Datadog Cluster Agent (RBAC) output
kubectl apply -f manifests/cluster-agent/rbac/rbac-cluster-
agent.yaml
clusterrole.rbac.authorization.k8s.io "dca" created
clusterrolebinding.rbac.authorization.k8s.io "dca" created
serviceaccount "dca" created

32. Create Datadog Cluster Agent & Services
In the Cluster Agent’s deployment manifest (cluster-agent.yaml)
add Datadog <API_KEY> and <APP_KEY>,
and set DD_EXTERNAL_METRICS_PROVIDER_ENABLED to true.
kubectl apply -f manifests/cluster-agent/cluster-agent.yaml
kubectl apply -f manifests/cluster-agent/datadog-cluster-
agent_service.yaml
kubectl apply -f manifests/cluster-agent/hpa-example/cluster-
agent-hpa-svc.yaml

34. Create Datadog Cluster Agent & Services
In the Cluster Agent’s deployment manifest (cluster-agent.yaml)
add Datadog <API_KEY> and <APP_KEY>,
and set DD_EXTERNAL_METRICS_PROVIDER_ENABLED to true.
kubectl apply -f manifests/cluster-agent/cluster-agent.yaml
kubectl apply -f manifests/cluster-agent/datadog-cluster-
agent_service.yaml
kubectl apply -f manifests/cluster-agent/hpa-example/cluster-
agent-hpa-svc.yaml

35. Create Datadog Cluster Agent & Services
In the Cluster Agent’s deployment manifest (cluster-agent.yaml)
add Datadog <API_KEY> and <APP_KEY>,
and set DD_EXTERNAL_METRICS_PROVIDER_ENABLED to true.
kubectl apply -f manifests/cluster-agent/cluster-agent.yaml
kubectl apply -f manifests/cluster-agent/datadog-cluster-
agent_service.yaml
kubectl apply -f manifests/cluster-agent/hpa-example/cluster-
agent-hpa-svc.yaml

36. Services in Kubernetes
Services == Load Balancer
Service == VIP

37. Verifying Cluster Agent
kubectl get pods, svc -l app=datadog-cluster-agent
PODS:
NAMESPACE NAME READY STATUS RESTARTS AGE
default datadog-cluster-agent-7b7f6d5547-cmdtc 1/1 Running 0 28m
SVCS:
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP
PORT(S) AGE
default datadog-custom-metrics-server ClusterIP 192.168.254.87 <none>
443/TCP 28m
default datadog-cluster-agent ClusterIP 192.168.254.197 <none>
5005/TCP 28m

38. Verifying Cluster Agent Output
kubectl get pods, svc -l app=datadog-cluster-agent
PODS:
NAMESPACE NAME READY STATUS RESTARTS AGE
default datadog-cluster-agent-7b7f6d5547-cmdtc 1/1 Running 0 28m
SVCS:
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP
PORT(S) AGE
default datadog-custom-metrics-server ClusterIP 192.168.254.87 <none>
443/TCP 28m
default datadog-cluster-agent ClusterIP 192.168.254.197 <none>
5005/TCP 28m

39. Register the External Metrics Provider
kubectl apply -f manifests/cluster-agent/hpa-example/rbac-hpa.yaml
clusterrolebinding.rbac.authorization.k8s.io "system:auth-delegator" created
rolebinding.rbac.authorization.k8s.io "dca" created
apiservice.apiregistration.k8s.io "v1beta1.external.metrics.k8s.io" created
clusterrole.rbac.authorization.k8s.io "external-metrics-reader" created
clusterrolebinding.rbac.authorization.k8s.io "external-metrics-reader" created

40. Register the External Metrics Provider Output
kubectl apply -f manifests/cluster-agent/hpa-example/rbac-hpa.yaml
clusterrolebinding.rbac.authorization.k8s.io "system:auth-delegator" created
rolebinding.rbac.authorization.k8s.io "dca" created
apiservice.apiregistration.k8s.io "v1beta1.external.metrics.k8s.io" created
clusterrole.rbac.authorization.k8s.io "external-metrics-reader" created
clusterrolebinding.rbac.authorization.k8s.io "external-metrics-reader" created

41. Datadog Cluster Agent running
kubectl get pods,svc
PODS
NAMESPACE NAME READY STATUS RESTARTS AGE
default datadog-agent-4c5pp 1/1 Running 0 14m
default datadog-agent-ww2da 1/1 Running 0 14m
default datadog-agent-2qqd3 1/1 Running 0 14m
[...]
default datadog-cluster-agent-7b7f6d5547-cmdtc 1/1 Running 0 16m

42. Example HPA Manifest
● is configured to scale a deployment called 'nginx'
● set a maximum number of replicas to create to 3, with a minimum of 1
● HPA triggers autoscaling off of the Datadog metric
'nginx.net.request_per_sec' over the scope
'kube_container_name: nginx'

43. Example HPA Manifest
● is configured to scale a deployment called 'nginx'
● set a maximum number of replicas to create to 3, with a minimum of 1
● HPA triggers autoscaling off of the Datadog metric
'nginx.net.request_per_sec' over the scope
'kube_container_name: nginx'

46. Creating an Autoscaling Deployment
# Nginx deployment
kubectl apply -f manifests/cluster-agent/hpa-
example/nginx.yaml
# HPA manifest
kubectl apply -f manifests/cluster-agent/hpa-example/hpa-
manifest.yaml
# Verify
kubectl get pods,svc

48. Creating an Autoscaling Deployment
# Nginx deployment
kubectl apply -f manifests/cluster-agent/hpa-
example/nginx.yaml
# HPA manifest
kubectl apply -f manifests/cluster-agent/hpa-example/hpa-
manifest.yaml
# Verify
kubectl get pods,svc

49. Creating an Autoscaling Deployment
# Nginx deployment
kubectl apply -f manifests/cluster-agent/hpa-
example/nginx.yaml
# HPA manifest
kubectl apply -f manifests/cluster-agent/hpa-example/hpa-
manifest.yaml
# Verify
kubectl get pods,svc

50. POD:
default nginx-6757dd8769-5xzp2 1/1 Running 0 3m
SVC:
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default nginx ClusterIP 192.168.251.36 none 8090/TCP 3m
HPAS:
NAMESPACE NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS
AGE
default nginxext Deployment/nginx 0/9 (avg) 1 3 1
3m

51. curl <NGINX_SVC>:8090/nginx_status
while true; do curl <NGINX_SVC>:8090/nginx_status; sleep 0.1;
done
Let’s Stress it out

52. +

53. kubectl get pods,svc
PODS:
NAMESPACE NAME READY STATUS RESTARTS AGE
default datadog-cluster-agent-7b7f6d5547-cmdtc 1/1 Running 0 9m
default nginx-6757dd8769-5xzp2 1/1 Running 0 2m
default nginx-6757dd8769-k6h6x 1/1 Running 0 2m
default nginx-6757dd8769-vzd5b 1/1 Running 0 29m
HPAS:
NAMESPACE NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
default nginxext Deployment/nginx 30/9 (avg) 1 3 3 29m

56. function yamllint() {
for i in $(find . -name '*.yml' -o -name '*.yaml');
do echo $i; ruby -e "require
'yaml';YAML.load_file("$i")"; done
}

57. In Conclusion

58. Datadog Cluster Agent: https://github.com/DataDog/datadog-agent/tree/master/Dockerfiles/manifests/cluster-agent (includes an
example)
HPA blog: https://www.datadoghq.com/blog/autoscale-kubernetes-datadog/
DCA blog: https://www.datadoghq.com/blog/datadog-cluster-agent/
HPA Docs: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/
Enable the Kubernetes Aggregation Layer: https://kubernetes.io/docs/tasks/access-kubernetes-api/configure-aggregation-layer/
Configure Agents to communicate securely with Cluster Agent: https://github.com/DataDog/datadog-
agent/blob/master/Dockerfiles/cluster-agent/README.md#security-premise
Datadog Cluster Agent Deployment Manifest: https://github.com/DataDog/datadog-agent/blob/master/Dockerfiles/manifests/cluster-
agent/cluster-agent.yaml
Datadog Example HPA: https://github.com/DataDog/datadog-agent/blob/master/Dockerfiles/manifests/cluster-agent/hpa-example/hpa-
manifest.yaml
Resources