Kubernetes has become the de facto Container orchestration platform over the last few years. For all the power and the scale that Kubernetes provides, it's still a complex platform to configure and manage. Managed Kubernetes service like GKE takes a lot of the pain out in managing Kubernetes. When we are using Kubernetes in production, it makes sense to follow the best practises around distributed system development targeted towards Kubernetes. In this session, I will talk about Kubernetes and GKE best practices around infrastructure, security and applications with specific focus on day-2 operations.
Best practises for using Kubernetes and GKE in production
1. Best practises for using
Kubernetes in production
Sreenivas Makam
August 2, 2020
2. About myself
● Application modernization specialist at Google Cloud. Previously at Cisco
and few startups
● Interest areas - Containers, Kubernetes, Networking, Cloud native
technologies
● Author of “Mastering CoreOS”, published 2016. Reviewed many technology
books
● Docker Captain from Oct 15 - Mar 18
● Active blogger and Community speaker
3. Agenda
● What is Kubernetes?
● What makes Kubernetes unique?
● Kubernetes day 2 operations
○ Cluster Management
○ Application Design
○ Security
● Kubernetes ecosystem and tools
5. What is Kubernetes
Kubernetes is a portable, extensible,
open-source platform for managing
containerized workloads and services,
that facilitates both declarative
configuration and automation
https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/
9. Kubernetes Uniqueness
● Declarative rather than imperative
● Extensible - custom resource, controllers, schedulers
● Meet the user where they are - (eg) read config, secrets from
applications
● Decouple distributed system application development
● Open source ecosystem friendly
13. Kubernetes operator
(eg: prometheus, etcd,
Spark, Airflow)
API server
Custom resources
Desired
State
Current
State
Operators manage the lifecycle of the custom application
Extensions -Kubernetes Operator
17. Init container
(Clone git repo and
generate config)
App container
(Web server)
Pod
Execution sequence
Specialized containers that runs to completion before application containers in a pod can
get started. This enforces sequence.
Pod patterns -Init Containers
18. Pod patterns -Sidecar
Sidecar containers extend and enhance the “main” container
Other examples:
Istio envoy proxy
Monitoring
Database config
19. Pod patterns -Adapter
Adapter containers standardize and normalize output so that external services can access
interface in a standard way(eg: Prometheus adapter)
20. Pod patterns - Ambassador
Ambassador containers proxies a local connection to the world and hides the complexity to
access external service.
Examples:
Accessing different kinds of
cache based on environment
Client side service discovery
using different mechanisms
21. Single app defined using Dockerfile and
multiple apps done using deployment
Config map and secrets
Service abstraction
and discovery
Stateless containers, stateful
dataset where needed
Services provides
different options for
port bindings
Autoscaler support
is comprehensive
Centralized log management with
third party integrations possible
Autohealing
Many ways to create and
manage clusters(cloud
provider, kops, kubeadm)
Map Twelve factor apps to Kubernetes
25. Use Namespaces and RBAC for isolation
https://cloud.google.com/solutions/prep-kubernetes-engine-for-prod
26. Multicluster handling
Need for multiple clusters - Different applications, teams, environments, regions
Central policy management using
Anthos config management(ACM)
Proximity based cluster routing
27. 4 wayAutoscaling in GKE
HPA
Autoscales pool of workers on
custom metrics
VPA
Recommends podspec
Actuates the adjustment
CA NAP+
Scale Nodepools
Create right nodes for the job
Gate changes by HPA + NAP
Workload
Infrastructure
29. Readiness and Liveness probe
https://cloud.google.com/blog/products/gcp/k
ubernetes-best-practices-setting-up-health-ch
ecks-with-readiness-and-liveness-probes
30. Graceful shutdown handling
Best practises
● Have handler for Prestop hooks or
SIGTERM and handle shutdown
gracefully
● Keep readiness check interval
aggressive
● Have client retry failed requests
https://dzone.com/articles/kubernetes-lifecycle-of-a-pod
31. Big Data on Kubernetes
● Kubernetes as a replacement for YARN for Big data workloads
● Spark and Flink operators for Kubernetes are available as beta
● Dataproc on GKE is available as beta
● Advantages
○ Single orchestrator for applications and Big Data
○ Better use of cluster resources
○ Big Data application dependency handled through containers
○ Use Kubernetes ecosystem for Big Data
32. MLworkloads with Kubeflow
● Deploying and managing ML models at scale using Kubernetes
● Build, train and serve models
● Components - Notebooks, UI, training, Serving, Pipelines
● Multiple frameworks supported for training as well as serving
● Advantages
○ Portable ML pipelines
○ Best of Kubernetes features used for Machine learning
36. Kubernetes Network policies
Topology Defined with Network Policy API:
● A first-class Kubernetes API
● Defines allowed traffic patterns
How does it work:
● K8s defines the API.
● User applies a policy.
● Network policy agents watch and enforce.
● Restricts pod to pod traffic.
K8sMaster
etcd
policy.yaml
spec:
podSelector:
matchLabels:
run: nginx
ingress:
- from:
- podSelector:
matchLabels:
run:
access
Frontend
Service A
Service B
Policy Agent
K8sNode
40. Practical Experiences
● Networking and security for clusters has to be pre-planned. These cannot be
changed later.
● Plan IP addresses before-hand. Kubernetes needs lot of addresses(Node, Pod,
Service)
● Use managed services when possible
● Keep separate environments for Dev, staging and production
● Isolate helper applications(CI/CD, Monitoring) from primary workloads
● Start with stateless workloads and then expand to stateful, big data and ML
● Invest in monitoring/logging/secret management solution
● Backup and DR is important for Kubernetes
● Make sure that every container has resource requests
41. References
● Kubernetes design principles video
● Kubernetes patterns video
● Kubernetes patterns slides
● Building Cloud native applications with Kubernetes and Istio - Kelsey
● Designing cloud native applications
● Extending Kubernetes