5. Container Camp London // @craigboxGoogle Cloud Platform 5
● Scheduling: Decide where my containers should run
● Lifecycle and health: Keep my containers running despite
failures
● Scaling: Make sets of containers bigger or smaller
● Naming and discovery: Find where my containers are now
● Load balancing: Distribute traffic across a set of containers
● Storage volumes: Provide data to containers
● Logging and monitoring: Track what’s happening with my
containers
● Debugging and introspection: Enter or attach to containers
● Identity and authorization: Control who can do things to my
containers
Kubernetes: a quick recap
6. Container Camp London // @craigboxGoogle Cloud Platform 6
● Kubernetes is one of the three legs of Cloud Native
○ Takes in container packaged apps
○ Emits microservices architectures
● Announced June 2014, in GA since June 2015
● 1.4 is due out in 1 week!
● Under half the code is now written by Google
● Stewarded by the Cloud Native Compute Foundation™
○ A Linux Foundation Collaborative Project™
Kubernetes: a quick recap
7. Container Camp London // @craigboxGoogle Cloud Platform 7
Kubernetes is stable
● Concrete ideas from 10 years of production experience
○ and mistakes!
● v1 API; breaking changes held until v2
● Alpha, Beta and GA tracks for new features
● Thorough end-to-end testing
● New work taking place outside of core
○ Volume & network plugins
○ Custom controllers
○ ThirdPartyResources
8. Container Camp London // @craigboxGoogle Cloud Platform 8
Kubernetes has a solid core
● Core primitives:
○ pods, services, volumes, labels, controllers, etc
● Continual improvement using these basic concepts
○ Ingress: connect a load balancer to a Service
○ ReplicaSet: fungible replicas
○ DaemonSet: put a pod on every node
○ Job: batch workloads
○ ScheduledJob: run a Job at a certain time (cron)
10. Container Camp London // @craigboxGoogle Cloud Platform 10
Kubernetes has great momentum
11. To host a similar set of services on our older Openstack environment
would require at least 2-3x the number of servers. The cost savings
isn't even the best part. Kubernetes has allowed us to build a
completely self-service pipeline for our devs and has taken the ops
team out of day-to-day app management. The nodes update
themselves with the latest OS and Kube shifts the workload around
as they do. This infrastructure is faster, more nimble, more
cost-effective and so much easier to run.
This is the best infrastructure I've ever used in twenty years
of doing ops and leading ops teams.
12. Since we started using kubernetes, we reduced our bill to 30%
of its original price, and it made everything easier and scalable
just as if we were using the costly [alternative]
14. laptop$ kubeadm --help
kubeadm: bootstrap a secure Kubernetes cluster easily.
/==========================================================
| KUBEADM IS ALPHA, DO NOT USE IT FOR PRODUCTION CLUSTERS! |
| |
| But, please try it out! Give us feedback at: |
| https://github.com/kubernetes/kubernetes/issues |
| and at-mention @kubernetes/sig-cluster-lifecycle |
==========================================================/
Example usage:
Create a two-machine cluster with one master (which controls the cluster),
and one node (where workloads, like pods and containers run).
On the first machine
====================
master# kubeadm init master
Your token is: <token>
On the second machine
=====================
node# kubeadm join node --token=<token> <ip-of-master>
16. Container Camp London // @craigboxGoogle Cloud Platform 16
The problems to be solved
1. Preserve individual identity for fungible entities
2. Provide predictable ordering and control as those entities change
3. To enable the software entities to identify and recognize the other entities
by those identities
4. To get access to a consistent storage mechanism (because their identity
also corresponds to data)
17. Container Camp London // @craigboxGoogle Cloud Platform 17
I apologise in advance for this horrible metaphor
18. Container Camp London // @craigboxGoogle Cloud Platform 18
Pets vs Cattle
It's so horrible it can only be written in Comic Sans
27. Container Camp London // @craigboxGoogle Cloud Platform 27
(In fairness, I can tell you the UDP joke, but I can't guarantee you will get it)
28. Container Camp London // @craigboxGoogle Cloud Platform 28
What is a pet?
A Pet Set ensures that a specified number of “pets” with unique identities are
running at any given time.
The identity of a Pet is comprised of:
● a stable hostname, available in DNS
● an ordinal index
● stable storage: linked to the ordinal & hostname
33. Container Camp London // @craigboxGoogle Cloud Platform 33
ReplicaSets
web-7ci7o
web-kzszj
web-qqcnn
Master
web-khku8
web-nacti
web-z9gth
at least i'm not
passing the butter
lol jk, 2 is plenty
34. Container Camp London // @craigboxGoogle Cloud Platform 34
ReplicaSets
web-7ci7o
Master
web-z9gth
at least i'm not
passing the butter
35. Container Camp London // @craigboxGoogle Cloud Platform 35
you're awesome
three please,
with storage
PetSets
Master
36. Container Camp London // @craigboxGoogle Cloud Platform 36
you're awesome
three please,
with storage
PetSets
db-0
Master
pvc-db-0pv-db-0
1:1 mapping
37. Container Camp London // @craigboxGoogle Cloud Platform 37
you're awesome
three please,
with storage
PetSets
db-0
db-1
Master
pvc-db-0
pvc-db-1
pv-db-0
pv-db-1
1:1 mapping
39. Container Camp London // @craigboxGoogle Cloud Platform 39
no robots shall
touch my pets
PetSets
db-0
db-1
db-2
Master
pvc-db-0
pvc-db-1
pvc-db-2
pv-db-0
pv-db-1
pv-db-2
scale down to
2, please
40. Container Camp London // @craigboxGoogle Cloud Platform 40
no robots shall
touch my pets
PetSets
db-0
db-1
Master
pvc-db-0
pvc-db-1
pvc-db-2
pv-db-0
pv-db-1
I might come in
useful some day
41. Container Camp London // @craigboxGoogle Cloud Platform 41
What other problems do I have?
● Discovery of peers for quorum
○ Sidecars and peer finder scripts
● Startup/teardown ordering
○ Init containers
○ Implicit ordering
42. Container Camp London // @craigboxGoogle Cloud Platform 42
InitContainers
db-0
Mount some
things
43. Container Camp London // @craigboxGoogle Cloud Platform 43
InitContainers
db-0
Mount some
things
pv-db-0
44. Container Camp London // @craigboxGoogle Cloud Platform 44
InitContainers
db-0
Copy some
stuff
pv-db-0
45. Container Camp London // @craigboxGoogle Cloud Platform 45
InitContainers
db-0
Write some
configs
pv-db-0
DNS
46. Container Camp London // @craigboxGoogle Cloud Platform 46
InitContainers
db-0
Be a database
pv-db-0
47. Container Camp London // @craigboxGoogle Cloud Platform 47
● InitContainers and PetSet introduced in 1.3
● InitContainers are Beta in 1.4
● PetSet remain in Alpha
"The real P0 beta blocker is solid prototypes that
increase our confidence in the core feature set."
https://github.com/kubernetes/charts/tree/master/incubator
Status: Alpha
Thanks to Christian and Matt from
49. Container Camp London // @craigboxGoogle Cloud Platform 49
Some terminology
● What is a cluster?
○ A bunch of machines on a high-speed network
● What is high-speed?
○ Generally "in the same building"
○ Same latency and throughput between any two machines
● How much is a bunch?
○ Enough to get the benefits of packing
○ Not too many to Accidentally Kill Everything
50. Container Camp London // @craigboxGoogle Cloud Platform 50
shared cell
(original)
shared cell
(compacted)
non-prod load
(compacted)
prod-only load
(compacted)
# machines
25% overhead
The bigger
the bin,
the better
the packing
51. Container Camp London // @craigboxGoogle Cloud Platform 51
How to separate
● Within a cluster
○ Use namespaces
● Within a region
○ Use NodePools to create "regional" cluster
● With multiple regions
○ Use cluster federation
52. Container Camp London // @craigboxGoogle Cloud Platform 52
etcd
scheduler
controllers
apiserver
Users Master Nodes
kubelet
kubelet
kubelet
CLI
UI
API
Single Kubernetes cluster
53. Container Camp London // @craigboxGoogle Cloud Platform 53
Container
Cluster
All you care about
API
54. Container Camp London // @craigboxGoogle Cloud Platform 54
kubelet
Control Plane
Users Control Plane Clusters
Federation
APICLI
UI
API
55. Container Camp London // @craigboxGoogle Cloud Platform 55
Cluster 2
us-central1-b
Cluster 1
us-east1-b
Cluster 3
europe-west1-b
Cluster 4
asia-east1-b
API CLI
UI
Create the clusters
API API API API
56. Container Camp London // @craigboxGoogle Cloud Platform 56
Federation consists of
● Namespace
● API Server Service with public VIP
● API Server Deployment with 2 replicas
● Controller Manager Pod with 1 replica
● Database key/value store
Familiar? --context=federation-cluster
Deploy the Federated Control Plane
Cluster 2
us-central1-b
Cluster 1
us-east1-b
Cluster 3
europe-west1-b
Cluster 4
asia-east1-b
API API API API
57. Container Camp London // @craigboxGoogle Cloud Platform 57
Add clusters to federation
Cluster 2
us-central1-b
Cluster 1
us-east1-b
Cluster 3
europe-west1-b
Cluster 4
asia-east1-b
API API API API
Federation Control Plane
kubectl --context=federation-cluster create -f clusters/gce-asia-east1.yaml
apiVersion: federation/v1beta1
kind: Cluster
metadata:
name: gce-asia-east1
spec:
serverAddressByClientCIDRs:
- clientCIDR: "0.0.0.0/0"
serverAddress: "https://257.100.194.68"
secretRef:
name: gce-asia-east1
58. Container Camp London // @craigboxGoogle Cloud Platform 58
Deploy a federated ReplicaSet
Cluster 2
us-central1-b
Cluster 1
us-east1-b
Cluster 3
europe-west1-b
Cluster 4
asia-east1-b
API API API API
Federation Control Plane
kubectl --context=federation-cluster create -f rs/nginx.yaml
apiVersion: extensions/v1beta1
kind: ReplicaSet
metadata:
name: nginx
spec:
replicas: 4
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.10
me again
59. Container Camp London // @craigboxGoogle Cloud Platform 59
Deploy a federated Service
Cluster 2
us-central1-b
Cluster 1
us-east1-b
Cluster 3
europe-west1-b
Cluster 4
asia-east1-b
API API API API
Federation Control Plane
kubectl --context=federation-cluster create -f service/nginx.yaml
apiVersion: v1
kind: Service
metadata:
labels:
app: nginx
name: nginx
spec:
ports:
- port: 80
protocol: TCP
targetPort: 80
name: http
selector:
app: nginx
type: LoadBalancer
60. Container Camp London // @craigboxGoogle Cloud Platform 60
Each service shard gets a load balancer
Cluster 1
us-east1-b
Cluster 2
us-central1-b
Cluster 3
europe-west1-b
Cluster 4
asia-east1-b
61. Container Camp London // @craigboxGoogle Cloud Platform 61
...and each service creates a DNS entry
70. Container Camp London // @craigboxGoogle Cloud Platform 70
Q2 2016 Q3 2016 Q4 2016 (**) 2017 and beyond (**)
Beta 1
● Public facing,
multi-region/Cloud,
cross-cluster
service discovery
(internal/external DNS)
● Service object API
support
Beta 2
● Replica Sets
● Multi-region Ingress (L7)
Load Balancing across
clusters for GCP only
Beta 3
● Cross-provider,
multi-region Ingress (L7)
Load Balancing
● GKE IAM Integration
GA!
● Non-public-facing
cross-cluster
service discovery
● Full support for
Kubernetes API objects
● UI support for
Federated Clusters
● Federated IAM
● GKE hosted control plane
(**) - this is a proposed roadmap. Items listed here are subject to change.
Status: Beta
71. Container Camp London // @craigboxGoogle Cloud Platform 71
● Kubernetes Cluster Federation Sneak Peak
● Kubernetes Cluster Federation using GKE
● Cluster Federation Admin Guide
● Cross Cluster Service Discovery Deployment Guide
● Cross Cluster Services - Achieving Higher Availability for your Kubernetes Applications
Also,
● Participate with us on the Kubernetes #sig-federation
● Post issues or feature requests on GitHub
● Join us in the #federation channel on Slack
Want to learn more?