Последние два года Booking.com решает задачу ускорения выхода новых продуктов на рынок. Частью нового подхода является построение внутреннего облака. Основу этого облака составляют 15 Kubernetes-кластеров. При их создании Booking.com отошел от общепринятых методов: здесь сетап включает shared-кластера, плоскую сеть, хитрым образом считаемые SLO/SLI для каждого кластера, набор тестов, которые в реальном времени проверяют функционал кластера и его интеграций. Booking.com не просто эксплуатирует Kubernetes, но еще и активно разрабатывает системные приложения, призванные улучшить экосистему. Например, был создан и выложен в опенсорс Shipper – набор контроллеров, обеспечивающих Kubernetes-native оркестрацию Сanary и Blue-Green деплойментов на несколько кластеров одновременно.

1. Kubernetes в Booking.com
Иван Круглов | 14.02.2019

2. Какую проблему
решаем?
Faster Time To Market
fast resource provisioning

3. 16 k8s кластеров
800 нод
shared/multitenant кластеры
проактивный мониторинг
Shipper
300+ сервисов
Факты

4. Архитектура

5. Тернии контейнеризованных приложений
и микросервисов
https://www.youtube.com/watch?v=jeEd8eiMwpY

7. ACI
AOS from Apstra
Big Cloud Fabric from Big Switch
Networks
Cilium
CNI-Genie from Huawei
Contiv
Contrail
DANM
Flannel
Google Compute Engine
Jaguar
Knitter
Kube-router
L2 networks and linux bridging
Multus
NSX-T
Nuage Networks VCS
OpenVSwitch
OVN (Open Virtual Networking)
Project Calico
Romana
Weave Net from Weaveworks
…
Kubernetes landscape
kubedns
coredns
kubeadm
rancher
kubespray
kubermatic
kublr
coreos
minikube
microk8s
docker-desktop
…
metrics-server
kube-state-metrics
kubelet
heapster
cadvisor
k8s dashboard
prometheus
kubewatch
fluentd
…
provision monitoring
network
DNS
storage
iscsi
nfs
trident
emptydir
hostpath
configmap
aws
gce
cephfs
flocker
glusterfs
…
containers
docker
containerd
rkt
cri-o
runc
lxd
kata
firecracker
gvisor
…
routing
kube-proxy
kube-router
---
iptables
ipvs
auth
certs
tokens
openid
auth-proxy
webhooks
rbac/abac
…
ingress
nginx
haproxy
istio
envoy
traefik
kong
f5
ambassaror
…

8. использовать максимальное количество
знакомых компонентов и наработанных подходов
максимально просто
для понимания
сложность придет сама собой
Принципы

10. маленький кластер = маленький failure domain

11. master master master
envoy envoy
gateway gateway
envoy envoy envoy
node node node
envoy envoy envoy
node node node
envoy envoy envoy
node node node
…
…
…
etcd
kube-apiserver
kube-scheduler
kube-controller-manager
coredns
booking controllers*
…
kubelet
docker
cadvisor
node-problem-detector
…
kube-proxy

12. bare-metal
puppet
systemd.unit
6 бинарников
разделяем инфраструктурные
и пользовательские компоненты
acceptance checklist + dashboard
conformance tests
Provisioning

13. цель – плоская и простая сеть
каждый Pod/Service получает 10.* IP-адрес
Bird + BGP + TO
CNI host-local plugin
https://github.com/containernetworking/plugins/tree/master/plugins/ipam/host
поощряем использование Pod IP-адресов
… экспортируем в Service Discovery
Сеть

14. Мониторинг

15. prometheus prometheus prometheus prometheus
prometheus prometheus
monitormonitor
observer
period: 1 min
retention: 2 years
period: 5 sec
retention: 7 days

16. API response rate
SLI
sum(rate(apiserver_request_count{code!~"5.."}[5m]))
/ sum(rate(apiserver_request_count[5m])) * 100
Pod startup latency
User Experience

17. pod
object
created
“Ready”
pulling image last
container
start
init container
Pod startup latency = time to start last container –
pulling image time –
init container(s) time
time to last container start

19. conformance tests
https://github.com/kubernetes/community/blob/master/contributors/devel/conformance-tests.md
используем после создания нового кластера
чем тестировать постоянно?
Как убедится, что k8s работает?

20. функциональное тестирование для k8s
периодически проверяет
способность кластера выполнять:
• базовые функции
• сетевое соединение
• интеграции
kube-probe

21. curl
ping
query-myself-in-kubernetes
send-events
send-logs
vault-init
ephemeral-storage
expose-service-to-dlb
expose-service-to-service-mesh
expose-service-to-zookeeper
persistent-storage-reuse-volume
persistent-storage-volume-claim
persistent-storage-claim-storage-classes
kube-probe

24. Shipper
https://shipper-k8s.io

25. complex deployment strategy
deployment objects are not enough
+ multi-cluster, multi-region, multi-cloud
= coordination scripts
problems:
• hard to debug
• inconsistent state
Why does Shipper exist?

26. application cluster
application cluster
management cluster
shipper

28. application cluster
application cluster
management cluster
shipper
1. staging
2. canary
3. full on
application.yaml

29. apiVersion: shipper.booking.com/v1alplha1
kind: Application
metadata:
name: bikerental
spec:
template:
clusterRequirements:
regions:
- name: us-west-1
- name: eu-west-1
chart:
name: java
version: 0.0.36
repoUrl: https://charts.prod.booking.com/charts
values:
replicaCount: 1
strategy:
steps:
- name: staging
capacity:
incumbent: 100
contender: 1
traffic:
incumbent: 100
contender: 0
- name: canary
capacity:
incumbent: 90
contender: 10
traffic:
incumbent: 90
contender: 10
- name: full on
capacity:
incumbent: 0
contender: 100
traffic:
incumbent: 0
contender: 100

30. apiVersion: shipper.booking.com/v1alplha1
kind: Application
metadata:
name: bikerental
spec:
template:
clusterRequirements:
regions:
- name: us-west-1
- name: eu-west-1
chart:
name: java
version: 0.0.36
repoUrl: https://charts.prod.booking.com/charts
values:
replicaCount: 1
strategy:
steps:
- name: staging
capacity:
incumbent: 100
contender: 1
traffic:
incumbent: 100
contender: 0
- name: canary
capacity:
incumbent: 90
contender: 10
traffic:
incumbent: 90
contender: 10
- name: full on
capacity:
incumbent: 0
contender: 100
traffic:
incumbent: 0
contender: 100

31. apiVersion: shipper.booking.com/v1alplha1
kind: Application
metadata:
name: bikerental
spec:
template:
clusterRequirements:
regions:
- name: us-west-1
- name: eu-west-1
chart:
name: java
version: 0.0.36
repoUrl: https://charts.prod.booking.com/charts
values:
replicaCount: 1
strategy:
steps:
- name: staging
capacity:
incumbent: 100
contender: 1
traffic:
incumbent: 100
contender: 0
- name: canary
capacity:
incumbent: 90
contender: 10
traffic:
incumbent: 90
contender: 10
- name: full on
capacity:
incumbent: 0
contender: 100
traffic:
incumbent: 0
contender: 100
chart:
name: java
version: 0.0.36
repoUrl: https://charts.prod.booking.com/charts
values:
replicaCount: 1

32. apiVersion: shipper.booking.com/v1alplha1
kind: Application
metadata:
name: bikerental
spec:
template:
clusterRequirements:
regions:
- name: us-west-1
- name: eu-west-1
chart:
name: java
version: 0.0.36
repoUrl: https://charts.prod.booking.com/charts
values:
replicaCount: 1
strategy:
steps:
- name: staging
capacity:
incumbent: 100
contender: 1
traffic:
incumbent: 100
contender: 0
- name: canary
capacity:
incumbent: 90
contender: 10
traffic:
incumbent: 90
contender: 10
- name: full on
capacity:
incumbent: 0
contender: 100
traffic:
incumbent: 0
contender: 100
clusterRequirements:
regions:
- name: us-west-1
- name: eu-west-1

33. apiVersion: shipper.booking.com/v1alplha1
kind: Application
metadata:
name: bikerental
spec:
template:
clusterRequirements:
regions:
- name: us-west-1
- name: eu-west-1
chart:
name: java
version: 0.0.36
repoUrl: https://charts.prod.booking.com/charts
values:
replicaCount: 1
strategy:
steps:
- name: staging
capacity:
incumbent: 100
contender: 1
traffic:
incumbent: 100
contender: 0
- name: canary
capacity:
incumbent: 90
contender: 10
traffic:
incumbent: 90
contender: 10
- name: full on
capacity:
incumbent: 0
contender: 100
traffic:
incumbent: 0
contender: 100
strategy:
steps:
- name: staging
capacity:
incumbent: 100
contender: 1
traffic:
incumbent: 100
contender: 0
- name: canary
capacity:
incumbent: 90
contender: 10
traffic:
incumbent: 90
contender: 10
- name: full on
capacity:
incumbent: 0
contender: 100
traffic:
incumbent: 0
contender: 100

34. apiVersion: shipper.booking.com/v1alplha1
kind: Application
metadata:
name: bikerental
spec:
template:
clusterRequirements:
regions:
- name: us-west-1
- name: eu-west-1
chart:
name: java
version: 0.0.36
repoUrl: https://charts.prod.booking.com/charts
values:
replicaCount: 1
strategy:
steps:
- name: staging
capacity:
incumbent: 100
contender: 1
traffic:
incumbent: 100
contender: 0
- name: canary
capacity:
incumbent: 90
contender: 10
traffic:
incumbent: 90
contender: 10
- name: full on
capacity:
incumbent: 0
contender: 100
traffic:
incumbent: 0
contender: 100
strategy:
steps:
- name: staging
capacity:
incumbent: 100
contender: 1
traffic:
incumbent: 100
contender: 0
- name: canary
capacity:
incumbent: 90
contender: 10
traffic:
incumbent: 90
contender: 10
- name: full on
capacity:
incumbent: 0
contender: 100
traffic:
incumbent: 0
contender: 100

35. apiVersion: shipper.booking.com/v1alplha1
kind: Application
metadata:
name: bikerental
spec:
template:
clusterRequirements:
regions:
- name: us-west-1
- name: eu-west-1
chart:
name: java
version: 0.0.36
repoUrl: https://charts.prod.booking.com/charts
values:
replicaCount: 1
strategy:
steps:
- name: staging
capacity:
incumbent: 100
contender: 1
traffic:
incumbent: 100
contender: 0
- name: canary
capacity:
incumbent: 90
contender: 10
traffic:
incumbent: 90
contender: 10
- name: full on
capacity:
incumbent: 0
contender: 100
traffic:
incumbent: 0
contender: 100
strategy:
steps:
- name: staging
capacity:
incumbent: 100
contender: 1
traffic:
incumbent: 100
contender: 0
- name: canary
capacity:
incumbent: 90
contender: 10
traffic:
incumbent: 90
contender: 10
- name: full on
capacity:
incumbent: 0
contender: 100
traffic:
incumbent: 0
contender: 100

36. apiVersion: shipper.booking.com/v1alplha1
kind: Application
metadata:
name: bikerental
spec:
template:
clusterRequirements:
regions:
- name: us-west-1
- name: eu-west-1
chart:
name: java
version: 0.0.36
repoUrl: https://charts.prod.booking.com/charts
values:
replicaCount: 1
strategy:
steps:
- name: staging
capacity:
incumbent: 100
contender: 1
traffic:
incumbent: 100
contender: 0
- name: canary
capacity:
incumbent: 90
contender: 10
traffic:
incumbent: 90
contender: 10
- name: full on
capacity:
incumbent: 0
contender: 100
traffic:
incumbent: 0
contender: 100
strategy:
steps:
- name: staging
capacity:
incumbent: 100
contender: 1
traffic:
incumbent: 100
contender: 0
- name: canary
capacity:
incumbent: 90
contender: 10
traffic:
incumbent: 90
contender: 10
- name: full on
capacity:
incumbent: 0
contender: 100
traffic:
incumbent: 0
contender: 100

37. application cluster
application cluster
management cluster
shipper
1. staging
2. canary
3. full on

38. Спасибо!
Вопросы?
Иван Круглов
ivan.kruglov@booking.com

39. https://github.com/bookingcom/shipper
https://github.com/ikruglov/kube-custom-monitor
https://github.com/ikruglov/kube-probe (TODO)
Links