How to Achieve Canary Deployment on Kubernetes

1. How to Achieve Canary
Deployment on Kubernetes
JohnChen
2019/11/30

2. Hello Everyone, My name is John
I was
• a System engineer
• a DevOps
I am
• a Site Reliability Engineer
1

3. Outline
• AWS Elastic Beanstalk
• Kubernetes
• Blue/Green deployment
• Canary deployment
• Component relationship
• Labels in Service and Deployment
• Kubernetes canary deployment
• Helm
• Jenkins
• Conclusions
• Questions
2

4. AWS Elastic Beanstalk
• AWS Elastic Beanstalk 是在 AWS 上啟動和執行 Web應用程式最快
也最簡單的方法。開發人員只要上傳他們的應用程式碼，服務就
會自動處理所有細節，如資源佈建、負載平衡、自動調整規模和
監控
• Elastic Beanstalk 最適合搭配您的 PHP、Java、Python、Ruby、
Node.js、.NET、Go 或 Docker Web 應用程式使用。Elastic
Beanstalk 可使用核心 AWS 服務，例如 Amazon EC2、Amazon
Elastic Container Service (Amazon ECS)、Auto Scaling 和 Elastic Load
Balancing，輕鬆支援需要擴展到服務數百萬位使用者的應用程式
3

5. Kubernetes (K8s)
• Kubernetes (K8s) is an open-source system for automating
deployment, scaling, and management of containerized applications.
• It groups containers that make up an application into logical units for
easy management and discovery. Kubernetes builds upon 15 years of
experience of running production workloads at Google, combined
with best-of-breed ideas and practices from the community.
4

6. Blue Green Deployment 1/3
• 藍綠部署是版本 B 同等數量的被並排部署在版本A旁邊。
• 當新版本滿足上線條件的測試後，流量在負載均衡層從版本 A 切
換到版本 B。
5

7. Blue Green Deployment 2/3
6

8. Blue Green Deployment 3/3
1. 連線是有狀態的
2. 變動是不相容於舊版本的
3. 叢集系統
4. 有持久性的資料儲存
7

9. Canary Deployment 1/3
• 金絲雀部署是指逐漸將生產環境流量從版本 A 切換到版本 B。
• 通常流量是按比例分配的。
• 例如：90% 的請求流向版本 A，10% 的流向版本 B。
8

10. Canary Deployment 2/3
9

11. Canary Deployment 3/3
1. 連線是無狀態的
2. 沒有 QA 進行測試
3. 服務之間的流量和延時問題
4. 大量併發時才會發現的問題
10

12. Original Deployment 1/8
www.chen.idv
www.chen.idv
blue.chen.idv weight 100
green.chen.idv weight 0
Original Version
New Version
11

13. Original Deployment 2/8
www.chen.idv
www.chen.idv
blue.chen.idv weight 100
green.chen.idv weight 0
staging.chen.idv staging.chen.idv
New Version
Original Version
Functional testing
12

14. Original Deployment 3/8
www.chen.idv
www.chen.idv
blue.chen.idv weight 95
green.chen.idv weight 5
Original Version
New Version
13

15. Original Deployment 4/8
www.chen.idv
www.chen.idv
blue.chen.idv weight 90
green.chen.idv weight 10
Original Version
New Version
14

16. Original Deployment 5/8
www.chen.idv
www.chen.idv
blue.chen.idv weight 75
green.chen.idv weight 25
Original Version
New Version
15

17. Original Deployment 6/8
www.chen.idv
www.chen.idv
blue.chen.idv weight 50
green.chen.idv weight 50
Original Version
New Version
16

18. Original Deployment 7/8
www.chen.idv
www.chen.idv
blue.chen.idv weight 25
green.chen.idv weight 75
Original Version
New Version
17

19. Original Deployment 8/8
www.chen.idv
www.chen.idv
blue.chen.idv weight 0
green.chen.idv weight 100
Original Version
New Version
18

20. Component relationship
Pod Container
PodReplicaSet
ReplicaSetDeployment
ServiceDeployment
1 N
N
N
N
1
1
1
19
Service Deployment1 N

21. Kubernetes canary deployment
You can tweak the number of replicas of the stable and canary releases to determine the ratio of
each release that will receive live production traffic (in this case, 3:1). Once you’re confident, you
can update the stable track to the new application release and remove the canary one.
20

22. Production Pod
metadata.labels:
app. envrionment: preprod
app.service: production
Production Pod
metadata.labels:
app. envrionment: production
app.service: production
Staging Deployment
spec.template.metadata.labels:
app.envrionment: staging
app.service: staging
Preprod Deployment
spec.template.metadata.labels:
app.envrionment: preprod
app.service: production
Production Deployment
spec.template.metadata.labels:
app.envrionment: production
app.service: production
Labels in Service and Deployment
Production Service
selector:
app.service: production
Staging Service
selector:
app.service: staging
Staging Pod
metadata.labels:
app. envrionment: staging
app.service: staging
21

23. Kubernetes canary deployment 1/6
22

24. Kubernetes canary deployment 2/6
23

25. Kubernetes canary deployment 3/6
24

26. Kubernetes canary deployment 4/6
25

27. Kubernetes canary deployment 5/6
26

28. Kubernetes canary deployment 6/6
27

29. Questions
• 原來用三個 deployment 和兩個 service 就可以達成金絲雀部署了
• 但每一次都用改 yaml 檔案， 再執行 kubectl apply –f xxx
• 這樣操作很複雜，會不會下錯指令？
• 有辦法讓 Developer 自己部署嗎？
28

30. The Four
29

31. Helm
• Helm helps you manage Kubernetes applications.
• Helm Charts help you define, install, and upgrade even the most
complex Kubernetes application.
30

32. Helm Chart
31

33. Jenkins
• 是一款由Java編寫的開源的持續整合工具。在與Oracle發生爭執後，專案從Hudson
專案復刻。
• 提供了軟體開發的持續整合服務。它執行在Servlet容器中（例如Apache Tomcat）。
它支援軟體組態管理（SCM）工具，可以執行基於Apache Ant和Apache Maven的專
案，以及任意的Shell指令碼和Windows批次處理命令。Jenkins的主要開發者是川口
耕介。Jenkins是在MIT許可證下發布的自由軟體。
32

34. Jenkins pipeline view
33

35. Kubernetes Canary Deployment 1/6
34

36. Kubernetes Canary Deployment 2/6
35

37. Kubernetes Canary Deployment 3/6
36

38. Kubernetes Canary Deployment 4/6
37

39. Kubernetes Canary Deployment 5/6
38

40. Kubernetes Canary Deployment 6/6
39

41. Conclusions
40
• Script 可以做到的事，Jenkins 就可以做到
• 要做 CI/CD 需要使用很多工具，但不要被工具限制
• 方法不在新舊，在於有用
• 先站穩腳跟，再向前邁進

42. Questions
• 為何 preprod pod 和 production pod 在和 1:1 時就切換 ？
• 為何流量還是沒有平均分配？
• 為何不用 ingress？
• 為何不用 Jenkins pipeline ？
41

43. Q＆A
42

44. Redis Server
6381 port
Redis Server
6379 port
Redis Server
6382 port
Redis Server
6380 port
client-testing-redis.chen.idv
AP 1 AP 2 AP 3 AP 4
EC2
Multi Redis Server in One VM

45. Redis Server
6379 port
Redis Server
6379 port
Redis Server
6379 port
Redis Server
6379 port
AP 1 AP 2 AP 3 AP 4
client-testing-redis.chen.idv:6381
client-testing-redis.chen.idv:6382
client-testing-redis.chen.idv:6379
client-testing-redis.chen.idv:6380
Kubernetes Pod
Kubernetes Service
One Service Connect to Multi Pod 1/2

46. Redis Service
selector:
app.service: redis
port:
- name: redis-6379
port: 6379
protocol: TCP
targetPort: redis-6379
- name: redis-6380
port: 6380
protocol: TCP
targetPort: redis-6380
- name: redis-6381
port: 6381
protocol: TCP
targetPort: redis-6381
- name: redis-6382
port: 6382
protocol: TCP
targetPort: redis-6382
Redis Pod 6379
metadata.labels:
app.service: redis
port:
- containerPort: 6379
name: redis-6379
protocol: TCP
Redis Pod 6380
metadata.labels:
app.service: redis
port:
- containerPort: 6379
name: redis-6380
protocol: TCPRedis Pod 6381
metadata.labels:
app.service: redis
port:
- containerPort: 6379
name: redis-6381
protocol: TCP
Redis Pod 6382
metadata.labels:
app.service: redis
port:
- containerPort: 6379
name: redis-6382
protocol: TCP
One Service Connect to Multi Pod 2/2