In a microservices world, applications consist of dozens, hundreds, or even thousands of components. Manually deploying and verifying deployment quality in production is virtually impossible. Kubernetes, which natively supports rolling updates, enables blue-green application deployments with Spinnaker. However, gradual rollouts is a feature that doesn't come out-of-the-box but can be achieved by adding Istio and Prometheus to the equation. During this meetup, Slava Koltovich, CEO of Kublr, and Oleg Atamanenko, Senior Software Architect, discussed canary release implementations on Kubernetes with Spinnaker, Istio, and Prometheus. They examined the role of each tool in the process and how they are all connected. During a demo, they demonstrated a successful and a failed canary release, and how these tools enable IT teams to properly roll out changes to their customer base without any downtime.

1. Gluing It All Together
Canary Releases on Kubernetes with Spinnaker,
Istio, and Prometheus

2. What’s Spinnaker?
• Continuous Delivery Platform
• Multi-cloud, multi-region deployment
capabilities
• Immutable pipelines to deliver reliably
and reproducible
• Deployment strategies enable unified
approach across projects and teams
• Zero-downtime and canary deployments
out-of-the-box*
• Chaos Engineering
• Open-sourced by Neflix in 2015
• Large community: Netflix, Target, Google,
Microsoft, AWS, Pivotal, Mirantis and
many others.
• Kubernetes support led by Google. Also
used by Google for internal deployments.

3. Nexus Prometheus
Kubernetes
Spinnaker in a CI/CD Process
Jenkins Spinaker
Kubernetes
Build
Run Unit Tests
Build Docker Image
Push Docker Image
Deploy to QA K8s
Run Int. Tests
Monitor
Deploy to Prod
Manage Traffic
Monitor
Monitor
Docker
Registry
Prod
Cluster
QA Cluster

4. Spinnaker Architecture

5. Spinnaker and Kubernetes
• Manifest-based deployment:
• Any k8s resource: Deployment,
Config, Secret, Custom Resource
(Routing rule etc.)
• May trigger pipeline execution
• Stored in Git, GCS, S3
• May be linked to each other
• Docker registry integration and ability
to run pipeline w/ new image is
pushed
Deploy vNext
Deploy new k8s deployment with a
new version. Run health checks
Increase traffic to vNext
Deploy Istio routing rule increasing
percentage of traffic to a new
version
Monitor
Monitor new version comparing
Prometheus metrics to a baseline or
specified threshold
Remove vNext
Remove vNext deployment or stop
sending traffic
Remove vCurrent
Remove previous deployment or
scale down to 0

6. Spinnaker and Kubernetes (cont.)
• Native k8s capabilities enable blue-
green deployments
• Istio or Linkerd enable Canary
deployments
• Prometheus and Datadog integrations
for monitoring
• Pipeline templates
• Helm as a template engine
Deploy vNext
Deploy new k8s deployment with a
new version. Run health checks
Increase traffic to vNext
Deploy Istio routing rule increasing
percentage of traffic to a new
version
Monitor
Monitor new version comparing
Prometheus metrics to a baseline or
specified threshold
Remove vNext
Remove vNext deployment or stop
sending traffic
Remove vCurrent
Remove previous deployment or
scale down to 0

7. Why Canary?
“Canary release is a technique to reduce the risk of
introducing a new software version in production by slowly
rolling out the change to a small subset of users before
rolling it out to the entire infrastructure and making it
available to everybody.”
https://martinfowler.com
Like what you hear? Tweet at us @kublr

8. Canary in Kubernetes

9. Canary in Kubernetes
APP A
POD
ReplicaSet
DEPLOYMENT
Version: 1.0
Version: 1.0APP B
POD
POD
APP B
APP B
SERVICE
@kublr, @vkoltovich

10. Canary in Kubernetes
APP A
POD
ReplicaSet
DEPLOYMENT
Version: 1.0
Version: 1.0APP B
POD
POD
APP B
APP B
SERVICE
ReplicaSet
POD
APP B
Version: 2.0
Track: canary
DEPLOYMENT
Version: 2.0
@kublr, @vkoltovich

11. ReplicaSet
POD
APP B
Canary in Kubernetes
APP A
POD
ReplicaSet
DEPLOYMENTVersion: 1.0
Version: 1.0
APP B
POD
POD
APP B
DEPLOYMENT
Version: 2.0
Version: 2.0
Track: canary
APP B
SERVICE
@kublr, @vkoltovich

12. Canary in Kubernetes
APP A
POD
ReplicaSet
DEPLOYMENT
Version: 2.0
Version: 2.0APP B
POD
POD
APP B
APP B
SERVICE
@kublr, @vkoltovich

13. Canary in Kubernetes
• Canary releases in Kubernetes are limited by k8s traffic
routing capabilities
• Traffic splitting is managed by manipulating a number of
pods in current and canary deployments
@kublr, @vkoltovich

14. Introducing Istio
Pilot
• Service discovery for Envoy and traffic routing
• Splitting: gradual (canary) rollout, A/B testing
• Fault injection
• Mirroring
• Failure recovery: circuit breakers, retries, timeouts
Mixer
• Per-request policies: access and usage control
Auth
• Request authentication and encryption
• Identity and credential management
Envoy
• Request routing and processing; attributes
Zipkin/Jaeger, Prometheus/Grafana
• Distributed request tracing
• Monitoring
@kublr, @vkoltovich

15. Canary with Istio
ReplicaSet
DEPLOYMENT
Version: 1.0
Version: 1.0
POD
POD
ReplicaSet
POD
Version: 2.0
Track: canary
DEPLOYMENT
Version: 2.0
POD
PILOT
Traffic rules
99%
1%
ENVOY
APP A
ENVOY
APP B
ENVOY
APP B
ENVOY
APP B
@kublr, @vkoltovich

16. Canary with Istio
ReplicaSet
DEPLOYMENT
Version: 1.0
Version: 1.0
POD
POD
ReplicaSet
POD
Version: 2.0
Track: canary
DEPLOYMENT
Version: 2.0
POD
PILOT
Traffic rules
50%
50%
ENVOY
APP A
ENVOY
APP B
ENVOY
APP B
ENVOY
APP B
@kublr, @vkoltovich

17. Canary with Istio
ReplicaSet
POD
Version: 2.0
ENVOY
APP B
DEPLOYMENT
Version: 2.0
POD
PILOT
Traffic rules
100%
ENVOY
APP A
@kublr, @vkoltovich

18. Canary with Istio
• Istio brings all necessary traffic routing capabilities
required to implement canary releases in Kubernetes
• A release process still has to be implemented outside
@kublr, @vkoltovich

19. Canary Pipeline

20. Canary Release Pipeline
Remove vCurrentMonitorIncrease traffic to vNextDeploy vNext
Remove vNext
Deploy new k8s deployment
with a new version.
Run health checks
Deploy Istio routing rule
increasing percentage of
traffic to a new version
Monitor new version
comparing Prometheus
metrics to a baseline or
specified threshold
Remove previous
deployment or scale down
to 0
Remove vNext deployment
or stop sending traffic
@kublr, @vkoltovich

21. Demo

22. Slava Koltovich | CEO
slava@kublr.com
@vkoltovich
Q&A
Oleg Atamanenko | Sr Software Engineer
oatamanenko@kublr.com
@real_atamanenko
Sign up for our newsletter on kublr.com
and stay in touch!