My talk from the Kubernetes Boston Meetup on Feb 20, 2018. I talked about how CloudHealth is using Kubernetes both in production and in development to increase velocity without sacrificing quality.

1. Using Kubernetes to increase
developer velocity
(without sacrificing quality)
Adam Schepis
Architect @ CloudHealth Technologies

2. About Me
● At CloudHealth I build things in the cloud
that help our customers to confidently
build things in the cloud.
● I love working on distributed systems with
high scalability requirements.
● I have met Spiderman.
@aschepis

3. Our Challenges
Growth

4. Our Challenges
Maturing Market

5. Our Challenges
Innovation in the Cloud
● AWS - 100+ feature announcements in 2018
● Azure - 13 announcements (chunkier)
● GCP - Next '18 in July (more than 100
announcements last year)

6. ● Started our Kubernetes journey in early
2017
● Running a number of production workloads
● Kubernetes is a key component of platform
overhaul in 2018
CloudHealth
and Kubernetes

7. 7 © 2018 CLOUDHEALTH® TECHNOLOGIES INC.
Our Stack
Helm
● Service Lifecycle
● Values templates
● Trivial rollback
● Canary
deployments were
a bit tricky (would
love suggestions)
Linkerd
● Service discovery
● Circuit breakers
● Metrics
● Distributed tracing
(via Zipkin)
Romana
● CNI
● Cloud Native
● Works well with
AWS
● Not a full mesh
network

8. 8 © 2018 CLOUDHEALTH® TECHNOLOGIES INC.
Primary Clusters
Development
● ~50 Nodes
● Namespace per
developer
● Devs given free
reign within their
namespace.
● Collaboration via
linkerd
Test/Staging
● ~20 Nodes
● Stable version of
each service
● Namespace per-
pull request
● Integration tests w/
new code + stable
services
Production
● ~50 Nodes
● Namespace per-
service group
● Tight restrictions on
access
● Distributed tracing

9. 9 © 2018 CLOUDHEALTH® TECHNOLOGIES INC.
Development Environments
● ch CLI tool
● Light wrapper around setup, dev, and service lifecycle
● simplifies and accelerates dev workflow
● commands
○ ch init
○ ch new service <foo>
○ ch build
○ ch deploy
○ ch run (build + deploy)
○ ch supervisor
Consistency drives both velocity and quality

10. 10 © 2018 CLOUDHEALTH® TECHNOLOGIES INC.
Development Environments
● Service endpoints are always in helm chart values
○ Injected as environment variables
○ can use any namespace https://auth/graphql for my namespace or
https://auth.david/graphql for David's namespace
● Test locally by using linkerd endpoint as http_proxy to reach remote services in
desired namespace
● Namespace for each developer
● Collaborate without deploying the world.
Collaborating in a shared dev cluster

11. 11 © 2018 CLOUDHEALTH® TECHNOLOGIES INC.
Our Build Pipeline
Pull Request
Human Gate
Staging
PR namespace
Prod
Canary
Prod
Svc Group
- Unit
- Integration
- Pact
(Contracts)
- Javadoc
- RDoc
- etc.
Published to
S3.

12. 12 © 2018 CLOUDHEALTH® TECHNOLOGIES INC.
When things go wrong
● Each PR has its own namespace to deploy service into
● Integration tests operate against stable versions of services it depends on
● When failure happens dev can:
○ Access resources in namespace through linkerd with customer header
○ Shell into a pod to check it out
○ Look at logs
○ Exercise failing service manually
○ Access UI of failing service (if one exists)
Failed Builds

13. 13 © 2018 CLOUDHEALTH® TECHNOLOGIES INC.
When things go wrong
● Human gates between canary and full prod deploy
● Canaries can be validated by
○ verifying that it is serving production requests
○ looking at error reporting service
○ using linkerd headers to ensure a request against canary
○ tailing logs
○ looking at performance and application metrics compared to current production
code
● Can temporarily scale down to 0.
● Rollback with helm is fast and trivial.
Bad Canaries

14. 14 © 2018 CLOUDHEALTH® TECHNOLOGIES INC.
● Consistency in tooling == velocity and quality
● Shared dev cluster
○ collaboration, shared understanding
● Namespace/deploy per PR
○ Velocity - faster to diagnose and fix test failures
○ Quality - easier to reach root cause via live debugging
● Canary Builds
○ Quickly detect bad deploys without heavy impact to customers
○ Confidence in deploys post-canary
In Summary

15. Thank you! Questions?
Adam Schepis
@aschepis