Slides from the presentation "From GitOps to an adaptable CI/CD Pattern for Kubernetes" at the Continuous Delivery NYC meetup, by Andrew Phillips. See https://www.meetup.com/ContinuousDeliveryNYC/events/255366708/ and https://www.youtube.com/watch?v=SYeZ0uIwbLc

1. From GitOps to an adaptable
CI/CD pattern for Kubernetes
Andrew Phillips
Continuous Delivery : NYC; Nov 1, 2018

2. From GitOps to an adaptable
CI/CD pattern...for everything?
Andrew Phillips
Continuous Delivery : NYC; Nov 1, 2018

3. The bio slide
● Been on most sides of this space: developer, infra builder, product owner,
evangelist and more
● Long-standing open-source contributor
● Author and regular conference and meetup presenter
● Co-organizer of ContainerDays Boston & NYC

4. Agenda
1. What’s the challenge?
2. CI/CD practices & patterns today
3. What does Kubernetes add into the mix?
4. A CI/CD pipeline for Kubernetes
5. Next steps for your scenario
6. Q&A

5. 1. The context
● Lots of organizations looking at Kubernetes right now
● Trying to use this also as an opportunity to “clean up” sub-optimal software
delivery pipelines
● How to do this right?

6. 1. The challenge
● Lots of new runtime-specific concepts to deal with
● “Kubernetes-native” best practices still very much in flux
● Wrapping your head around “new” practices (e.g. GitOps) is hard
● Extrapolating from new practices in theory to actual, working implementations
is even harder
● Figuring out how all this new stuff relates to accepted industry practices and
tools is harder still

7. Failure mode: CCAOA
Cargo Cult All Over Again

8. 1. The approach
● Recall good practices that we’d like to retain
● Incorporate refinements related to Kubernetes to create a straw man setup
● Develop a mental model to understand advantages and shortfalls of the setup
● Refine the setup based on tradeoffs to be made related to each user’s
particular situation
● Implement using appropriate tools

9. 2. Existing good practice
● Reproducible builds
● Store source and derived artifacts appropriately
● Minimize duplication, especially around environment config
● Keep the business process flexible and the env automation robust
● Support 4 related processes:
○ Application update
○ Environment (config) update
○ Environment spin-up/restore
○ Environment drift detection and remediation
● Specifically, support env-specific (e.g. log settings change) and cross-env
(e.g. new app version rollout) processes concisely

10. Deployment execution: business process vs. technical process
Release pipeline A business process, represented as a sequence, possibly very specific to a service
Test environment Staging environment
Technical components with
interdependencies, defined
“as-code”. To be
automatically sequenced if
possible
Production environment
Deploy to Test Review Approve Deploy to Prod...
App
Endpoint
Config
App
Endpoint
Config’
Old app version
Namespace

11. 2. The four related processes
● Application update
○ “I want to validate a new release candidate and promote it through envs to prod”
● Environment (config) update
○ “I want to change the attributes of a particular env only”
● Environment spin-up/restore
○ “I want to (re-)create an environment from scratch, with config from a specific checkpoint/point
in time”
● Environment diff, drift detection and remediation
○ “I want to understand how the actual config of an env relates to the intended config, also
potentially across different environments”

12. 3. What’s new with Kubernetes
● “as-code” description of what an environment should look like
(“environments-as-code”)
● Actuation based on reconciliation engine built into runtime with continuous
enforcement
○ Interest in “pull-based”, async invocation via repo-watching
● Out-of-the-box support for some types of rollout via Deployment object, as
well as CRDs to define your own
○ But can also manipulate underlying objects directly
● GitOps ~ environments-as-code + async invocation + repo workflow for
business process

13. Deployment execution: adding in Kubernetes
Release pipeline A business process, represented as a sequence, possibly very specific to a service
Test environment Staging environment
Technical components with
interdependencies, defined
“as-code”. To be
automatically sequenced if
possible
Production environment
“environments-as-code” repositories
Deploy to Test Review Approve Deploy to Prod...
App
Endpoint
Config
App
Endpoint
Config’
Old app version
Namespace

14. TL;DR
Release pipeline A business process, represented as a sequence, possibly very specific to a service
Test environment Staging environment
Technical components with
interdependencies, defined
“as-code”. To be
automatically sequenced if
possible
Production environment
“environments-as-code” repositories
Deploy to Test Review Approve Deploy to Prod...
App
Endpoint
Config
App
Endpoint
Config’
Old app version
Namespace
Imperative pipeline across environments
Declarative spec for each environment
(with support for some imperative “cheating” where necessary)

15. 3. The straw man

16. team1-repo
env-repo
applied-staging-branch applied-prod-branch
master-
branch
service
config
staging-
branch
deployme
nt
prod-
branch
deployme
nt
team2-repo
master-
branch
custom-s
ettings
staging-
branch
config
prod-
branch
config

17. App1 update

18. team1-repo
env-repo
applied-staging-branch applied-prod-branch
master-
branch
service
config
staging-
branch
deployme
nt
prod-
branch
deployme
nt
team2-repo
master-
branch
custom-s
ettings
staging-
branch
config
prod-bra
nch
config
apiVersion: v1
kind: Service
metadata:
name: app1-service
...
spec:
type: ClusterIP
ports:
- port: 808080
targetPort: http
protocol: TCP
name: http
selector:
app: app1
env: {{ environment }}

19. team1-repo
env-repo
applied-staging-branch
app1.yaml
applied-prod-branch
master-
branch
service
config
staging-
branch
deployme
nt
prod-
branch
deployme
nt
team2-repo
master-
branch
custom-s
ettings
staging-
branch
config
prod-bra
nch
config
apiVersion: v1
kind: Service
metadata:
name: app1-service
...
spec:
type: ClusterIP
ports:
- port: 808080
targetPort: http
protocol: TCP
name: http
selector:
app: app1
env: {{ environment }}
Step 1
apiVersion: v1
kind: Service
metadata:
name: app1-service
...
spec:
type: ClusterIP
ports:
- port: 8080
targetPort: http
protocol: TCP
name: http
selector:
app: app1
env: staging
---
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: app1
namespace: staging
...
---
apiVersion: v1
kind: ConfigMap
...

20. team1-repo
env-repo
applied-staging-branch
app1.yaml
applied-prod-branch
app1.yaml
master-
branch
service
config
staging-
branch
deployme
nt
prod-
branch
deployme
nt
team2-repo
master-
branch
custom-s
ettings
staging-
branch
config
prod-bra
nch
config
apiVersion: v1
kind: Service
metadata:
name: app1-service
...
spec:
type: ClusterIP
ports:
- port: 808080
targetPort: http
protocol: TCP
name: http
selector:
app: app1
env: {{ environment }}
Step 2
apiVersion: v1
kind: Service
metadata:
name: app1-service
...
spec:
type: ClusterIP
ports:
- port: 8080
targetPort: http
protocol: TCP
name: http
selector:
app: app1
env: production
---
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: app1
namespace: production
...
---
apiVersion: v1
kind: ConfigMap
...

21. App2 update

22. team1-repo
env-repo
applied-staging-branch
app1.yaml
applied-prod-branch
app1.yaml
master-
branch
service
config
staging-
branch
deployme
nt
prod-
branch
deployme
nt
team2-repo
master-
branch
custom-s
ettings
staging-
branch
config
prod-
branch
config
app.version: 0.0.34
port: 8090
image: gcr.io/image@sha256:{{
container.image.digest }}

23. team1-repo
env-repo
applied-staging-branch
app1.yaml
app2.yaml
app2-env.yaml
applied-prod-branch
app1.yaml
master-
branch
service
config
staging-
branch
deployme
nt
prod-
branch
deployme
nt
team2-repo
master-
branch
custom-s
ettings
staging-
branch
config
prod-
branch
config
Step 1app.version: 0.0.34
port: 8090
image: gcr.io/image@sha256:{{
container.image.digest }}
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: app2
namespace: staging
...
spec:
replicas: 1
...
image:
gcr.io/image@sha256:abcdef
ports:
- name: http
containerPort: 90
---
...
---
apiVersion: v1
kind: ConfigMap
...
data:
config.yaml: |
'app.version': 0.0.4

24. team1-repo
env-repo
applied-staging-branch
app1.yaml
app2.yaml
app2-env.yaml
applied-prod-branch
app1.yaml
app2.yaml
app2-env.yaml
master-
branch
service
config
staging-
branch
deployme
nt
prod-
branch
deployme
nt
team2-repo
master-
branch
custom-s
ettings
staging-
branch
config
prod-
branch
config
Step 2
app.version: 0.0.34
port: 8090
image: gcr.io/image@sha256:{{
container.image.digest }}
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: app2
namespace: production
...
spec:
replicas: 1
...
image:
gcr.io/image@sha256:abcdef
ports:
- name: http
containerPort: 90
---
...
---
apiVersion: v1
kind: ConfigMap
...
data:
config.yaml: |
'app.version': 0.0.4

25. App2 prod config change

26. team1-repo
env-repo
applied-staging-branch
app1.yaml
app2.yaml
app2-env.yaml
applied-prod-branch
app1.yaml
app2.yaml
app2-env.yaml
master-
branch
service
config
staging-
branch
deployme
nt
prod-
branch
deployme
nt
team2-repo
master-
branch
custom-s
ettings
staging-
branch
config
prod-
branch
config
apiVersion: v1
kind: ConfigMap
metadata:
name: app2-env-config
namespace: production
labels:
app: app2
env: production
data:
config.yaml: |
'log.level': INFOTRACE

27. team1-repo
env-repo
applied-staging-branch
app1.yaml
app2.yaml
app2-env.yaml
applied-prod-branch
app1.yaml
app2.yaml
app2-env.yaml
master-
branch
service
config
staging-
branch
deployme
nt
prod-
branch
deployme
nt
team2-repo
master-
branch
custom-s
ettings
staging-
branch
config
prod-
branch
config
Step 1
apiVersion: v1
kind: ConfigMap
metadata:
name: app2-env-config
namespace: production
labels:
app: app2
env: production
data:
config.yaml: |
'log.level': INFOTRACE
apiVersion: v1
kind: ConfigMap
metadata:
name: app2-env-config
namespace: production
labels:
app: app2
env: production
data:
config.yaml: |
'log.level': TRACE

28. 3. The straw man
● Source config in app repo, “compiled” (a.k.a. hydrated) config in env repo(s)
○ Cf. source code in app repo, compiled code in artifact repository
● One app repo per app/team, shared env repos for environments
● Use branches where possible to represent different environments
○ If greater separation via access control is needed, use different repos or consider other
storage
● Keep config in app repo unless it needs to be independent of the app lifecycle
or you really, really need to be able to change it for one env
○ Allows you to use templating/overriding to cut down on duplication
○ The fewer places to keep track of config, the better
● Source config doesn’t have to be “raw” YAML, can be more suitable
abstraction!
○ Cf. higher-level language source code vs. low-level assembly code

29. 3. The straw man
● Prefer explicit invocation over “repo-watching”
○ Avoids “root-level” process running inside cluster, and provides more flexibility for
multi-step rollouts
○ “Repo-watching” makes visualizing current status harder, and requires some sort of
feedback mechanism to distinguish successful from failed deployments in the repo
○ Hard to support phased/multi-step application
○ Easier to reproduce/simulate

30. 3. The straw man
● Prefer explicit invocation over “repo-watching”
○ Avoids “root-level” process running inside cluster, and provides more flexibility for
multi-step rollouts
○ “Repo-watching” makes visualizing current status harder, and requires some sort of
feedback mechanism to distinguish successful from failed deployments in the repo
○ Hard to support phased/multi-step application
○ Easier to reproduce/simulate
● Commit after successful application, not before
○ Avoids having to distinguish attempted from successful deployments in the repo
○ Allows for richer pre-application validation than code diff in a PR (e.g. three-way diff
against actual environment)
○ Avoids commit rights to repo being equivalent to deploy rights to env (and your
automation will need commit rights to make pull requests, unless you used forked repos)
○ Harder if pull requests are used for the business process (as in vanilla GitOps) - requires
multiple branches or “on approve” deployment

31. 4. The mental model
● Understand how your app is updated across two dimensions:
● code change promoted through to
prod
● common externalized config setting,
e.g. localized title
dependent on app version
independent
of
environment

32. 4. The mental model
● Understand how your app is updated across two dimensions:
● code change promoted through to
prod
● common externalized config setting,
e.g. localized title
● adding debug logging to staging
● configuring a scaling policy in prod
based on a new metric
dependent on app version
independent
of
environment
dependent
on
environment

33. 4. The mental model
● Understand how your app is updated across two dimensions:
● code change promoted through to
prod
● common externalized config setting,
e.g. localized title ● rotating database credentials for
prod
● updating discovery service
endpoint
● adding debug logging to staging
● configuring a scaling policy in prod
based on a new metric
dependent on app version
independent
of
environment
dependent
on
environment
independent of app version

34. 4. The mental model
● Understand how your app is updated across two dimensions:
Application update
Static environment config
update
(App-linked) environment config
update
dependent on app version
independent
of
environment
dependent
on
environment
independent of app version

35. 4. The mental model
● Understand how your app is updated across two dimensions:
Application update
Static environment config update
(App-linked) environment config
update
dev responsibility ~ app repo platform responsibility ~ env repo

36. 4. The mental model
1. Application update
a. “I want to validate a new release candidate and promote it through envs to prod”
2. Environment (config) update
a. “I want to change the attributes of a particular env only”
3. Environment spin-up/restore
a. “I want to (re-)create an environment from scratch, with config from a specific checkpoint/point
in time”
4. Environment diff, drift detection and remediation
a. “I want to understand how the actual config of an env relates to the intended config, also
potentially across different environments”

37. ● App repo is driving deployment, env repo is snapshotting cluster state
○ Think source code in github is to docker image in registry as template in app repo is to
manifest in env repo
● Env repo is just a checkpoint in time, cluster can evolve
○ Kubernetes applies changes to manifests
○ Strategies like exponential rollouts or traffic shifting apply changes over time
● Env repo is not a guaranteed healthy state
○ We can defer snapshotting until some health/success metric… but rollback has no silver bullet
● Deletion from either repo ≠ deletion from cluster
○ Challenge is: not trivial to know if cluster depends, or will depend on manifest not submitted to
repo
○ kubectl apply --prune attempts to solve this with a lot of (scary) caveats
4. The mental model

38. 5. Tuning for your scenario
● Appropriate level of abstraction? How much “raw” Kubernetes YAML
should our developers have access to?
● Where should the abstraction live? In templates? In CRDs? In the
automation tool?
● What to use for templating? Token replacement or overrides?
● When do you snapshot/publish to the env repo? On every change to the
cluster? Or when a desired end-state is reached? (think multi-step rollout)
● Access control: how many repos do you need? Are code repositories right
for your use case, or e.g. a blobstore better?
● Am I distributing or deploying? Helm is much better suited for distribution
than deployment
○ Its templating capability is often used as part of deployment flows, though

39. 5. Implement
● Choose storage implementations and partitioning strategies for your
environments-as-code
● Define the appropriate level of abstraction for your developers and choose
tools to support it
● Choose a flexible automation tool for your deployment business process
● Define an appropriate definition of deployment health/success to determine
when a deployment is “good”
● Decide which of the four processes - app update, env config update, env
restore and env drift detection - you want to support
● Build pipelines
● Done!