JS Fest 2019. Peter Braun. DevOps in the Cluster: A deep dive into operators

CONTINUOUS DELIVERY.
CONTINUOUS DEVOPS.
Professional conference on DevOps practices
6APRIL 2019
KYIV, UKRAINE
th

Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINE
Peter Braun | pbraun@redhat.com | Github: pb82
DevOps in the Cluster: a deep dive into
Kubernetes Operators
th

Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth
Agenda
1. What is this talk about?
2. The Situation today
3. What is an Operator?
4. Kubernetes Controllers
5. Operator Tooling
6. Tips for developing Operators
7. Demo

What this talk is about:
Kubernetes Applications and how to manage them.

So what is a ‘Kubernetes Application’?
A Kubernetes application is an application that is both
deployed on Kubernetes and managed using the
Kubernetes APIs and kubectl tooling.

What’s available today?
● Templates
● Helm

Existing Tools: Templates
● YAML / JSON files
● Kubernetes Resources are purely declarative
● Parameterization is possible
○ Kustomize
○ Openshift Templates
● No concept of dependencies

Existing Tools: Helm
● The Package Manager for Kubernetes, it can do:
○ Parameterization,
○ Dependency Resolution
○ and even Version Management

So what’s missing?
● Templates and Helm help with installation.
● Neither of them allow you to manage the Application.

The Solution? Operators!

Let’s start from zero: what exactly is an Operator?
An Operator is a method of packaging, deploying and
managing a Kubernetes application.

Let’s start from zero: what exactly is an Operator?
An Operator is a method of packaging, deploying and
managing a Kubernetes application using custom
resources.

Why is that distinction important?
Operators are Kubernetes Controllers
for Custom Resources.

Kubernetes Controllers
● Kubernetes resources are backed by Controllers
● The purpose of a Controller is to synchronize
○ the cluster state...
○ ...with the desired state (resource definition).

An Example: Deployment Controller
Desired state
Cluster state
Analyze
Update

Kubernetes Controllers: Detailed Look
Informer (Cache)
List
Watch
ns/name
Add/Update/Delete
Work QueueAPI Server Reconcile
Analyze
Update
Pop
Push back

Operators are Kubernetes Controllers
for Custom Resources.

Custom Resources
● Extension to the Kubernetes API
● Created with a Custom Resource Definition
● Let’s you define your own types

Custom Resources (Example)

Custom Resources (Commands)
$ kubectl get crds
$ kubectl get <CRD name>

Operator Characteristics
● Typically written in Golang
● Standalone applications
● Deployed to a namespace
● Come bundled with their Custom Resource Definitions

So how to start developing Operators?

Operator Tooling
Operator
Framework
CoreOS Operator Framework: https://coreos.com/operators/

Operator Framework
Operator
SDK
CoreOS Operator Framework: https://coreos.com/operators/

Operator SDK
● CLI Tool to bootstrap new operators
● SDK to abstract controller facilities
● Testing and build

Operator SDK Development Workflow
1. Bootstrap new operator
2. Add API
3. Add Controller
4. Code, run locally, repeat
5. Build & Push image

Operator SDK Development Workflow

Operator SDK (project structure)
Structs to represent the custom
resource(s) in Golang
Often one controller per custom
resource
YAML resources (CRD, RBAC)

Adding an API to the Operator
operator-sdk add api --api-version=app.example.com/v1alpha1 --kind=AppService

Adding a Controller to the Operator
operator-sdk add controller --api-version=app.example.com/v1alpha1 --kind=AppService
● Creates a new controller
● Sets up the Informer
● Adds a Reconcile function
● Ready to implement your Logic

Controller Implementation: Reconcile Function
● Called when resources change
● Only gets the resource name
● Must figure out what changes to the cluster are required
● Return value can be used to reschedule the resource

Controller Implementation: Creating Resources
● Operator-SDK provides an API
● Programmatically or by parsing templates

Cleanup and Garbage Collection
● Operators should allow clean uninstallation
● Finalizers and Owner References can help
● Use both with care
How to handle deprovision / deletion of the CR?

Finalizers
Finalizer has been set. Resource
won’t be deleted until removed.

Finalizers
Resource has
Finalizer(s)?
Delete
Resource
No
Set Delete
Timestamp
Reschedule
Yes
Done
Delete
Request

Owner References
Resource is owned by another
resource.

Owner References
Sets resource owner to the custom resource that triggered it’s creation

Tips for developing Operators
1. Be careful when setting the Owner Reference
2. Be careful with Finalizers
3. Allow for deletion in any Operator state
4. Use the Phase Pattern
5. Don’t (over)use the Kubernetes API
6. Never rely on local testing only

Be careful when setting the Owner Reference
Why?
● Cascaded Deletion is great
● Best case: no delete logic needed in Operator
● There are cases where it’s problematic
Example:
● Backup/Restore
● References an owner that now has a different UID
or is not yet created.

Tips for developing Operators
1. Always set the Owner Reference
2. Be careful with Finalizers
3. Allow for deletion in any Operator state
4. Use the Phase Pattern
5. Don’t (over)use the Kubernetes API
6. Never rely on local testing only

Be careful with finalizers
Why?
● Halts deletion
● Can propagate to parent resources (e.g.
namespaces)
Example:
● Uninstall script that removes all namespaces
● Blocked by finalizer in one resource
● Needs to be manually resolved now

Allow for deletion in any Operator state
Why?
● Operators can get stuck
● Give users a chance to make them unstuck
Example:
● Operator fails to create a resource (e.g. permissions)
● User requests deprovision
● Operator still in install phase

Allow for deletion regardless of the Operator Phase
How to fix this?
● No problem without finalizers
● Check the ‘deletionTimestamp’ in every state
● Always service finalizers

Use the Phase Pattern
What is it? ● Operators are state machines
● Every task can be seen as a state
● Clear control flow
Install Reconcile
Uninstall
(service finalizers)

Use the Phase Pattern
Check Phase
Take Action
Update Phase

Don’t (over) use the Kubernetes API
Why?
● Bypasses the Cache
● Causes performance issues
Example:
● Operator that use get/list excessively
● Should rely on the Informer instead
● Sometimes inevitable, e.g. resource creation

Never rely on local testing only
Why?
● You can test your Operator without deploying it
● Permissions will be different though
Example:
● Operator-sdk’s up local
● Starts the Operator locally without deploying it

What else is there?
● Dependency resolution
○ Operators can (and are expected to) deploy other operators
○ An example will be in the Demo

What else is there?
A better way to do dependency resolution?
Operator
Lifecycle Manager

What else is there?
OperatorHub: https://operatorhub.io/

Demo
(from zero to a monitoring stack)

Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINE
Thank you!
th

JS Fest 2019. Peter Braun. DevOps in the Cluster: A deep dive into operators

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a JS Fest 2019. Peter Braun. DevOps in the Cluster: A deep dive into operators

Similar a JS Fest 2019. Peter Braun. DevOps in the Cluster: A deep dive into operators (20)

Más de DevOps_Fest

Más de DevOps_Fest (20)

Último

Último (20)

JS Fest 2019. Peter Braun. DevOps in the Cluster: A deep dive into operators