Sebastien Thomas, System Architect at Coyote Amerique, gave a presentation on operator frameworks. His talk covered how Operator SDK can be used to create Kubernetes Operators with Go.

1. https://github.com/operator-framework/operator-sdk
K8s / CNCF Meetup - 2019/02/25
Operator Framework
From Github Project :
The Operator Framework is an open source toolkit to manage Kubernetes
native applications, called Operators, in an effective, automated, and scalable
way.
prune@lecentre.net

2. Agenda
1. Overview of Operators
2. Operator SDK usage
3. Operators workflow
4. Hands On
5. Conclusion

3. COYOTE SYSTEM
Who are we ?
A leading provider of community-based driving assistant systems
Founded in 2005 350 employees
1M daily users on a payed
subscription model
2 billion kilometers
traveled every month (1,24
billion miles)
50M members declaration
analyzed every month
Specific know-how in big
data and automotive market
protected by 13 patents

4. COYOTE SYSTEM
How to be part of the community ?
• Available on a range of Products and Apps,
• … but also with embedded car solutions
COYOTE mini
COYOTE S
COYOTE NAV+
Smartphone APPs

5. COYOTE SYSTEM
Where does it work ?
… almost everywhere in Europe !
› France
› Belgium
› Netherlands
› Luxembourg
› Italy
› Spain
› Germany
› Poland
› Portugal

6. Who I am ?
20+ years in Computers / Network / Admin / Devops / Woodworker
Work at Coyote https://www.moncoyote.com/ as System Architect
Github : https://github.com/prune998
Blog (sort of) : https://medium.com/@prune998
Coyote Lab Blog (more to come there) : https://www.mycoyote.ca/blog
Contact : Sebastien “Prune” THOMAS - prune@lecentre.net

7. What’s an Operator ?
An Operator is an application that deals with the Kubernetes API and Custom
Resources to create/operate new Resources.
It’s an intelligent piece of software that embed the templating to deploy your
resources.
The Operator watch events on the K8s API and react (ex : re-create a pod,
change Labels, update a Secret, Remove a Service…)

8. What are Custom Resource Definition
CRD are new Resources, like Pods, Deployments, Secrets that you can create.
They are managed through the K8s API the same way as official resources
kubectl get crd
certificates.certmanager.k8s.io 2019-01-25T15:56:53Z
certmerges.certmerge.lecentre.net 2019-01-25T15:57:10Z
prometheuses.monitoring.coreos.com 2019-01-25T16:05:42Z
prometheusrules.monitoring.coreos.com 2019-01-25T16:05:44Z
virtualservices.networking.istio.io 2019-01-25T16:09:16Z
...

9. Example Custom Resource
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: certmerges.certmerge.lecentre.net
spec:
group: certmerge.lecentre.net
names:
kind: CertMerge
listKind: CertMergeList
plural: certmerges
singular: certmerge
scope: Namespaced
version: v1alpha1
apiVersion: certmerge.lecentre.net/v1alpha1
kind: CertMerge
metadata:
name: "test-certmerge-labels"
spec:
selector:
- labelselector:
matchLabels:
env: "dev"
certmerge: "true"
namespace: default
name: test-cert-labels
namespace: default
Custom Resource Definition (CRD) Custom Resource Manifest (CR)

10. Before Operators

11. With Operators

12. Operators 1
2
3
4
5
6
7
8
9

13. Operators

14. Difference with other tools
- Helm / Jsonnet / Ksonnet
They are templating tools. Create a template, set some variables, generate
the Manifests. Once deployed they have no control (tiller does not count).
- StatefulSets / Deployments / Pods
They are K8s Resources. Some minimal feedback to scale/restart, no
dependency between them, no intelligence in management.
- Operators
Watch the K8s API and react in real time. Can have a better control to
scale/restart/configure the target application, with richer features than
Readyness/Liveness Probes

15. Who needs Operators ?
You may need an Operator if :
- you need to use many times the same Application. ex : deploying one EtcD
cluster in each Namespace
- You need to automate some Resource creation. ex : create some SSL
Certificates inside Secrets (cert-manager), create Prometheus scraping rules
- You need more intelligence in the management. ex : the Etcd-Operator create
and manage Pods directly instead of using a Deployment or StatefulSets

16. Helm Chart to deploy an Operator ?
- Operators are usually easy to deploy
- use whatever mean you have to deploy them (Helm, Jsonnet, plain manifest
from the Operator creator)
- Once the Operator is running, use the Custom Resources to trigger its power

17. Existing Operators ?
- Etcd-Operator : https://github.com/coreos/etcd-operator
- Kafka : https://github.com/strimzi/strimzi-kafka-operator
- Nats : https://github.com/nats-io/nats-operator
- Prometheus : https://github.com/coreos/prometheus-operator
- SSL Certificates : https://github.com/jetstack/cert-manager
- RBAC-Manager : https://github.com/reactiveops/rbac-manager
and a lot more, growing...
(check https://github.com/operator-framework/awesome-operators)

18. Operator all the thing ?
An Operator embed the knowledge and the deployments “templates”.
Don’t create an operator :
- if your application deployment is not stable !
- to deploy one application per cluster (it’s easier to template it)
Create an Operator :
- if you have many users in need to use your resource
- you have a complicated workflow to handle your resource
- you want to (learn to) code in GO (or check other languages operators too)

19. Operator Creation
Operator SDK (Go) : https://github.com/operator-framework/operator-sdk
- High level APIs and abstractions to write the operational logic more intuitively
- Tools for scaffolding and code generation to bootstrap a new project fast
- Extensions to cover common operator use cases
- Base on official Kubernetes API packages
- Provide common package for leader election for HA Operators

20. CertMerge Operator - github.com/prune998/certmerge-operator

21. Install (fast)
mkdir -p $GOPATH/src/github.com/operator-framework
cd $GOPATH/src/github.com/operator-framework
git clone https://github.com/operator-framework/operator-sdk
cd operator-sdk
git checkout v0.4.0
make dep
make install
operator-sdk --version
operator-sdk version v0.4.0+git

22. Create your operator
mkdir -p $GOPATH/src/github.com/prune998/
cd $GOPATH/src/github.com/prune998/
operator-sdk new certmerge-operator --cluster-scoped
INFO[0000] Create pkg/apis/apis.go
INFO[0000] Create pkg/controller/controller.go
INFO[0000] Create version/version.go
INFO[0000] Create .gitignore
INFO[0000] Create Gopkg.toml
INFO[0000] Run dep ensure ...
INFO[0068] Run dep ensure done
INFO[0068] Run git init ...
INFO[0074] Run git init done
INFO[0074] Project creation complete.
INFO[0000] Creating new Go operator 'certmerge-operator'.
INFO[0000] Create cmd/manager/main.go
INFO[0000] Create build/Dockerfile
INFO[0000] Create build/bin/entrypoint
INFO[0000] Create build/bin/user_setup
INFO[0000] Create deploy/service_account.yaml
INFO[0000] Create deploy/role.yaml
INFO[0000] Create deploy/role_binding.yaml
INFO[0000] Create deploy/operator.yaml

23. Add API
# Add a new API for the custom resource AppService
operator-sdk add api
--api-version=certmerge.lecentre.net/v1alpha1
--kind=CertMerge
This is the basic operation to create the CRD.
It creates files in pkg/apis/certmerge/v1alpha1 including certmerge_types.go which holds the definition of the
CRD :
…
type CertMerge struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec CertMergeSpec `json:"spec,omitempty"`
Status CertMergeStatus `json:"status,omitempty"`
}
...

24. Add Controler
# Add a new controller that watches for AppService
operator-sdk add controller
--api-version=certmerge.lecentre.net/v1alpha1
--kind=CertMerge
Creates files in pkg/controller/certmerge. This is where all your watch and reconcile logic happens
Check doc reference at
https://godoc.org/github.com/kubernetes-sigs/controller-runtime/pkg#hdr-Controller

25. type CertMerge struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec CertMergeSpec `json:"spec,omitempty"`
Status CertMergeStatus `json:"status,omitempty"`
}
type CertMergeSpec struct {
SecretName string `json:"name"`
Selector []SecretSelector `json:"selector"`
SecretNamespace string `json:"namespace"`
SecretList []SecretDefinition `json:"secretlist"`
}
type SecretSelector struct {
LabelSelector metav1.LabelSelector `json:"labelselector"`
Namespace string `json:"namespace"`
}
Types (API)

26. // add adds a new Controller to mgr with r as the reconcile.Reconciler
func add(mgr manager.Manager, r reconcile.Reconciler) error {
// Create a new controller
c, err := controller.New("certmerge-controller", mgr, controller.Options{Reconciler: r})
if err != nil { return err }
// Watch for changes to primary resource CertMerge
err = c.Watch(&source.Kind{Type: &certmergev1alpha1.CertMerge{}}, &handler.EnqueueRequestForObject{})
if err != nil { return err }
// TODO(user): Modify this to be the types you create that are owned by the primary resource
// Watch for changes to secondary resource Pods and requeue the owner CertMerge
err = c.Watch(&source.Kind{Type: &corev1.Secret{}}, &handler.EnqueueRequestForOwner{
IsController: true,
OwnerType: &certmergev1alpha1.CertMerge{},
})
if err != nil { return err }
Watchers

27. func (r *ReconcileCertMerge) Reconcile(request reconcile.Request) (reconcile.Result, error) {
…
// Fetch the CertMerge instance that triggered this Reconsile
instance := &certmergev1alpha1.CertMerge{}
err := r.client.Get(context.TODO(), request.NamespacedName, instance)
if err != nil {
if errors.IsNotFound(err) {
// Request object not found, could have been deleted after reconcile request.
// Owned objects are automatically garbage collected. For additional cleanup logic use finalizers.
// Return and don't requeue ( by sending `nil` in the error field)
return reconcile.Result{}, nil
}
// Error reading the object - requeue the request. (by sending a non-nil error)
return reconcile.Result{}, err
}
… do some stuff for your operator (see next slide)
}
Reconcile

28. // Define a new Secret object
secret := newSecretForCR(instance)
// create the DATA for the new secret based on the CertMerge request
certData := make(map[string][]byte)
// Set CertMerge instance as the owner and controller (for garbage collection)
if err := controllerutil.SetControllerReference(instance, secret, r.scheme); err != nil {
return emptyRes, err
}
// build the Cert Data from the secret List provided in the CertMerge Custom Resource
if len(instance.Spec.SecretList) > 0 {
for _, sec := range instance.Spec.SecretList {
secContent, err := r.searchSecretByName(ctx, sec.Name, sec.Namespace)
...
certData[sec.Name+".crt"] = secContent.Data["tls.crt"]
certData[sec.Name+".key"] = secContent.Data["tls.key"]
}
}
// add the Data to the secret
secret.Data = certData
// create the new secret
if err := r.client.Create(ctx, secret); err != nil {...}
Reconcile 2

29. Generate and build
# re-generate all the files that depend on the CRD API
operator-sdk generate k8s
# re-generate the CRD Manifest (rarely used, when you change your API name)
operator-sdk generate openapi
# build the operator (aka go build)
operator-sdk build prune/certmerge-operator:v0.0.1

30. Operator workflow (easy)

31. Operator workflow (harder with Predicate)

32. Hands-on
switch to the console / code

33. Conclusion
● Operator SDK make it really easy
● using K8s primitives (and go-client), not “vendor” dependent
● Operators can be declined in Controlers (admission)
● You need to learn a little bit of the K8s API to get to cool stuff

34. References
● https://github.com/operator-framework/operator-sdk
● https://github.com/operator-framework/operator-sdk/blob/master/doc/user-guide.md
● https://godoc.org/github.com/kubernetes-sigs/controller-runtime/pkg#hdr-Controller
● https://github.com/operator-framework/operator-sdk/blob/master/doc/user/event-filtering.md