A presentation on using Scala to build a K8s Operator that was given to the Nov 2018 Dublin Scala meet up.

1. Kubernetes Operators
Nov 2018Iris Team
With Scala

2. About Me
Bill
Javier
Peter Barron
Principal Engineer
IRIS
Conor
Located in Dublin,
IRIS is a technical
unit in Zalando that
is focused on ML and
Data Intensive
Systems.

3. What is an Operator?
“An Operator is a method of packaging,
deploying and managing a Kubernetes
application.” [1]
“Conceptually, an Operator takes human
operational knowledge and encodes it
into software that is more easily packaged
and shared with consumers.” [1]
[1] https://coreos.com/operators
[2] https://github.com/operator-framework/awesome-operators

4. Custom Resource Definitions
[1] https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: predictiondeployments.zalando.org
spec:
additionalPrinterColumns:
- JSONPath: .spec.scope
name: Scope
type: string
- JSONPath: .status.phase
name: Phase
type: string
- JSONPath: .metadata.creationTimestamp
description: Age of the stack
name: Age
type: date
group: zalando.org
version: v1alpha1
scope: Namespaced
names:
plural: predictiondeployments
singular: predictiondeployment
kind: PredictionDeployment
shortNames:
- pd
categories:
- all
validation:
openAPIV3Schema:
properties:
spec:
properties:
scope:
type: string
….
subresources:
status: {}

5. How does an Operator work?
Kind: KafkaCluster
Replicas: 2
…

6. So how do you build an
Operator in Scala?

7. Basic Building Block
implicit val system: ActorSystem = ActorSystem("Operator")
implicit val mat: ActorMaterializer = ActorMaterializer()
implicit val ec: ExecutionContext = system.dispatcher
val k8s: client.RequestContext = k8sInit
val stream: Future[(UniqueKillSwitch, Future[Done])] =
k8s.list[PredictionDeployment].map { l =>
k8s.watchAllContinuously[PredictionDeployment](Some(l.resourceVersion))
.viaMat(KillSwitches.single)(Keep.right)
.groupBy(10, we => we._object.name)
.map{ we =>
we._type match {
case ADDED => reconcile(we)
case MODIFIED => reconcile(we)
case DELETED => we
case ERROR => we
}
}.async.mergeSubstreams
.toMat(Sink.ignore)(Keep.both)
.run()
}
Determine
starting point
Create a Akka
Stream Source of
watch events
Process events
SKUBER
Create Sub Streams to
process in parallel

8. On Startup - Committer
implicit val system: ActorSystem = ActorSystem("Operator")
implicit val mat: ActorMaterializer = ActorMaterializer()
implicit val ec: ExecutionContext = system.dispatcher
val streamId = StreamId("pd")
val k8s: client.RequestContext = k8sInit
val committer: Committer = buildCommiter()
val stream = committer.retrieve[PredictionDeployment](streamId,Position.Latest).map(offset =>
k8s.watchAllContinuously[PredictionDeployment](offset)
.viaMat(KillSwitches.single)(Keep.right)
.groupBy(10, we => we._object.name)
.map { we =>
we._type match {
case ADDED => reconcile(we)
case MODIFIED => reconcile(we)
case DELETED => we
case ERROR => we
}
}.async.mergeSubstreams.map { we =>
committer.commit(streamId, Offset(we._object.metadata.resourceVersion))
}.toMat(Sink.ignore)(Keep.both) .run()
)
Create Kafka like
committer
Retrieve offset,
defaulting to latest
position is non exist
Commit offset

9. On Startup - Alternative
implicit val system: ActorSystem = ActorSystem("Operator")
implicit val mat: ActorMaterializer = ActorMaterializer()
implicit val ec: ExecutionContext = system.dispatcher
val k8s: client.RequestContext = k8sInit
val stream= k8s.list[PredictionDeploymentList].map { l =>
Source(l.items.map(l => WatchEvent(MODIFIED, l))).concat(
k8s.watchAllContinuously[PredictionDeployment](Some(l.resourceVersion))
).viaMat(KillSwitches.single)(Keep.right)
.groupBy(10, we => we._object.name)
.map { we =>
we._type match {
case ADDED => reconcile(we)
case MODIFIED => reconcile(we)
case DELETED => we
case ERROR => we
}
}.async.mergeSubstreams
.toMat(Sink.ignore)(Keep.both).run()
}
Retrieve current state of
Prediction Deployment
resources.
Use the resource version
from the list operator to
subsequent changes

10. Reconciling
The reconciler function ensures that
the state of the system matches the
specification defined by the resource.
Derive
Expected
State
Determine
Current
State
Calculate
Difference
Apply
changes

11. Reconciling
EitherT(
k8s.create[O](resource).map(Right(_)).recoverWith {
case ex: K8SException if ex.status.code.contains(409) =>
k8s.get[O](resource.name).flatMap { existing =>
if (existing.metadata.ownerReferences.contains(ORef(pd))) {
if (!resource.subsetOf(existing)) {
k8s.update[O](existing.merge(resource))
.map(Right(_))
.recoverWith(K8sErrorHandler())
} else {
Future.successful(Right(existing))
}
} else {
Future.successful(Left(DuplicateService))
}
}
}.recoverWith(
K8sErrorHandler()
)
)
}
Optimistically attempt to
create the resource.
If that fails resolve the
conflict and update if
necessary
Subset is an implicit
function that determines
if the resource needs to
be updated.
Merge is an implicit
function that combines
the resources.

12. Working with Multiple Resource Types
val pdSource = k8s.list[PredictionDeploymentList].map { l =>
Source(l.items.map(l => WatchEvent(MODIFIED, l))).concat(
k8s.watchAllContinuously[PredictionDeployment](Some(l.resourceVersion))
)
}
val dSource = k8s.list[DeploymentList].map { l =>
k8s.watchAllContinuously[Deployment](Some(l.resourceVersion))
}
Future.sequence(List(pdSource, dSource)).map(s =>
Source.combine(s.head, s(1))(Merge(_))
.viaMat(KillSwitches.single)(Keep.right)
.groupBy(10, we => we._object.name)
.map {
case `WatchEvent[PredictionDeployment]`(we) => we
case `WatchEvent[Deployment]`(we) => we
}.async.mergeSubstreams
.toMat(Sink.ignore)(Keep.both).run()
)
Combine sources into a
single source
Use shapeless ‘TypeCase’
to avoid type erasure

13. Demo

14. Demo - Outline
Package
Deploy
Publish

15. Questions