The document discusses the Spark Operator, which allows deploying, managing, and monitoring Spark clusters on Kubernetes. It describes how the operator extends Kubernetes by defining custom resources and reacting to events from those resources, such as SparkCluster, SparkApplication, and SparkHistoryServer. The operator takes care of common tasks to simplify running Spark on Kubernetes and hides the complexity through an abstract operator library.

1. WIFI SSID:Spark+AISummit | Password: UnifiedDataAnalytics

2. Jiri Kremser, Red Hat
Spark Operator
Deploy, Manage and Monitor Spark clusters
on Kubernetes
#UnifiedDataAnalytics #SparkAISummit

3. 3#UnifiedDataAnalytics #SparkAISummit

4. 4#UnifiedDataAnalytics #SparkAISummit
Deployment
StatefulSet
Job
Pod
Service
ReplicationController

5. Manifest Nightmares
5#UnifiedDataAnalytics #SparkAISummit

6. Operator Pattern
• Extends Kubernetes
• Resources and Controllers
• Custom Resource Definitions (CRD)
• Reacts on events when resource is CRUDed
• Sometimes referred as Custom Controllers
6#UnifiedDataAnalytics #SparkAISummit

7. Operator<X> - example
7#UnifiedDataAnalytics #SparkAISummit
Operator K8s API
I am listening on CR<X>
CR<X> …. CustomResource representing the desired configuration of X

8. Operator<X> - example
8#UnifiedDataAnalytics #SparkAISummit
Operator K8s API
OK, whatever
¯_( ツ )_/¯

9. Operator<X> - example
9#UnifiedDataAnalytics #SparkAISummit
Operator K8s API
Hey! New resource

10. Operator<X> - example
10#UnifiedDataAnalytics #SparkAISummit
Operator K8s API
Beep!Beep!
Boop!Zzzz!
⚡⚡

11. Comparison
Operator can be seen merely as deployment
mechanism, but it can do much more
• Kubernetes manifests
• Helm Chart
• Ansible
• Kustomize
• Ksonnet
11#UnifiedDataAnalytics #SparkAISummit

12. Operator minimal example
namespace=${WATCH_NAMESPACE:-default}
base=http://localhost:8001
ns=namespaces/$namespace
curl -N -s $base/api/v1/${ns}/configmaps?watch=true |
while read -r event
do
# ...
done
12#UnifiedDataAnalytics #SparkAISummit

13. Spark Operator
• Started as toy project
• Adopted by AI-CoE project OpenDataHub.io
• Compatible with Spark operator from Google to
avoid vendor lock-in
• Available also in operatorhub.io or Helm chart
or using ansible role
13#UnifiedDataAnalytics #SparkAISummit

14. Spark Operator
14#UnifiedDataAnalytics #SparkAISummit
Reacts on events from these custom resources:
• SparkCluster
• SparkApplication
• SparkHistoryServer

15. Spark Operator
15#UnifiedDataAnalytics #SparkAISummit
Reacts on events from these custom resources:
• SparkCluster
• SparkApplication
• SparkHistoryServer
Full schema captured by JSON schema

16. 16#UnifiedDataAnalytics #SparkAISummit

17. 17#UnifiedDataAnalytics #SparkAISummit

18. Spark Operator
18#UnifiedDataAnalytics #SparkAISummit
Reacts on events from these custom resources:
• SparkCluster
• SparkApplication
• SparkHistoryServer

19. Fabric8 Kubernetes client
Fluent API
Type-safety
Takes the credentials from:
• kube config file
• service account token & mounted CA cert
19#UnifiedDataAnalytics #SparkAISummit

20. Abstract Operator Library
• Automates the common tasks
• User has to only extend the class and override
couple of methods.
• Supports JSON schema as the representation
of the configuration.
• CRDs and CMs supported
20#UnifiedDataAnalytics #SparkAISummit

21. Dependencies
21#UnifiedDataAnalytics #SparkAISummit
operator-parent-pom
spark-operator abstract-operator kubernetes-client
depends on
has parent

22. Tooling
22#UnifiedDataAnalytics #SparkAISummit
• Soit – Python CLI that verifies if container
image is “operator compliant”
• Ansible role – it supports also deploying
Prometheus together with the operator
• Oshinko-temaki – CLI that produces valid
yamls with custom resources for the operator
All the tools are available in the readme file

23. Metrics
23#UnifiedDataAnalytics #SparkAISummit
• Endpoints for Prometheus
• Operator metrics (including JVM metrics)
• Metrics from deployed Spark clusters

24. 24#UnifiedDataAnalytics #SparkAISummit
22

25. Takeaways
25#UnifiedDataAnalytics #SparkAISummit
• Spark on K8s can be easy
• Operator can hide complexity
• Operators can be done in any language
• Hopefully in Spark:
http://bit.ly/spark-op-pr

26. DON’T FORGET TO RATE
AND REVIEW THE SESSIONS
SEARCH SPARK + AI SUMMIT