This presentation describes the journey we went through in containerizing Spark workload into multiple elastic Spark clusters in a multi-tenant kubernetes environment. Initially we deployed Spark binaries onto a host-level filesystem, and then the Spark drivers, executors and master can transparently migrate to run inside a Docker container by automatically mounting host-level volumes. In this environment, we do not need to prepare a specific Spark image in order to run Spark workload in containers. We then utilized Kubernetes helm charts to deploy a Spark cluster. The administrator could further create a Spark instance group for each tenant. A Spark instance group, which is akin to the Spark notion of a tenant, is logically an independent kingdom for a tenant’s Spark applications in which they own dedicated Spark masters, history server, shuffle service and notebooks. Once a Spark instance group is created, it automatically generates its image and commits to a specified repository. Meanwhile, from Kubernetes’ perspective, each Spark instance group is a first-class deployment and thus the administrator can scale up/down its size according to the tenant’s SLA and demand. In a cloud-based data center, each Spark cluster can provide a Spark as a service while sharing the Kubernetes cluster. Each tenant that is registered into the service gets a fully isolated Spark instance group. In an on-prem Kubernetes cluster, each Spark cluster can map to a Business Unit, and thus each user in the BU can get a dedicated Spark instance group. The next step on this journey will address the resource sharing across Spark instance groups by leveraging new Kubernetes’ features (Kubernetes31068/9), as well as the Elastic workload containers depending on job demands (Spark18278). Demo: https://www.youtube.com/watch?v=eFYu6o3-Ea4&t=5s

1. Haohai Ma, IBM
Khalid Ahmed, IBM
RUNNING SPARK INSIDE
CONTAINERS

2. Myself
• “How High”
• Software Architect
• IBM Spectrum Computing
• Toronto Canada
2IBM Spectrum Computing

3. Agenda
• Why container?
• Migrate spark workload to container
• Spark instance on Kubernetes
– Architecture
– Workflow
– Multi-tenancy
• Future work
3IBM Spectrum Computing

4. Why use containers?
• To enforce the CPU and memory bounds.
– CPU shares are proportional to the allocated slots
– spark.driver.memory & spark.executor.memroy
• To completely isolate the file system
– Solve the dependency conflicts
• To create and ship images
– Develop once and run everywhere
4IBM Spectrum Computing

5. No prebuilt Spark image
• A running container needs an application image
– Independent to Spark versions
• Seamlessly migrate Spark workloads to a
container based environment
– Assume: Spark is distributed onto the host file system
5IBM Spectrum Computing

6. Host Filesystem
Spark
installation
Regular Spark workload
IBM Spectrum Computing
Spark Master
JVM
Spark Submit
Spark Driver
Spark Executor

7. Host Filesystem
Spark
installation
Running in containers
IBM Spectrum Computing
Spark Master
Spark Executor
container: ubuntu
JVM
Spark Submit
Spark Driver
container: ubuntu
container: image

8. Creating a container definition for an application
image
IBM Spectrum Computing
Extra dependency from
host file system

9. Submitting workload with the container
definition
9IBM Spectrum Computing
spark-submit --class<main-class> --master<master-url> --deploy-mode cluster
--conf spark.ego.driver.docker.definition= MyAppDef
--conf spark.ego.executor.docker.definition= MyAppDef
<application-jar>
[application-arguments]
Cluster Mode:
Define container specifications for
the drivers and executors

10. Host Filesystem
Spark
installation
Running in containers
IBM Spectrum Computing
Spark Master
Spark Executor
Container: myappimage:v1
Spark Submit
Spark Driver
container: myappimage:v1
Infobatch
lib

11. Spark Instance on Kubernetes
• Increase resource utilization
– Share nodes between Spark and surrounding ecosystem
• Isolation between tenants and apply resource
enforcement
– Each tenant gets a dedicated Spark working instance
– Tenant price plan can directly map to its resource quota
• Simplify deployment and roll out
11IBM Spectrum Computing

12. Architecture
IBM Spectrum Computing
IBM Spectrum Conductor with Spark
Spark Instance
Group
Spark Instance
Group
Spark
Master
History
Server
Notebook
Shuffle
Service
Spark
Master
History
Server
Notebook
Shuffle
Service
SparkaaS level
Tenant level
Spark instance group
Admin Portal
Spark Information Hub Image and deployment

13. Architecture
IBM Spectrum Computing
IBM Spectrum Conductor with Spark
Spark Instance
Group
Spark Instance
Group
Spark
Master
History
Server
Notebook
Shuffle
Service
Spark
Master
History
Server
Notebook
Shuffle
Service
Spark Service level
Tenant level
Spark Information Hub Image and deployment
Spark instance group
Admin Portal
• A Spark instance group is an independent
deployment in Kubernetes.
• A docker image is built automatically based on the
Spark version, configuration, notebook edition, and
user application dependencies.
• Initially only one container for Spark services of the
Spark instance group.
• Dynamic scalability based on workload demand.

14. Architecture
IBM Spectrum Computing
IBM Spectrum Conductor with Spark
Spark Instance
Group
Spark Instance
Group
Spark
Master
History
Server
Notebook
Shuffle
Service
Spark
Master
History
Server
Notebook
Shuffler
Service
SparkaaS level
Tenant level
Spark Information Hub Image and deployment
Spark instance group
Admin Portal
• IBM Spectrum Conductor with Spark - End points
• Admin manages Spark instance group life cycle
• Tenant accesses Spark workloads and notebooks
• Deploy by a helm chart and expose as a service with
one single container: CWS master
• Multiple deployments in a Kubernetes cluster
• Cloud: One deployment for one Spaas
• On-Prem: One deployment for one BU

15. Creating a master container Kubernetes

16. Creating a master container Kubernetes
Namespace: ns4bu1
container: CWS
spaas4bu1_cwsmaster

17. Creating a Spark instance group Kubernetes
Namespace: ns4bu1
container: CWS
spaas4bu1_cwsmaster
Registry
tenant1
Image

18. Deploying a Spark instance group Kubernetes
Namespace: ns4bu1
container: CWS
spaas4bu1_cwsmaster
container: tenant1
spaas4bu1_tenant1
Registry
tenant1
Image

19. Kubernetes
Namespace: ns4bu1
container: tenant1
container: tenant1 container: tenant1
Scaling the Spark instance group based on
workload demands
container: tenant1
scheduler
API Server
K8s master
Spark Master
Spark Driver
Spark ExecutorSpark Executor

20. Performance
• Without Dynamic Scaling

21. Performance
• With Dynamic Scaling
• Without Dynamic Scaling

22. Multitenancy with Spark instance groups Kubernetes
Namespace: ns4bu1
container: CWS
spaas4bu1_cwsmaster
Registry
container: tenant1
spaas4bu1_tenant1
container: tenant1
spaas4bu1_tenant1
container: tenant1
spaas4bu1_tenant1
container: tenant1
spaas4bu1_tenant1
container: tenant2
spaas4bu1_tenant2
container: tenant2
spaas4bu1_tenant2
container: tenant2
spaas4bu1_tenant2
tenant1
Image
tenant2
Image

23. Multi-Spaas Kubernetes

24. Registry
tenant1
Image
tenant2
Image
tenant3
Image
tenant4
Image
Multi-Spaas Kubernetes
Namespace: ns4bu1
container: CWS
spaas4bu1_cwsmaster
container: tenant1
spaas4bu1_tenant1
container: tenant1
spaas4bu1_tenant1
container: tenant1
spaas4bu1_tenant1
container: tenant1
spaas4bu1_tenant1
container: tenant2
spaas4bu1_tenant2 container: tenant2
spaas4bu1_tenant2
Namespace: ns4bu2
container: CWS
spaas4bu2_cwsmaster
container: tenant3
spaas4bu2_tenant3
container: tenant4
spaas4bu2_tenant4

25. Survey: Spark on Kubernetes
SPARK-18278 Standalone
IBM Spectrum Conductor
with Spark on
Kubernetes
Dynamic allocation on
demand
Yes Static Yes
K8s interaction granularity Job level Instance level – static Instance level –
dynamic
Deployment Automation
• Simple deploy by helm
charts
No Yes Yes
Spark instance per tenant
• Multi-job/workflow/user
• Image with user
applications
• Security
No limited Yes

26. Future work
• Integration with Kubernetes batch workload
scheduler
– Kube-arbitrator (https://github.com/kubernetes-
incubator/kube-arbitrator)
• Performance comparation with other Spark on
Kubernetes solutions
26IBM Spectrum Computing

27. www.ibm.com/spectrum-conductor
hma@ca.ibm.com
Thank You