4/4 Webinar: How to Automate Deployment and Orchestration of Application (MiCADO introduction) This part of the webinar covers the software JQueuer, with JQueuer the computing a large number of jobs results in significant overall execution time. It was presented by Amjad Ullah (University of Westminster). The webinar took place on the 26th of September 2019. If you would like to have more information visit: https://micado-scale.eu MiCADO is open-source and a highly customisable multi-cloud orchestration and auto-scaling framework for Docker containers, orchestrated by Kubernetes. Developed by Project COLA funded by the European Commission (grant agreement no: 731574). https://project-cola.eu

1. Execution of deadline-
constrained workloads using
jQueuer and MiCADO
Amjad Ullah
University of Westminster
September 26th 2019

2. Webinar on MiCADO [26/09/2019]
Agenda
• Introduction
1. Jozsef Kovacs (SZTAKI): What is MiCADO (good for)? [15 mins]
2. James Deslauriers (WMIN): How to develop an application in MiCADO? [15 mins]
• Questions [5 mins]
• Use cases
3. Matt Kendon (Outlandish): Wordpress [10 mins]
4. Amjad Ullah (WMIN): JQueuer [10 mins]
• Questions [5 mins]

3. Multi-job experiments
www.project-cola.eu 3
• Large number of jobs results in significant overall execution time
• Restricted to complete all jobs by a deadline
• Where to put the jobs?
• How to distribute?
• How to execute (in containers)
• How to liaise with deadline?
MICADO
MASTER
ADT:
infrastructure
and scaling
rules
End user
MiCADO
Submitter
Policy
Keeper
(Scaling
logic)
cqueue
worker
MICADO
WORKER
cqueue
workerJobs
Scale up/
down
jQUEUER
MASTER jQueuer
Agent
Jobs
experiment
.json
Container and
Cloud
Orchestrator
MICADO
Queuing system
jobs
September 26th 2019

4. JQueuer
www.project-cola.eu 4
• Asynchronous Distributed Task Queue
• Master/Server Component
• Runs externally
• Queue & monitoring
• Agent Component
• Runs on worker VMs
• Fetch & execute jobs
MICADO
MASTER
ADT:
infrastructure
and scaling
rules
End user
MiCADO
Submitter
Policy
Keeper
(Scaling
logic)
cqueue
worker
MICADO
WORKER
cqueue
workerJobs
Scale up/
down
jQUEUER
MASTER jQueuer
Agent
Jobs
experiment
.json
Container and
Cloud
Orchestrator
MICADO
MICADO
MASTER
ADT:
infrastructure
and scaling
rules
End user
MiCADO
Submitter
Policy
Keeper
(Scaling
logic)
cqueue
worker
MICADO
WORKER
cqueue
workerJobs
Scale up/
down
jQUEUER
MASTER jQueuer
Agent
Jobs
experiment
.json
Container and
Cloud
Orchestrator
MICADO
September 26th 2019

5. Demo: Agent based simulation framework
September 26th 2019 www.project-cola.eu 5
• Experiment: Determining the impact of changes in behaviour on the
spread of a disease across a population
• Tool: Repast Simphony
• Total jobs: 200
• Deadline: 1 hour
• Max VMs to use: 10

6. Execution using jQueuer + MiCADO
September 26th 2019 www.project-cola.eu 6
experiment.json JQueuer
Manager
MiCADO Worker 1
JQueuer
Agent
Repast 1
MiCADO
Master
200
jobs
MiCADO Worker 2
JQueuer
Agent
Repast 2
MiCADO Worker n
JQueuer
Agent
Repast n
1-hour
deadline
Calculated by scaling policy

7. jQueuer deployment using ADT
September 26th 2019 www.project-cola.eu 7

8. Deadline-based scaling policy
September 26th 2019 www.project-cola.eu 8

9. Results
September 26th 2019 www.project-cola.eu 9
• Dynamic allocation of
variable length jobs
• Results in better use of
cloud resources
Manually: 5 VMs MiCADO: 3.86 VMs

10. Thanks and Questions!
September 26th 2019 www.project-cola.eu 10
• github.com/micado-scale
• micado-scale.eu/
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731574