Luca Abeni - Real-Time Virtual Machines with Linux and kvm

Real-Time Virtual
Machines with Linux
and kvm
Luca Abeni
luca.abeni@santannapisa.it

Real-Time Application
LinuxLab 2018 Luca Abeni – 2 / 36
■ The time when a result is produced matters
◆ A correct result produced too late is equivalent to a
wrong result (or to no result)
■ What does “too late” mean, here?
◆ Applications characterised by temporal constraints
■ Temporal constraints are modelled through deadlines
◆ Finish some activity before a time (deadline)
◆ Generate some data before a deadline
◆ Terminate some process/thread before a deadline
◆ ...

Real-Time Task
Task (process or thread): sequence of actions characterized
by deadlines and maximum execution time. Example:
periodic task with period 8 and maximum execution time 3.
0 2 4 6 8 10 12 14 16 18 20 22 24
τ1
Note: while the ﬁrst and the third jobs execute for 3 time
units the second one executes for only 2 time units.

Formal Task Model
■ Real-Time task τi: stream of jobs (or instances) Ji,k
■ Each job Ji,k = (ri,k, ci,k, di,k):
◆ Arrives at time ri,k (activation time)
◆ Executes for a time ci,k
◆ Finishes at time fi,k
◆ Should ﬁnish within an absolute deadline di,k
ri,k
fi,k
di,k
ci,k

Periodic Tasks
Periodic task τi = (Ci, Di, Pi): stream of jobs Ji,k, with
ri,k+1 = ri,k + Pi
di,k = ri,k + Di
Ci = max
k
{ci,k}
■ Pi is the task period;
■ Di is the task relative deadline;
■ Ci is the task worst-case execution time (WCET);
■ Ri is the worst-case response time:
Ri = maxk{ρi,k} = maxk{fi,k − ri,k};
◆ Constraints respected ⇔ Ri ≤ Di.

Periodic Task Model
■ A periodic task has a regular structure (cycle):
◆ activate periodically (period Pi)
◆ execute a computation
◆ suspend waiting for the next period
void ∗ PeriodicTask ( void ∗ arg )
{
<i n i t i a l i z a t i o n >;
<s t a r t p e r i o d i c timer , period = P>;
while ( cond ) {
<do something . . . > ;
<wait next a c t i v a t i o n >;
}
}

Real-Time Scheduling
■ A real-time task τi = (Ci, Di, Pi) (or τi = (Ci, Pi) if
Di = Pi) is properly served if all jobs respect their
deadline...
■ ...Appropriate scheduling is important!
◆ The scheduler must somehow know the temporal
constaints of the tasks...
◆ ...In order to schedule them so that such temporal
constraints are respected
■ How to schedule real-time tasks? (scheduling algorithm)
■ Is it possible to respect all the deadlines?
■ Does Linux provide an appropriate scheduling algorithm?

Fixed Priorities
■ The Linux fixed-priority scheduler is often used for
real-time tasks...
■ ...Is this ok in general, or only in some cases?
◆ Given a set of real-time tasks Γ = {τi}, can a fixed
priority scheduler allow to respect all the deadlines?
◆ Is it possible to know in advance if some deadline will
be missed?
◆ How to assign the priorities?
■ If fixed priorities are enough, the SCHED FIFO and
SCHED RR policies can be used!

Schedulability Analysis
■ Given a set of tasks, is it possible to know in advance if
deadlines will be missed?
◆ A task is schedulable if it will not miss any deadline
■ Depends on the task
■ Depends on the scheduler
■ Depends on the other tasks, ...
■ Schedulability test: mathematical test to check if
τ = (C, D, P) will miss any deadline
◆ Possible idea: compute the amount of time needed
by all the tasks to respect all their deadlines in an
interval (t0, t1) and compare with t1 − t0

Partitioned Scheduling
■ How to schedule tasks on multiple CPUs / cores?
◆ First idea: partitioned scheduling
■ Statically assign tasks to CPUs
■ Reduce the problem of scheduling on M CPUs to M
instances of uniprocessor scheduling
CPU CPU CPU CPU
M

Global Scheduling
■ One single task queue, shared by M CPUs
◆ The first M ready tasks from the queue are selected
◆ What happens using fixed priorities (or EDF)?
◆ Tasks are not bound to specific CPUs
◆ Tasks can often migrate between different CPUs
■ Problem: schedulers designed for UP do not work well
M
CPU CPU CPU CPU
{M

Fixed Priorities in Linux
■ SCHED FIFO and SCHED RR use ﬁxed priorities
◆ They can be used for real-time tasks, to implement
RM and DM
◆ Real-time tasks have priority over non real-time
(SCHED OTHER) tasks
■ Diﬀerence between the two policies: visible when more
tasks have the same priority
■ The administrator can use sched setscheduler() (or
similar) to schedule real-time tasks
◆ Might be a little bit impractical, but can be used
■ However...

Periodic Task Example
■ Consider a periodic task
/∗ . . . ∗/
while (1) {
/∗ Job body ∗/
c l o c k n a n o s l e e p (CLOCK REALTIME ,
TIMER ABSTIME , &r , NULL ) ;
timespec add us (&r , period ) ;
}
■ The task expects to be executed at time r (= r0 + jT)...
■ ...But is sometimes delayed to r0 + jT + δ

Theoretical Schedule
0 2 4 6 8 10 12 14 16 18 20 22
τ1
τ2

Actual Schedule
0 2 4 6 8 10 12 14 16 18 20 22
τ1
τ2
■ What happens if the 2nd
job of τ1 arrives a little bit
later???
◆ The 2nd
job of τ2 misses a deadline!!!

Theory vs Real Schedule
■ The delay δ in scheduling a task is due to kernel latency
■ Kernel latency can be modelled as a blocking time
■ In real world, high priority tasks often suﬀer from
blocking times coming from the OS (more precisely, from
the kernel)
◆ Why?
◆ How?
◆ What can we do?
■ To answer the previous questions, we need to recall how
the hardware and the OS work...

Latency
■ Latency: measure of the diﬀerence between the
theoretical and actual schedule
◆ Task τ expects to be scheduled at time t . . .
◆ . . . but is actually scheduled at time t′
◆ ⇒ Latency L = t′
− t
■ The latency L can be modelled as a blocking time ⇒
aﬀects the guarantee test
◆ Similar to what done for shared resources
◆ Blocking time due to latency, not to priority inversion

Eﬀects of the Latency
■ Upper bound for L? If not known, no schedulability
tests!!!
◆ The latency must be bounded: ∃Lmax
: L < Lmax
■ If Lmax
is too high, only few task sets result to be
schedulable
◆ Large blocking time experienced by all tasks!
◆ The worst-case latency Lmax
cannot be too high

Latency in Linux
■ Tool (cyclictest) to measure the latency
■ Vanilla kernel: depends on the conﬁguration
◆ Can be tens of milliseconds
■ PREEMPT RT patchset
(https://wiki.linuxfoundation.org/realtime):
reduce latency to less than 100 microseconds
◆ Tens of microseconds on well-tuned systems!
■ So, scheduling real-time tasks in Linux is not an issue
anymore...

Real-Time in VMs???
■ So, serving real-time applications in Linux-based systems
is not a problem today...
■ ...Can real-time applications run in virtual machines?
◆ Real-Time in Virtual Machines??? But... Why?
■ Component-Based Development
◆ Complex applications: sets of smaller components
◆ Both functional and temporal interfaces
■ Security
◆ Isolate real-time applications in a VM
■ Easy deployment
■ ...

Real-Time Components
■ Assume to have N components...
◆ Each component is described by its interface...
◆ ...But a description of its temporal behaviour is also
somehow provided...
■ Component are assumed to respect some temporal
constraints
◆ Some kind of “contract” provided by real-time
guarantees
■ Is it possible to combine the components so that the
timing of the system is predictable?
◆ Can real-time guarantees be combined? How?

Combining Real-Time Apps
?
■ Real-time analysis for each application / in each VM...
■ What about the resulting system?

1 Real-Time Task per VM
■ Single real-time thread/process in a VM → combining
VMs is easy...
◆ VMs can be modelled as real-time tasks
◆ Simple (C, D, T) model, or something more
complex...
■ What about VMs containing multiple real-time tasks?
◆ Model for a VM?
◆ How to summarize the VM’s temporal requirements?
◆ Scheduler in the VM? How to model it?

More Complex VMs
■ Example: component Ci
composed by ni
tasks
■ More complex model... How to handle it?
◆ We only know how to schedule single tasks...
◆ And we need to somehow “summarise” the
requirements of a component!
Ci
= {(Ci
0, Di
0, Ti
0), (Ci
1, Di
1, Ti
1), . . . , (Ci
ni, Di
ni, Ti
ni)}
■ So, 2 main issues:
1. Describe the temporal requirements of a VM in a
simple way
2. Schedule the VMs, and somehow “combine” their
temporal guarantees

Practical Issues
■ VMs can contain sets of real-time tasks
◆ The scheduler only sees a VM per taskset, but
cannot see the tasks inside it
◆ Para-virtualization (of the OS scheduler) could be
used to address this issue, but it is not so simple...
■ So, how to schedule VMs?
■ Two-level hierarchical scheduling system
◆ Host (global / root) scheduler, scheduling VMs
◆ Each VM contains its (local / 2nd level) scheduler

2-Levels Hierarchy
1τ 1τ
1ττn
τn
Local Scheduler Local Scheduler Local Scheduler
2
τn
τ
Global
Scheduler
■ Global Scheduler assigns CPU to virtual machines
■ Local Schedulers assign CPU to single tasks

Hierarchical Scheduling
■ The global scheduler does not “see” all the tasks
■ Each VM has its own scheduler (fixed priorities for
real-time tasks)
◆ Global scheduler: host / hypervisor scheduler
◆ Local scheduler: guest scheduler
◆ No problems in assigning fixed priorities to tasks!
■ Problem: what to use as a global scheduler?
◆ We must have a model for it
◆ Must allow to compose the “local guarantees”
◆ This time, fixed priorities are not enough :(

Schedulability Analysis
■ (Over?)Simplifying things a little bit...
■ ...Suppose to know the amount of time needed by a VM
to respect its temporal constraints and the amount of
time provided by the global scheduler
■ A VM is “schedulable” if
demanded time ≤ supplied time
◆ “demanded time”: amount of time (in a time
interval) needed by a VM
◆ “supplied time”: amount of time (in a time interval)
given by the global scheduler to a VM
■ Of course the devil is in the details

Supplied Time
■ Description of the root scheduler temporal behaviour
■ More formally:
◆ Depends on the time interval t we are considering
◆ Depends on the root scheduler S
■ Minimum amount of time given by S to a VM in a time
interval of size s
◆ Given all the time interval (t0, t1) : t1 − t0 = s...
◆ ...Compute the size of the sub-interval in which the
VM is scheduled...
◆ ...And then ﬁnd the minimum!
■ Fixed priorities do not guarantee a minimum time to the
VM!!!

Host Scheduling
■ Issue: how to guarantee a minimum amount of CPU
time to each VM?
◆ As said, SCHED FIFO / SCHED RR are not ok...
■ SCHED DEADLINE policy: schedule a thread / process
for an amount of time Q every period P
◆ Q: runtime
◆ P: reservation period
■ This is what we need!!! Allows to compute a supply
function!

In Practice...
■ Real-Time applications running in a VM?
◆ As for OSs, two diﬀerent aspects
■ Resource allocation/management (scheduling)
■ Latency (host and guest)
◆ CPU allocation: interesting issues with multiple
vCPUs
■ Virtualisation: full hw virtualisation or OS-level
virtualisation
◆ Full hw virtualisation: focus on kvm
◆ OS-level virtualisation: containers (lxc, Docker, ...)

kvm-Based VMs
■ kvm: Linux driver to use the CPU virtualisation
extensions (Intel VT-X and similar technologies)
◆ User-space VMM (qemu, kvmtool, TinyEMU,
Amazon’s ﬁrecracker...) for virtual devices, etc...
◆ A thread per virtual CPU (vCPU thread)
■ Use SCHED DEADLINE for the vCPU threads
◆ Real-Time theory provides some (boring) algorithms
to dimension Q and P
■ Global guest scheduler ← para-virtualised scheduling!!!
◆ To always schedule on physical CPUs the highest
priority guest tasks

Implementation
■ Use PREEMPT RT for the host kernel
■ Use PREEMPT RT for the guest kernel
■ Use kvm-based VMs
◆ qemu as a user-space VMM (you can use kmvtool or
similar tools, too)
■ After starting the VM, use SCHED DEADLINE for the
vCPU threads
◆ Runtime and period computed according to the
real-time load of the VM
◆ There are existing tools to compute the values
■ Use partitioned ﬁxed priority scheduling in the guest

Scheduler Setup
Host Scheduling
■ CARTS: https://rtg.cis.upenn.edu/carts
■ Retina Tools:
retinaproject.eu/publications-deliverables#products
Guest Scheduling
■ Use ﬁxed priorities (SCHED FIFO / SCHED RR)
■ Avoid global scheduling (set aﬃnities to single CPUs)
◆ The Retina Tools provide support for partitioning
real-time tasks
Starting the VM
■ Retina Tools → example scripts using qemu/kvm

Conclusions
■ Deterministic scheduling of real-time tasks in VMs is
possible
◆ But can be tricky; you need to know what to do
■ Use real-time support provided by Linux community →
PREEMPT RT
■ Use theory developed in real-time research →
SCHED DEADLINE
■ Use the virtualization support provided by Linux → kvm,
virtio, ...
◆ Some tools can help you in putting everything
together

Demo?
■ “I have a truly marvelous demo of this, which the margin
of these slides is too narrow to contain...”
■ So, let’s try a less marvelos demo...

Luca Abeni - Real-Time Virtual Machines with Linux and kvm

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Luca Abeni - Real-Time Virtual Machines with Linux and kvm

Similar a Luca Abeni - Real-Time Virtual Machines with Linux and kvm (20)

Más de linuxlab_conf

Más de linuxlab_conf (18)

Último

Último (20)

Luca Abeni - Real-Time Virtual Machines with Linux and kvm