The Xen hypervisor is a great fit for embedded, automotive and avionics, thanks to its lean architecture, small code base (even smaller on ARM), flexibility and security. Low interrupt latency is an essential requirement for many embedded use cases. Unavoidably, virtualization increases interrupt latency, but different Xen configurations lead to very different results.
This talk will explain how to achieve minimal interrupt latency. It will cover how scheduling interacts with interrupt delivery in Xen. It will introduce the new Xen "null" scheduler, designed for maximum determinism and minimum overhead on embedded platforms. Findings will be corroborated by showing interrupt latency numbers taken on Xen on ARM in various configurations, such as vCPU pinning, Credit 1 and 2, and null schedulers.
19. A vCPU Wake-Up in Credit1
pCPU0
pCPU1
pCPU2
pCPU3
vcpu vcpu vcpuvcpuvcpurunq0
vcpu vcpurunq1
runq2
vcpu vcpu vcpurunq3
Case a:
1. vcpu goes in a runq1
2. where can vcpu run?
Hey, pCPU2 is idle!
3. put vcpu in runq2
4. pCPU2 picks up vcpu
from its runqueue
Case b:
1. vcpu goes in a runq3
2. hey, vcpu can prempt
what’s running on
pCPU3!
3. context switch
vcpu
vcpu
vcpu
vcpu
vcpu
vcpu
vcpu
(1)
(2)
(3)
(4)
vcpu
vcpu
(1)
(2)
(3)
20. A vCPU Wake-Up in Credit2
pCPU0
pCPU1
pCPU2
pCPU3
vcpu vcpu vcpurunqA
runqB
vcpu
vcpu
vcpu
vcpu
vcpu
vcpu
vcpu
vcpu
vcpu
Case a:
1. vcpu goes in a runqA
2. load-balancer moves
vcpu to a less loaded
runq
3. pCPU2 picks up vcpu
from its runqueue
Case b:
1. vcpu goes in runqB
2. pCPU3 picks up vcpu
21. A vCPU Wake-Up in Credit20.000971102 irq_enter
0.000971102 irq_direct, vec fa, handler = apic_timer_interrupt
0.000971649 raise_softirq TIMER_SOFTIRQ
0.000971962 irq_exit, in_irq = 0
0.000974070 softirq_handler TIMER_SOFTIRQ
0.000976010 tasklet_schedule fn=hvm_assert_evtchn_irq, sched_on=6 (softirq)
0.000976010 tasklet_enqueue fn=hvm_assert_evtchn_irq
0.000976510 raise_softirq TASKLET_SOFTIRQ on cpu 6
0.000978213 softirq_handler TASKLET_SOFTIRQ
0.000981562 tasklet_do_work fn=hvm_assert_evtchn_irq
0.000982017 vcpu_wake d1v1
0.000982437 runstate_change d1v1 blocked->runnable
0.000982987 csched2:update_load
0.000983230 csched2:update_rq_load rq# 0, load = 1, avgload = 34.830%
0.000983430 csched2:update_vcpu_load d1v1, vcpu_load = 11.824%
0.000983735 csched2:runq_insert d1v1, position 0
0.000984060 csched2:runq_tickle_new d1v1, processor = 6, credit = 5292567
0.000984490 csched2:runq_tickle cpu 6
0.000984842 raise_softirq SCHEDULE_SOFTIRQ on cpu 6
0.000985500 softirq_handler SCHEDULE_SOFTIRQ
0.000988941 csched2:schedule cpu 6, rq# 0, idle, SMT idle, tickled
0.000989344 csched2:runq_cand_check d1v1
0.000989611 csched2:runq_candidate d1v1, credit = 5292567
0.000990881 sched_switch prev idle, run for 344.6us
0.000991199 sched_switch next d1v1, was runnable for 5.862us, next slice 1000.0us
0.000991377 sched_switch prev idle next d1v1
0.000991697 runstate_change idle running->runnable
0.000991979 runstate_change d1v1 runnable->running
vcpu wakes-up;
goes in runq
Scheduler triggered
on CPU 6
CPU 6 schedules
vcpu runs
Interrupt arrives
Scheduling
Introduced latency:
9.9620 us
BEST CASE
22. _This_ _is_ _all_ _good_ ...
… Because, thanks to this, we can offer VMs/users:
• Overcommitting (i.e., having more vCPUs than pCPUs)
• Weighted fair share of pCPU time
• Hard and soft affinity
• Cache and NUMA awareness
• Caps and reservations on pCPU time
… but it _comes_ _at_ _a_ _price_
24. A scheduler that does nothing
If we want features & flexibility, we must pay the price :-(
What if we don’t want (or need) them, e.g.:
• Static environments (some embedded usecases)
• Systems (or cpupools, where we know we’ll never have
overcommit)
• For testing/benchmarking (as reference)
Then you can use the ‘null’ scheduler
25. Runqueues in ‘null’. No, wait...
pCPU0
pCPU1
pCPU2
pCPU3
vcpu
vcpu
vcpu
vcpu
There are no runqs at all!
• vCPUs are statically assigned to pCPUs
• Only 1 vCPUs per pCPU
• Overcommit is possible (i.e.: the system won’t
explode), but use only if you really know what
you’re doing (i.e.: the VMs will likely explode!)
26. A vCPU Wake-Up in ‘null’
pCPU0
pCPU1
pCPU2
pCPU3
1. vcpu wakes up and run
:-)vcpu
vcpuvcpu
(1)
27. A vCPU Wake-Up in ‘null’
0.636884641 irq_enter
0.636884641 irq_direct, vec fa, handler = 0xffff82d080267ec4
0.636885492 raise_softirq TIMER_SOFTIRQ
0.636885922 irq_exit, in_irq = 0
0.636889583 softirq_handler TIMER_SOFTIRQ
0.636892021 tasklet_schedule fn=hvm_assert_evtchn_irq, sched_on=5 (softirq)
0.636892021 tasklet_enqueue fn=hvm_assert_evtchn_irq
0.636892836 raise_softirq TASKLET_SOFTIRQ on cpu 5
0.636895074 softirq_handler TASKLET_SOFTIRQ
0.636895607 tasklet_do_work fn=hvm_assert_evtchn_irq
0.636896202 vcpu_wake d1v1
0.636896712 runstate_change d1v1 blocked->runnable
0.636897197 raise_softirq SCHEDULE_SOFTIRQ on cpu 5
0.636898470 softirq_handler SCHEDULE_SOFTIRQ
0.636899465 null:schedule cpu 5, vcpu d1v1
0.636899720 sched_switch prev idle, run for 999936.973us
0.636899970 sched_switch next d1v1, was runnable for 2.411us
0.636900155 sched_switch prev idle next d1v1
0.636900448 runstate_change idle running->runnable
0.636900738 runstate_change d1v1 runnable->running
vcpu wakes-up;
goes in runq
Scheduler triggered
on CPU 6
CPU 6 schedules
vcpu runs
Interrupt arrives
Scheduling
Introduced latency:
4.5360 us
(less than half of
Credit2)
NORMAL CASE