Today XEN comes to embedded systems, where it needs to be much closer to a hardware. In one hand hypervisor needs to mediate calls to Trusted Zone, control power, provide drivers for coprocessors, on other hand it needs to remain as small and as secure as possible. So natural approach is to offload all these tasks to something else (like stubdomain or native application). ARM platform allow hypervisor to act as a common kernel by handling system calls from userspace. In this talk Volodymyr will describe idea of native applications, compare them with stubdomains and share results of his Native Apps PoC.

2. Approach to Native Applications
in XEN on ARM
Volodymyr “Vova” Babchuk
Lead Embedded Developer
EPAM Systems, Inc.

3. What we are working on
Xen
Native EL0 apps / stub domains
Real time scheduling
Heterogenous big.LITTLE support
PMF (cpufreq, cpuidle, thermal, vcoprocpm)
SCF
IOMMUF & IPMMU support
SMC/HVC bridge
PV frontends
Xen apps
PM governor +SoC drivers
TEE manager +OP-TEE driver
GPU mediator +SGX driver
OP-TEE Mullti-domain support
Integration
Android HALs
Sound/Display managers
PV backends
Certification ISO 61508 path 3s
CI Build/release system
See us at:
https://github.com/xen-troops

4. Agenda
1. What “native application” is
2. Why we could need it
3. How it is implemented
4. Testing and benchmarking results

5. What Native Application is

6. Features similar to Linux kernel usermode
helper:
● No MMU
● No interrupts
● Syscalls instead of hypercalls
● Most unprivileged mode (EL0 or USR)
● No main loop
● Called when hypervisor needs it

7. Why do we need
something like that

8. Use case: emulator
● Device emulator (like PL011 UART)
● VM System Specification for ARM Processors:
“Serial console: The hypervisor must provide an emulated
UART meeting the minimum requirements in SBSA UART”

10. Use case: TEE mediator
• TEE is like TPM (Trusted Platform Module) on x86, but
runs in Secure Mode in an ARM processor.
• Multiple guests can work with TEE.
• Hypervisor should handle this.
• We don’t want TEE mediator code in hypervisor

12. Use case: SCF driver
• Shared Coprocessor Framework allows coprocessors
virtualization (GPUs, HW multimedia encoders, etc).
• SCF driver is responsible for context switching.
• We can’t run it in XEN due to various reasons.

13. Goal: Isolation
Aiming to prevent a hypervisor crash, native application
should be isolated from hypervisor as from any other domain.

14. Goal: Speed
Context switch to an app (HYP -> Native app) should be
faster than context switch to a domain.
* Numbers will follow in “Results” section

15. There are stubdoms already. Why not to use
them?

16. Goal: No scheduler overhead

17. Goal: No scheduler overhead

18. Goal: Simpler than stubdomain
No boilerplate code for:
• Interrupt handlers
• MMU
• Task management, etc.

19. How I did this

20. TGE bit
“Trap General Exceptions, from Non-secure EL0”
Any exception will take us into EL2 (HYP mode):
● Syscall (SVC instruction)
● Interrupt request
● Data abort, Prefetch abort, etc.

21. My native app is domain
Interesting fact: You need to create 6 constans if you are adding
new domain type:
• “guest_type_el0” enum in hypervisor code
• “DOMCRF_el0” flag in domctl interface
• “XEN_DOMINF_el0” flag in domctl interface
• “XEN_DOMCTL_CDF_app_domain” flag in domctl interface
• “XC_DOM_APP_CONTAINER” define in libxc
• “LIBXL_DOMAIN_TYPE_APP” define in libxl

22. Entry point: app side
void __noreturn __app_entry(unsigned long func,
struct app_params *up)
{
int res = 0;
do_log("Hello from EL0 app!");
app_return(res);
}

23. Syscalls
Like Linux (or any other OS) syscalls, called with SVC
instruction.
• void app_return(unsigned long ret) __noreturn;
• void app_log(const void *buf, size_t len);

24. How to switch to app
● Pause current vCPU
● Set stage 2 MMU
● Set saved PC to entry point: __app_entry() function
● Set TGE bit
● Set saved processor state to EL0
● Do standard hypervisor context switch

25. How to switch back
• Trap syscall: app_exit in this case
• Disable TGE bit
• Set stage 2 MMU back to calling guest mapping
• Unpause calling vCPU
• Do standard hypervisor context switch to calling vCPU

26. Testing & Benchmarking
Results

27. This is not proper benchmarking setup
But
● it is reproducible
and
● it shows some numbers :-)

28. Benchmarking setup: what to benchmark
Secure Monitor Call instruction (SMC)
SMC “handling”:
void handle_smc()
{
return;
}

29. Benchmarking setup: what I compared
● Handling right in hypervisor
● Handling using app
● Handling in stubdomain
● Handling in baremetal

30. Handling in hypervisor
void do_trap_smc(regs)
{
advance_pc(regs);
}

31. Handling in the app
void __noreturn __app_entry(unsigned long func,
struct app_params *up)
{
int res = 0;
app_return(res);
}

32. ● MiniOS is not ready for ARM and especially for ARM64
● Initial porting was done by Chen Baozi
● MiniOS served as “Monitor” for DomU
● Initialization:
event_id = register_as_monitor(domId);
register_event_handler(event_id)
Handling in stubdomain

33. Handling in stubdomain (cont.)
● Handling:
void monitor_handler(evtchn_port, struct pt_regs)
{
vm_event_control(_domid, XEN_VM_EVENT_RESUME,
XEN_DOMCTL_VM_EVENT_OP_MONITOR, 0);
}

34. Handling in the ARM Trusted Firmware
● Without hypervisor at all, so we can see virtualization overhead
uintptr_t handle_runtime_svc(smc_fid, ...)
{
...
if (smc_fid == 0)
return 0;
…
}

35. How I measured: kernel driver
● /proc/smc_bench entry
static int s_show_bench(struct seq_file)
{
for (int i = 0; i < 1000*1000*10; i++)
arm_smccc_smc(...);
}

36. How I measured: actual measurements
# time cat /proc/smc_bench
real 1m3.525s
user0m0.000s
sys 1m3.516s

37. Device used for benchmarking
Renesas R-Car Gen3 ES2.0 (r8a7795) chip:
● Four Cortex A57 cores
● 4 GB of RAM

38. Results
Total time
(seconds)
Avg call time
(us)
Relative value
Hypervisor 10.8 1.08 1
Native app 63.5 6.35 5.9
Stubdom 111.4 11.114 10.35
ARM TF 1.5 0.15 0.13

39. Useful Links
My e-mail: volodymyr_babchuk@epam.com
Native app: https://github.com/lorc/xen_app_stub
MiniOS: https://github.com/lorc/mini-os
My patch series for hypervisor:
https://github.com/lorc/xen/tree/el0_app

40. Thank you!