In 2013, the Xen Hypervisor will be 10 years old: when Xen was designed, we anticipated a world, which now is known as cloud computing. Today, Xen powers the largest clouds in production and is the basis for several commercial virtualization products. In this talk we will give on overview of Xen and related projects, cover hot developments in the Xen community and outline what comes next. The talk is intended for users and developers that are familiar with virtualization: no deep knowledge is required. We will start with an architectural overview and cover topics such as: Xen and Linux, how to secure your cloud using disaggregation, SELinux and XSM/FLASK, the evolution of Paravirtualization, Xen on ARM and common challenges for open source hypervisors. We will explore the potential of Open Mirage for testing hypervisors. The talk will conclude with an outlook to the future of Xen.

1. 10 Years of Xen and beyond …
Lars Kurth
Xen Project Community Manager
lars.kurth@xen.org
@lars_kurth
FREENODE: lars_kurth

2. Xen.org becomes XenProject.org
• Teams aka sub-projects
– Hypervisor
– XAPI
– ARM Hypervisor (for Servers as well as Mobile Devices)
– Mirage OS
• Governance : mixture between Linux Kernel and Apache
– Consensus decision making
– Sub-project life-cycle (aka incubator)
– PMC style structure for team leadership

3. Xen contributor community is diversifying
100%
90%
• The number of “significant”
80% active vendors is increasing
70%
60%
50% • New feature development driving
40%
30%
new participation
20%
10%
0%
2010 2011 2012
Citrix UPC
SUSE Amazon
University AMD
GridCentric Individual
NSA Intel
Fujitsu iWeb
Misc Oracle
Spectralogic University of British Columbia

4. Hypervisor Architecture

5. Hypervisor Architectures
Type 1: Bare metal Hypervisor
A pure Hypervisor that runs directly on the
hardware and hosts Guest OS’s.
VMn
VM1
VM0
Guest OS
and Apps
Scheduler Hypervisor
Device Drivers/Models MMU
Host HW
I/O Memory CPUs
Provides partition isolation + reliability,
higher security

6. Hypervisor Architectures
Type 1: Bare metal Hypervisor Type 2: OS ‘Hosted’
A pure Hypervisor that runs directly on the A Hypervisor that runs within a Host OS and hosts
hardware and hosts Guest OS’s. Guest OS’s inside of it, using the host OS services
to provide the virtual environment.
VMn User-level VMM VMn
VM1 User
VM1
Apps
VM0 Device Models
VM0
Guest OS Guest OS
and Apps and Apps
Host OS
Scheduler Hypervisor
Ring-0 VM Monitor
Device Drivers/Models Device Drivers “Kernel “
MMU
Host HW Host HW
I/O Memory CPUs I/O Memory CPUs
Provides partition isolation + reliability, Low cost, no additional drivers
higher security Ease of use & installation

7. Xen: Type 1 with a Twist
Type 1: Bare metal Hypervisor
VMn
VM1
VM0
Guest OS
and Apps
Scheduler Hypervisor
Device Drivers/Models MMU
Host HW
I/O Memory CPUs

8. Xen: Type 1 with a Twist
Type 1: Bare metal Hypervisor Xen Architecture
VMn
VM1 VMn
VM0 VM1
Guest OS VM0
and Apps
Guest OS
and Apps
Scheduler Hypervisor
Device Drivers/Models MMU Scheduler MMU Hypervisor
Host HW Host HW
I/O Memory CPUs I/O Memory CPUs

9. Xen: Type 1 with a Twist
Type 1: Bare metal Hypervisor Xen Architecture
Control domain
(dom0)
VMn
VM1 Device Models VMn
VM0 VM1
Guest OS
Drivers
VM0
and Apps
Guest OS
Linux & BSD and Apps
Scheduler Hypervisor
Device Drivers/Models MMU Scheduler MMU Hypervisor
Host HW Host HW
I/O Memory CPUs I/O Memory CPUs

10. Xen Project and Linux
• Xen Hypervisor is not in the Linux kernel
• BUT: everything Xen and Xen Guests need to run is!
• Xen packages are in all Linux distros (except RHEL6)
– Install Dom0 Linux distro
– Install Xen package(s) or meta package
– Reboot
– Config stuff: set up disks, peripherals, etc.
More info: wiki.xen.org/wiki/Category:Host_Install

11. Basic Xen Concepts
Console
• Interface to the outside world
Control Domain aka Dom0
VMn
• Dom0 kernel with drivers
Control domain VM1
(dom0) • Xen Management Toolstack
VM0
Guest OS
Guest Domains
Dom0 Kernel and Apps • Your apps
Scheduler MMU XSM Hypervisor Driver/Stub/Service Domain(s)
• A “driver, device model or control
Host HW service in a box”
I/O Memory CPUs
• De-privileged and isolated
• Lifetime: start, stop, kill
Trusted Computing Base 11

12. Basic Xen Concepts
Console
Console • Interface to the outside world
Control Domain aka Dom0
VMn
• Dom0 kernel with drivers
Control domain VM1
(dom0) • Xen Management Toolstack
VM0
Toolstack
Guest OS
Guest Domains
Dom0 Kernel and Apps • Your apps
Scheduler MMU XSM Hypervisor Driver/Stub/Service Domain(s)
• A “driver, device model or control
Host HW service in a box”
I/O Memory CPUs
• De-privileged and isolated
• Lifetime: start, stop, kill
Trusted Computing Base 12

13. Basic Xen Concepts
Console
Console • Interface to the outside world
Control Domain aka Dom0
VMn
• Dom0 kernel with drivers
Control domain VM1
(dom0) • Xen Management Toolstack
One or more VM0
Toolstack driver, stub or
service domains Guest OS
Guest Domains
Dom0 Kernel and Apps • Your apps
Scheduler MMU XSM Hypervisor Driver/Stub/Service Domain(s)
• A “driver, device model or control
Host HW service in a box”
I/O Memory CPUs
• De-privileged and isolated
• Lifetime: start, stop, kill
Trusted Computing Base 13

14. Xen Variants for Server & Cloud
Hypervisor Xen Hypervisor
14

15. Xen Variants for Server & Cloud
Hypervisor Xen Hypervisor
Toolstack / Console Default / XL (XM) Libvirt / VIRSH XAPI / XE
Increased level of functionality and integration with other components
Single Host Single Host
Basic Functions Additional Functionality
Multiple Hosts
Additional Functionality
15

16. Xen Variants for Server & Cloud
Hypervisor Xen Hypervisor
Toolstack / Console Default / XL (XM) Libvirt / VIRSH XAPI / XE
Increased level of functionality and integration with other components
Single Host Single Host
Basic Functions Additional Functionality
Multiple Hosts
Additional Functionality

17. Xen Variants for Server & Cloud
Project Xen Hypervisor
Toolstack / Console Default / XL (XM) Libvirt / VIRSH XAPI / XE
Increased level of functionality and integration with other components
Products Oracle VM Huawei UVP Citrix XenServer
17

18. Xen Variants for Server & Cloud
Project Xen Hypervisor
Toolstack / Console Default / XL (XM) Libvirt / VIRSH XAPI / XE
Increased level of functionality and integration with other components
Products Oracle VM Huawei UVP Citrix XenServer
Used by …
18

19. Xen : Types of Virtualization

20. PV Domains
Technology:
Control domain Guest VMn
(dom0) • Paravirtualization
Apps Linux PV guests have limitations:
• limited to a subset of set of virtual HW
PV Back Ends PV Front Ends
Advantages
HW Drivers • Fast
Dom0 Kernel Guest OS
• Works on any system
(even without virt extensions)
Xen Hypervisor
Host HW
I/O Memory CPUs
20

21. PV Domains & Driver Domains
Technology:
Control domain Guest VMn Driver Domain
(dom0) e.g. • Paravirtualization
Apps • Disk Linux PV guests have limitations:
• Network
• limited to a subset of virtual HW
PV Back Ends PV Front Ends PV Back End
Advantages
HW Drivers HW Driver • Fast
Dom0 Kernel Guest OS Dom0 Kernel*
• Works on any system
(even without virt extensions)
Xen Hypervisor Driver Domains
• Security
Host HW • Isolation
I/O Memory CPUs
• Reliability and Robustness
*) Can be MiniOS
21

22. HVM & Stub Domains
Technology:
Dom0 Guest VMn
• Shows emulation using QEMU/Device
Model (SW Virtualization)
IO Emulation • In other situation HW can be used
Device Model Disadvantages
IO Event
• Emulation slower than PV
(mainly I/O devices)
Dom0 Kernel VMEXIT
Advantages
Xen Hypervisor • No kernel support needed
Host HW
I/O Memory CPUs
22

23. HVM & Stub Domains
Technology:
Dom0 Guest VMn Stubdomn Guest VMn
• Shows emulation using QEMU/Device
Model (SW Virtualization)
IO Emulation IO Emulation • In other situation HW can be used
Device Model Device Model Disadvantages
IO Event
• Emulation slower than PV
IO Event
(mainly I/O devices)
Dom0 Kernel VMEXIT Mini OS VMEXIT
Advantages
Xen Hypervisor • No kernel support needed
Stub Domains
Host HW • Security
I/O Memory CPUs
• Isolation
• Reliability and Robustness
23

24. The Virtualization Spectrum
VS Virtualized (SW)
VH Virtualized (HW)
P Paravirtualized
Fully Virtualized (FV) VS VS VS VH
FV with PV for disk & network P VS VS VH HVM mode/domain
PVHVM P P VS VH
PVH Xen 4.4 P P P VH
PV mode/domain
Fully Paravirtualized (PV) P P P P

25. The Virtualization Spectrum
Optimal performance
Scope for improvement
Poor performance
Fully Virtualized (FV) VS VS VS VH
FV with PV for disk & network P VS VS VH HVM mode/domain
PVHVM P P VS VH
PVH Xen 4.4 P P P VH
PV mode/domain
Fully Paravirtualized (PV) P P P P

26. The Virtualization Spectrum
Important: Xen automatically picks the best
option based on HW & OS capabilities and
Optimal performance available drivers.
As a Xen user I chose a HVM or PV domain.
Scope for improvement
Poor performance
Fully Virtualized (FV) VS VS VS VH
FV with PV for disk & network P VS VS VH HVM mode/domain
PVHVM P P VS VH
PVH Xen 4.4 P P P VH
PV mode/domain
Fully Paravirtualized (PV) P P P P

27. XAPI, XCP and XCP-XAPI

28. XAPI, XCP and XCP-XAPI : What is it?
Hypervisor Xen Hypervisor
Toolstack / Console Default / XL (XM) Libvirt / VIRSH XAPI / XE
Increased level of functionality and integration with other components
Single Host Single Host
Basic Functions Additional Functionality
Multiple Hosts
Additional Functionality

29. XAPI : What do I get?
• VM lifecycle: live snapshots, checkpoint, migration
Xen Hypervisor
• Storage XenMotion: Migrate VMs between hosts or pools
XAPI / XE without shared storage (while the VM is running)
• Resource pools: flexible storage and networking
• Event tracking: progress, notification
Multiple Hosts • Upgrade and patching capabilities
Additional Functionality
• Real-time performance monitoring and alerting
• Templates for Windows and Linux guests
• Open vSwitch support built-in (default)
More info: wiki.xen.org/wiki/XCP_Release_Features

30. XAPI : two variants!
Xen Hypervisor
XAPI / XE
XCP ISO (at v1.6) XCP-XAPI packages
Multiple Hosts
Xen 4.1.3 + XAPI Additional Functionality Debian Wheezy
CentOS 5.3 Ubuntu 12.04 LTS
Kernel (v2.6.32.43)
OVS 1.4.2 Others in progress …

31. XAPI : Orchestration and UIs
Xen Hypervisor
XAPI / XE
Multiple Hosts
Additional Functionality

32. Challenges for FOSS hypervisors

33. “Security and QoS/Reliability are amongst
the top 3 blockers for cloud adoption”
www.colt.net/cio-research

34. System characteristics cloud users care about:
“Robustness, Performance, Scalability & Security”
Results XCP User Survey 2012 – 90% of users quoted these as most important attributes

35. Disaggregation
Split Control Domain into Driver,
Stub and Service Domains
– See: ”Breaking up is hard to do” @ Xen Papers
– See: “Domain 0 Disaggregation for XCP and XenServer”
Used today by Qubes OS and Citrix XenClient XT
Prototypes for XAPI
See qubes-os.org
Different windows run
in different VMs

36. Benefits of Disaggregation
More Security
Increased serviceability and flexibility
Better Robustness
Better Performance Ability to safely restart parts of the system
(e.g. just 275ms outage from failed Ethernet driver)
Better Scalability

37. Next: XAPI Architecture Diagram
Before and After Disaggregation

38. User VM User VM
NF BF NF BF
NB gntdev NB gntdev gntdev
Dom0 Network NFS/
Dom0 Qemu xapi Qemu Network NFS/ Local
drivers iSCSI drivers iSCSI storage
Domain
manager
drivers . drivers drivers
qemu
. qemu
healthd storaged storaged storaged
networkd . networkd
xenopsd tapdisk syslogd . tapdisk tapdisk
libxl vswitch blktap3 vswitch blktap3 blktap3
xapi xapi
eth eth
Dom0 eth eth scsi
Xen Xen
CPU CPU
NIC NIC RAM RAM NIC NIC
(or SR- (or SR- (or SR- (or SR- RAID
IOV VF) IOV VF) IOV VF) IOV VF)

39. User VM User VM
NF BF NF BF
NB gntdev NB gntdev gntdev
Dom0 Network NFS/ Qemu xapi D Logging Network NFS/ Local
driver iSCSI domain domain domain driver iSCSI storage
Domain o
manager
domain driver
m
. domain driver driver
domain domain domain
healthd storaged
qemu 0 . storaged storaged
networkd networkd
xenopsd tapdisk
. tapdisk tapdisk
libxl vswitch blktap3 xapi syslogd vswitch blktap3 blktap3
dbus over v4v dbus over v4v
eth eth eth eth scsi
Xen Xen
CPU CPU
NIC NIC RAM RAM NIC NIC
(or SR- (or SR- (or SR- (or SR- RAID
IOV VF) IOV VF) IOV VF) IOV VF)

40. Xen Security Advantages
• Even without Advanced Security Features
– Well-defined trusted computing base (much smaller than on type-2 HV)
– Minimal services in hypervisor layer
• Xen Security Modules (or XSM) and FLASK
– XSM is Xen equivalent of LSM
– FLASK is Xen equivalent of SELinux
– Developed, maintained and contributed to Xen by NSA
– Compatible with SELinux (tools, architecture)
– XSM object classes maps onto Xen features
More info: http://www.slideshare.net/xen_com_mgr/
a-brief-tutorial-on-xens-advanced-security-features
40

41. User VM User VM
NF BF NF BF
NB gntdev NB gntdev gntdev
Dom0 Network NFS/ Qemu xapi D Logging Network NFS/ Local
driver iSCSI domain domain domain driver iSCSI storage
Domain o
manager
domain driver
m . domain driver driver
domain domain domain
healthd storaged
qemu 0 . storaged storaged
networkd networkd
xenopsd tapdisk
. tapdisk tapdisk
libxl vswitch blktap3 xapi syslogd vswitch blktap3 blktap3
dbus over v4v dbus over v4v
eth eth eth eth scsi
FLASK policy
restricting access Xen Xen
CPU CPU
NIC NIC RAM RAM NIC NIC
(or SR- (or SR- (or SR- (or SR- RAID
IOV VF) IOV VF) IOV VF) IOV VF)

42. ARM Hypervisor

43. Xen 4.3 for ARM Servers
Fully functional for ARM v7 & v8
ARM v7: Versatile Express, Arndale &
Samsung Chromebook
ARM v8: Fast Model

44. Xen + ARM = a perfect Match
ARM SOC ARM Architecture Features for Virtualization
User mode : EL0
Device Tree describes …
Kernel mode : EL1
I/O
Hypercall Interface HVC
GIC 2 stage
GT
v2 MMU Hypervisor mode : EL2

45. Xen + ARM = a perfect Match
ARM SOC ARM Architecture Features for Virtualization
EL0
Device Tree describes …
EL1
I/O
HVC
GIC 2 stage
GT
v2 MMU EL2
Xen Hypervisor

46. Xen + ARM = a perfect Match
ARM SOC ARM Architecture Features for Virtualization
Any Xen Guest VM (including Dom0)
EL0
Device Tree describes … User Space
Kernel
EL1
I/O
HVC
GIC 2 stage
GT
v2 MMU EL2
Xen Hypervisor

47. Xen + ARM = a perfect Match
ARM SOC ARM Architecture Features for Virtualization
Dom0 Any Xen Guest VM (including Dom0)
only EL0
Device Tree describes … User Space
Kernel
PV PV
back front EL1
I/O
HVC
GIC 2 stage
GT
v2 MMU EL2
Xen Hypervisor

48. One mode to rule them all
Optimal performance
Scope for improvement
x86: PVHVM P P VS VH HVM mode/domain
x86: PVH P P P VH PV mode/domain
ARM v7 & v8 P VH VH VH

49. Code Size of x86 and ARM Hypervisors
X86 Hypervisor 100K -120K LOC Any x86 CPU
ARM Hypervisor for 60K LOC ARM v5 – v7
mobile Devices (no virt extensions)
(extra code for RT)
ARM Hypervisor for 17K LOC ARM v7+
Servers (virt extensions)

50. Mirage OS

51. Library Operating Systems
Application stacks only running on Xen APIs
Control domain Guest VMn
Works on any Xen based cloud or hosting service (dom0)
Apps
Examples
PV Back Ends
– ErlangOnXen.org : Erlang Library OS
embedded
– HalVM : Haskell HW Drivers in Language
run-time
– Mirage OS : Ocaml Dom0 Kernel
Benefits: Xen
– Small footprint Host HW
– Low startup latency
– Extremely fast migration of VMs

52. Mirage OS
• Recently added to Xen Project incubator
• In beta stage : first release on its way
• Clean-slate protocols implementations, e.g.
– TCP/IP, DNS, SSH, Openflow (switch/controller), HTTP, XMPP, ...
– New applications using next generation XAPI
(disaggregated XAPI architecture)
More info: http://www.slideshare.net/xen_com_mgr/
mirage-extreme-specialisation-of-virtual-appliances

53. What’s Next?

54. Coming in Xen 4.3 (Q2 2013)
• Code Freeze now
• Xen ARM for Servers
• Extend scope of Xen Security Modules
• qxl Spice support for 3d acceleration
• Updated and improved libvirt drivers for Xen
• Lots of other stuff:
– scalability, performance, better NUMA support, …
More info: blog.xen.org/index.php/2013/02/11/xen-4-3-mid-release-roadmap-update

55. What’s next (and already happening)
• Establish a shared test infrastructure
– Most major contributors are duplicating effort
– Mirage OS provides interesting opportiunities
• Usability and better distro-integration
– Example: Xen + XAPI in CentOS 6.4
• More focus on downstreams
– Examples: OpenStack and Xen Orchestra
• Disaggregation in Xen toolstacks (e.g XAPI)
• Better Libvirt and virt-manager integration
– Embed Xen more into the Linux eco-system and provide benefits for the
wider Linux community

56. Getting Started with Xen Projects
• Document Days
• Test Days
• Mailing Lists and IRC
• Find me and I can get you hooked up!
Xen Hackathon, May 16-17, Dublin, Ireland
@Google

57. • News: blog.xenproject.org
• Web: xenproject.org
– Help for IRC, Lists, …
– Stackoverflow like Q&A
• Wiki: wiki.xenproject.org
• Presentations: slideshare.net/xen_com_mgr
• Videos: vimeo.com/channels/xen
Thank You! @lars_kurth
FREENODE: lars_kurth
Slides available under CC-BY-SA 3.0
From www.slideshare.net/xen_com_mgr