This is a talk presented at the OpenStack DC Meetup #9 of GPU pass-through of an Nvidia GRID K2 card with XenServer, Microsoft Hyper-V, and open source Xen hypervisors. We looked at

1. Page 1
GPU-Accelerated VDI on OpenStack
NIMBIS SERVICES, INC.
GPU-Accelerated VDI on OpenStack
OpenStack DC MeetUp
June 18, 2013
Nimbis Services, Inc.
http://www.nimbisservices.com
Brian Schott, CTO
Brian.Schott@NimbisServices.com

2. Page 2
GPU-Accelerated VDI on OpenStack
NIMBIS SERVICES, INC.
About Nimbis Services
Nimbis Services is lowering the barriers to high-
performance computing (HPC) and offering a secure
marketplace for technical computing in the cloud.
Nimbis partners with leading HPC & cloud providers, software
vendors, and domain experts to configure and broker secure
on-demand pay-as-you-go access to applications.

3. Page 3
GPU-Accelerated VDI on OpenStack
NIMBIS SERVICES, INC.
Challenges for GPU-Accelerated Virtual Desktop Infrastructure (VDI)
on OpenStack
 Making the GPU available to a virtual machine.
(3 basic approaches, ok, maybe 4).
 Bare Metal / Containers (lxc)
 PCI Pass-Through
 Para-virtualization (proxy device drivers)
 Remote Display Protocols (X, NX, VirtualGL, WebGL)
 Remote desktop protocol that supports GPU acceleration.
 Host and Guest Combinations: Linux or Windows
 Remote Clients: Linux, Windows, OSX, Web Browser, Tablet
 Networking: “The Joys of Firewalls”, or “Any Port as Long as it is 443”
 Licenses: “This is too good to be free.”
 Provisioning GPU resources from within OpenStack.
 Nova compute virtualization driver (plugin).
 GPU instance types, resource discovery, and scheduling.
 This is still under development in the community.
 Full disclosure: Nimbis hand-deployed the GPU instances for testing.

4. Page 4
GPU-Accelerated VDI on OpenStack
NIMBIS SERVICES, INC.
Nimbis Testbed Hardware
 “Production” Cisco System
 (1) UCS C220 M3 – dual quad-core 3.3GHz
Intel Xeon E5-2643, 192GB SDRAM, 2TB
SAS-7K, dual channel 10GbE interface.
 (2) UCS C220 M3 – dual quad-core 2.4GHz
Intel Xeon E5-2609, 192GB SDRAM, 2TB
SAS-7K, dual channel 10GbE interface.
 (3) UCS C240 M3 – dual quad-core 2.4GHz
Intel Xeon E5-2609, 192GB SDRAM, 6TB
SAS-7K, dual-channel 10GbE.
• (1) Nvidia GRID K2 Board
 (2) Nexus 3064 10GbE Switches
 (2) ASA 5520 Adaptive Security Appliances
 (2) Cisco Catalyst 2960S 1GbE switches
 “Development” Cisco System
 (3) Cisco UCS C200 M2
 (1) Nexus 3064 10GbE Switch
 (1) ASA 5520 Adaptive Security Appliance
 (1) Cisco Catalyst 2960S 1GbE switch
Cores: 72
Memory: 1440 GB
Disk: 36 TB
Servers: 9*
Switches: 9*
* What do you
expect with a
Cisco testbed?

5. Page 5
GPU-Accelerated VDI on OpenStack
NIMBIS SERVICES, INC.
Nvidia GRID GPU Virtualization (Kepler Architecture)
Source: http://www.nvidia.com/object/grid-boards.html
Source: 1http://www.nvidia.com/content/PDF/kepler/NV_DS_Tesla_KCompute_Arch_May_2012_LR.pdf
GPU Virtualization¹
GRID boards feature
the NVIDIA Kepler
architecture that, for the
first time, allows
hardware virtualization
of the GPU. This
means multiple users
can share a single GPU,
improving user density
while providing true
PC performance and
compatibility.

6. Page 6
GPU-Accelerated VDI on OpenStack
NIMBIS SERVICES, INC.
GPU-Accelerated VDI Approaches Considered
 Customer Requirements
 Combination of Windows and Linux guest images.
 Applications using OpenGL on both Windows and Linux.
 Variety of Windows (95, 7, 8) clients, OSX clients, and web browsers
(IE7, IE8, Firefox, Chrome, Safari).
 Variety of firewall and IT/FSO software challenges.
 Technologies Tested
 Microsoft Hyper-V and RemoteFX
 XenServer with VNC / RDP / Guacamole
 Open Source Xen (libvirt) on Ubuntu Linux with VNC
 Possible Alternatives
 Linux Containers (lxc)
• USC/ISI team has the blueprint and code targeted for Havana.
• No Windows support.
 XenDesktop with HDX 3D Pro (not tested)
• Requires a custom client, Citrix Receiver, OpenGL support unclear.
• Does not integrate with OpenStack.

7. Page 7
GPU-Accelerated VDI on OpenStack
NIMBIS SERVICES, INC.
Microsoft Hyper-V and RemoteFX
 Technology
 Windows Host GPU device with para-virtualized driver for windows guests.
 Host / Hypervisor
 Microsoft Windows Server 2012 Datacenter Edition (RemoteFX license).
 Latest Nvidia driver for Windows.
 Guests
 No GPU pass-through available or driver for Linux guests.
 Accelerates Windows DirectX calls, does software OpenGL rendering.
 Clients
 Good performance with standard Windows RDC > 8.0.
 Good performance from FreeRDP* OSX client, not Microsoft client.
 iOS and Android? Not tested, but FreeRDP clients exist on both.
 OpenStack Status
 Integrates with OpenStack Hyper-V nova compute drivers.
 Nova scheduler support for Hyper-V extra specs not on blueprint radar.
 Possible to use today with custom nova scheduler plugin.
* http://www.freerdp.com

8. Page 8
GPU-Accelerated VDI on OpenStack
NIMBIS SERVICES, INC.
Hyper-V and RemoteFX Benchmarks with Nvidia GRID K2
GFXBench Test Metric
RDP Without
RemoteFX
NVidia
Quadro 600
(~$150)*
RDP With
RemoteFX
NVidia
Quadro K5000
(~$1800)*
Spark Lines
Fill rate - Offscreen (1080p) M Texels/s 76 3291 34507 39776
Fill rate - Onscreen M Texels/s 60 2324 8760 7949
GFXBench 2.5 Egypt HD C24Z16 - Offscreen (1080p) Frames 124 4274 6610 70733
GFXBench 2.5 Egypt HD C24Z16 - Offscreen (1080p) Frames / s 1.1 76.3 58.5 626
GFXBench 2.5 Egypt HD C24Z16 - Onscreen Frames 74 2622 4682 6776
GFXBench 2.5 Egypt HD C24Z16 - Onscreen Frames / s 0.7 46.8 41.4 60
GFXBench 2.7 T-Rex HD C24Z16 - Offscreen (1080p) Frames 26 3124 3719 24434
GFXBench 2.7 T-Rex HD C24Z16 - Offscreen (1080p) Frames / s 0.5 55.8 66.4 436.3
GFXBench 2.7 T-Rex HD C24Z16 - Onscreen Frames 15 4274 2708 3358
GFXBench 2.7 T-Rex HD C24Z16 - Onscreen Frames / s 0.3 76.3 48.4 60
GFXBench 2.7 T-Rex HD Fixed Timestep - Offscreen (1080p) ms 1199830 56022 9117 1546
GFXBench 2.7 T-Rex HD Fixed Timestep - Offscreen (1080p) Frames / s 0.5 56022 61.4 1546
GFXBench 2.7 T-Rex HD Fixed Timestep - Onscreen ms 1699985 10 11734 362.2
GFXBench 2.7 T-Rex HD Fixed Timestep - Onscreen Frames / s 0.3 6761 47.7 9361
Triangle throughput: Textured - Offscreen (1080p) M Triangles/s 1 59.8 2209 59.8
Triangle throughput: Textured - Onscreen M Triangles/s 0 303 6246 2098
Triangle throughput: Textured, fragment lit - Offscreen (1080p) M Triangles/s 1 300 1770 2049
Triangle throughput: Textured, fragment lit - Onscreen M Triangles/s 0 303 4263 1678
Triangle throughput: Textured, vertex lit - Offscreen (1080p) M Triangles/s 1 299 1780 1665
Triangle throughput: Textured, vertex lit - Onscreen M Triangles/s 0 300 7310 1672
Host: Microsoft Windows Server 2012 Datacenter Edition,
Guest: Microsoft Windows Server 2012 Standard Evaluation,
Client: FreeRDP on Mac OSX * http://gfxbench.com/

9. Page 9
GPU-Accelerated VDI on OpenStack
NIMBIS SERVICES, INC.
XenServer with VNC / RDP / Guacamole
 Technology
 GPU pass-through of Nvidia GRID K2 to Windows and Linux guests.
 Two GPU devices available for two separate instances.
 Physical GPU virtualization drivers promised by end of the year (n=64?).
 Host / Hypervisor
 XenServer 6.1 Platinum Edition (enables GPU license).
 Guests
 Linux: specify Windows 7 as server template then standard CentOS 6.4
install process. Good GPU performance using VNC and VirtualGL.
 Windows: tried both RDP and TigerVNC do software OpenGL rendering.
• Unable to VNC or RDP to “console” display, possibly no monitor?
 Clients
 Windows and OSX Clients comparable to remote desktop to native boxes.
 Tried RDP, VNC, and Guacamole (HTML5 VNC client).
 OpenStack Status
 XenServer Integrates with OpenStack using XenServer plugins.
 Nova scheduler support for XenServer not on blueprint radar.

10. Page 10
GPU-Accelerated VDI on OpenStack
NIMBIS SERVICES, INC.
Open Source Libvirt/Xen with VNC / RDP / Guacamole
 Technology
 GPU pass-through of Nvidia GRID K2 to Windows and Linux guests.
 Two GPU devices available for two separate instances.
 Unsure of GPU virtualization drivers status for n > 2.
 Host / Hypervisor
 Ubuntu 12.04, Xen (Xen hypervisor 4.1.0), libvirt (0.98)
 Xen HVM guest cfgfile in /etc/xen/<guest> to enable VGA passthrough:
gfx_passthru=1
pci=['yy:zz.n’]
 Guests
 Linux: Tested CentOS 6.4 image with latest Nvidia drivers (310.44).
Verified device functioned. Expect similar results to XenServer.
 Windows: Did not test an image. Expect similar results to XenServer.
 OpenStack Status
 Xen supported within OpenStack using nova libvirt driver.
 Nova specs support for libvirt exists (i.e. this host has a GPU).
 Feasible to have support included in H or I release given lxc work.
http://wiki.xen.org/wiki/Xen_VGA_Passthrough

11. Page 11
GPU-Accelerated VDI on OpenStack
NIMBIS SERVICES, INC.
XenServer Linux Virtual Machine Installation Hack

12. Page 12
GPU-Accelerated VDI on OpenStack
NIMBIS SERVICES, INC.
Specify GPU Type

13. Page 13
GPU-Accelerated VDI on OpenStack
NIMBIS SERVICES, INC.
VirtualGL Architecture (VirtualGL.org)

14. Page 14
GPU-Accelerated VDI on OpenStack
NIMBIS SERVICES, INC.
GLXINFO for Linux Instance on XenServer

15. Page 15
GPU-Accelerated VDI on OpenStack
NIMBIS SERVICES, INC.
Guacamole HTML5 Browser VNC

16. Page 16
GPU-Accelerated VDI on OpenStack
NIMBIS SERVICES, INC.
Windows Server 2012 on XenServer

17. Page 17
GPU-Accelerated VDI on OpenStack
NIMBIS SERVICES, INC.
Heterogeneous Architecture Support Blueprints
https://wiki.openstack.org/wiki/Heter
ogeneousInstanceTypes
https://wiki.openstack.org/wiki/Sc
heduleHeterogeneousInstances
https://wiki.openstack.org/wiki/Heterog
eneousGpuAcceleratorSupport

18. Page 18
GPU-Accelerated VDI on OpenStack
NIMBIS SERVICES, INC.
Summary
 GPU-accelerated VDI pools within OpenStack are quite feasible.
 Code support is very rough around the edges, not production ready.
 We really need to get open community support organized for this feature.
 Several vertical vendor solutions out there, but Xen and KVM baselines
should be possible as open-source.
 Questions?