In era of cloud computing, security is becoming more and more critical for customers. In existing HW/SW architecture hypervisor does not protect tenants against the cloud provider and thus the supplied operating system and hardware. Intel Software Guard Extension (SGX) provides a mechanism that addresses this scenario. It aims at protecting user-level software from attacks from other processes, the operating system, and even physical attackers. Intel SGX makes such protection possible through the use of enclave, which is a protected area in userspace application where the code/data cannot be accessed directly by any software from outside. This presentation intends to give you an introduction of Intel SGX technology, including what it is, how it works, and the existing SW stack to enable SGX for customers, followed by introduction of our work to support SGX virtualization in Xen hypervisor, including the high-level design, current status and future plan.

1. INTEL RESTRICTED SECRET1 SSG System Software Division
Introduction to SGX (Software Guard
Extensions) and SGX Virtualization
Kai Huang, Jun Nakajima (Speaker)
July 12, 2017

2. INTEL RESTRICTED SECRET2 SSG System Software Division
Agenda
• SGX Introduction
• Xen SGX Virtualization Support
• Backup

3. INTEL RESTRICTED SECRET3 SSG System Software Division
SGX: Reduce TCB to “HW + Enclave”
App App App
OS
VMM
Hardware
App
OS
VMM
Hardware
Enclave
Attack Surface Today Attack Surface with Enclave
Enclave:
• A protected container in App’s address space (ring 3).
• Even privileged SW cannot access enclave directly.
• Reduce TCB to “HW + Enclave”
è App gets its own capability of protection

4. INTEL RESTRICTED SECRET4 SSG System Software Division
SGX: Prevent Memory Snooping Attacks
• Security perimeter is the CPU
package boundary
• Data and code unencrypted inside
CPU package
• Data outside CPU package is
encrypted and/or integrity checked
• External memory reads and bus
snooping only see encrypted data
CPU
MEE12345678 jco3lks937w
Snoop
Snoop
Memory Bus
System
Memory
*MEE: SGX Memory Encryption Engine

5. INTEL RESTRICTED SECRET5 SSG System Software Division
SGX Enclave
• Enclave
– Trusted Execution Environment embedded in application
– Provides confidentiality and/or integrity
– With it’s own code/data.
– With controlled entry points
– Multiple threads supported
• EPC (Enclave Page Cache)
– Trusted Memory to commit enclave (via page table)
– With additional access check
– Typically reserved by BIOS as Processor Reserved Memory
– Along with EPCM with limited size (ex, 32M, 64M, 128M)
• New SGX instructions to manage/access Enclave
– ENCLS, ENCLU.
OS
Enclave
APP stack
APP code
Enclave stack
Enclave heap
Enclave code
Memory EPC
Page table
mapping
EPCM
* EPCM (Enclave Page Cache Map) :
Used by HW to track EPC status (not-visible to SW)

6. INTEL RESTRICTED SECRET6 SSG System Software Division
Instruction Behavior Changes in Enclave
• Invalid Instructions
• Behavior Changes
– RDTSC, RDTSCP: Only legal when SGX2 is available.
– RDRAND, RDSEED, PAUSE: May cause VMEXIT
– INVD: #UD in enclave
– INT3

7. INTEL RESTRICTED SECRET7 SSG System Software Division
SGX Application Flow
2. Create enclave
3. CallTrusted ()
4. Process
secrets
5. Return
6. Cont
Untrusted part
Trusted part
(Enclave)
Call Bridge1. Define and partition App to untrusted
and trusted part.
2. App creates enclave
3. Trusted function is called;
4. Code in enclave process secrets.
5. Trusted function returns.
6. App continues normal execution.
1. Partition SGX App
OS, VMM, BIOS, SMM,…

8. INTEL RESTRICTED SECRET8 SSG System Software Division
Agenda
• SGX Introduction
• Xen SGX Virtualization Support
• Backup

9. INTEL RESTRICTED SECRET9 SSG System Software Division
Virtualizing SGX
• General Enabling – Pretty Straightforward
– Discover SGX and Manage EPC in Hypervisor
– Expose part of the host EPC to guest
– Size: configurable from user
– Base: calculated internally
– SGX CPUID/MSR emulation
– Setup EPT mapping for guest EPC and host EPC.
– ENCLS and ENCLU runs perfectly in non-root mode.
• EPC Virtualization Approaches
– Static Partitioning
– Oversubscription
– Ballooning

10. INTEL RESTRICTED SECRET10 SSG System Software Division
SGX Interaction with VMX
• New ENCLS VMEXIT
– New bit in secondary exec control to enable ENCLS VMEXIT
– New 64-bit bitmap to control which ENCLS leaves will trigger VMEXIT
• New bits to indicate whether VMEXIT (any) is from Enclave
– Bit 27 in exit_reason
– Bit 4 in GUEST_INTERRUPTBILITY_INFO.

11. INTEL RESTRICTED SECRET11 SSG System Software Division
Enclave
SGX App
ACPI/CPUID/MSR
EPT mapping
Xen
SGX App
Notify
- EPC base
- EPC size
Enclave
Xen SGX Virtualization Support
Guest EPC
SGX driver
* EPT
Violation
* ENCLS
Emulation
Dom0 HVM
XL Tools
memory = 1024
epc = 64M
…
Kernel
EPT Violation
handler
ENCLS Emulation
handler
Host EPCEPC Management
Exposing EPCPopulate EPC
* ENCLS emulation & EPT violation may not needed, depending on implementation of EPC virtualization
Hypervisor Changes

12. INTEL RESTRICTED SECRET12 SSG System Software Division
EPC Virtualization Approaches (3)
Pros Cons
Static Partitioning • Easy implementation (no ENCLS
trapping/emulation, No EPT violation)
• No hypervisor overhead
Potential inefficient use of EPC
Ballooning • Pros of “static partitioning”
• More efficient use of EPC
Require ballooning driver in guest
Oversubscription More efficient use of EPC • Complicated implementation
• Higher hypervisor overhead
• We have preliminary patches on github with “static partitioning” implemented.
• Oversubscription vs Ballooning?

13. INTEL RESTRICTED SECRET13 SSG System Software Division
Questions?