PVH is a new guest type that is designed to take the best features of both PV and HVM guests. This talk will be a technical deep-dive: explaining exactly which features of HVM and PV guests are used, and which ones are discarded, and why. The primary goal will be to inform developers about the details of PVH to make it easier for them to interact with the code.

1. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH technical deep-dive
George Dunlap
Edinburg – 21-23 October, 2013
PVH Issues
Spare slides

2. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Intro: PVH
Edinburg – 21-23 October, 2013
PVH technical deep-dive
2 / 25

3. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Goal
Give you a technical overview of PVH, so that...
Edinburg – 21-23 October, 2013
PVH technical deep-dive
3 / 25

4. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Goal
Give you a technical overview of PVH, so that...
you can understand the characteristics, advantages, and
disadvantages...
Edinburg – 21-23 October, 2013
PVH technical deep-dive
3 / 25

5. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Goal
Give you a technical overview of PVH, so that...
you can understand the characteristics, advantages, and
disadvantages...
and approach the code to improve and fix
Edinburg – 21-23 October, 2013
PVH technical deep-dive
3 / 25

6. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Overview
PV, HVM, and PVH
Edinburg – 21-23 October, 2013
PVH technical deep-dive
4 / 25

7. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Overview
PV, HVM, and PVH
PVH from Xen’s perspective
Edinburg – 21-23 October, 2013
PVH technical deep-dive
4 / 25

8. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Overview
PV, HVM, and PVH
PVH from Xen’s perspective
PVH from Linux’s perspective
Edinburg – 21-23 October, 2013
PVH technical deep-dive
4 / 25

9. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Overview
PV, HVM, and PVH
PVH from Xen’s perspective
PVH from Linux’s perspective
Issues in PVH
Edinburg – 21-23 October, 2013
PVH technical deep-dive
4 / 25

10. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
PV and HVM
Issues with PV
Edinburg – 21-23 October, 2013
PVH technical deep-dive
5 / 25

11. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
PV and HVM
Issues with PV
PVMMU in Linux
Edinburg – 21-23 October, 2013
PVH technical deep-dive
5 / 25

12. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
PV and HVM
Issues with PV
PVMMU in Linux
64-bit hypercalls
Edinburg – 21-23 October, 2013
PVH technical deep-dive
5 / 25

13. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
PV and HVM
Issues with PV
PVMMU in Linux
64-bit hypercalls
Issues with HVM (PVHVM)
Edinburg – 21-23 October, 2013
PVH technical deep-dive
5 / 25

14. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
PV and HVM
Issues with PV
PVMMU in Linux
64-bit hypercalls
Issues with HVM (PVHVM)
Qemu process
Edinburg – 21-23 October, 2013
PVH technical deep-dive
5 / 25

15. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
PV and HVM
Issues with PV
PVMMU in Linux
64-bit hypercalls
Issues with HVM (PVHVM)
Qemu process
Legacy boot
Edinburg – 21-23 October, 2013
PVH technical deep-dive
5 / 25

16. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
PV and HVM
Issues with PV
PVMMU in Linux
64-bit hypercalls
Issues with HVM (PVHVM)
Qemu process
Legacy boot
Devices emulated in Xen
Edinburg – 21-23 October, 2013
PVH technical deep-dive
5 / 25

17. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
PVH
“PV in an HVM container”
Edinburg – 21-23 October, 2013
PVH technical deep-dive
6 / 25

18. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
PVH
“PV in an HVM container”
Goal: take the best aspects of PV and HVM
Edinburg – 21-23 October, 2013
PVH technical deep-dive
6 / 25

19. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
PVH
“PV in an HVM container”
Goal: take the best aspects of PV and HVM
Written by Mukesh Rathor @ Oracle
Edinburg – 21-23 October, 2013
PVH technical deep-dive
6 / 25

20. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
PVH
“PV in an HVM container”
Goal: take the best aspects of PV and HVM
Written by Mukesh Rathor @ Oracle
Significant revisions by George Dunlap @ Citrix
Edinburg – 21-23 October, 2013
PVH technical deep-dive
6 / 25

21. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
PVH from Xen’s perspective
Begin with HVM guest
Edinburg – 21-23 October, 2013
PVH technical deep-dive
7 / 25

22. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
PVH from Xen’s perspective
Begin with HVM guest
Disable HVM-specific things not needed
Edinburg – 21-23 October, 2013
PVH technical deep-dive
7 / 25

23. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
PVH from Xen’s perspective
Begin with HVM guest
Disable HVM-specific things not needed
Start & pin in 64-bit paged mode
Edinburg – 21-23 October, 2013
PVH technical deep-dive
7 / 25

24. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
PVH from Xen’s perspective
Begin with HVM guest
Disable HVM-specific things not needed
Start & pin in 64-bit paged mode
Enable PV path for a handful of things
Edinburg – 21-23 October, 2013
PVH technical deep-dive
7 / 25

25. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Xen: Things disabled
Device model (qemu)
Edinburg – 21-23 October, 2013
PVH technical deep-dive
8 / 25

26. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Xen: Things disabled
Device model (qemu)
Emulated hardware (APIC, PIT, etc)
Edinburg – 21-23 October, 2013
PVH technical deep-dive
8 / 25

27. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Xen: Things disabled
Device model (qemu)
Emulated hardware (APIC, PIT, etc)
Nested HVM
Edinburg – 21-23 October, 2013
PVH technical deep-dive
8 / 25

28. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Xen: Things disabled
Device model (qemu)
Emulated hardware (APIC, PIT, etc)
Nested HVM
MSIX
Edinburg – 21-23 October, 2013
PVH technical deep-dive
8 / 25

29. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Xen: Pinning in 64-bit
Set initial values ofr cr0, cr4, EFER
Edinburg – 21-23 October, 2013
PVH technical deep-dive
9 / 25

30. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Xen: Pinning in 64-bit
Set initial values ofr cr0, cr4, EFER
Certain paging things have to happen at start of day
Edinburg – 21-23 October, 2013
PVH technical deep-dive
9 / 25

31. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Xen: Pinning in 64-bit
Set initial values ofr cr0, cr4, EFER
Certain paging things have to happen at start of day
Disable writes to EFER
Edinburg – 21-23 October, 2013
PVH technical deep-dive
9 / 25

32. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Xen: Pinning in 64-bit
Set initial values ofr cr0, cr4, EFER
Certain paging things have to happen at start of day
Disable writes to EFER
Don’t allow guest to change paging-related cr0 bits
Edinburg – 21-23 October, 2013
PVH technical deep-dive
9 / 25

33. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Xen: PV paths
Enable PV hypercalls
Edinburg – 21-23 October, 2013
PVH technical deep-dive
10 / 25

34. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Xen: PV paths
Enable PV hypercalls
PV e820 map
Edinburg – 21-23 October, 2013
PVH technical deep-dive
10 / 25

35. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Xen: PV paths
Enable PV hypercalls
PV e820 map
PVH VCPU boot
Edinburg – 21-23 October, 2013
PVH technical deep-dive
10 / 25

36. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Xen: PV paths
Enable PV hypercalls
PV e820 map
PVH VCPU boot
PV CPUID
Edinburg – 21-23 October, 2013
PVH technical deep-dive
10 / 25

37. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Xen: PV paths
Enable PV hypercalls
PV e820 map
PVH VCPU boot
PV CPUID
PV PIO
Edinburg – 21-23 October, 2013
PVH technical deep-dive
10 / 25

38. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
PVH from Linux’s perspective
xen hvm domain() false, xen pv domain() true
Edinburg – 21-23 October, 2013
PVH technical deep-dive
11 / 25

39. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
PVH from Linux’s perspective
xen hvm domain() false, xen pv domain() true
Act natural (get rid of PV special cases)
Edinburg – 21-23 October, 2013
PVH technical deep-dive
11 / 25

40. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
PVH from Linux’s perspective
xen hvm domain() false, xen pv domain() true
Act natural (get rid of PV special cases)
Autotranslate side effects
Edinburg – 21-23 October, 2013
PVH technical deep-dive
11 / 25

41. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
PVH from Linux’s perspective
xen hvm domain() false, xen pv domain() true
Act natural (get rid of PV special cases)
Autotranslate side effects
Use PVHVM callback vector setup
Edinburg – 21-23 October, 2013
PVH technical deep-dive
11 / 25

42. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
PVH from Linux’s perspective
xen hvm domain() false, xen pv domain() true
Act natural (get rid of PV special cases)
Autotranslate side effects
Use PVHVM callback vector setup
VCPU bring-up
Edinburg – 21-23 October, 2013
PVH technical deep-dive
11 / 25

43. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Linux: Things disabled
No PV IDT
Edinburg – 21-23 October, 2013
PVH technical deep-dive
12 / 25

44. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Linux: Things disabled
No PV IDT
No PV irq ops
Edinburg – 21-23 October, 2013
PVH technical deep-dive
12 / 25

45. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Linux: Things disabled
No PV IDT
No PV irq ops
No PV CPUID
Edinburg – 21-23 October, 2013
PVH technical deep-dive
12 / 25

46. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Linux: Things disabled
No PV IDT
No PV irq ops
No PV CPUID
Native Syscall/sysenter
Edinburg – 21-23 October, 2013
PVH technical deep-dive
12 / 25

47. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Linux: Things disabled
No PV IDT
No PV irq ops
No PV CPUID
Native Syscall/sysenter
No VM assists
Edinburg – 21-23 October, 2013
PVH technical deep-dive
12 / 25

48. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Linux: Things disabled
No PV IDT
No PV irq ops
No PV CPUID
Native Syscall/sysenter
No VM assists
No event / failsafe callbacks
Edinburg – 21-23 October, 2013
PVH technical deep-dive
12 / 25

49. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Linux: Things disabled
No PV IDT
No PV irq ops
No PV CPUID
Native Syscall/sysenter
No VM assists
No event / failsafe callbacks
No need to set IOPL
Edinburg – 21-23 October, 2013
PVH technical deep-dive
12 / 25

50. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Linux: Things disabled
MMUOPS:
No need to pin pagetables
Edinburg – 21-23 October, 2013
PVH technical deep-dive
13 / 25

51. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Linux: Things disabled
MMUOPS:
No need to pin pagetables
No pfn/mfn conversion
Edinburg – 21-23 October, 2013
PVH technical deep-dive
13 / 25

52. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Linux: Things disabled
MMUOPS:
No need to pin pagetables
No pfn/mfn conversion
No need to special-case PT protections
Edinburg – 21-23 October, 2013
PVH technical deep-dive
13 / 25

53. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Linux: Things disabled
MMUOPS:
No need to pin pagetables
No pfn/mfn conversion
No need to special-case PT protections
Only pvmmu op: flush tlb others
Edinburg – 21-23 October, 2013
PVH technical deep-dive
13 / 25

54. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Linux: Autotranslate
PV
Edinburg – 21-23 October, 2013
PVH technical deep-dive
14 / 25

55. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Linux: Autotranslate
PV
Pagetables controlled by Xen
Edinburg – 21-23 October, 2013
PVH technical deep-dive
14 / 25

56. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Linux: Autotranslate
PV
Pagetables controlled by Xen
Real MFN in pagetables
Edinburg – 21-23 October, 2013
PVH technical deep-dive
14 / 25

57. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Linux: Autotranslate
PV
Pagetables controlled by Xen
Real MFN in pagetables
“Map this page here” hypercall
Edinburg – 21-23 October, 2013
PVH technical deep-dive
14 / 25

58. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Linux: Autotranslate
PV
Pagetables controlled by Xen
Real MFN in pagetables
“Map this page here” hypercall
PVH
Edinburg – 21-23 October, 2013
PVH technical deep-dive
14 / 25

59. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Linux: Autotranslate
PV
Pagetables controlled by Xen
Real MFN in pagetables
“Map this page here” hypercall
PVH
Pagetables controlled by guest
Edinburg – 21-23 October, 2013
PVH technical deep-dive
14 / 25

60. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Linux: Autotranslate
PV
Pagetables controlled by Xen
Real MFN in pagetables
“Map this page here” hypercall
PVH
Pagetables controlled by guest
gpfn in pagetables
Edinburg – 21-23 October, 2013
PVH technical deep-dive
14 / 25

61. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Linux: Autotranslate
PV
Pagetables controlled by Xen
Real MFN in pagetables
“Map this page here” hypercall
PVH
Pagetables controlled by guest
gpfn in pagetables
Every mapped page must be in the p2m
Edinburg – 21-23 October, 2013
PVH technical deep-dive
14 / 25

62. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Linux: Autotranslate
“Special pages”: grant frame, make a hole in the p2m
Edinburg – 21-23 October, 2013
PVH technical deep-dive
15 / 25

63. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Linux: Autotranslate
“Special pages”: grant frame, make a hole in the p2m
Need to add p2m entries to map foreign pages
Edinburg – 21-23 October, 2013
PVH technical deep-dive
15 / 25

64. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Linux: Autotranslate
“Special pages”: grant frame, make a hole in the p2m
Need to add p2m entries to map foreign pages
Grant table: similar to HVM (but not the same)
Edinburg – 21-23 October, 2013
PVH technical deep-dive
15 / 25

65. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Linux: VCPU bring-up
Brought up via hypercalls (PV)
Edinburg – 21-23 October, 2013
PVH technical deep-dive
16 / 25

66. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Linux: VCPU bring-up
Brought up via hypercalls (PV)
Guest in control of IDT, Xen can’t guarantee loading properly
Edinburg – 21-23 October, 2013
PVH technical deep-dive
16 / 25

67. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Linux: VCPU bring-up
Brought up via hypercalls (PV)
Guest in control of IDT, Xen can’t guarantee loading properly
Only set GS by default
Edinburg – 21-23 October, 2013
PVH technical deep-dive
16 / 25

68. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Linux: VCPU bring-up
Brought up via hypercalls (PV)
Guest in control of IDT, Xen can’t guarantee loading properly
Only set GS by default
All other state must be set in on-cpu boot code
Edinburg – 21-23 October, 2013
PVH technical deep-dive
16 / 25

69. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
PVH: Not yet working
32-bit
Edinburg – 21-23 October, 2013
PVH technical deep-dive
17 / 25

70. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
PVH: Not yet working
32-bit
vtsc
Edinburg – 21-23 October, 2013
PVH technical deep-dive
17 / 25

71. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
PVH: Not yet working
32-bit
vtsc
shadow mode
Edinburg – 21-23 October, 2013
PVH technical deep-dive
17 / 25

72. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
PVH: Not yet working
32-bit
vtsc
shadow mode
vcpu hotplug
Edinburg – 21-23 October, 2013
PVH technical deep-dive
17 / 25

73. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Issues: HVM paths
Original idea: “Lightweight” container
Edinburg – 21-23 October, 2013
PVH technical deep-dive
18 / 25

74. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Issues: HVM paths
Original idea: “Lightweight” container
Reality: Large amount of code necessary for minimum
functionality
Edinburg – 21-23 October, 2013
PVH technical deep-dive
18 / 25

75. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Issues: HVM paths
Original idea: “Lightweight” container
Reality: Large amount of code necessary for minimum
functionality
Very large amount of code duplication
Edinburg – 21-23 October, 2013
PVH technical deep-dive
18 / 25

76. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Issues: HVM paths
Original idea: “Lightweight” container
Reality: Large amount of code necessary for minimum
functionality
Very large amount of code duplication
Current patch: use HVM paths, with special cases for PVH
Edinburg – 21-23 October, 2013
PVH technical deep-dive
18 / 25

77. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Issues: PIO instructions
Direct access permitted
Edinburg – 21-23 October, 2013
PVH technical deep-dive
19 / 25

78. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Issues: PIO instructions
Direct access permitted
pv PIT
Edinburg – 21-23 October, 2013
PVH technical deep-dive
19 / 25

79. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Issues: PIO instructions
Direct access permitted
pv PIT
cmos idx
Edinburg – 21-23 October, 2013
PVH technical deep-dive
19 / 25

80. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Issues: PIO instructions
Direct access permitted
pv PIT
cmos idx
pci config space
Edinburg – 21-23 October, 2013
PVH technical deep-dive
19 / 25

81. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Issues: PIO instructions
Direct access permitted
pv PIT
cmos idx
pci config space
“Abusing” IO with SMM
Edinburg – 21-23 October, 2013
PVH technical deep-dive
19 / 25

82. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Issues: PIO instructions
Direct access permitted
pv PIT
cmos idx
pci config space
“Abusing” IO with SMM
Execute exact instruction with guest GPRs
Edinburg – 21-23 October, 2013
PVH technical deep-dive
19 / 25

83. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Issues: PIO instructions
Problems
Edinburg – 21-23 October, 2013
PVH technical deep-dive
20 / 25

84. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Issues: PIO instructions
Problems
Ugly changes to allow PV paths to emulate for PVH guests
Edinburg – 21-23 October, 2013
PVH technical deep-dive
20 / 25

85. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Issues: PIO instructions
Problems
Ugly changes to allow PV paths to emulate for PVH guests
Race condition because of double checking
Edinburg – 21-23 October, 2013
PVH technical deep-dive
20 / 25

86. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Issues: PIO instructions
Problems
Ugly changes to allow PV paths to emulate for PVH guests
Race condition because of double checking
Possibilities
Edinburg – 21-23 October, 2013
PVH technical deep-dive
20 / 25

87. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Issues: PIO instructions
Problems
Ugly changes to allow PV paths to emulate for PVH guests
Race condition because of double checking
Possibilities
Need 2 sets of access controls: For user procs, and for guest
Edinburg – 21-23 October, 2013
PVH technical deep-dive
20 / 25

88. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Issues: PIO instructions
Problems
Ugly changes to allow PV paths to emulate for PVH guests
Race condition because of double checking
Possibilities
Need 2 sets of access controls: For user procs, and for guest
PV only has one
Edinburg – 21-23 October, 2013
PVH technical deep-dive
20 / 25

89. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Issues: PIO instructions
Problems
Ugly changes to allow PV paths to emulate for PVH guests
Race condition because of double checking
Possibilities
Need 2 sets of access controls: For user procs, and for guest
PV only has one
PVH: Have two, so may not need to emulate anymore
Edinburg – 21-23 October, 2013
PVH technical deep-dive
20 / 25

90. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
HAP vs Shadow
Page table update
Edinburg – 21-23 October, 2013
PVH technical deep-dive
21 / 25

91. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
HAP vs Shadow
Page table update
HAP: Memory write
Edinburg – 21-23 October, 2013
PVH technical deep-dive
21 / 25

92. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
HAP vs Shadow
Page table update
HAP: Memory write
Shadow: Trip through Xen
Edinburg – 21-23 October, 2013
PVH technical deep-dive
21 / 25

93. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
HAP vs Shadow
Page table update
HAP: Memory write
Shadow: Trip through Xen
TLB effectiveness
Edinburg – 21-23 October, 2013
PVH technical deep-dive
21 / 25

94. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
HAP vs Shadow
Page table update
HAP: Memory write
Shadow: Trip through Xen
TLB effectiveness
HAP: Superpages allow 2MiB / 1GiB TLB entries
Edinburg – 21-23 October, 2013
PVH technical deep-dive
21 / 25

95. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
HAP vs Shadow
Page table update
HAP: Memory write
Shadow: Trip through Xen
TLB effectiveness
HAP: Superpages allow 2MiB / 1GiB TLB entries
Shadow: Still only 4k TLB entries
Edinburg – 21-23 October, 2013
PVH technical deep-dive
21 / 25

96. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
HAP vs Shadow
Page table update
HAP: Memory write
Shadow: Trip through Xen
TLB effectiveness
HAP: Superpages allow 2MiB / 1GiB TLB entries
Shadow: Still only 4k TLB entries
TLB replacement cost
Edinburg – 21-23 October, 2013
PVH technical deep-dive
21 / 25

97. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
HAP vs Shadow
Page table update
HAP: Memory write
Shadow: Trip through Xen
TLB effectiveness
HAP: Superpages allow 2MiB / 1GiB TLB entries
Shadow: Still only 4k TLB entries
TLB replacement cost
Shadow: Worst case, 4 memory reads
Edinburg – 21-23 October, 2013
PVH technical deep-dive
21 / 25

98. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
HAP vs Shadow
Page table update
HAP: Memory write
Shadow: Trip through Xen
TLB effectiveness
HAP: Superpages allow 2MiB / 1GiB TLB entries
Shadow: Still only 4k TLB entries
TLB replacement cost
Shadow: Worst case, 4 memory reads
HAP: 9-16 memory reads
Edinburg – 21-23 October, 2013
PVH technical deep-dive
21 / 25

99. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
HAP vs Shadow
Page table update
HAP: Memory write
Shadow: Trip through Xen
TLB effectiveness
HAP: Superpages allow 2MiB / 1GiB TLB entries
Shadow: Still only 4k TLB entries
TLB replacement cost
Shadow: Worst case, 4 memory reads
HAP: 9-16 memory reads
Does this matter?
Edinburg – 21-23 October, 2013
PVH technical deep-dive
21 / 25

100. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
HAP vs Shadow
Page table update
HAP: Memory write
Shadow: Trip through Xen
TLB effectiveness
HAP: Superpages allow 2MiB / 1GiB TLB entries
Shadow: Still only 4k TLB entries
TLB replacement cost
Shadow: Worst case, 4 memory reads
HAP: 9-16 memory reads
Does this matter?
Kernel build: Shadow 30% slower than HAP
Edinburg – 21-23 October, 2013
PVH technical deep-dive
21 / 25

101. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
HAP vs Shadow
Page table update
HAP: Memory write
Shadow: Trip through Xen
TLB effectiveness
HAP: Superpages allow 2MiB / 1GiB TLB entries
Shadow: Still only 4k TLB entries
TLB replacement cost
Shadow: Worst case, 4 memory reads
HAP: 9-16 memory reads
Does this matter?
Kernel build: Shadow 30% slower than HAP
SpecJBB: Shadow 30% faster than HAP
Edinburg – 21-23 October, 2013
PVH technical deep-dive
21 / 25

102. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Overview
PV, HVM, and PVH
Edinburg – 21-23 October, 2013
PVH technical deep-dive
22 / 25

103. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Overview
PV, HVM, and PVH
PVH from Xen’s perspective
Edinburg – 21-23 October, 2013
PVH technical deep-dive
22 / 25

104. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Overview
PV, HVM, and PVH
PVH from Xen’s perspective
PVH from Linux’s perspective
Edinburg – 21-23 October, 2013
PVH technical deep-dive
22 / 25

105. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Overview
PV, HVM, and PVH
PVH from Xen’s perspective
PVH from Linux’s perspective
Issues in PVH
Edinburg – 21-23 October, 2013
PVH technical deep-dive
22 / 25

106. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Goal
Give you a technical overview of PVH, so that...
Edinburg – 21-23 October, 2013
PVH technical deep-dive
23 / 25

107. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Goal
Give you a technical overview of PVH, so that...
you can understand the characteristics, advantages, and
disadvantages...
Edinburg – 21-23 October, 2013
PVH technical deep-dive
23 / 25

108. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Goal
Give you a technical overview of PVH, so that...
you can understand the characteristics, advantages, and
disadvantages...
and approach the code to improve and fix
Edinburg – 21-23 October, 2013
PVH technical deep-dive
23 / 25

109. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Questions
Questions?
Feedback? george.dunlap@citrix.com
Check out our blog: http://blog.xen.org/
Edinburg – 21-23 October, 2013
PVH technical deep-dive
24 / 25

110. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Xen: PVH hypercalls
memory op
Edinburg – 21-23 October, 2013
PVH technical deep-dive
25 / 25

111. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Xen: PVH hypercalls
memory op
console io
Edinburg – 21-23 October, 2013
PVH technical deep-dive
25 / 25

112. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Xen: PVH hypercalls
memory op
console io
vcpu op
Edinburg – 21-23 October, 2013
PVH technical deep-dive
25 / 25

113. Intro
PV and HVM
PVH and Xen
PVH and Linux
PVH Issues
Spare slides
Xen: PVH hypercalls
memory op
console io
vcpu op
mmuext op
Edinburg – 21-23 October, 2013
PVH technical deep-dive
25 / 25