Koichi Onoue: Controlling System Calls and Protecting Application Data in Virtual Machines

1. Controlling System Calls and
Protecting Application Data
in Virtual Machines
Koichi Onoue* Yoshihiro Oyama** Akinori Yonezawa*
* The University of Tokyo
** The University of Electro-Communications

2. Protection for Applications
• Security systems has been widely applied to provide
secure computing environments
– Sandboxing systems
– Intrusion detecion/prevension systems (IDSes/IPSes)
– Anti-virus tools Confidential
data
Security Application
control
system
OS kernel

3. Security Systems Can Also Be
Compromised！
• Security systems and the other applications are
running in the same execution space
Application 1 was
compromised ! Confidential
data
Application 1
Security Application 2
control
system
OS kernel

4. Advantages of Virtual Machine Monitor
(VMM) in Terms of Security
• VMM can provide strong
isolation between VMs Application Application
– VMM prevents a
compromised VM from OS OS
attacking to the other VMs （Guest OS） （Guest OS）
• VMM can control access Virtual machine Virtual machine
to physical computing (VM) (VM)
resources such as a
physical memory and a
disk
Virtual machine monitor (VMM)
– VMM is running at the
higher privileged level than
VMs Hardware

5. Our Goal
• Enhancing application security from outside of
VMs
– In cooperation with VMM, security systems
control behaviors of application and
protect application data
Target VM
Control VM
Application
Security
system
VMM

6. Our Approach
• Controlling system calls from outside of target VMs
– We confine behaviors of application processes
• Controlling memory and file operations related with
target applications in a VMM and a control VM
– We prevent non-target programs from leaking target
application data and tampering with them
 Our system control only the target application which
users specify
We extend a para-virtualization version of Xen

7. Controlling System Calls from
Outside of target VMs

8. Comparison between
“w/o VMM” and “w/ VMM”
Security Security system in
systems cooperation with VMM
(“w/o VMM”) (“w/ VMM”)
× ○
Attack against
security systems Not hard Hard
Execution states ○ ×
obtained by OS-level Hardware-level
security systems

9. Goal for Controlling System Calls
Security Security systems in
systems cooperation with VMM
× ○
Attack against
security systems Not hard Hard
Execution states ○ ×
obtained by OS-level Hardware-level
security systems
Semantic gap
Our goal

10. Approach for Controlling System Calls
• Controlling system calls from outside of VMs
– Using information on target OSes
• Conforming to security policies
Target VM
Control VM
Security
policy
application
Security
system
VMM

11. Bridging the Semantic Gap
• What a VMM can observe
– Events ：Privileged instructions, Interrupts, …
– Execution States：Registers, Memory pages, …
Semantic gap
• What security systems require
– Events ：System calls, …
– Execution states： Process ID, System call number, …

12. Security Policy
• Specifies invoked system calls with
pattern matching
...
open default: allow
fileEq(“/etc/passwd”)
or filePrefixEq(“/etc/cron.d”)
deny(EPERM)
...

13. Controlling Memory and File Operations
Related with Application Data

14. Goal for Protecting Application Data
• Prevent compromised programs from leaking
target data and tampering with them
– Attackers use ptrace system call and kernel
modules, etc.
Memory
Virtual disk
Target
Target
Confidential
data
Tampering
Leaking
Application
OS kernel
VM
Compromised program

15. Approach for Protecting Application Data
• Hiding “real” application data on a memory and
a virtual disk from compromised programs
– Compromised programs include target OS
• Application data on a memory
– Code region, data region, stack region, etc.
➔VMM multiplexes target physical pages
• Application data on a virtual disk
– Executables, configuration files, etc.
➔ Control VM manages them

16. OS Memory Management
Virtual address Physical address
space space
OS address
translation
Application
Application

17. VMM Memory Management
Target VM virtual Target VM VMM physical
address space physical address address space
gPA →
space
（gVA） （hPA）
hPA
（gPA）
Application
Application
application
gVA → hPA
VMM address translation
17

18. Approach for Protecting Application
Memory Data (1/2)
• According to the operational mode, a VMM
switches accessible physical pages
Target VM virtual Target VM VMM physical
address space physical address address space
space
（gVA） （hPA）
（gPA）
Application
Dummy data
Application
“Real” data
Accessible page at user-level
Accessible page at kernel-level
– Overshadow[Chen et al., 2008]
– [Rosenblum et al.,2008]

19. Approach for Protecting Application
Memory Data (2/2)
• When the operational mode were changed,
a VMM switch the page tables
– Exception/Interrupt handling
– System call handling

20. Approach for Protecting Application
File Data (1/5)
• Control VM manages “real” target files
– Executables, configuration and data base files, etc.
– Security policy specifies target files
Control VM Target VM
Security
Dummy
policy Security system Application configuration
“Real” file
configuration file Dummy
“Real” executable executable
VMM

21. Approach for Protecting Application
File Data (2/5)
Control VM Target VM
Memory
Security system
Application read a
configuration file
Configuration
file
Application
VMM

22. Approach for Protecting Application
File Data (3/5)
Control VM Target VM
Memory
Security system
VMM intercepts
Configuration
“read” system call
file
Application
VMM

23. Approach for Protecting Application
File Data (4/5)
Security system Control VM Target VM
emulates “read”
Memory
Security system
Configuration
file
Application
VMM

24. Approach for Protecting Application
File Data (5/5)
Control VM Target VM
Memory
Security system
Configuration VMM notifies
file application of a Application
result of “read”
VMM

25. Conclusion
• We have proposed security system that
enhances applications inside target VMs
– Controlling of application behaviors
• Controlling of system calls from outside of target VMs
– Protecting application data on a memory and
a virtual disk
• Application memory data: VMM multiplexes target
physical pages
• Application file data: Control VM manages them

26. Fin