1. Autor: 2013/09/20
Disk I/O Performance Comparison
Xen v.s. KVM

2. Autor: 2013/09/20
Agenda
1. About This Research
2. Benchmark Configuration
3. Test Result with HDD
4. Test Result with SSD
5. Consideration

3. Autor: 2013/09/20
1. About This Research

4. Autor: 2013/09/20
Goal
Get enough information on I/O performance
characteristic of Xen and KVM so that we can rightly
judge which shoud be adopted for a use case.

5. Autor: 2013/09/20
Premise
This document is writen on a premise that you know
what Xen and KVM is.
The explanation about them is skipped mainly because
author's laziness.

6. Autor: 2013/09/20
2. Benchmark Configuration

7. Autor: 2013/09/20
Benchmarked Virtulization Softwares
Two famous open source virtulization softwares are
tested.
● KVM
● Xen 4.2.2
● XenServer 6.2
→ Could not test because the installer did not find
any disk to install...

8. Autor: 2013/09/20
VM Host Server
VM host server specification:
CPU model Core i 5 2500K 3.3GHz 2Core4Thread
CPU settings Hyper-Threading enabled
Turbo Core disabled
Power Saving disabled
Memory 16GB DDR3 1,333MHz dual channel
Disk 1 80GB HDD 3.5inch 7,200rpm
Hitachi HDS721680PLA380
Disk 2 128GB SSD
CFD CSSD-S6T128MHG5Q (Toshiba HG5Q)
OS CentOS 6.4
filesystem Ext4

9. Autor: 2013/09/20
VM Guest Server
VM guest server common specification:
VCPU 2
Memory 512MB
Disk Size 10GB
Thin Provisioning No
OS CentOS 6.4
Disk Driver Para-Virtulized Driver
filesystem Ext4

10. Autor: 2013/09/20
Benchmark Tool
Flexible I/O benchmark tool "fio" is used.
http://freecode.com/projects/fio
● Setting item exmaples:
read/write, sequential/random,
direct false/true (use file cache/do not use)...
● Abalable data examples:
bandwidth, IOPS, latency distribution,
CPU load, IO %util ...

11. Autor: 2013/09/20
Benchmark Configurations
Tested Configuration:
I/O size(byte) Load Type Direct Test Limit
11 random read false 1GB or 180sec
random write false 1GB or 180sec
512 random read true 1GB or 180sec
random write true 1GB or 180sec
4k random read true 3GB or 180sec
random write true 3GB or 180sec
32k random read true 3GB or 180sec
random write true 3GB or 180sec
512k random read true 3GB or 180sec
random write true 3GB or 180sec
1m sequential read true 3GB or 180sec
sequential write true 3GB or 180sec

12. Autor: 2013/09/20
KVM disk cache
Two benchmarked disk cache configurations:
・ writethrough/default
Defalut setting by virt-manager
on Ubuntu 12.10
・ none
Defalut setting by virt-manager
on CentOS 6.4
http://infoliser.com/a-guide-to-kvm-guest-disk-cache/

13. Autor: 2013/09/20
KVM disk cache
Other disk cache configurations:
Not benchmarked in this test.
・ writeback ・ diretsync
http://infoliser.com/a-guide-to-kvm-guest-disk-cache/

14. Autor: 2013/09/20
Criteria
● IOPS: Higher is better
● Bandwidth: Higher is better (I/O size * IOPS)
● Latency: Lower is better, lower variance is better
● CPU usage: Lower is better
Emuration cost is
in KVM: host CPU – guest CPU
in Xen: Domain0 CPU?
● I/O %util

15. Autor: 2013/09/20
Benchmark Scripts
KVM/Xen installation and benchmarks are done by
scripts published here
https://github.com/nknytk/disk-performance-xen-
kvm

16. Autor: 2013/09/20
3. Test Result with HDD

17. Autor: 2013/09/20
Tested Machines
● Host
● KVM guest1: disk cache = writethrough
● KVM guest2: disk cache = none
● Xen guest
In all machines, disk scheduler is "cfq."

18. Autor: 2013/09/20
Summary
● KVM with cache="writethrough" performs well
only in small size reading.
● Xen generally performs best. Even better than host.
● Xen's unique latency distribution seem to be
resulting from some optimization.
● Performance of KVM with cache=“none“ is slightly
worse than host, but the delay is within 10%.
● KVM with cache=“none“ has very similar latency
distribution to host.

19. Autor: 2013/09/20
Detalied Results
● Only typical data is represented in this presentation.
● You can get full data from the repository bellow.
https://github.com/nknytk/disk-performance-xen-
kvm/tree/master/result_hdd_examples

20. Autor: 2013/09/20
IOPS iosize = 11byte
xen
kvm2
kvm1
host
0 100 200 300 400 500 600 700 800 900 1000
206
139
680116
143
11byte direct=false
randomread IOPS
xen
kvm2
kvm1
host
0 100 200 300 400 500 600 700 800 900 1000
335
126
372087
127
11byte direct=false
randomwrite IOPS

21. Autor: 2013/09/20
Latency Distribution iosize = 11byte0.002
0.004
0.010
0.020
0.050
0.100
0.250
0.500
0.750
1.000
2.000
4.000
10.000
20.000
50.000
100.000
250.000
1000.000
0
20
40
60
80
100
11byte direct=false
random read latency distribution
host
kvm1
kvm2
xen
latency(msec)
distribution(%)
0.002
0.004
0.010
0.020
0.050
0.100
0.250
0.500
0.750
1.000
2.000
4.000
10.000
20.000
50.000
100.000
250.000
500.000
750.000
1000.000
0
20
40
60
80
100
11byte direct=false
random write latency distribution
host
kvm1
kvm2
xen
latency(msec)
distribution(%)

22. Autor: 2013/09/20
IOPS iosize = 4kbyte
xen
kvm2
kvm1
host
0 20 40 60 80 100 120 140 160 180
133
119
156
126
4kbyte direct=true
randomread IOPS
xen
kvm2
kvm1
host
0 200 400 600 800 1000 1200
1122
164
97
185
4kbyte direct=true
randomwrite IOPS

23. Autor: 2013/09/20
Latency Distribution iosize = 4kbyte0.1
0.25
0.5
0.75
1
2
4
10
20
50
100
250
500
0
10
20
30
40
50
60
70
80
90
100
4kbyte direct=true
random read latency distribution
host
kvm1
kvm2
xen
latency(msec)
distribution(%)
0.05
0.1
0.25
0.5
0.75
1
2
4
10
20
50
100
250
500
0
10
20
30
40
50
60
70
80
90
100
4kbyte direct=true
random read latency distribution
host
kvm1
kvm2
xen
latency(msec)
distribution(%)

24. Autor: 2013/09/20
IOPS iosize = 512kbyte
xen
kvm2
kvm1
host
0 10 20 30 40 50 60 70
50
61
41
64
512kbyte direct=true
randomread IOPS
xen
kvm2
kvm1
host
0 10 20 30 40 50 60 70 80 90
85
61
46
60
512kbyte direct=true
randomwrite IOPS

25. Autor: 2013/09/20
Latency Distribution iosize = 512kbyte0.25
0.5
0.75
4
10
20
50
100
250
500
1000
2000
0
10
20
30
40
50
60
70
80
90
100
512kbyte direct=true
random read latency distribution
host
kvm1
kvm2
xen
latency(msec)
distribution(%)
0.5
0.75
1
2
4
10
20
50
100
250
500
1000
1000
500
0
10
20
30
40
50
60
70
80
90
100
512kbyte direct=true
random read latency distribution
host
kvm1
kvm2
xen
latency(msec)
distribution(%)

26. Autor: 2013/09/20
IOPS iosize = 1mbyte
xen
kvm2
kvm1
host
0 10 20 30 40 50 60 70 80
69
62
57
68
1mbyte direct=true
sequential read IOPS
xen
kvm2
kvm1
host
0 10 20 30 40 50 60 70 80
74
71
41
70
1mbyte direct=true
sequential write IOPS

27. Autor: 2013/09/20
Latency Distribution iosize = 1mbyte0.25
0.5
1
2
10
20
50
100
250
500
750
0
10
20
30
40
50
60
70
80
90
100
1mbyte direct=true
sequential read latency distribution
host
kvm1
kvm2
xen
latency(msec)
distribution(%)
0.75
1
2
4
10
20
50
100
250
0
10
20
30
40
50
60
70
80
90
100
1mbyte direct=true
sequential read latency distribution
host
kvm1
kvm2
xen
latency(msec)
distribution(%)

28. Autor: 2013/09/20
Read Bandwidth Comparison
11(direct=false) 512 4k 32k 512k 1m(sequential)
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
Read Bandwidth Comparison (host=1)
host
kvm1
kvm2
xen
I/O size

29. Autor: 2013/09/20
Write Bandwidth Comparison
11(direct=false) 512 4k 32k 512k 1m(sequential)
0
1
2
3
4
5
6
Write Bandwidth Comparison (host=1)
host
kvm1
kvm2
xen
I/O size

30. Autor: 2013/09/20
4. Test Result with SSD

31. Autor: 2013/09/20
Tested Machines
● Host
● KVM guest1
- disk cache = "writethrough"
- Another process consumes so much memory on
host that only 200mb is left for page cache
● KVM guest2
- disk cache = "none"
● Xen guest
In all machines, disk scheduler is "noop."

32. Autor: 2013/09/20
Summary
● All VMs are unignorably delayed in small size I/O.
● Both KVM and Xen cost nearly 100% of CPU
for emuration at the worst case.
● KVM's performance with cache=“writethrough“ is
lower than KVM with cache=“none“ by up to 90%.
● KVM with cache=“none“ has similar latency
distribution to host.

33. Autor: 2013/09/20
Detalied Results
● Only typical data is represented in this presentation.
● You can get full data at the repository bellow.
https://github.com/nknytk/disk-performance-xen-
kvm/tree/master/result_ssd_examples

34. Autor: 2013/09/20
IOPS iosize = 11byte
xen
kvm2
kvm1
host
0 100000 200000 300000 400000 500000 600000 700000 800000
5263
4514
231262
737753
11byte direct=false
randomread IOPS
xen
kvm2
kvm1
host
0 50000 100000 150000 200000 250000 300000 350000 400000 450000 500000
3006
3146
927
447745
11byte direct=false
randomwrite IOPS

35. Autor: 2013/09/20
CPU Usage iosize = 11byte
xen
kvm2
kvm1
host
0 20 40 60 80 100 120 140
11byte direct=false
random read CPU usage (100%=1core)
test_server host
xen
kvm2
kvm1
host
0 20 40 60 80 100 120
11byte direct=false
random write CPU usage (100%=1core)
test_server host

36. Autor: 2013/09/20
IO %util iosize = 11byte
xen
kvm2
kvm1
host
0 10 20 30 40 50 60 70 80 90 100
11byte direct=false
random read IO %util
test_server host
xen
kvm2
kvm1
host
0 10 20 30 40 50 60 70 80 90 100
11byte direct=false
random write IO %util
test_server host

37. Autor: 2013/09/20
Latency Distribution iosize = 11byte0.002
0.004
0.010
0.020
0.050
0.100
0.250
0.500
0.750
1.000
2.000
4.000
10.000
20.000
0
10
20
30
40
50
60
70
80
90
100
11byte direct=false
random read latency distribution
host
kvm1
kvm2
xen
latency(msec)
distribution(%)
0.002
0.004
0.010
0.020
0.050
0.100
0.250
0.500
0.750
1.000
2.000
4.000
10.000
20.000
50.000
100.000
250.000
500.000
0
10
20
30
40
50
60
70
80
90
100
11byte direct=false
random write latency distribution
host
kvm1
kvm2
xen
latency(msec)
distribution(%)

38. Autor: 2013/09/20
IOPS iosize = 4kbyte
xen
kvm2
kvm1
host
0 2000 4000 6000 8000 10000 12000
3139
3029
9894
4260
4kbyte direct=true
randomread IOPS
xen
kvm2
kvm1
host
0 2000 4000 6000 8000 10000 12000 14000
11833
6246
581
10198
4kbyte direct=true
randomwrite IOPS

39. Autor: 2013/09/20
CPU Usage iosize = 4kbyte
xen
kvm2
kvm1
host
0 10 20 30 40 50 60 70
4kbyte direct=true
random read CPU usage (100%=1core)
test_server host
xen
kvm2
kvm1
host
0 20 40 60 80 100 120 140 160
4kbyte direct=true
random write CPU usage (100%=1core)
test_server host

40. Autor: 2013/09/20
IO %util iosize = 4kbyte
xen
kvm2
kvm1
host
0 10 20 30 40 50 60 70 80 90 100
4kbyte direct=true
random read IO %util
test_server host
xen
kvm2
kvm1
host
0 10 20 30 40 50 60 70 80 90 100
4kbyte direct=true
random write IO %util
test_server host

41. Autor: 2013/09/20
Latency Distribution iosize = 4kbyte0.002
0.004
0.010
0.020
0.050
0.100
0.250
0.500
0.750
1.000
2.000
4.000
10.000
20.000
50.000
250.000
0
10
20
30
40
50
60
70
80
90
100
4kbyte direct=true
random read latency distribution
host
kvm1
kvm2
xen
latency(msec)
distribution(%)
0.05
0.1
0.25
0.5
0.75
1
2
4
10
20
50
100
250
0
10
20
30
40
50
60
70
80
90
100
4kbyte direct=true
random read latency distribution
host
kvm1
kvm2
xen
latency(msec)
distribution(%)

42. Autor: 2013/09/20
IOPS iosize = 512kbyte
xen
kvm2
kvm1
host
0 100 200 300 400 500 600 700 800
345
660
338
748
512kbyte direct=true
randomread IOPS
xen
kvm2
kvm1
host
0 100 200 300 400 500 600 700 800 900
748
678
454
800
512kbyte direct=true
randomwrite IOPS

43. Autor: 2013/09/20
CPU Usage iosize = 512kbyte
xen
kvm2
kvm1
host
0 10 20 30 40 50 60 70 80
512kbyte direct=true
random read CPU usage (100%=1core)
test_server host
xen
kvm2
kvm1
host
0 10 20 30 40 50 60 70 80 90 100
512kbyte direct=true
random write CPU usage (100%=1core)
test_server host

44. Autor: 2013/09/20
IO %util iosize = 512kbyte
xen
kvm2
kvm1
host
0 10 20 30 40 50 60 70 80 90 100
512kbyte direct=true
random read IO %util
test_server host
xen
kvm2
kvm1
host
0 10 20 30 40 50 60 70 80 90 100
512kbyte direct=true
random write IO %util
test_server host

45. Autor: 2013/09/20
Latency Distribution iosize = 512kbyte2
4
10
20
50
100
250
0
10
20
30
40
50
60
70
80
90
100
512kbyte direct=true
random read latency distribution
host
kvm1
kvm2
xen
latency(msec)
distribution(%)
0.5
0.75
1
2
4
10
20
50
100
250
0
10
20
30
40
50
60
70
80
90
100
512kbyte direct=true
random read latency distribution
host
kvm1
kvm2
xen
latency(msec)
distribution(%)

46. Autor: 2013/09/20
IOPS iosize = 1mbyte
xen
kvm2
kvm1
host
0 50 100 150 200 250 300 350 400 450 500
315
434
254
440
1mbyte direct=true
sequential read IOPS
xen
kvm2
kvm1
host
0 50 100 150 200 250 300 350 400 450
369
364
259
412
1mbyte direct=true
sequential write IOPS

47. Autor: 2013/09/20
CPU Usage iosize = 1mbyte
xen
kvm2
kvm1
host
0 10 20 30 40 50 60 70 80 90 100
1mbyte direct=true
sequential read CPU usage (100%=1core)
test_server host
xen
kvm2
kvm1
host
0 20 40 60 80 100 120
1mbyte direct=true
sequential write CPU usage (100%=1core)
test_server host

48. Autor: 2013/09/20
IO %util iosize = 1mbyte
xen
kvm2
kvm1
host
0 10 20 30 40 50 60 70 80 90 100
1mbyte direct=true
sequential read IO %util
test_server host
xen
kvm2
kvm1
host
0 10 20 30 40 50 60 70 80 90 100
1mbyte direct=true
sequential write IO %util
test_server host

49. Autor: 2013/09/20
Latency Distribution iosize = 1mbyte4
10
20
100
250
0
10
20
30
40
50
60
70
80
90
100
1mbyte direct=true
sequential read latency distribution
host
kvm1
kvm2
xen
latency(msec)
distribution(%)
0.75
1
2
4
10
20
50
100
250
0
10
20
30
40
50
60
70
80
90
100
1mbyte direct=true
sequential read latency distribution
host
kvm1
kvm2
xen
latency(msec)
distribution(%)

50. Autor: 2013/09/20
Read Bandwidth Comparison
11(direct=false) 512 4k 32k 512k 1m(sequential)
0
0.5
1
1.5
2
2.5
3
Read Bandwidth Comparison (host=1)
host
kvm1
kvm2
xen
I/O size

51. Autor: 2013/09/20
Write Bandwidth Comparison
11(direct=false) 512 4k 32k 512k 1m(sequential)
0
0.2
0.4
0.6
0.8
1
1.2
1.4
Read Bandwidth Comparison (host=1)
host
kvm1
kvm2
xen
I/O size

52. Autor: 2013/09/20
5. Consideration

53. Autor: 2013/09/20
Use Cases
● If your VM host server's disks are slow, Xen will
offer better I/O performance than KVM.
● KVM with cache="none" is relatively appropriate
for performance simulation of physical server
because of the similarity of latency distribution.
● Both Xen and KVM are not suitable for high
random I/O load on super fast device,
e.g. OLTP DB server with ioDrive.

54. Autor: 2013/09/20
Questions
● In this test, I/O load is single-threaded.
What if prarel I/O load by many guests?
● Is Xen's data safe?
It seems to me that
more optimization
→ longer time in mermory
→ larger data loss in server fault