1. Inspecting a multi-everything Linux system
Understand your system
Frédéric Descamps
Original slides and idea by Aurimas Mikalauskas
Percona Live MySQL Conference & Expo
April, 2014

2. AurimasAurimas
dev/ops 1999
dba 2002
consultant 2005
Percona 2007
@ Percona:
- Architect
- Focus: High performance, troubleshooting, full text search,
data recovery

3. Me (@lefred)Me (@lefred)
dev/ops 1998
consultant 2001
Percona 2011
@ Percona:
- Senior Consultant
- Focus: High performance, high availability, troubleshooting
- devops believer

4. - MySQL Consulting
- Development
- Percona Server
- Percona Toolkit
- Percona Xtrabackup
- Percona XtraDB Cluster
- Remote DBA
- MySQL support
- Training
Percona

5. IT’S JUSTIT’S JUSTa system.a system.

6. I/O spikes
#ts wr_s wr_rt busy
1.0 13.0 8.7 7%
1.0 20.0 9.0 7%
1.0 27.0 8.4 7%
1.0 27.0 9.0 8%
1.0 283.0 40.9 21%
1.0 603.0 10.6 45%
1.0 327.0 10.7 28%
1.0 52.0 7.7 12%
1.0 26.0 7.9 6%
1.0 12.0 7.3 6%
1.0 7.0 9.0 6%
1.0 10.0 9.8 7%
1.0 24.0 7.5 8%
1.0 10.0 7.4 7%
1.0 298.0 71.0 30%
1.0 487.0 246.3 100%
1.0 279.0 275.1 74%
1.0 40.0 9.3 7%
1.0 15.0 7.4 8%
1.0 62.0 8.4 9%
1.0 23.0 7.4 5%
1.0 19.0 8.8 6%
1.0 27.0 8.0 7%
1.0 15.0 9.3 6%
1.0 570.0 151.8 80%
1.0 402.0 341.6 100%
1.0 33.0 214.4 26%
1.0 35.0 9.8 8%
1.0 15.0 11.1 8%

7. 10s10s

9. 16kb16kb

12. 0.0001s0.0001s

13. 0.01s0.01s

14. 7.2krpm7.2krpm

18. IT’S JUSTIT’S JUSTa system.a system.

19. GNU LinuxGNU Linux

20. MySQL
MyISAM
Squid
MacOS X
SAN
InnoDB
httpd
BIG data
FreeBSD
MongoDB
DRBD
varnish
DATA
Linux
XFS

21. STANDARD I/OSTANDARD I/O
librarylibrary

22. load average: 1.00, 1.00, 1.00

24. STANDARD I/OSTANDARD I/O
librarylibrary

25. what aboutwhat about
UNBUFFERED I/O?UNBUFFERED I/O?

26. Writing big file with dd
# dd if=/dev/fioa of=/data/opt/testfile.bin bs=4096 count=7864320
...
32212254720 bytes (32 GB) copied, 70.7952 s, 455 MB/s
// strace
...
open("/dev/fioa", O_RDONLY) = 3
dup2(3, 0) = 0
close(3) = 0
lseek(0, 0, SEEK_CUR) = 0
open("/data/opt/testfile.bin", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
dup2(3, 1) = 1
close(3) = 0
read(0, "353H220202163202740260270002163302163003732760"..., 4096) = 4096
write(1, "353H220202163202740260270002163302163003732760"..., 4096) = 4096
read(0, "00000000000000000000000000000000"..., 4096) = 4096
write(1, "00000000000000000000000000000000"..., 4096) = 4096
...
close(0) = 0
close(1) = 0

27. meanwhile...
# vmstat 1
procs ­­­­­­­­­­­memory­­­­­­­­­­ ­­­swap­­ ­­­­­io­­­­ ­­system­­ ­­­­­cpu­­­­­
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 64979240 6457316 48724 0 0 0 0 726 177 0 0 100 0 0
1 0 0 64736104 6578076 168964 0 0 121456 0 1769 5468 0 2 98 0 0
1 0 0 63733356 7065556 656456 0 0 487424 0 6133 21445 0 7 93 1 0
1 0 0 62660484 7586900 1177756 0 0 521344 12 6051 22913 0 8 92 0 0
1 0 0 61592424 8105460 1696316 0 0 518656 0 6082 23024 0 7 93 0 0
1 0 0 60440380 8665556 2256852 0 0 559932 44 6266 26413 0 7 92 1 0
1 0 0 59119376 9306964 2898172 0 0 641348 0 7015 31229 0 6 93 0 0
1 0 0 58010208 9845716 3436984 0 0 538752 0 6158 25078 0 7 93 0 0
1 0 0 57018712 10327256 3918336 0 0 481536 24 6004 21398 0 7 93 0 0
1 0 0 55895652 10872924 4464236 0 0 545664 0 6280 25376 0 6 93 0 0
3 0 0 54591312 11506264 5097260 0 0 633344 8 6985 31139 0 8 90 1 0
2 0 0 53263628 12151256 5742488 0 0 644480 0 6971 31668 0 6 93 1 0
1 0 0 51964508 12782296 6373724 0 0 631424 0 7030 31187 0 8 91 1 0
1 0 0 51054968 13224664 6815484 0 0 442212 0 7619 27035 0 5 93 2 0
0 1 0 50891056 13304412 6895208 0 0 79644 12 8448 18104 0 2 92 6 0
0 1 0 49882144 13794140 7385144 0 0 489984 229598 9420 31855 0 6 92 2 0
1 0 0 48985468 14228012 7819192 0 0 433920 434756 10934 30660 0 5 93 1 0
1 1 0 48024344 14694492 8285256 0 0 466432 471500 10914 33897 0 12 84 3 0
1 0 0 47064056 15160924 8751692 0 0 466432 467400 10979 33871 0 7 90 2 0
... many samples skipped ...
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 0 1940860 37169660 30761252 0 0 455680 459200 14158 27724 0 6 91 2 0
0 1 0 920864 37665532 31257060 0 0 495744 500200 14708 28307 0 6 92 2 0
2 0 0 3304476 35016696 31506636 0 0 248832 250136 10781 14505 0 9 89 1 0
1 0 0 9349772 28971956 31506812 0 0 0 0 3503 294 0 6 94 0 0

28. Reads kick in. Writes?
# vmstat 1
procs ­­­­­­­­­­­memory­­­­­­­­­­ ­­­swap­­ ­­­­­io­­­­ ­­system­­ ­­­­­cpu­­­­­
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 64979240 6457316 48724 0 0 0 0 726 177 0 0 100 0 0
1 0 0 64736104 6578076 168964 0 0 121456 0 1769 5468 0 2 98 0 0
1 0 0 63733356 7065556 656456 0 0 487424 0 6133 21445 0 7 93 1 0
1 0 0 62660484 7586900 1177756 0 0 521344 12 6051 22913 0 8 92 0 0
1 0 0 61592424 8105460 1696316 0 0 518656 0 6082 23024 0 7 93 0 0
1 0 0 60440380 8665556 2256852 0 0 559932 44 6266 26413 0 7 92 1 0
1 0 0 59119376 9306964 2898172 0 0 641348 0 7015 31229 0 6 93 0 0
1 0 0 58010208 9845716 3436984 0 0 538752 0 6158 25078 0 7 93 0 0
1 0 0 57018712 10327256 3918336 0 0 481536 24 6004 21398 0 7 93 0 0
1 0 0 55895652 10872924 4464236 0 0 545664 0 6280 25376 0 6 93 0 0
3 0 0 54591312 11506264 5097260 0 0 633344 8 6985 31139 0 8 90 1 0
2 0 0 53263628 12151256 5742488 0 0 644480 0 6971 31668 0 6 93 1 0
1 0 0 51964508 12782296 6373724 0 0 631424 0 7030 31187 0 8 91 1 0
1 0 0 51054968 13224664 6815484 0 0 442212 0 7619 27035 0 5 93 2 0
0 1 0 50891056 13304412 6895208 0 0 79644 12 8448 18104 0 2 92 6 0
0 1 0 49882144 13794140 7385144 0 0 489984 229598 9420 31855 0 6 92 2 0
1 0 0 48985468 14228012 7819192 0 0 433920 434756 10934 30660 0 5 93 1 0
1 1 0 48024344 14694492 8285256 0 0 466432 471500 10914 33897 0 12 84 3 0
1 0 0 47064056 15160924 8751692 0 0 466432 467400 10979 33871 0 7 90 2 0
... many samples skipped ...
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 0 1940860 37169660 30761252 0 0 455680 459200 14158 27724 0 6 91 2 0
0 1 0 920864 37665532 31257060 0 0 495744 500200 14708 28307 0 6 92 2 0
2 0 0 3304476 35016696 31506636 0 0 248832 250136 10781 14505 0 9 89 1 0
1 0 0 9349772 28971956 31506812 0 0 0 0 3503 294 0 6 94 0 0

29. disk stats?
# pt­diskstats
#ts device wr_s wr_avkb wr_mb_s wr_mrg wr_cnc wr_rt busy in_prg
{1} sda5 0.0 0.0 0.0 0% 0.0 0.0 0% 0
{1} sda5 0.0 0.0 0.0 0% 0.0 0.0 0% 0
{1} sda5 0.0 0.0 0.0 0% 0.0 0.0 0% 0
{1} sda5 0.0 0.0 0.0 0% 0.0 0.0 0% 0
{1} sda5 0.0 0.0 0.0 0% 0.0 0.0 0% 0
{1} sda5 0.0 0.0 0.0 0% 0.0 0.0 0% 0
{1} sda5 0.0 0.0 0.0 0% 0.0 0.0 0% 0
{1} sda5 0.0 0.0 0.0 0% 0.0 0.0 0% 0
{1} sda5 0.0 0.0 0.0 0% 0.0 0.0 0% 0
{1} sda5 0.0 0.0 0.0 0% 0.0 0.0 0% 0
{1} sda5 0.0 0.0 0.0 0% 0.0 0.0 0% 0
{1} sda5 0.0 0.0 0.0 0% 0.0 0.0 0% 0
{1} sda5 0.0 0.0 0.0 0% 0.0 0.0 0% 0
{1} sda5 0.0 0.0 0.0 0% 0.0 0.0 0% 0
{1} sda5 0.0 0.0 0.0 0% 0.0 0.0 0% 0
{1} sda5 595.0 455.6 264.7 0% 1.3 2.2 20% 0
{1} sda5 950.5 456.9 424.1 0% 1.7 1.8 29% 0
{1} sda5 1031.0 456.0 459.1 0% 1.9 1.8 32% 0
{1} sda5 1027.4 456.0 457.5 0% 1.9 1.8 32% 0
{1} sda5 1027.4 456.0 457.5 0% 1.9 1.8 32% 0
... many samples skipped ...
{1} sda5 1045.2 456.0 465.4 0% 1.7 1.7 32% 0
{1} sda5 1072.6 456.4 478.0 0% 2.0 1.9 33% 0
{1} sda5 0.0 0.0 0.0 0% 0.0 0.0 0% 0
{1} sda5 0.0 0.0 0.0 0% 0.0 0.0 0% 0
{1} sda5 0.0 0.0 0.0 0% 0.0 0.0 0% 0

30. Let’s call sync
# sync &
... and while sync is running:
# vmstat 1
procs ­­­­­­­­­­­memory­­­­­­­­­­ ­­­swap­­ ­­­­­io­­­­ ­­system­­ ­­­cpu­­­­
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 31864892 6457368 31506576 0 0 0 0 626 193 0 0 100 0 0
1 0 0 31865272 6457372 31506572 0 0 28 139452 1521 305 0 2 97 0 0
0 1 0 31863884 6457372 31506608 0 0 0 855552 4480 264 0 2 93 5 0
... 8 samples skipped ...
0 1 0 31864424 6457372 31506632 0 0 0 656384 3135 237 0 0 92 7 0
0 0 0 31864556 6457372 31506636 0 0 0 174909 3556 260 0 2 97 1 0
0 0 0 31864572 6457372 31506636 0 0 0 0 629 162 0 0 100 0 0
1 0 0 31864944 6457372 31506640 0 0 0 0 566 149 0 0 100 0 0
0 0 0 31866232 6457372 31506640 0 0 0 0 464 140 0 0 100 0 0

31. Let’s call sync
# sync &
... and while sync is running:
# vmstat 1
procs ­­­­­­­­­­­memory­­­­­­­­­­ ­­­swap­­ ­­­­­io­­­­ ­­system­­ ­­­cpu­­­­
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 31864892 6457368 31506576 0 0 0 0 626 193 0 0 100 0 0
1 0 0 31865272 6457372 31506572 0 0 28 139452 1521 305 0 2 97 0 0
0 1 0 31863884 6457372 31506608 0 0 0 855552 4480 264 0 2 93 5 0
... 8 samples skipped ...
0 1 0 31864424 6457372 31506632 0 0 0 656384 3135 237 0 0 92 7 0
0 0 0 31864556 6457372 31506636 0 0 0 174909 3556 260 0 2 97 1 0
0 0 0 31864572 6457372 31506636 0 0 0 0 629 162 0 0 100 0 0
1 0 0 31864944 6457372 31506640 0 0 0 0 566 149 0 0 100 0 0
0 0 0 31866232 6457372 31506640 0 0 0 0 464 140 0 0 100 0 0
# pt­diskstats
#ts device wr_s wr_avkb wr_mb_s wr_mrg wr_cnc wr_rt busy in_prg
{1} sda5 0.0 0.0 0.0 0% 0.0 0.0 0% 0
{1} sda5 296.5 456.8 132.2 0% 8.5 28.7 9% 0
{1} sda5 1829.4 512.0 914.7 0% 130.4 71.3 96% 0
{1} sda5 1316.7 512.0 658.4 0% 143.0 108.6 100% 0
... 7 samples skipped ...
{1} sda5 1313.8 512.0 656.9 0% 142.9 108.8 100% 0
{1} sda5 1094.2 511.3 546.4 0% 119.3 109.0 83% 0
{1} sda5 9.7 3.7 0.0 9% 0.0 0.3 0% 0
{1} sda5 0.0 0.0 0.0 0% 0.0 0.0 0% 0

32. What if file is small?
# dd if=/dev/zero of=/data/opt/testfile.bin bs=4096 count=262144
1073741824 bytes (1.1 GB) copied, 0.957317 s, 1.1 GB/s
# vmstat 1
procs ­­­­­­­­­­­memory­­­­­­­­­­ ­­­swap­­ ­­­­io­­­ ­­system­ ­­­­­cpu­­­­­
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 64976544 6458268 49352 0 0 0 0 701 178 0 0 100 0 0
0 0 0 64976584 6458268 49356 0 0 0 0 488 160 0 0 100 0 0
1 0 0 64491596 6458268 534368 0 0 0 0 4022 191 0 4 96 0 0
0 0 0 63926396 6458276 1097560 0 0 0 12 3315 160 0 5 94 0 0
# pt­diskstats
#ts device wr_s wr_avkb wr_mb_s wr_mrg wr_cnc wr_rt busy in_prg
{1} sda5 0.0 0.0 0.0 0% 0.0 0.0 0% 0
{1} sda5 0.0 0.0 0.0 0% 0.0 0.0 0% 0
{1} sda5 0.0 0.0 0.0 0% 0.0 0.0 0% 0
{1} sda5 0.0 0.0 0.0 0% 0.0 0.0 0% 0
{1} sda5 0.0 0.0 0.0 0% 0.0 0.0 0% 0

33. Synchronize!
# sync &
...
# vmstat 1
procs ­­­­­­­­­­­memory­­­­­­­­­­ ­­­swap­­ ­­­­io­­­ ­­system­ ­­­­­cpu­­­­­
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 63928704 6458288 1097956 0 0 0 0 480 154 0 0 99 0 0
1 0 0 63928076 6458288 1097960 0 0 0 106628 1765 307 0 1 99 0 0
0 0 0 63927488 6458288 1097964 0 0 0 942027 4311 273 0 1 93 6 0
0 0 0 63927480 6458288 1097964 0 0 0 34 819 231 0 1 99 0 0
# pt­diskstats
#ts device wr_s wr_avkb wr_mb_s wr_mrg wr_cnc wr_rt busy in_prg
{1} sda5 0.0 0.0 0.0 0% 0.0 0.0 0% 0
{1} sda5 200.1 462.5 90.4 0% 5.7 28.7 7% 0
{1} sda5 1656.0 510.5 825.6 0% 114.2 69.0 85% 0
{1} sda5 166.2 484.9 78.7 2% 16.8 100.9 12% 0
{1} sda5 0.0 0.0 0.0 0% 0.0 0.0 0% 0

34. CACHED
Reads are
other kernel caches and structures
OS page cache

35. Caching in action
# vmstat 1
procs ­­­­­­­­­­­memory­­­­­­­­­­ ­­­swap­­ ­­­­­io­­­­ ­­system­­ ­­­­­cpu­­­­­
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 64979240 6457316 48724 0 0 0 0 726 177 0 0 100 0 0
1 0 0 64736104 6578076 168964 0 0 121456 0 1769 5468 0 2 98 0 0
1 0 0 63733356 7065556 656456 0 0 487424 0 6133 21445 0 7 93 1 0
1 0 0 62660484 7586900 1177756 0 0 521344 12 6051 22913 0 8 92 0 0
1 0 0 61592424 8105460 1696316 0 0 518656 0 6082 23024 0 7 93 0 0
1 0 0 60440380 8665556 2256852 0 0 559932 44 6266 26413 0 7 92 1 0
1 0 0 59119376 9306964 2898172 0 0 641348 0 7015 31229 0 6 93 0 0
1 0 0 58010208 9845716 3436984 0 0 538752 0 6158 25078 0 7 93 0 0
1 0 0 57018712 10327256 3918336 0 0 481536 24 6004 21398 0 7 93 0 0
1 0 0 55895652 10872924 4464236 0 0 545664 0 6280 25376 0 6 93 0 0
3 0 0 54591312 11506264 5097260 0 0 633344 8 6985 31139 0 8 90 1 0
2 0 0 53263628 12151256 5742488 0 0 644480 0 6971 31668 0 6 93 1 0
1 0 0 51964508 12782296 6373724 0 0 631424 0 7030 31187 0 8 91 1 0
1 0 0 51054968 13224664 6815484 0 0 442212 0 7619 27035 0 5 93 2 0
0 1 0 50891056 13304412 6895208 0 0 79644 12 8448 18104 0 2 92 6 0
0 1 0 49882144 13794140 7385144 0 0 489984 229598 9420 31855 0 6 92 2 0
1 0 0 48985468 14228012 7819192 0 0 433920 434756 10934 30660 0 5 93 1 0
1 1 0 48024344 14694492 8285256 0 0 466432 471500 10914 33897 0 12 84 3 0
1 0 0 47064056 15160924 8751692 0 0 466432 467400 10979 33871 0 7 90 2 0
... many samples skipped ...
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 0 1940860 37169660 30761252 0 0 455680 459200 14158 27724 0 6 91 2 0
0 1 0 920864 37665532 31257060 0 0 495744 500200 14708 28307 0 6 92 2 0
2 0 0 3304476 35016696 31506636 0 0 248832 250136 10781 14505 0 9 89 1 0
1 0 0 9349772 28971956 31506812 0 0 0 0 3503 294 0 6 94 0 0
4 0 0 14827068 23494436 31506704 0 0 0 0 11120 181 0 14 86 0 0

36. load average: ?
How’s our load test?

37. Load average
load average: 0.5

38. Load average
load average: 1.0

39. Load average
load average: 1.8

40. Quiz
There’s a server:
- it has 2 6-core CPUs
- Hyperthreading is off
- 32GB of RAM
- 8 disks in RAID10
What load average would indicate that server is saturated?
a) 4.0
b) 8.0
c) 12.0
d) 16.0
e) 20.0

41. Answer
Either of the them:
a) 4.0
b) 8.0
c) 12.0
d) 16.0
e) 20.0
- 4 processes constantly writing
- 8 processes constantly reading
- 12 processes constantly computing
- 12 computing & 4 writing processes
- 12 computing & 8 reading processes

43. Latency

44. Random -vs- Sequential

45. Random -vs- Sequential
Random -vs- Sequential
Read this post from Chris Buckel (@flashdba) : http://flashdba.com/2013/04/15/understanding­io­random­vs­sequential/

46. Random -vs- Sequential
# pt­diskstats
wr_s wr_avkb wr_mb_s wr_mrg
13149.5 4.7 60.0 0%
13261.9 4.7 60.5 0%
13324.2 4.7 60.8 0%
# pt­diskstats
wr_s wr_avkb wr_mb_s wr_mrg
1031.0 456.0 459.1 0%
1027.4 456.0 457.5 0%
1027.4 456.0 457.5 0%
# pt­diskstats
wr_s wr_avkb wr_mb_s wr_mrg
212.0 20.7 4.3 40%
54.1 55.9 3.0 79%
221.2 19.2 4.2 37%
39.3 25.4 1.0 82%

47. Random IO
# pt­diskstats
wr_s wr_avkb wr_mb_s wr_mrg
13149.5 4.7 60.0 0%
13261.9 4.7 60.5 0%
13324.2 4.7 60.8 0%
# pt­diskstats
wr_s wr_avkb wr_mb_s wr_mrg
1031.0 456.0 459.1 0%
1027.4 456.0 457.5 0%
1027.4 456.0 457.5 0%
# pt­diskstats
wr_s wr_avkb wr_mb_s wr_mrg
212.0 20.7 4.3 40%
54.1 55.9 3.0 79%
221.2 19.2 4.2 37%
39.3 25.4 1.0 82%

48. Sequential IO (BIG random reads)
# pt­diskstats
wr_s wr_avkb wr_mb_s wr_mrg
13149.5 4.7 60.0 0%
13261.9 4.7 60.5 0%
13324.2 4.7 60.8 0%
# pt­diskstats
wr_s wr_avkb wr_mb_s wr_mrg
1031.0 456.0 459.1 0%
1027.4 456.0 457.5 0%
1027.4 456.0 457.5 0%
# pt­diskstats
wr_s wr_avkb wr_mb_s wr_mrg
212.0 20.7 4.3 40%
54.1 55.9 3.0 79%
221.2 19.2 4.2 37%
39.3 25.4 1.0 82%

49. Mixed
# pt­diskstats
wr_s wr_avkb wr_mb_s wr_mrg
13149.5 4.7 60.0 0%
13261.9 4.7 60.5 0%
13324.2 4.7 60.8 0%
# pt­diskstats
wr_s wr_avkb wr_mb_s wr_mrg
1031.0 456.0 459.1 0%
1027.4 456.0 457.5 0%
1027.4 456.0 457.5 0%
# pt­diskstats
wr_s wr_avkb wr_mb_s wr_mrg
212.0 20.7 4.3 40%
54.1 55.9 3.0 79%
221.2 19.2 4.2 37%
39.3 25.4 1.0 82%

50. IOPSIOPShow manyhow many can your hardware deliver?can your hardware deliver?

51. IOPS
The formula to calculate IOPS is:
IOPS = 1
[average latency in sec + average read/write seek time in dec ]

52. RAID Penalty
RAID configuration Penalty
RAID 0 1
RAID 1 2
RAID 10 2
RAID 5 4
RAID 6 6

53. Rotation Latency
HDD Spindle [rpm] Average rotational latency [ms]
4,200 7.14
5,400 5.56
7,200 4.17
10,000 3.00
15,000 2.00

54. Workload system IOPS formula
The formula to calculate workload system
IOPS (with the RAID) is:
1(Total Workload IOPS * Percentage of workload read
operations)
+
( (Total Workload IOPS * Percentage of workload
write operations) / RAID IOPS Penalty )

55. Example
Disk : SEAGATE ST3300655LC (279GB)
RAID : 5
# of disks: 4
Spin Speed (RPM): 15k
Average latency : 2.0ms
Random read seek time : 3.50ms
Random write seek time : 4.0ms

56. Example
IOPS = 1 / [ 0.002 + (0.0035+0.004)/2 ]
= 173.91
Total Workload IOPS = 4 x 173.91 = 695 (695.64)
Let's assume 80% reads, 20% writes:
Workload system IOPS = (695 x 0.8)
+ ( (695 x 0.2)/4 )
= 590.75 IOPS
RAID : 5 ­> penalty = 4
# of disks: 4
Spin Speed (RPM): 15k
Average latency : 2.0ms
Random read seek time : 3.50ms
Random write seek time : 4.0ms

57. Mirroring
Striping across 4 stripes
Mirroring Mirroring Mirroring
BufferingBuffering
8 disk RAID 10

58. %util explained
# pt­diskstats
rd_s rd_avkb rd_mb_s rd_mrg rd_cnc rd_rt busy in_prg
238.4 16.0 3.7 0% 1.0 4.2 100% 1
231.5 16.0 3.6 0% 1.0 4.3 99% 1
246.0 16.0 3.8 0% 1.0 4.1 99% 1

59. %util explained
# pt­diskstats
rd_s rd_avkb rd_mb_s rd_mrg rd_cnc rd_rt busy in_prg
238.4 16.0 3.7 0% 1.0 4.2 100% 1
231.5 16.0 3.6 0% 1.0 4.3 99% 1
246.0 16.0 3.8 0% 1.0 4.1 99% 1
479.6 16.0 7.5 0% 2.0 4.1 100% 2
472.0 16.0 7.4 0% 2.0 4.2 100% 2
456.2 16.0 7.1 0% 2.0 4.4 100% 2

60. %util explained
# pt­diskstats
rd_s rd_avkb rd_mb_s rd_mrg rd_cnc rd_rt busy in_prg
238.4 16.0 3.7 0% 1.0 4.2 100% 1
231.5 16.0 3.6 0% 1.0 4.3 99% 1
246.0 16.0 3.8 0% 1.0 4.1 99% 1
479.6 16.0 7.5 0% 2.0 4.1 100% 2
472.0 16.0 7.4 0% 2.0 4.2 100% 2
456.2 16.0 7.1 0% 2.0 4.4 100% 2
840.7 16.0 13.1 0% 4.0 4.7 100% 4
857.9 16.0 13.4 0% 4.0 4.6 100% 4
837.9 16.0 13.1 0% 4.0 4.8 100% 4
1375.4 16.0 21.5 0% 8.0 5.8 100% 8
1386.2 16.0 21.7 0% 8.0 5.8 100% 8
1435.3 16.0 22.4 0% 8.0 5.6 100% 8
2147.4 16.0 33.6 0% 16.0 7.4 100% 16
2177.7 16.0 34.0 0% 16.0 7.3 100% 16
2105.1 16.0 32.9 0% 16.0 7.6 100% 16
4057.4 16.0 63.4 0% 63.8 15.7 100% 64
4038.3 16.0 63.1 0% 63.7 15.8 100% 64
4020.8 16.0 62.8 0% 64.3 16.0 100% 64

61. iostat pitfalls
# iostat ­kdx 1
rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq­sz avgqu­sz await svctm %util
0.00 0.00 22.00 2974.00 352.00 47584.00 32.00 46.30 15.53 0.33 99.90
0.00 0.00 16.00 2868.00 256.00 45888.00 32.00 47.53 16.52 0.35 100.10
0.00 0.00 22.00 3025.00 352.00 48400.00 32.00 45.84 15.05 0.33 100.00

62. iostat pitfalls
# iostat ­kdx 1
rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq­sz avgqu­sz await svctm %util
0.00 0.00 22.00 2974.00 352.00 47584.00 32.00 46.30 15.53 0.33 99.90
0.00 0.00 16.00 2868.00 256.00 45888.00 32.00 47.53 16.52 0.35 100.10
0.00 0.00 22.00 3025.00 352.00 48400.00 32.00 45.84 15.05 0.33 100.00

63. alternative - pt-diskstats
# iostat ­kdx 1
rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq­sz avgqu­sz await svctm %util
0.00 0.00 22.00 2974.00 352.00 47584.00 32.00 46.30 15.53 0.33 99.90
0.00 0.00 16.00 2868.00 256.00 45888.00 32.00 47.53 16.52 0.35 100.10
0.00 0.00 22.00 3025.00 352.00 48400.00 32.00 45.84 15.05 0.33 100.00
# pt­diskstats
rd_s rd_avkb rd_mb_s rd_mrg rd_cnc rd_rt wr_s wr_avkb wr_mb_s wr_mrg wr_cnc wr_rt busy
13.5 16.0 0.2 0% 0.9 68.7 2921.3 16.0 45.6 0% 47.1 16.1 100%
15.5 16.0 0.2 0% 1.0 66.4 2939.5 16.0 45.9 0% 44.8 15.2 100%
26.1 16.0 0.4 0% 1.0 39.3 2986.5 16.0 46.7 0% 46.2 15.5 100%

64. alternative - pt-diskstats
# iostat ­kdx 1
rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq­sz avgqu­sz await svctm %util
0.00 0.00 22.00 2974.00 352.00 47584.00 32.00 46.30 15.53 0.33 99.90
0.00 0.00 16.00 2868.00 256.00 45888.00 32.00 47.53 16.52 0.35 100.10
0.00 0.00 22.00 3025.00 352.00 48400.00 32.00 45.84 15.05 0.33 100.00
# pt­diskstats
rd_s rd_avkb rd_mb_s rd_mrg rd_cnc rd_rt wr_s wr_avkb wr_mb_s wr_mrg wr_cnc wr_rt busy
13.5 16.0 0.2 0% 0.9 68.7 2921.3 16.0 45.6 0% 47.1 16.1 100%
15.5 16.0 0.2 0% 1.0 66.4 2939.5 16.0 45.9 0% 44.8 15.2 100%
26.1 16.0 0.4 0% 1.0 39.3 2986.5 16.0 46.7 0% 46.2 15.5 100%

65. pt-diskstats
wget percona.com/get/pt-diskstats
or visit: http://www.percona.com/software/percona-toolkit

66. FLASHFLASH
a word ona word on

67. Reads on Samsung 830 256GB

68. Flash

70. mpstat
[root@x1 percona]# mpstat ­P ALL 1
Linux 2.6.32­358.11.1.el6.x86_64 (x1) 08/22/2013 _x86_64_ (32 CPU)
05:04:24 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
05:04:25 PM all 13.60 0.00 2.86 0.09 0.00 0.22 0.00 0.00 83.23
05:04:25 PM 0 56.12 0.00 11.22 1.02 0.00 6.12 0.00 0.00 25.51
05:04:25 PM 1 13.13 0.00 4.04 0.00 0.00 0.00 0.00 0.00 82.83
05:04:25 PM 2 39.39 0.00 7.07 1.01 0.00 0.00 0.00 0.00 52.53
05:04:25 PM 3 15.15 0.00 3.03 0.00 0.00 0.00 0.00 0.00 81.82
05:04:25 PM 4 36.73 0.00 6.12 0.00 0.00 0.00 0.00 0.00 57.14
05:04:25 PM 5 8.08 0.00 1.01 0.00 0.00 0.00 0.00 0.00 90.91
05:04:25 PM 6 32.99 0.00 6.19 0.00 0.00 0.00 0.00 0.00 60.82
05:04:25 PM 7 3.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 96.00
05:04:25 PM 8 26.00 0.00 6.00 0.00 0.00 0.00 0.00 0.00 68.00
05:04:25 PM 9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
05:04:25 PM 10 21.00 0.00 5.00 1.00 0.00 0.00 0.00 0.00 73.00
05:04:25 PM 11 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
05:04:25 PM 12 15.15 0.00 4.04 0.00 0.00 0.00 0.00 0.00 80.81
05:04:25 PM 13 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
05:04:25 PM 14 9.00 0.00 3.00 0.00 0.00 0.00 0.00 0.00 88.00
05:04:25 PM 15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
05:04:25 PM 16 39.80 0.00 11.22 1.02 0.00 0.00 0.00 0.00 47.96
05:04:25 PM 17 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 96.00
05:04:25 PM 18 12.00 0.00 3.00 0.00 0.00 0.00 0.00 0.00 85.00
05:04:25 PM 19 1.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.99
05:04:25 PM 20 14.14 0.00 3.03 0.00 0.00 0.00 0.00 0.00 82.83
05:04:25 PM 21 3.96 0.00 0.99 0.00 0.00 0.00 0.00 0.00 95.05
05:04:25 PM 22 2.04 0.00 0.00 0.00 0.00 0.00 0.00 0.00 97.96
05:04:25 PM 23 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
05:04:25 PM 24 5.00 0.00 2.00 0.00 0.00 0.00 0.00 0.00 93.00
...

71. UNDERSTANDUNDERSTANDhow your application uses CPUhow your application uses CPU

72. vmstat
procs ­­­­­­­­­­­memory­­­­­­­­­­­­ ­­­swap­­ ­­­­­io­­­­ ­­­­sys tem­­ ­­­­cpu­­­­
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 108340 320216 818012 18975152 0 0 48 19260 25853 86397 14 13 72 0
6 0 108340 319620 818016 18976468 0 0 0 9868 23850 83340 14 12 74 0
2 0 108340 317752 818016 18977748 0 0 0 7760 22928 75023 16 13 71 0
2 0 108340 316900 818016 18978948 0 0 0 4256 22449 74763 16 11 72 0
6 0 108340 316024 818020 18980208 0 0 0 7372 24713 85416 17 14 69 0
1 0 108340 314164 818020 18981584 0 0 0 9284 24591 84896 17 14 68 0
0 0 108340 312676 818020 18982940 0 0 0 16156 23294 77961 14 12 74 0
3 0 108340 311940 818020 18984188 0 0 16 4808 22080 74513 18 13 69 0
5 0 108340 310576 818020 18985376 0 0 0 8344 23050 76195 17 10 72 0
1 0 108340 308760 818020 18986728 0 0 0 4872 24791 84684 19 15 66 0
3 0 108340 308220 818020 18988016 0 0 16 14924 14481 36277 18 6 76 0
1 0 108340 306972 818020 18989776 0 0 0 11988 7604 8062 15 4 81 0
1 0 108340 305260 818020 18990436 0 0 0 7996 7660 8060 16 2 81 0
2 0 108340 304020 818020 18992128 0 0 0 8736 8014 8788 13 3 84 0
3 0 108340 302880 818020 18993580 0 0 0 9964 8098 8616 12 4 84 0
2 0 108340 301888 818020 18994844 0 0 16 9532 7751 8245 15 3 82 0
1 0 108340 300176 818020 18996104 0 0 0 5428 7798 8679 16 3 81 0
2 0 108340 299308 818020 18997448 0 0 16 8588 7248 7901 13 3 84 0
11 0 108340 307468 818564 18966788 0 0 540 14804 9712 58947 18 24 56 2
0 0 108336 326348 818564 18967956 0 0 0 9416 24360 129017 17 26 57 0
1 0 108336 326008 818564 18969240 0 0 0 10208 24056 83679 18 14 68 0
3 0 108336 323948 818564 18970512 0 0 0 8824 23329 80613 17 14 69 0
0 0 108336 323064 818564 18971736 0 0 0 8084 22668 77074 13 12 75 0
1 0 108336 321708 818564 18973468 0 0 16 7700 23822 84581 15 12 72 0
5 0 108336 320104 818568 18974776 0 0 16 17252 24826 85009 14 14 72 0
0 0 108336 319236 818568 18975740 0 0 0 8656 23709 81355 18 13 69 0

73. vmstat
procs ­­­­­­­­­­­memory­­­­­­­­­­­­ ­­­swap­­ ­­­­­io­­­­ ­­­­sys tem­­ ­­­­cpu­­­­
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 108340 320216 818012 18975152 0 0 48 19260 25853 86397 14 13 72 0
6 0 108340 319620 818016 18976468 0 0 0 9868 23850 83340 14 12 74 0
2 0 108340 317752 818016 18977748 0 0 0 7760 22928 75023 16 13 71 0
2 0 108340 316900 818016 18978948 0 0 0 4256 22449 74763 16 11 72 0
6 0 108340 316024 818020 18980208 0 0 0 7372 24713 85416 17 14 69 0
1 0 108340 314164 818020 18981584 0 0 0 9284 24591 84896 17 14 68 0
0 0 108340 312676 818020 18982940 0 0 0 16156 23294 77961 14 12 74 0
3 0 108340 311940 818020 18984188 0 0 16 4808 22080 74513 18 13 69 0
5 0 108340 310576 818020 18985376 0 0 0 8344 23050 76195 17 10 72 0
1 0 108340 308760 818020 18986728 0 0 0 4872 24791 84684 19 15 66 0
3 0 108340 308220 818020 18988016 0 0 16 14924 14481 36277 18 6 76 0
1 0 108340 306972 818020 18989776 0 0 0 11988 7604 8062 15 4 81 0
1 0 108340 305260 818020 18990436 0 0 0 7996 7660 8060 16 2 81 0
2 0 108340 304020 818020 18992128 0 0 0 8736 8014 8788 13 3 84 0
3 0 108340 302880 818020 18993580 0 0 0 9964 8098 8616 12 4 84 0
2 0 108340 301888 818020 18994844 0 0 16 9532 7751 8245 15 3 82 0
1 0 108340 300176 818020 18996104 0 0 0 5428 7798 8679 16 3 81 0
2 0 108340 299308 818020 18997448 0 0 16 8588 7248 7901 13 3 84 0
11 0 108340 307468 818564 18966788 0 0 540 14804 9712 58947 18 24 56 2
0 0 108336 326348 818564 18967956 0 0 0 9416 24360 129017 17 26 57 0
1 0 108336 326008 818564 18969240 0 0 0 10208 24056 83679 18 14 68 0
3 0 108336 323948 818564 18970512 0 0 0 8824 23329 80613 17 14 69 0
0 0 108336 323064 818564 18971736 0 0 0 8084 22668 77074 13 12 75 0
1 0 108336 321708 818564 18973468 0 0 16 7700 23822 84581 15 12 72 0
5 0 108336 320104 818568 18974776 0 0 16 17252 24826 85009 14 14 72 0
0 0 108336 319236 818568 18975740 0 0 0 8656 23709 81355 18 13 69 0

74. Learn vmstat
procs ­­­­­­­­­­­memory­­­­­­­­­­ ­­­swap­­ ­­­­­io­­­­ ­­system­­ ­­­­­cpu­­­­­­
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 7 140 157964 59124 394676 0 0 12312 5180 4006 9755 32 5 35 28 0
6 4 140 159260 57188 394416 0 0 11352 16 3324 9965 36 4 40 20 0
3 7 140 161832 54280 395472 0 0 9292 3648 3522 10218 67 7 12 14 0
24 4 140 160468 54400 396220 0 0 10356 2608 3547 12196 62 5 20 13 0
14 4 140 157368 54604 399152 0 0 11192 2048 2941 11666 58 7 21 14 0
7 3 140 156360 54656 400092 0 0 7608 2068 4555 10273 43 5 37 15 0
6 5 140 158828 54680 399180 0 0 8340 3832 3914 9622 54 5 29 13 0
3 4 140 157996 55084 398408 0 0 8856 8688 3327 9929 43 5 35 16 0
1 5 140 173740 55168 383104 0 0 7488 1992 2338 10352 46 6 34 14 0
1 3 140 170748 55464 384260 0 0 6840 2560 2545 12245 57 6 25 13 0
2 8 140 168516 55692 387740 0 0 4952 3112 2085 11066 42 5 41 12 0
2 5 140 167236 55840 388140 0 0 6352 716 2134 11347 42 5 35 18 0
4 6 140 165056 55984 389764 0 0 5896 2036 2043 8925 35 3 38 24 0
7 6 140 164024 56032 390944 0 0 5428 1672 1944 9485 43 4 34 20 0
6 4 140 161500 56100 393116 0 0 4496 1588 2038 13300 54 5 26 15 0
3 6 140 158312 56184 395904 0 0 4824 2204 1989 12303 59 6 15 20 0
4 6 140 158068 56300 396412 0 0 5632 4208 2238 10127 50 4 24 22 0
5 8 140 155828 56460 398960 0 0 4104 2152 1815 7972 28 2 43 26 0
4 4 140 158680 55092 395920 0 0 7616 1764 2010 8047 27 2 54 17 0
10 5 140 157588 55156 396780 0 0 3076 2208 1906 10372 36 3 31 29 0
5 4 140 158332 55200 396576 0 0 5008 1868 1936 9832 46 3 35 16 0

75. vmstat - procs
procs ­­­­­­­­­­­memory­­­­­­­­­­ ­­­swap­­ ­­­­­io­­­­ ­­system­­ ­­­­­cpu­­­­­­
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 7 140 157964 59124 394676 0 0 12312 5180 4006 9755 32 5 35 28 0
6 4 140 159260 57188 394416 0 0 11352 16 3324 9965 36 4 40 20 0
3 7 140 161832 54280 395472 0 0 9292 3648 3522 10218 67 7 12 14 0
24 4 140 160468 54400 396220 0 0 10356 2608 3547 12196 62 5 20 13 0
14 4 140 157368 54604 399152 0 0 11192 2048 2941 11666 58 7 21 14 0
7 3 140 156360 54656 400092 0 0 7608 2068 4555 10273 43 5 37 15 0
6 5 140 158828 54680 399180 0 0 8340 3832 3914 9622 54 5 29 13 0
3 4 140 157996 55084 398408 0 0 8856 8688 3327 9929 43 5 35 16 0
1 5 140 173740 55168 383104 0 0 7488 1992 2338 10352 46 6 34 14 0
1 3 140 170748 55464 384260 0 0 6840 2560 2545 12245 57 6 25 13 0
2 8 140 168516 55692 387740 0 0 4952 3112 2085 11066 42 5 41 12 0
2 5 140 167236 55840 388140 0 0 6352 716 2134 11347 42 5 35 18 0
4 6 140 165056 55984 389764 0 0 5896 2036 2043 8925 35 3 38 24 0
7 6 140 164024 56032 390944 0 0 5428 1672 1944 9485 43 4 34 20 0
6 4 140 161500 56100 393116 0 0 4496 1588 2038 13300 54 5 26 15 0
3 6 140 158312 56184 395904 0 0 4824 2204 1989 12303 59 6 15 20 0
4 6 140 158068 56300 396412 0 0 5632 4208 2238 10127 50 4 24 22 0
5 8 140 155828 56460 398960 0 0 4104 2152 1815 7972 28 2 43 26 0
4 4 140 158680 55092 395920 0 0 7616 1764 2010 8047 27 2 54 17 0
10 5 140 157588 55156 396780 0 0 3076 2208 1906 10372 36 3 31 29 0
5 4 140 158332 55200 396576 0 0 5008 1868 1936 9832 46 3 35 16 0

76. vmstat - memory
procs ­­­­­­­­­­­memory­­­­­­­­­­ ­­­swap­­ ­­­­­io­­­­ ­­system­­ ­­­­­cpu­­­­­­
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 7 140 157964 59124 394676 0 0 12312 5180 4006 9755 32 5 35 28 0
6 4 140 159260 57188 394416 0 0 11352 16 3324 9965 36 4 40 20 0
3 7 140 161832 54280 395472 0 0 9292 3648 3522 10218 67 7 12 14 0
24 4 140 160468 54400 396220 0 0 10356 2608 3547 12196 62 5 20 13 0
14 4 140 157368 54604 399152 0 0 11192 2048 2941 11666 58 7 21 14 0
7 3 140 156360 54656 400092 0 0 7608 2068 4555 10273 43 5 37 15 0
6 5 140 158828 54680 399180 0 0 8340 3832 3914 9622 54 5 29 13 0
3 4 140 157996 55084 398408 0 0 8856 8688 3327 9929 43 5 35 16 0
1 5 140 173740 55168 383104 0 0 7488 1992 2338 10352 46 6 34 14 0
1 3 140 170748 55464 384260 0 0 6840 2560 2545 12245 57 6 25 13 0
2 8 140 168516 55692 387740 0 0 4952 3112 2085 11066 42 5 41 12 0
2 5 140 167236 55840 388140 0 0 6352 716 2134 11347 42 5 35 18 0
4 6 140 165056 55984 389764 0 0 5896 2036 2043 8925 35 3 38 24 0
7 6 140 164024 56032 390944 0 0 5428 1672 1944 9485 43 4 34 20 0
6 4 140 161500 56100 393116 0 0 4496 1588 2038 13300 54 5 26 15 0
3 6 140 158312 56184 395904 0 0 4824 2204 1989 12303 59 6 15 20 0
4 6 140 158068 56300 396412 0 0 5632 4208 2238 10127 50 4 24 22 0
5 8 140 155828 56460 398960 0 0 4104 2152 1815 7972 28 2 43 26 0
4 4 140 158680 55092 395920 0 0 7616 1764 2010 8047 27 2 54 17 0
10 5 140 157588 55156 396780 0 0 3076 2208 1906 10372 36 3 31 29 0
5 4 140 158332 55200 396576 0 0 5008 1868 1936 9832 46 3 35 16 0

77. vmstat - swap
procs ­­­­­­­­­­­memory­­­­­­­­­­ ­­­swap­­ ­­­­­io­­­­ ­­system­­ ­­­­­cpu­­­­­­
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 7 140 157964 59124 394676 0 0 12312 5180 4006 9755 32 5 35 28 0
6 4 140 159260 57188 394416 0 0 11352 16 3324 9965 36 4 40 20 0
3 7 140 161832 54280 395472 0 0 9292 3648 3522 10218 67 7 12 14 0
24 4 140 160468 54400 396220 0 0 10356 2608 3547 12196 62 5 20 13 0
14 4 140 157368 54604 399152 0 0 11192 2048 2941 11666 58 7 21 14 0
7 3 140 156360 54656 400092 0 0 7608 2068 4555 10273 43 5 37 15 0
6 5 140 158828 54680 399180 0 0 8340 3832 3914 9622 54 5 29 13 0
3 4 140 157996 55084 398408 0 0 8856 8688 3327 9929 43 5 35 16 0
1 5 140 173740 55168 383104 0 0 7488 1992 2338 10352 46 6 34 14 0
1 3 140 170748 55464 384260 0 0 6840 2560 2545 12245 57 6 25 13 0
2 8 140 168516 55692 387740 0 0 4952 3112 2085 11066 42 5 41 12 0
2 5 140 167236 55840 388140 0 0 6352 716 2134 11347 42 5 35 18 0
4 6 140 165056 55984 389764 0 0 5896 2036 2043 8925 35 3 38 24 0
7 6 140 164024 56032 390944 0 0 5428 1672 1944 9485 43 4 34 20 0
6 4 140 161500 56100 393116 0 0 4496 1588 2038 13300 54 5 26 15 0
3 6 140 158312 56184 395904 0 0 4824 2204 1989 12303 59 6 15 20 0
4 6 140 158068 56300 396412 0 0 5632 4208 2238 10127 50 4 24 22 0
5 8 140 155828 56460 398960 0 0 4104 2152 1815 7972 28 2 43 26 0
4 4 140 158680 55092 395920 0 0 7616 1764 2010 8047 27 2 54 17 0
10 5 140 157588 55156 396780 0 0 3076 2208 1906 10372 36 3 31 29 0
5 4 140 158332 55200 396576 0 0 5008 1868 1936 9832 46 3 35 16 0

78. vmstat - I/O
procs ­­­­­­­­­­­memory­­­­­­­­­­ ­­­swap­­ ­­­­­io­­­­ ­­system­­ ­­­­­cpu­­­­­­
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 7 140 157964 59124 394676 0 0 12312 5180 4006 9755 32 5 35 28 0
6 4 140 159260 57188 394416 0 0 11352 16 3324 9965 36 4 40 20 0
3 7 140 161832 54280 395472 0 0 9292 3648 3522 10218 67 7 12 14 0
24 4 140 160468 54400 396220 0 0 10356 2608 3547 12196 62 5 20 13 0
14 4 140 157368 54604 399152 0 0 11192 2048 2941 11666 58 7 21 14 0
7 3 140 156360 54656 400092 0 0 7608 2068 4555 10273 43 5 37 15 0
6 5 140 158828 54680 399180 0 0 8340 3832 3914 9622 54 5 29 13 0
3 4 140 157996 55084 398408 0 0 8856 8688 3327 9929 43 5 35 16 0
1 5 140 173740 55168 383104 0 0 7488 1992 2338 10352 46 6 34 14 0
1 3 140 170748 55464 384260 0 0 6840 2560 2545 12245 57 6 25 13 0
2 8 140 168516 55692 387740 0 0 4952 3112 2085 11066 42 5 41 12 0
2 5 140 167236 55840 388140 0 0 6352 716 2134 11347 42 5 35 18 0
4 6 140 165056 55984 389764 0 0 5896 2036 2043 8925 35 3 38 24 0
7 6 140 164024 56032 390944 0 0 5428 1672 1944 9485 43 4 34 20 0
6 4 140 161500 56100 393116 0 0 4496 1588 2038 13300 54 5 26 15 0
3 6 140 158312 56184 395904 0 0 4824 2204 1989 12303 59 6 15 20 0
4 6 140 158068 56300 396412 0 0 5632 4208 2238 10127 50 4 24 22 0
5 8 140 155828 56460 398960 0 0 4104 2152 1815 7972 28 2 43 26 0
4 4 140 158680 55092 395920 0 0 7616 1764 2010 8047 27 2 54 17 0
10 5 140 157588 55156 396780 0 0 3076 2208 1906 10372 36 3 31 29 0
5 4 140 158332 55200 396576 0 0 5008 1868 1936 9832 46 3 35 16 0

79. vmstat - system
procs ­­­­­­­­­­­memory­­­­­­­­­­ ­­­swap­­ ­­­­­io­­­­ ­­system­­ ­­­­­cpu­­­­­­
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 7 140 157964 59124 394676 0 0 12312 5180 4006 9755 32 5 35 28 0
6 4 140 159260 57188 394416 0 0 11352 16 3324 9965 36 4 40 20 0
3 7 140 161832 54280 395472 0 0 9292 3648 3522 10218 67 7 12 14 0
24 4 140 160468 54400 396220 0 0 10356 2608 3547 12196 62 5 20 13 0
14 4 140 157368 54604 399152 0 0 11192 2048 2941 11666 58 7 21 14 0
7 3 140 156360 54656 400092 0 0 7608 2068 4555 10273 43 5 37 15 0
6 5 140 158828 54680 399180 0 0 8340 3832 3914 9622 54 5 29 13 0
3 4 140 157996 55084 398408 0 0 8856 8688 3327 9929 43 5 35 16 0
1 5 140 173740 55168 383104 0 0 7488 1992 2338 10352 46 6 34 14 0
1 3 140 170748 55464 384260 0 0 6840 2560 2545 12245 57 6 25 13 0
2 8 140 168516 55692 387740 0 0 4952 3112 2085 11066 42 5 41 12 0
2 5 140 167236 55840 388140 0 0 6352 716 2134 11347 42 5 35 18 0
4 6 140 165056 55984 389764 0 0 5896 2036 2043 8925 35 3 38 24 0
7 6 140 164024 56032 390944 0 0 5428 1672 1944 9485 43 4 34 20 0
6 4 140 161500 56100 393116 0 0 4496 1588 2038 13300 54 5 26 15 0
3 6 140 158312 56184 395904 0 0 4824 2204 1989 12303 59 6 15 20 0
4 6 140 158068 56300 396412 0 0 5632 4208 2238 10127 50 4 24 22 0
5 8 140 155828 56460 398960 0 0 4104 2152 1815 7972 28 2 43 26 0
4 4 140 158680 55092 395920 0 0 7616 1764 2010 8047 27 2 54 17 0
10 5 140 157588 55156 396780 0 0 3076 2208 1906 10372 36 3 31 29 0
5 4 140 158332 55200 396576 0 0 5008 1868 1936 9832 46 3 35 16 0

80. vmstat - CPU
procs ­­­­­­­­­­­memory­­­­­­­­­­ ­­­swap­­ ­­­­­io­­­­ ­­system­­ ­­­­­cpu­­­­­­
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 7 140 157964 59124 394676 0 0 12312 5180 4006 9755 32 5 35 28 0
6 4 140 159260 57188 394416 0 0 11352 16 3324 9965 36 4 40 20 0
3 7 140 161832 54280 395472 0 0 9292 3648 3522 10218 67 7 12 14 0
24 4 140 160468 54400 396220 0 0 10356 2608 3547 12196 62 5 20 13 0
14 4 140 157368 54604 399152 0 0 11192 2048 2941 11666 58 7 21 14 0
7 3 140 156360 54656 400092 0 0 7608 2068 4555 10273 43 5 37 15 0
6 5 140 158828 54680 399180 0 0 8340 3832 3914 9622 54 5 29 13 0
3 4 140 157996 55084 398408 0 0 8856 8688 3327 9929 43 5 35 16 0
1 5 140 173740 55168 383104 0 0 7488 1992 2338 10352 46 6 34 14 0
1 3 140 170748 55464 384260 0 0 6840 2560 2545 12245 57 6 25 13 0
2 8 140 168516 55692 387740 0 0 4952 3112 2085 11066 42 5 41 12 0
2 5 140 167236 55840 388140 0 0 6352 716 2134 11347 42 5 35 18 0
4 6 140 165056 55984 389764 0 0 5896 2036 2043 8925 35 3 38 24 0
7 6 140 164024 56032 390944 0 0 5428 1672 1944 9485 43 4 34 20 0
6 4 140 161500 56100 393116 0 0 4496 1588 2038 13300 54 5 26 15 0
3 6 140 158312 56184 395904 0 0 4824 2204 1989 12303 59 6 15 20 0
4 6 140 158068 56300 396412 0 0 5632 4208 2238 10127 50 4 24 22 0
5 8 140 155828 56460 398960 0 0 4104 2152 1815 7972 28 2 43 26 0
4 4 140 158680 55092 395920 0 0 7616 1764 2010 8047 27 2 54 17 0
10 5 140 157588 55156 396780 0 0 3076 2208 1906 10372 36 3 31 29 0
5 4 140 158332 55200 396576 0 0 5008 1868 1936 9832 46 3 35 16 0

81. All-in-1
procs ­­­­­­­­­­­memory­­­­­­­­­­ ­­­swap­­ ­­­­­io­­­­ ­­system­­ ­­­­­cpu­­­­­­
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 7 140 157964 59124 394676 0 0 12312 5180 4006 9755 32 5 35 28 0
6 4 140 159260 57188 394416 0 0 11352 16 3324 9965 36 4 40 20 0
3 7 140 161832 54280 395472 0 0 9292 3648 3522 10218 67 7 12 14 0
24 4 140 160468 54400 396220 0 0 10356 2608 3547 12196 62 5 20 13 0
14 4 140 157368 54604 399152 0 0 11192 2048 2941 11666 58 7 21 14 0
7 3 140 156360 54656 400092 0 0 7608 2068 4555 10273 43 5 37 15 0
6 5 140 158828 54680 399180 0 0 8340 3832 3914 9622 54 5 29 13 0
3 4 140 157996 55084 398408 0 0 8856 8688 3327 9929 43 5 35 16 0
1 5 140 173740 55168 383104 0 0 7488 1992 2338 10352 46 6 34 14 0
1 3 140 170748 55464 384260 0 0 6840 2560 2545 12245 57 6 25 13 0
2 8 140 168516 55692 387740 0 0 4952 3112 2085 11066 42 5 41 12 0
2 5 140 167236 55840 388140 0 0 6352 716 2134 11347 42 5 35 18 0
4 6 140 165056 55984 389764 0 0 5896 2036 2043 8925 35 3 38 24 0
7 6 140 164024 56032 390944 0 0 5428 1672 1944 9485 43 4 34 20 0
6 4 140 161500 56100 393116 0 0 4496 1588 2038 13300 54 5 26 15 0
3 6 140 158312 56184 395904 0 0 4824 2204 1989 12303 59 6 15 20 0
4 6 140 158068 56300 396412 0 0 5632 4208 2238 10127 50 4 24 22 0
5 8 140 155828 56460 398960 0 0 4104 2152 1815 7972 28 2 43 26 0
4 4 140 158680 55092 395920 0 0 7616 1764 2010 8047 27 2 54 17 0
10 5 140 157588 55156 396780 0 0 3076 2208 1906 10372 36 3 31 29 0
5 4 140 158332 55200 396576 0 0 5008 1868 1936 9832 46 3 35 16 0

82. Load indices
● On busy systems, the processes queue length is a better way
to evaluate the total load.
● For example, on two strongly charged host where the use of
the processor continuously reach a maximum of 100%, the
system with the higher amount of processes in standby can
be considered as more charged.

83. To reCAP
1. VMSTAT 1
2. MPSTAT -P ALL 1
3. PT-DISKSTATS

84. To reCAP
1. VMSTAT 1
2. MPSTAT -P ALL 1
3. PT-DISKSTATS

85. IT’S JUSTIT’S JUSTa system.a system.

86. Even more tools ?

87. ??

88. Aurimas & lered, percona.comAurimas & lered, percona.com