My presentation at OSCON 2013 on file system tuning and leveraging cooperative tools like inotify to build robust workflows.

1. Exploiting Your File System to Build Robust &
Efficient Workflows
Jason Johnson
jajohnson@softlayer.com
Exploiting Your File System to Build Robust &
Efficient Workflows
Jason Johnson
jajohnson@softlayer.com

3. What is /dev/sdc, anyway?

4. The Hard Disk Drive

5. Basic Platter Geometry
Cylinder-Head-Sector (obsolete)
Logical Block Addressing, LBA

6. What is /dev/sdc, anyway?

7. The Disk Array Controller
● Adaptec 5405Z
● PCIe x8
● 1.2 GHz Dual Core RAID on Chip (ROC)
● 128-1024 MB Battery-Backed DDR
● 1-4 GB NAND
● Up to 256 SATA or SAS HDD's
● arcconf

8. Write Caching

9. “...you *must* disable the
individual hard disk write cache in
order to ensure to keep the file
system intact after a power
failure.”
XFS.org FAQ

10. Initial 1MB Sector Alignment
Sector Size Starting Sector Drive Type
512 B 2048 SATA & SAS
2 KB 512 SSD
4 KB 256 Advanced Format & SSD
blockdev --getpbsz /dev/sdc
blockdev --getss /dev/sdc
“Aligning IO on a hard disk RAID”
http://www.mysqlperformanceblog.com/2011/06/09/aligning-io-on-a-hard-disk-raid-the-theory/

11. (s)gdisk

12. Tuning the File System
● Disable Caching
● Tools: sysbench, iozone, iostat, vmstat
● Start Simple
● Apply Increasing Parallel I/O
● ext2, ext3, ext4, xfs, btrfs, zfs?
● Graph Everything

13. arcconf
arcconf
create 1 logicaldrive
stripesize 256
wcache wt
rcache roff
max
0
0 3
0 4
...
0 18

14. sysbench, fileio
sysbench
--num-threads=[8-1024]
--test=fileio
--file-total-size=10G
--file-test-mode=rndwr
--file-fsync-all=on
--file-num=64
--file-block-size=16384
[prepare|run|cleanup]

15. EXT4, mkfs
mkfs.ext4 /dev/sdc1
mke2fs
-b 4096
-O journal_dev
/dev/sdb1 32768
mkfs.ext4
-b 4096
-E stride=4,stripe_width=16
-J device=/dev/sdb1
/dev/sdc1
mount -o
noatime,stripe=16
/dev/sdc1
/mnt/data

16. I/O Requests per Second, EXT4

17. Latency, EXT4

18. XFS, mkfs
mkfs.xfs /dev/sdc1
mkfs.xfs
-d sw=16,su=16k
-l
logdev=/dev/sdb1,
size=128m,
su=256k
/dev/sdc1
mount -o
noatime,
logdev=/dev/sdb1,
logbufs=8,logbsize=256k
/dev/sdc1
/mnt/data

19. I/O Requests per Second, XFS

20. Latency, XFS

21. What are we looking for?
I/O Request per Second... per Drive
&
Reasonable Latency

22. XFS vs. EXT4, Latency

23. XFS vs. EXT4, per Drive

24. Scenario 1, Efficiency
MySQL

25. MySQL Write Pattern

26. MySQL Configuration
System Variable Value
innodb_io_capacity 5000
innodb_thread_concurrency 256
innodb_write_io_threads 192
innodb_read_io_threads 64
innodb_log_file_size 32M
innodb_log_files_in_group 32
innodb_buffer_pool_size 10GB
innodb_buffer_pool_instances 10
“MySQL System Variables”

27. sysbench, mysql
sysbench
--num-threads=[32|64|128|256]
--test=oltp
--oltp-test-mode=nontrx
--oltp-nontrx-mode=insert
--oltp-table-size=100000
--max-requests=10000000
[prepare|run|cleanup]

28. Transactions per Second
+96.28%
+125.37%
+102.29%
+69.43%
15,962.79/s @ 16ms
9,421.29/s @ 27ms

30. inotify
Event Mask Fired when...
IN_ACCESS File was accessed (read)
IN_ATTRIB Metadata changed
IN_CLOSE_WRITE File opened for writing was closed
IN_CLOSE_NOWRITE File not opened for writing was closed
IN_CREATE File/directory created in watched directory
IN_DELETE File/directory deleted from watched directory
IN_DELETE_SELF Watched file/directory was itself deleted
IN_MODIFY File was modified
IN_MOVE_SELF Watched file/directory was itself moved
IN_MOVED_FROM File moved out of watched directory
IN_MOVED_TO File moved into watched directory
IN_OPEN File was opened

32. inotify in [language]
Language Source
Python pip install pyinotify
PHP pecl install inotify
Go go's exp repository
Ruby gem install rb-inotify
C #include <sys/inotify.h>

33. Scenario 2, Robustness
A Custom Message Queue

34. Message Queue Architecture

35. I/O Serialization

36. I/O Serialization
hash(queue_id) % num_threads

37. Message Queue Architecture
64K

38. Summary
● Caching
● File system choice
● Benchmarking w/ sysbench
● Efficiency through proper configuration
● Robustness through cooperation & decoupling
● Discovering & understanding your write pattern
● Benchmark & Graph everything
● Never Assume Anything (atime, stripe width, etc.)

39. Jason Johnson jajohnson@softlayer.com
https://github.com/jasonjohnson
http://www.slideshare.net/jasonajohnson
“A Case for Redundant Arrays of Inexpensive Disks (RAID)”
http://www.cs.cmu.edu/~garth/RAIDpaper/Patterson88.pdf
“Practical File System Design”
http://www.nobius.org/~dbg/practical-file-system-design.pdf
“XFS Papers and Documentation”
http://xfs.org/index.php/XFS_Papers_and_Documentation
“Kernel Documentation on File Systems”
https://www.kernel.org/doc/Documentation/filesystems/
“MySQL Performance Blog”
http://www.mysqlperformanceblog.com/
“MySQL DBA”
http://mysqldba.blogspot.com/
“MySQL Server System Variables”
http://dev.mysql.com/doc/refman/5.5/en/server-system-variables.html