Customizing Linux Server - administration changes to improve performance

1. Refining Linux

2. Jason Murray, D.CS
Cornwall-Lebanon SD
 jasonmurray72@gmail.com
 @jasonmurray72
 slideshare.net/jasonmurray72

3. Agenda
 Linux Intro
 What is Optimization?
 Performance
 Server Optimization
 Performance Monitoring
 System Monitoring Tools
 Benchmark Tools

6. Kernels
 Linux – Monolithic Kernel Architecture: provides all
services
 Windows – Micro Kernel Architecture: small core group of
services that executes other services and applications

7. Recompile the Kernel
 make mrproper - This target cleans up the build environment of any stale
files and dependencies that might have been left over from a previous kernel
build. All previous kernel configurations will be cleaned (deleted) from the
build environment.
 make clean - This target does not do as thorough a job as the “mrproper”
target. It only deletes most generated files. It does not delete the kernel
configuration file (.config).
 make menuconfig - This target invokes a text-based editor interface with
menus, option lists, and text-based dialog boxes for configuring the kernel.
 make xconfig - This is an X Window System–based kernel configuration tool
that relies on the Qt graphical development libraries. These libraries are used
by KDE-based applications.
 make gconfig - This target also invokes an X Window System–based kernel
configuration tool, but it relies on the GTK2 (GIMP) toolkit. This GTK2 toolkit
is heavily used in the GNOME desktop world.
 make help - This target will show you all the other possible make targets and
also serves as a quick online help system.

8. What is Optimization?
 Server Design
 Stability
 Availability
 Performance
 General Server Response
 Hardware Response
 Service Response

9. Performance
 Hardware
 Safe Clean Environment
 Software
 Enhancement in parameter improvements can compensate and
improve older hardware performance
 Avoid a GUI
 At least unload the GUI after using it

10. Server Optimization
 Fine Tuning
 Services
 Hardware
 Kernel
 Network
 Security
 Database
 Storage
 Memory

11. Recommendations
 Custom installs
 Only load the essentials
 Partitioning Planning
 Know the purpose of your server

12. Performance Monitoring
 CPU
 Network
 I/O
 Memory

13. CPU Performance Metrics
 Context Switch
 Run Queue
 CPU Utilization
 Load Average

14. Context Switch
 Context Switch - CPU moves from one thread to another
 During context switch, the kernel stores current thread state in
memory
 Simultaneously, the kernel retrieves previously stored thread state
from memory & loads it to the CPU
 Context switching affords multitasking of CPU
 High levels of context switching can cause performance degradation
 Example: pidstat 2 5
 Display five reports of CPU statistics for every active task in the system at
two second intervals.

15. Run Queue
 Run Queue – total number of active processes in current
queue for CPU
 Processes in sleep state of I/O wait state are not in the
run queue
 High number of processes in the run queue can cause
performance degradation
 Example: vmstat
 The r column in the table represents run queue processes

16. CPU Utilization
 CPU Utilization – percentage that the CPU is being used
 Higher percentage of CPU utilization can cause
performance degradation
 Example: top
 Real time view of cpu utilization

17. Load Average
 Load Average – average CPU load over a specified time period
 Load average is displayed for the last 1 minute, 5 minutes, and 15
minutes
 Easy monitoring for overall load
 Load Average is calculated by combining the total number of
processes in the run queue and the total number of processes in the
uninterruptable task status
 Example: uptime
 “0.25 1.20 1.90” – System load coming down
 .25 - Load average for last 1 minute
 1.20 – Load average for last 5 minutes
 1.90 – Load average for last 15 minutes

18. Disk I/O Optimization
 4 different I/O schedulers
 CFQ (default) – distributes all I/O bandwidth evenly among all
processes requesting I/O
 NOOP – acts as a basic FIFO queue expecting the hardware controller to
handle the performance operations of the requests; attempts to use as
little CPU as possible
 Anticipatory – designed to increase performance on systems that have
slow disks; reorders all disk I/O operations to optimize disk seeks
 Deadline – places I/O requests in a priority queue so each is guaranteed
to run within a certain time; often used in real-time operating

19. Changing Schedulers
 Which schedules do you have compiled
 /sys/block/sda/queue # cat scheduler
 anticipatory deadline [cfq]
 Compile noop
 /sys/block/sda/queue # modprobe noop-iosched
 anticipatory deadline [cfq] noop
 Change
 /sys/block/sda/queue # echo anticipatory > scheduler
 http://www.linuxhowtos.org/System/iosched.htm

20. Scheduler
 Agility
 Changing schedulers on the fly allows for benchmark
testing of your specific applications and hardware
 Notes
 The change will not be instant. All current processes in
run queue must complete first.
 Once you find the optimal scheduler for your specific
situation, apply the change for subsequent reboots

21. Scheduler Suggestions
 Use NOOP or Deadline on SSD drives
 Benchmark each scheduler to determine your optimal
solution
 CFQ is not always the best scheduler for your system
 Example: Your system is running a raid

22. I/O Optimization
 Consistent high I/O wait indicates an issue with the disk
subsystem
 Monitor BI & BO (Block in & Block out)
 Reads/second & writes/second can also identify trouble areas
 Check TPS (transactions/second) – total of BI & BO
 Example: vmstat

23. vmstat
 Procs
 r: The number of processes
waiting for run time.
 b: The number of processes in
uninterruptible sleep.
 Memory
 swpd: the amount of virtual
memory used.
 free: the amount of idle
memory.
 buff: the amount of memory
used as buffers.
 cache: the amount of memory
used as cache.
 inact: the amount of inactive
memory. (-a option)
 active: the amount of active
memory. (-a option)
 Swap
 si: Amount of memory swapped
in from disk (/s).
 so: Amount of memory swapped
to disk (/s).
 IO
 bi: Blocks received from a block
device (blocks/s).
 bo: Blocks sent to a block
device (blocks/s).
 System
 in: The number of interrupts per
second, including the clock.
 cs: The number of context
switches per second.
 CPU
 These are percentages of total
CPU time.
 us: Time spent running non-
kernel code. (user time,
including nice time)
 sy: Time spent running kernel
code. (system time)
 id: Time spent idle. Prior to
Linux 2.5.41, this includes IO-
wait time.
 wa: Time spent waiting for IO.
Prior to Linux 2.5.41, included
in idle.
 st: Time stolen from a virtual
machine. Prior to Linux 2.6.11,
unknown.

24. Disk I/O Optimization
File
System
Max File
Size
Max Partition
Size
Journaling Notes
Fat16 2 GB 2 GB No Legacy
Fat32 4 GB 8 TB No Legacy
NTFS 2 TB 256 TB Yes
(For Windows Compatibility) NTFS-3g is
installed by default in Ubuntu, allowing
Read/Write support
ext2 2 TB 32 TB No Legacy
ext3 2 TB 32 TB Yes
Standard linux filesystem for many years. Best
choice for super-standard installation.
ext4 16 TB 1 EB Yes
Modern iteration of ext3. Best choice for new
installations where super-standard isn't
necessary.
reiserFS 8 TB 16 TB Yes No longer well-maintained.
JFS 4PB 32PB
Yes
(metadata)
Created by IBM - Not well maintained.
XFS 8 EB 8 EB
Yes
(metadata)
Created by SGI. Best choice for a mix of
stability and advanced journaling.

25. Disk I/O Optimization
 Disk Cache Policy
 hdparm -a /dev/sda
 Readahead value at least 1024
 Write Policy
 Writeback
 Read Policy
 Adaptive readahead
 Penalties
 Calculate the IOPS

26. Disk Optimization
 Solutions
 Bonnie++
 Hdparm
 Upgrades
 BIOS
 Firmware

27. Network
 Monitoring
 Total number of packets
 Received
 Sent
 Dropped

28. TCP Optimization –
Large Production Servers
 Port Availability
 Echo 1024 65000 > /proc/sys/net/ipv4/ip_local_port_range
 Memory Allocation with Socket Buffers
 Echo 262143 > /proc/sys/net/core/rmem_max
 Echo 262143 > /proc/sys/net/core/rmem_default
 TCP Stack Workload Reduction
 Echo 0 >/proc/sys/net/ipv4/tcp_sack
 Echo 0 > /proc/sys/net/ipv4/tcp_timestamps

29. Memory Optimization
 Virtual Memory – swap space available on the disk & physical memory
(user space & kernel space)
 Kernel’s file system cache takes advantage of the unused RAM
 During a swap, Linux writes the least used memory pages from the
physical memory to the swap space on the disk
 High levels of swapping will degrade performance, because the disk is
much slower
 Vmstat command
 Recommendation
 Benchmark tests prove that ECC RAM is faster

30. Additional Helpful Commands
 Netstat – lists network sockets
 Ps – snapshot of current processes
 Atop – interactive monitor for the load averages
 Mtop – displays mysql threads
 Iostat – reports cpu & I/O statistics
 Xosview – displays CPU usage, load average, memory
usage, swap space usage, network usage, interrupts, and
serial port status

31. Kernel Optimization
 The first thing to get loaded is the kernel. Keep in mind that no operating system
exists in memory at this point, and PCs (by their unfortunate design) have no easy
way to access all of their memory. Thus, the kernel must load completely into the
first megabyte of available random access memory (RAM). In order to accomplish
this, the kernel is compressed. The head of the file contains the code necessary to
bring the CPU into protected mode (thereby removing the memory restriction) and
decompress the remainder of the kernel.
 Kernel Execution. With the kernel in memory, it can begin executing. It knows only
whatever functionality is built into it, which means any parts of the kernel
compiled as modules are useless at this point. At the very minimum, the kernel
must have enough code to set up its virtual memory subsystem and root file system
(usually, the ext3 file system). Once the kernel has started, a hardware probe
determines what device drivers should be initialized. From here, the kernel can
mount the root file system. (You could draw a parallel of this process to that of
Windows being able to recognize and access its C drive.) The kernel mounts the
root file system and starts a program called init.

32. Boot Loaders
 MBR – starting on page 142 of Linux Administration
 GRUB
 Modify boot record & kernel initiation
 Splash screen
 Hidden menus
 Title
 Root
 Initrd – preload modules in kernel
 LILO

33. Database Optimization
 MySQL – use MySQL tuning scripts
 Optimize table

34. OpenLDAP Optimization
 Important Decision – Which attributes will you build indexes on
 Optimal Parameters – change before data entry
 /etc/openldap.conf
 Cachesize 10000
 Dbcachesize 100000
 Sizelimit 10000
 Loglevel 0
 dbcacheNoWsync
 Index cn,uid
 Index uidnumber
 Index gid
 Index gidnumber
 Index mail

35. Apache Optimization
 Good Benchmark Scores – many initial daemons
 Example
 MinSpareServers 20
 MaxSpareServers 80
 StartServers 32
 If Apache is recompiled
 MaxClients 256
 MaxRequestsPerChild 10000
 Notes – A high number of max servers and an honest
number of spare servers will work fine. For the most part
the StartServers help with the instant load that
benchmarks generate.

36. Apache Management
 Default Directory - /var/www/html
 Apache Config File - /etc/httpd/httpd.conf
 /etc/httpd/conf.d/phpMyAdmin.conf
 PHP Config File - /etc/php.ini
 /etc/init.d/httpd start|stop|restart
 Service httpd start|stop|restart
 /var/log/httpd/error_log
 /var/log/httpd/access_log

37. Poor Performance Website
 Consider
 Optimizers
 Memcache
 Adjusting Apache
 Minimizing Apache modules
 Alternative to apache - nginx

38. Benchmarking
 Benchmarking is helping; HOWEVER,
 Its impossible to duplicate real world situations
 Understand the GOAL for benchmarking
 Measuring the performance of ONE particular element very
accurately
 Benchmarks are excellent tools if you understand what
they are doing and trying to accomplish

39. Benchmark Tools
 Bonnie++ - file system benchmarking tool
 Dbench – I/O workload generator to a file system or
networked CIFS or NFS server
 Http_load – throughput tester for a web server
 Dkftpbench – FTP download benchmark analyzer
 Tiobench – multi-threaded I/O benchmark tool
 Ttcp – network throughput measuring tool
 Netperf – network performance tester

40. More Helpful Tools
 Lsof – list open files
 Tcpdump – network packet analyzer
 Htop – process monitoring
 Iotop – monitors disk I/O
 IPTraf – real time IP LAN monitoring
 Psacct – monitor user activity
 Monit – process & services monitoring
 Nethogs – monitor per process network bandwidth
 Iftop – network bandwidth monitoring
 Monitorix – system & network monitoring

41. Troubleshooting Methodology
 Clearly understand the problem
 Monitor & collect data
 Eliminate and narrow issues
 Make one change at a time & OBSERVE
 Document everything!

42. References
 Davor Guttierrez - @dguttierrez
 Soyinka, W. (2008). Linux Administration: A Beginner's Guide.
McGraw-Hill, Inc..
 http://www.tecmint.com/command-line-tools-to-monitor-linux-
performance/
 http://www.thegeekstuff.com/2011/07/iostat-vmstat-mpstat-
examples/