A presentation for DrupalCon Prague 2013

1. Have you been stalking
your servers?

2. Have you been stalking
your servers?
Marji Cermak
Sysadmin & DevOps Engineer at Morpht
marji@morpht.com
@cermakm

3. The rule of 3 things
picture: http://www.flickr.com/photos/helenaperezgarcia/5692392667/

4. The rule of 3 things
1. What is monitoring and why do you want to
monitor
2. Some monitoring tools available for you
3. It is easy to start with monitoring.

5. Part 1
What is monitoring and why do you want to
monitor

6. photo: http://www.flickr.com/photos/tiagopadua/7903366470/

7. Monitoring
Monitoring is an intermittent (regular or
irregular) series of observations in time,
carried out to show the extent of compliance
with a formulated standard or degree of
deviation from an expected norm.
J. M. Hellawell (1991), modified by A. Brown
(2000), http://jncc.defra.gov.uk/page-2268
nature conservation area

8. Why you need to monitor
● to know about the bad news before your
customers (or your boss)

9. Why you need to monitor
● to know about the bad news before your
customers (or your boss)
● to scale up your server in advance

10. Why you need to monitor
● to know about the bad news before your
customers (or your boss)
● to scale up your server in advance
● to tune up your app

11. Why you need to monitor (cont.)
● to prove your uptime of 99.999 :)

12. The fun of the nines
Source: http://en.wikipedia.org/wiki/High_availability
Nines: http://en.wikipedia.org/wiki/List_of_unusual_units_of_measurement#Nines

13. Why you need to monitor (cont.)
● to prove your uptime of 99.999 :)
● to minimise downtime (expensive)

14. Why you need to monitor (cont.)
● to prove your uptime of 99.999 :)
● to minimise downtime (expensive)
● to capture customer information

15. Why you need to monitor (cont.)
● to have data / metrics to diagnose

16. Diagnosing your collected data
watch out for:
● trends

17. Diagnosing your collected data
watch out for:
● trends
● spikes

18. Diagnosing your collected data
watch out for:
● trends
● spikes
● irregularities

19. Diagnosing your collected data
watch out for:
● trends
● spikes
● irregularities
● thresholds

20. Areas to monitor
● network
photo: http://www.flickr.com/photos/misja_klimov/2120956405/

21. Areas to monitor
● network
● server
photo: http://www.flickr.com/photos/johnjack/3666997634/

22. Areas to monitor
● network
● server
● services
photo: http://www.flickr.com/photos/agustingodet/3691794089/

23. Areas to monitor
● network
● server
● services
photo: http://www.flickr.com/photos/agustingodet/3691792393/

24. Areas to monitor
● network
● server
● services
● applications
photo: http://www.flickr.com/photos/cheerfulstoic/942211994/

25. Areas to monitor
● network
● server
● services
● applications
● users
photo: http://www.flickr.com/photos/jimmysmith/99528596/

26. Drupal Areas to monitor?
● network
● server
● services
● applications
● users

27. Drupal Areas to monitor
● network
● server
● services
● applications
● users

28. Drupal Areas to monitor
● network
● server
● services
● applications
● users

29. Drupal Areas to monitor
● network
● server
● services
○ webserver
○ database
● applications
● users

30. Drupal Areas to monitor
● network
● server
● services
○ webserver
○ database
● applications - your Drupal site(s)
● users

31. Drupal Areas to monitor
● network
● server
● services
○ webserver
○ database
● applications - your Drupal site(s)
● users

32. Part 2
Some monitoring tools available for you

33. Meet Nagios, Munin and others
● Nagios
● Munin
● APC dashboard
● related Drupal modules

35. Nagios /ˈnɑːɡiːoʊs/
● system, network and infrastructure
monitoring software application
● monitors and alerts
● many plugins

36. Nagios /ˈnɑːɡiːoʊs/
Name and Pronunciation:
● NetSaint -> "Nagios Ain't Gonna Insist On
Sainthood"
● Agios' a transliteration of the Greek word
άγιος (saint)

37. Nagios /ˈnɑːɡiːoʊs/
● alerts by email/pager/IM...
● alerts to different contacts
● notification escalation
● service / host dependencies
● soft / hard states

38. Nagios /ˈnɑːɡiːoʊs/

39. Drupal and Nagios

41. Munin
● network/system monitoring application
● outputs graphs through a web interface
● many plugins

42. Munin
● master / node architecture
● connects to all nodes at regular intervals
● it uses the RRDtool (round robin database
tool, handles time-series data)

45. Munin Example

46. Drupal and Munin

47. Drupal and Munin

48. ● they complement each other
● nagios normally alerts on one “service”
● munin can be used to correlate different
things
Nagios & Munin

49. APC - what is it?
The Alternative PHP Cache (APC) is a free
and open opcode cache for PHP.

50. APC - what is it?
The Alternative PHP Cache (APC) is a free
and open opcode cache for PHP.
Its goal is to provide a free, open, and robust
framework for caching and optimising PHP
intermediate code.
Inside your webserver (not a webcache)

51. Monitoring APC
Memory Usage, Hit & Misses

52. Monitoring APC
Fragmentation

53. Monitoring APC
memory usage

54. Monitoring APC
files in cache

55. Other monitoring tools
● Collectd
● Graphite
● Shinken
● Sensu
● NewRelic
● Pingdom

56. Part 3
It is easy to start with monitoring.

57. How to install these tools?
Munin
sudo apt-get install munin munin-node
Nagios
sudo apt-get install nagios3
APC dashboard
php.apc script from php-apc package

58. How to configure these?
● It is a bit fiddly
● There are many guides targeting beginners
● You don’t want to do it again and again

59. puppet – a quick way to start
system for automating system administration
tasks

60. puppet – a quick way to start
● a declarative language for expressing
system configuration,

61. puppet – a quick way to start
● a declarative language for expressing
system configuration,
● a client and server for distributing it

62. puppet – a quick way to start
● a declarative language for expressing
system configuration,
● a client and server for distributing it
● and a library for realising the configuration.

63. puppet – a quick way to start
package { 'munin-node': ensure => installed }
service { 'munin-node':
enable => true,
ensure => running,
require => Package['munin-node'],
}

64. puppet – a quick way to start
1. clone the stalk-your-box repo
2. run puppet apply on the code
3. monitor!

65. A quick way to start
$ git clone
git://github.com/morpht/stalk-your-box.git
/tmp/stalk-your-box
Cloning into '/tmp/stalk-your-box'...
remote: Counting objects: 23, done.
remote: Compressing objects: 100% (19/19), done.
remote: Total 23 (delta 1), reused 23 (delta 1)
Receiving objects: 100% (23/23), 11.35 KiB, done.
Resolving deltas: 100% (1/1), done.

66. A quick way to start
$ cd /tmp/stalk-your-box/
$ sudo puppet apply
--modulepath=modules manifest.pp
notice: /Stage[main]/Nagios::Server/Package[nagios3]/ensure: ensure changed 'purged' to 'present'
notice: /Stage[main]/Nagios::Server/File[/etc/nagios3/htpasswd.users]/ensure: created
notice: /Stage[main]/Nagios::Server/Exec[update-nagios-htpasswd]/returns: Adding password for user nagiosadmin
notice: /Stage[main]/Nagios::Server/Exec[update-nagios-htpasswd]/returns: executed successfully
notice: /Stage[main]/Munin::Node/Package[libcache-cache-perl]/ensure: ensure changed 'purged' to 'present'
notice: /Stage[main]/Munin::Node/Package[munin-node]/ensure: ensure changed 'purged' to 'present'
notice: /Stage[main]/Munin::Node/File[munin-node.conf]/content: content changed '{md5}
e486786f866d7d7e025dea401c300e7b' to '{md5}dbf97a87a8da86ef68155815ecae3c1c'
notice: /Stage[main]/Munin::Server/Service[apache2]: Triggered 'refresh' from 1 events
notice: Finished catalog run in 44.26 seconds

67. What this gives you

68. What this gives you

69. What this gives you

70. Manifest.pp
# Execute apt-get update before any package is installed:
exec { 'apt-update':
command => 'apt-get update',
# but don't execute it more than once a day:
unless => 'test $(find /var/cache/apt/pkgcache.bin -mtime 0 | wc -l ) -eq 1',
}
Exec['apt-update'] -> Package <| |>
# Include minimal apache2 installation. Munin server, nagios
# and APC dashboard depend on it.
include 'apache2'

71. Manifest.pp
# Install munin node and munin server:
class { 'munin::node': }
class { 'munin::server':
htuser => 'munin', # Username for basic access auth.
htpass => 'Prague2013' # Password for basic access auth.
}
# Install nagios:
class { 'nagios::server':
contact_email => 'root@localhost', # Email to send alerts to.
htpass => 'Prague2013', # Password for the nagiosadmin username.
}

72. Manifest.pp
# Deploys APC dashboard - install php-apc package and
# deploy the apc.php script from it.
package { 'php-apc': ensure => installed }
exec { 'deploy-apc-dashboard':
path => '/bin:/usr/bin',
command => 'gzip -dc /usr/share/doc/php-apc/apc.php.gz > /var/www/apc.php',
notify => Service['apache2'],
unless => '[ -f /var/www/apc.php ]',
require => [ Package['php-apc'], Package['apache2'] ]
}

73. Summary
It is easy to start with monitoring.

74. The fun part - what’s wrong?
What’s wrong here?

75. The fun part - what’s wrong?

76. Questions
Here is the get started monitoring repo:
https://github.com/morpht/stalk-your-box
Marji Cermak
Sysadmin & DevOps Engineer at Morpht
marji@morpht.com
@cermakm

77. Resources
Rule of Three: en.wikipedia.org/wiki/Rule_of_three_(writing)
Nagios: http://www.nagios.org/
Munin: http://munin-monitoring.org/
Nagios module: https://drupal.org/project/nagios
Munin module: https://drupal.org/project/munin
Munin plugins (experimental): https://drupal.org/sandbox/murrayw/2084281
Sensu: http://sensuapp.org
MySQLTuner: http://MySQLTuner.pl

78. THANK YOU!
WHAT DID YOU THINK?
Locate this session at the
DrupalCon Prague website:
http://prague2013.drupal.org/schedule
Click the “Take the survey” link