Leland Lammert's presentation on Nagios in a Multi-Platform Enviornment.
The presentation was given during the Nagios World Conference North America held Sept 20-Oct 2nd, 2013 in Saint Paul, MN. For more information on the conference (including photos and videos), visit: http://go.nagios.com/nwcna
Axa Assurance Maroc - Insurer Innovation Award 2024
Nagios Conference 2013 - Leland Lammert - Nagios in a Multi-Platform Enviornment
1. Nagios in a Multi-Platform
Environment
lvl@omnitec.net
Leland V. Lammert, PhD
Chief Scientist
Omnitec Corporation
2. The Problem
Different OSs can require
– Different type of connection
– Different installation procedure
Nagios checks or an agent require a
connection to the remote machine
(i.e. from the Nagios server to the remote
machine)
No issue inside the firewall
There are issues for machines outside the
firewall
3. The Solution – SSH
SSH forward connections are a good solution
for monitoring inside the firewall
An ssh reverse tunnel is a good solution for
monitoring outside the firewall
Active checks can then be used on all systems
– No Agents
– No Complex Installation
– A shell is always available for
troubleshooting
6. SSH History
SSH [Secure Shell] is a data exchange protocol
that uses an encrypted connection between two
network devices
SSH replaced Telnet and other insecure remote
shells which send information, most notably
passwords, in plaintext
SSH encryption provides confidentiality and
integrity of data over an insecure network
Connection Types
– Normal [Forward]
– Tunnel [Reverse]
7. SSH Advantages
Secure [ssh2]
– Host Identity Verified
– User Authorization
– Secure Data Transmission
Ubiquitous ('Nix, Windows [Cygwin], OSX)
Reverse Tunnels require no firewall changes
Allows Active Checks
No agent to install or configure
Simplified testing, GUI still available [X or RDP]
8. SSH Details
Authentication is verified with SSH keys
[Forward] ssh connections work for hosts
behind the firewall
[Reverse or Tunnel] ssh connections work for
hosts anywhere else
The tunnel passes traffic TO the remote
machine FROM the Nagios Server (reverse
operation)
SSH Connections can be used to standardize
connections to ALL machines!
9. Forward connections
Requires
– Static IP
– Open port
A static IP may not available at external
locations
An open port for an incoming connection is a
BIG security problem
Even when an agent is used [NSCA], there is
no access to the remote machine for testing
and troubleshooting
10. Reverse Connections - Tunnels
A Tunnel is packet encapsulation using a
network protocol
The data payload protocol is then
encapsulated in a delivery protocol.
– L2TP (layer 2) Tunneling Protocol)
– SSH
– Socks
Reverse Tunnels - an ssh connection from
client to server, where data is transmitted from
server to client (i.e. reverse)
11. Connection Process
Startup
– Start Session
– Exchange Server key
– Generate Session key
Authentication
– Exchange Client key
– If ACK, continue
14. SSH uses Asymetric Encryption
Public/Private Keypair
A Keypair verifies identity for:
– Machine
– User
Currently keys are 2048 bits in length, usual
RSA
Each Key has two parts:
– Public and Private
– AKA the keypair
Keys are created with ssh-keygen
16. Host and User Credentials
Host Key
Verifies that the host/server at the other end of
the connection is the same one seen
previously
User Keys
Verifies that the user is authorized for the
connection
Why not Passwords?
● Requires manual entry
● Cannot be automated without storing as
plaintext
● Much less secure than key
21. Basic ssh command
ssh Base command
-f Run in background
-X Enable X Forwarding ('Nix)
-n Prevent reading from STDIN
-N No remote command
-R Reverse Operation
ssh -f -X -n -N -R
22. Ports
REMOTE:127.0.0.1:LOCAL*
REMOTE Port on remote machine [from]
127.0.0.1 localhost
LOCAL Port on local machine [to]
* Remember, this is being done at the external machine –
so the “Remote” port here is actually on the Nagios server!
23. Authentication
$USER_NAME@$REMOTE_HOST
USER_NAME User ID on remote machine
REMOTE _HOST Hostname or IP of
remote hostt
-p LOGIN_PORT
LOGIN_PORT Port for login on remote
machine
-i IDENTITY_FILE
IDENTITY_FILE Private key file to use
24. Three Steps to Create a Tunnel
1. Remote to Nagios
Connect remote machine to Nagios server and
create Tunnel
2. Make it permanent
– cron
– autossh
– launchctl/launchd
3. Nagios to Remote
Setup access from Nagios user or process to
remote machine
25. Step 1. Remote to Nagios
Connect remote machine to Nagios server
and create Tunnel
26. Step 1 – 'Nix
Create keypair on remote system and copy to
Nagios Server
– ssh-keygen
– cat ~/.ssh/id_rsa.pub
● <highlight>
– ssh <nagios server>
– vim ~/.ssh/authorized_keys
– G o <shift insert>
– :x
<exit> test
27. Step 1 – Windows
Install Cygwin (cygwin.com/install.html)
– Install in c:program filesCygwin
– Add autossh, rsync, bash, perl
Create keypair on remote system and copy to
Nagios Server [same as 'Nix]
28. Step 1 – OSX
Create keypair on remote system and copy to
Nagios Server [same as 'Nix]
30. Step 2 – 'Nix cron
Connection variables
REMOTE_HOST="nagios._______.com"
USER_NAME="________"
REMOTE_PORT="____"
LOCAL_PORT="____" *
LOGIN_PORT="____" *
IDENTITY_KEY="/home/nagios/.ssh/id_rsa"
* NOTE: The ssh port should be changed for security, do NOT use the standard port 22.
31. Step 2 – 'Nix cron
Command to create the link
COMMAND="ssh -f -n -N -R
$REMOTE_PORT:127.0.0.1:$LOCAL_PORT
$USER_NAME@$REMOTE_HOST -p$LOGIN_PORT -i
$IDENTITY_KEY"
32. Step 2 – 'Nix cron
Running? If not, start it
pgrep -f -x "$COMMAND" > /dev/null 2>&1 || $COMMAND
Working? Login to Nagios and check from the
other side
ssh -i$IDENTITY_KEY -p$LOGIN_PORT
$USER_NAME@$REMOTE_HOST netstat -an | egrep
"tcp.*127.0.0.1:$REMOTE_PORT.*LISTEN" > /dev/null 2>&1
If not working, kill and restart.
if [ $? -ne 0 ] ; then
pkill -f -x "$COMMAND"
$COMMAND
fi
33. Step 2 – Windows autossh
Install service
cygrunsrv -I AutoSSH -f "nagios_link" -p
/usr/bin/autossh -a "
-M <port + 1000>:<port + 2000> -N -R
<port>:127.0.0.1:<ssh port> <user>@<nagios
server>"
Use services.msc to set restart options and
credentials:
– cyg_server
– <created Nagios user>
– Local Administrator
34. Step 2 – OSX
Install autossh with homebrew
Command
/usr/local/bin/autossh -M <port + 1000>:<port + 2000> -N -R -p 2206 -g
4000:127.0.0.1:22 <user>@<nagios server> -tt
Launchctl
Can install AutoSSH as System Service
Autostart at boot
36. Step 3 – Nagios to Remote
Setup access from Nagios user or process
to remote machine
37. Step 3 - 'Nix
Copy Nagios user public key to remote
Note: Nagios may use different UID than the
one for testing
– cat ~/.ssh/id_rsa.pub
– <highlight>
– ssh <remote server>
– vim .ssh/authorized_keys
– G o <shift insert>
– :x
40. What was just created?
Take note – at the current time there are
working ssh connections to:
– Any 'Nix
– Any Windows*
– OSX
These connections can be used for active
checks, as well as troubleshooting and setting
up an RDP or X session if needed.
* There sometimes are authentication issues with Windows
Domain servers that must be handled uniquely.
42. Normal [forward] checks
Local checks
define service{
use local-service
host_name Nagios
service_description Root Partition
check_command check_local_disk!20%!10%!/
}
Checks via ssh connection
define host{
use openbsd-server
host_name mx1
alias mail_mx1_server
address 206.197.251.200
icon_image envelope.gif
statusmap_image envelope.gif
check_command check_smtp
}
43. Reverse Check
Same as other checks, except command format
includes additional data
define service{
use generic-service
host_name hanley
service_description SSH Check Proc
normal_check_interval 15
retry_check_interval 5
notifications_enabled 1
check_command check_by_ssh_reverse!-p2210!proc
}
Additional data required in command
– ssh Private key
– ssh Port
– Host address
47. X apps run natively
Be sure to include “-X” in tunnel setup
Connect to remote with “ssh -X”
Launch app
Possibly explicit invocation options, a la Firefox:
firefox –no-remote
Start firefox, but run on the remote machine,
forwarding the display to the Nagios server.
48. RDP on demand
On remote machine
#
# Reverse ssh link for RDP tunnel
#
ssh -f -n -N -R 3389:127.0.0.1:3389
<user>@<nagios server> -p <ssh port>
Tunnel 3389 on remote machine to 3389 on
Nagios Server
Check for open port
netstat -an | grep 3389
Use one connection at a time for sanity!
50. Common Problems
Host Key Verification Failed
– The Host Key on the sending side is not the
same as seen last time
– Possible cause - the remote machine has
been rebuilt
Troubleshooting checks
– $64K Secret – tmux
– Very step-by-step
– Paremeters often not required
Useful aliases
51. ●Hostkey verification failed
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
The ECDSA host key for [storm]:2206 has changed,
and the key for the corresponding IP address [10.0.0.1]:2206
is unknown. This could either mean that DNS SPOOFING is happening or the IP
address for the host and its host key have changed at the same time.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
6b:da:2e:50:a9:ea:b0:b1:3d:c1:b8:4a:a3:a5:56:87.
Please contact your system administrator.
Add correct host key in /home/lvl/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/lvl/.ssh/known_hosts:5
You can use following command to remove all keys for this IP:
ssh-keygen -R storm -f /home/lvl/.ssh/known_hosts
Host key verification failed.
52. tmux
tmux is a terminal multiplexer
A terminal multiplexer permits switching
between several programs in one terminal,
detach them (they keep running in the
background) and reattach them to a different
terminal.
Essential for troubleshooting checks!