Collecting Metrics With Ganglia and Friends

Collecting Metrics
With Ganglia and Friends

Cambridge Geek Night 28th March 2011

gareth rushgrove | morethanseven.net http://www.ﬂickr.com/photos/memestate/45986749

Gareth Rushgrove

gareth rushgrove | morethanseven.net

freeagentcentral.com

Work at FreeAgent


Blog at morethanseven.net


Curate devopsweekly.com


- Capacity planning metrics
- Metrics for your application
- Business analytics
- Having everything in one place

Covering (Business Version)


- Ganglia Store metrics and view graphs
- Logster Get log ﬁles into Ganglia
- Gmetric Get anything into Ganglia
- Syslog Using Loggly to view individual log items

Covering (Tech Version)


Everyone Uses Something Like?


Use Something Like This Too


“Ganglia is a scalable distributed monitoring
system for high-performance computing
systems such as clusters and Grids.
ganglia.sourceforge.net

What is Ganglia?


Example: vagrantbox.es


Load Averages


CPU


Aggregate Graphs


Across Entire Cluster


“A strategy for anticipating future workloads
of your computers, with the aim of creating
a computing environment that can handle
future workload
IBM

Predicting When Your System Will Fail


Disk Space


Monitoring Your Application


86.26.7.33 - - [26/Mar/2011:20:39:53 +0000] "GET / HTTP/1.0" 200 5970 "-" "FunkLoad/1.14.0"
86.26.7.33 - - [26/Mar/2011:20:39:53 +0000] "GET / HTTP/1.1" 200 2081 "-" "Mozilla/5.0
(Macintosh; U; Intel Mac OS X 10_6_7; en-us) AppleWebKit/533.20.25 (KHTML, like Gecko)
Version/5.0.4 Safari/533.20.27"

Web Server Logs

Logster from Etsy


Tail a log file and filter each line to generate metrics that can be sent to
common monitoring packages.

Options:
-p METRIC_PREFIX, --metric-prefix=METRIC_PREFIX
Add prefix to all published metrics. This is for
people that may multiple instances of same service on
same host.
--gmetric-options=GMETRIC_OPTIONS
Options to pass to gmetric such as -d 180 -c
/etc/ganglia/gmond.conf (default). These are passed
directly to gmetric.
--graphite-host=GRAPHITE_HOST
Hostname and port for Graphite collector, e.g.
graphite.example.com:2003
-s STATE_DIR, --state-dir=STATE_DIR
Where to store the logtail state file. Default
location /var/run
-d, --dry-run Parse the log file but send stats to standard output.
-D, --debug Provide more verbose logging for debugging.

Logster


logster SampleGangliaLogster /../access.log

Logster Command Line


HTTP Responses with a 2xx Status Code


The Ganglia Metric Client (gmetric) announces a metric
on the list of defined send channels defined in a configuration file

Usage: gmetric [OPTIONS]...
-V, --version Print version and exit
-c, --conf=STRING The configuration file to use for finding send channels
(default='/etc/ganglia/gmond.conf')
-n, --name=STRING Name of the metric
-v, --value=STRING Value of the metric
-t, --type=STRING Either
string|int8|uint8|int16|uint16|int32|uint32|float|double
-u, --units=STRING Unit of measure for the value e.g. Kilobytes, Celcius
(default='')
-s, --slope=STRING Either zero|positive|negative|both (default='both')
-x, --tmax=INT The maximum time in seconds between gmetric calls
(default='60')
-d, --dmax=INT The lifetime in seconds of this metric (default='0')
-S, --spoof=STRING IP address and name of host/device (colon separated) we
are spoofing (default='')
-H, --heartbeat spoof a heartbeat message (use with spoof option)

Gmetric


Gmetric Scripts for Common Applications


gmetric -n sales -v 200 -t float

Gmetric Command Line


Our Custom Metric in Ganglia


import subprocess

from bottle import route, run, abort, default_app

@route('/:name/:value')
def index(name, value):
try:
cmd = 'gmetric -n %s -v %s -t float' % (name, value)
subprocess.check_call(
cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
return "Success: %s" % cmd
except subprocess.CalledProcessError:
abort(500, "Error")

app = default_app()

Gmetric HTTP Interface


http://../sales/200

Gmetric URL


import subprocess
import SocketServer

class GmetricTCPHandler(SocketServer.BaseRequestHandler):

def handle(self):
self.data = self.request.recv(1024).strip()
items = self.data.split(' ')
try:
cmd = 'gmetric -n %s -v %s -t float' % (items[0], items[1])
subprocess.check_call(
cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
return "Success: %s" % cmd
except Exception:
return "Error"

if __name__ == "__main__":
HOST, PORT = "0.0.0.0", 8001
server = SocketServer.TCPServer((HOST, PORT), GmetricTCPHandler)
server.serve_forever()

Gmetric TCP Interface


sales 200

Gmetric TCP


“Syslog is a standard for logging program
messages. It allows separation of the
software that generates messages from the
system that stores them and the software
that reports and analyzes them.
Wikipedia

Syslog


Loggly - Logging as a Service


View logs


Logstash


Graylog2


- Database table sizes
- Cache hits
- Time taken for test runs
- Codebase size
- Signups, sales, subscriptions
- Twitter followers

Other Things You Could Monitor


- Wikipedia http://ganglia.wikimedia.org/
- Install Ganglia deb and rpm packages available
- Add system metrics web servers, databases
- Add business metrics users, sales, tweets
- Try Loggly or at least investigate syslog

What Next?


Reading


CBGN11

2 months free on FreeAgent


Questions?

gareth rushgrove | morethanseven.net http://ﬂickr.com/photos/psd/102332391/

Collecting Metrics With Ganglia and Friends

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Destacado

Destacado (9)

Similar a Collecting Metrics With Ganglia and Friends

Similar a Collecting Metrics With Ganglia and Friends (20)

Más de Gareth Rushgrove

Más de Gareth Rushgrove (20)

Último

Último (20)

Collecting Metrics With Ganglia and Friends