Greg Lindahl's lightning talk from http://www.meetup.com/SF-Bay-Area-Large-Scale-Production-Engineering/calendar/15464663/

1. Asset
and
Configura/on
Tracking:
the
most
boring
topic
in
the
Universe
Greg
Lindahl
CTO,
blekko

2. The
problem(s)
• 100s
of
servers,
1000s
of
disks,
100s
of
desktops,
100s
of
laptops
• Some
of
the
servers
are
leased.
Disks
move
around
constantly.
Your
financial
people
want
to
do
an
audit.
Your
startup
finally
is
old
enough
to
need
asset
tags.
blah
blah
blah…
• How
many
hosts
got
that
new
kernel,
and
when?
What
soOware
is
installed
on
that
host
that’s
down?
What’s
the
up/me
history
of
the
en/re
cluster?
• Did
my
integrator
screw
up
and
return
broken
parts
instead
of
sending
them
for
repair?
1/3

3. System
requirements
• Based
on
tools
I
already
have
• Dead
simple
• Support
date
searches,
showing
when
a
string
appears
and
disappears
from
the
database
• I
know
what
you’re
thinking…
I
mean
something
simpler
than
that.

4. Global
part
• DVCS
-­‐-­‐
mercurial,
in
my
case
• Collect
text
files
for
every
host
in
a
separate
directory
• Check
in
files
daily
• hg
log
-­‐p
to
see
changes,
global
or
per-­‐host
• hg
grep
-­‐-­‐all
-­‐f
-­‐d
6QG06ZQN
• (put
the
repo
on
an
SSD…)
1/3

5. Local
part:
every
server
• Create
a
script
to
generate
text
files
containing:
– MAC
addrs,
running
kernel
version,
rpms
installed,
disk
serial
#s,
dmidecode,
installed
sysctl,
installed
firewall
config,
ipmi
info,
vendor
info
(hpdiscovery)
• Important:
these
files
should
only
change
if
there’s
a
change.
Don’t
store
up/me,
store
the
boot
/me.
• Commit
these
files
locally
1/3

6. Tie
them
together
• Copy
the
local
files
to
a
central
server,
check
in
there.
Push
to
several
backup
repos.
• See,
now
isn’t
this
simpler
than
what
you
were
thinking?
NoSQL.

7. What
about
laptops?
• Macs
are
Unix,
done.
• mercurial
runs
under
Windoze,
too
• They
need
their
own
data
collectors.
• The
only
trick
is
delivering
files
to
the
mothership
on
a
regular
basis.
1/2

8. slash
the
web!
• yes,
we’re
hiring
• vote
for
us
in
the
Crunchies
• check
out:
/linux,
/python,
/ruby,
/…