Hacking the data out of Puppet. Presentation from Puppetconfg

1. Hacking
The Data
out of Puppet
Dan Bode| Puppet Labs
dan@puppetlabs.com

2. Who is this talk for?
• People who already understand Puppet
• Developers or people who are dev-curious
# puppetconf # puppetize @ puppetlabs

3. Shameless plug
# puppetconf # puppetize @ puppetlabs

4. What is it about?
• Deconstructing Puppet to data
# puppetconf # puppetize @ puppetlabs

5. Puppet as Data
Dissecting a Puppet Run

6. Facter, who am I?
Hi! your facts are:
kernel=linux
ipaddress=10.0.0.3
Agent
macaddress=…
# puppetconf # puppetize @ puppetlabs

7. facts
Hi Mr. Master,
I need a catalog. Here
Agent
are my facts
http://www.dgcomputers.org/testimonials.php
# puppetconf # puppetize @ puppetlabs

8. facts
Thanks for you facts.
Agent
store them in
I’ll just
PuppetDB
PuppetDB
# puppetconf # puppetize @ puppetlabs

9. Mr. ENC, is this host
defined as an external Yep, he should be an
node?
apache server. Here is
the definition
nodes
ENC
Agent
PuppetDB
facts
# puppetconf # puppetize @ puppetlabs

10. catalog
Just compiled your
Agent
One sec while
catalog. PuppetDB
I store it in PuppetDB.
facts
# puppetconf # puppetize @ puppetlabs

11. Here is your
catalog. Send me
a report and let
me know how it
went!
catalog
Agent
PuppetDB
catalog
facts
# puppetconf # puppetize @ puppetlabs

12. I hate to be a
bother, but can
you compute
the md5sums of
a few files?
catalog
Agent
PuppetDB
catalog
facts
# puppetconf # puppetize @ puppetlabs

13. report
Just finished applying.
Agent
are the results.
Here PuppetDB
catalog
facts
# puppetconf # puppetize @ puppetlabs

14. Termini and the indirector
Interacting with Puppet’s Data

15. facts find from
terminus facter
Agent
# puppetconf # puppetize @ puppetlabs

16. facts
catalog find from
terminus rest
Agent
http://www.dgcomputers.org/testimonials.php
# puppetconf # puppetize @ puppetlabs

17. facts
facts save to terminus
Agent
puppetdb
PuppetDB
# puppetconf # puppetize @ puppetlabs

18. node find from
terminus exec (or
ldap)
nodes
ENC
Agent
PuppetDB
facts
# puppetconf # puppetize @ puppetlabs

19. catalog
catalog find from
Agent
compiler
terminus PuppetDB
facts
# puppetconf # puppetize @ puppetlabs

20. catalog
catalog save to
Agent
puppetdb
terminus PuppetDB
facts
# puppetconf # puppetize @ puppetlabs

21. Disecting a Puppet Run
Facter
Facts
ENC
Nodes/
Manifest
Compiler
Reports
Config
Catalogs
# puppetconf # puppetize @ puppetlabs

22. CLI commands
puppet
facts
find
puppet
node
find
puppet
catalog
find
# puppetconf # puppetize @ puppetlabs

23. CLI Puppet Facts
# mkdir –p /tmp/yaml/facts
# puppet facts find node_name --render-as yaml
> /tmp/yaml/facts/node_name.yaml
# puppetconf # puppetize @ puppetlabs

24. Creating a node (optional):
# puppet node find node_name
--node_terminus=exec
--external_nodes=/etc/puppet/nodes.sh
--facts_terminus=yaml
--clientyamldir=/tmp/yaml/ --render-as=yaml
> /tmp/yaml/nodes/node_name.yaml
# puppetconf # puppetize @ puppetlabs

25. Creating a catalog:
# puppet catalog find node_name
--facts_terminus=yaml
# puppet catalog find node_name
--node_terminus=yaml
--manifest=/etc/puppet/manifest/site.pp
--modulepath=/etc/puppet/modules/
--clientyamldir=/tmp/yaml/ > /tmp/catalog.yaml
# puppetconf # puppetize @ puppetlabs

26. Fun with IRB
Puppet::Node::Facts.indirection.find
facts
Puppet::Node.new
nodes
Puppet::Catalog.indirection.find
catalog
# puppetconf # puppetize @ puppetlabs

27. IRB Facts
irb:> require ‘puppet/face’
> facts=Puppet::Face[:facts, :current].find('node’)
# puppetconf # puppetize @ puppetlabs

28. Access a Fact value (irb):
…
> facts.values['ipaddress']
=> "10.0.2.15"
# puppetconf # puppetize @ puppetlabs

29. Creating a node (from irb):
…
> node=Puppet::Node.new('node_name',
{:classes => {:foo => {:bar => :baz}}})
>node.merge(facts.values)
# puppetconf # puppetize @ puppetlabs

30. Creating a catalog:
…
irb> catalog=Puppet::Resource::Catalog.indirection.
find('node_name’, :node => node)
# puppetconf # puppetize @ puppetlabs

31. Use Cases
Interacting with Puppet’s Data

32. Inspecting the catalog:
• What types are in the catalog?
irb> catalog.resources.collect do |r| r.type end.uniq
• Gimme a resource:
irb>catalog.resource(‘Package[httpd]’)
# puppetconf # puppetize @ puppetlabs

33. Rspec Puppet:
let :facts do
{:operatingsystem => ‘Redhat’}
end
let :params do
{:bind_address => ‘0.0.0.0’
end
it { should contain_file(‘/tmp/foo.conf’) }
# puppetconf # puppetize @ puppetlabs

34. Thundering Herd
Pre-compile catalogs for faster auto-scaling
# puppetconf # puppetize @ puppetlabs

35. Applying pre-compiled
catalogs:
puppet apply --catalog /tmp/catalog.json –server
puppet-fileserver
# puppetconf # puppetize @ puppetlabs

36. DMZ
tcp over USB
# puppetconf # puppetize @ puppetlabs

37. Use Cases
Interacting with Puppet’s Data

38. Hacking reports
Everything in Puppet is a state transition
User[‘dan’] : absent -> present
User[‘dan’][‘shell’] -> ‘/sbin/nologin’ -> /bin/bash
# puppetconf # puppetize @ puppetlabs

39. Setting up the agent:
[agent]
report=true
# puppetconf # puppetize @ puppetlabs

40. Archive reports in your
yamldir
[master]
reports = store
# puppetconf # puppetize @ puppetlabs

41. Puppet reports
$ cd `puppet config print reportdir`
$ ls
node1 node2 node3
$ ls node1
# puppetconf # puppetize @ puppetlabs

42. Every report from every run
ever
$ ls node1
201206060256.yaml 201206060303.yaml
201206060519.yaml 201206122349.yaml
201206122354.yaml 201206130002.yaml
# puppetconf # puppetize @ puppetlabs

43. Lets crack one open!
Irb > require ‘yaml’
>reports=YAML.load_file('201206130002.yaml')
# puppetconf # puppetize @ puppetlabs

44. Have a look
>(reports.methods - Object.methods).sort
Notice the following methods:
# puppetconf # puppetize @ puppetlabs

45. High level data
> reports.exit_status
⇒ 0
> reports.status
=> "unchanged"
> reports.host
⇒ ”node1”
# puppetconf # puppetize @ puppetlabs

46. metrics
> reports.metrics.keys
⇒ ["resources", "events", "changes", "time"]
> reports.metrics['resources']
⇒ [‘failed’, 0],[ ‘changed’, ‘7’]
# puppetconf # puppetize @ puppetlabs

47. And the awesome sauce
> reports.resource_statuses.keys
=> ["Package[xinetd]", "File[/srv/node/1]",
"Package[swift]", "Exec[compile fragments]",
"Package[swift-container]", "File[/var/opt/lib/pe-
puppet/concat/_etc_swift_object-server.conf]",
"File[/etc/rsync.d/frag-account]”]
# puppetconf # puppetize @ puppetlabs

48. And the awesome sauce
> status = reports.resource_statuses
> status.keys
=> ["Package[xinetd]", "File[/srv/node/1]",
"Package[swift]", "Exec[compile fragments]",
"Package[swift-container]", "File[/var/opt/lib/pe-
puppet/concat/_etc_swift_object-server.conf]",
"File[/etc/rsync.d/frag-account]”]
# puppetconf # puppetize @ puppetlabs

49. And the awesome sauce
>events = status["File[/etc/swift/swift.conf]"].events
> events.first.status
⇒ "success”
> events.first.desired_value
⇒ :present
> events.first.previous_value
=> :absent
# puppetconf # puppetize @ puppetlabs

50. Thank You
Dan Bode| Puppet Labs
dan@puppetlabs.com