Here are the slides from Peter Souter's PuppetConf 2016 presentation called Nice and Secure: Good OpSec Hygiene With Puppet!. Watch the videos at https://www.youtube.com/playlist?list=PLV86BgbREluVjwwt-9UL8u2Uy8xnzpIqa
4. @petersouter
My feelings on Q&A
http://bit.ly/why_no_talk_qa
● Tweet me @petersouter
● Come up after this talk
● Meet me in the hallway
4
When will this QA
be over so I can
leave?
We’ve got lots to cover - No Q&A!
11. @petersouter
“Operations Security, or OPSEC, is the process by which
we protect unclassified information that can be used
against us. OPSEC challenges us to look at ourselves
through the eyes of an adversary (individuals, groups,
countries, organizations). Essentially, anyone who can
harm people, resources, or mission is an adversary.”
11
Department of Defense Education Activity
http://www.dodea.edu/offices/safety/opsec.cfm
12. @petersouter 12
● Keeping your code clear of
sensitive information
● Approaches to secrets
management with the Puppet
toolchain
● Making sure security is part of
your workflow, rather than an
afterthought
What are we going to cover?
https://flic.kr/p/7LcF2W
14. @petersouter 14
What are secrets in IT?
Radioactive
Consequences are
dire from a leak
Examples
Passwords, API
Keys, SSH Keys,
SSL Certs...
Small
A few kb at most
Required
The infrastructure
won't work without
them!
https://flic.kr/p/dHrwpb
15. @petersouter
Easiest to hardest
● Avoid exposing secrets in logs
● Remove data from code and into
the data layer (hiera)
● Encryption
15
How do we avoid exposing secrets in Puppet?
https://flic.kr/p/aCJZrf
18. @petersouter 18
root@homebox:~# puppet agent --show_diff
Notice: Compiled catalog for homebox.home in environment production in 0.10 seconds
Notice: /Stage[main]/Main/File[/etc/sensitive]/content:
--- /etc/sensitive 2016-08-14 23:01:37.036863915 +0100
+++ /tmp/puppet-file20160814-24654-ak1ywd 2016-08-14 23:01:56.852882307 +0100
@@ -1 +1 @@
-Not Secret
No newline at end of file
+SECRET-CONTENT
No newline at end of file
Notice: /Stage[main]/Main/File[/etc/sensitive]/content: content changed
'{md5}2ab96390c7dbe3439de74d0c9b0b1767' to '{md5}44c7be48226ebad5dca8216674cad62b'
Notice: Applied catalog in 0.20 seconds
How it looks...
19. @petersouter
Anywhere reports go:
● syslog
● interactive terminal output
● PE Console
● ENC
● report processors
19
Where does the information from show_diff go?
20. @petersouter 20
file { ‘/etc/secrets.txt’:
ensure => 'file',
owner => 'root',
mode => '0600',
content => 'hunter2',
show_diff => false,
}
Setting show_diff to false at the resource level
21. @petersouter 21
An example from a Supported Module: mysql
file { "${::root_home}/.my.cnf":
content => template('mysql/my.cnf.pass.erb'),
owner => 'root',
mode => '0600',
}
# show_diff was added with puppet 3.0
if versioncmp($::puppetversion, '3.0') >= 0 {
File["${::root_home}/.my.cnf"] { show_diff => false }
}
https://github.com/puppetlabs/puppetlabs-mysql/blob/d58a100fa67bc99b4388d4ea3921b11647d483d7/manifests/server/root_password.pp#L39
22. @petersouter
Setting show_diff to false at resource scope
show_diff = false
22
root@homebox:~# puppet apply secret.pp
Notice: Compiled catalog for homebox.home in environment production in 0.10 seconds
Notice: /Stage[main]/Main/File[/etc/sensitive]/content: content changed '{md5}d3b07384d113edec49eaa6238ad5ff00' to
'{md5}44c7be48226ebad5dca8216674cad62b'
Notice: Applied catalog in 0.19 seconds
33. @petersouter 33https://github.com/binford2k/binford2k-node_encrypt
● Master encrypts secrets for each node using their own certificate
● Secret can only be decrypted with the node's private key
● Uses built-in Puppet CA, so the base case is zero-config
node_encrypt::file {'/etc/company_app/credentials':
ensure => file,
owner => 'root',
content => 'hunter2', # transparently encrypted
}
How does it work?
34. @petersouter
node_encrypt
34
$ puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for master.puppetlabs.vm
Info: Applying configuration version '1450109738'
Notice: /Stage[main]/Main/Node[default]/Node_encrypt::File[/tmp/foo]/Node_encrypted_file[/tmp/foo]/ensure: created
Notice: Applied catalog in 9.33 seconds
$ echo blah > /tmp/foo
$ puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for master.puppetlabs.vm
Info: Applying configuration version '1450109821'
Notice: /Stage[main]/Main/Node[default]/Node_encrypt::File[/tmp/foo]/Node_encrypted_file[/tmp/foo]/content: content changed '<<encrypted>>' to
'<<encrypted>>'
Notice: Applied catalog in 7.61 seconds
35. @petersouter
So data is no longer exposed in logs
But the data is still visible in the code!
35
42. @petersouter
Allows you to set organisational defaults in your
roles and profiles
42
● Keep organisational specific data in hiera
● Move organisational specific setup into role and profile wrappers
Advantage:
Not only more secure, cleaner code that’s more reusable!
55. @petersouter 55
This is actually a tenet of
12 Factor Apps...
Apps sometimes store config as constants in the code. This is a violation of
twelve-factor, which requires strict separation of config from code. Config varies
substantially across deploys, code does not.
A litmus test for whether an app has all config correctly factored out of the
code is whether the codebase could be made open source at any moment,
without compromising any credentials.
Note that this definition of “config” does not include internal application config, such
as config/routes.rb in Rails, or how code modules are connected in Spring. This
type of config does not vary between deploys, and so is best done in the code.
http://12factor.net/config
57. @petersouter 57
Meeting the Digital Service Standard
To meet point 8 (understand security and privacy issues) you must:
● Make all new source code open and reusable
● Publish code under an appropriate licence
● Explain your reasoning for any code you haven’t made open
You’ll have to explain how you did this at your service assessments.
https://www.gov.uk/service-manual/technology/making-source-code-open-and-reusable
58. @petersouter 58
Meeting the Digital Service Standard
When GOV.UK was first set up we were unable to publish our Puppet
repository because our code and secrets were tied together. This goes
against patterns like the 12-factor app which “requires strict separation
of config from code”
This wasn’t true for our Puppet repository, but we gradually moved our
credentials into a separate repository (rotating them as we did so).
“A litmus test for whether an app has all config correctly factored out of the
code is whether the codebase could be made open source at any moment,
without compromising any credentials.”
59. @petersouter 59
$ strings modules/**/*.pp | tr ' '
'n' | sort -n | uniq | view -
Check code for unique strings that look secret-y
Note: Requires zsh for the strings function!
73. @petersouter
The idea is that Puppet will natively
support encrypted data in the future
Follow this ticket for the roadmap view: PUP-1974
73
74. @petersouter
● Transcrypt
Git-Crypt
Blackbox
● Turtles All The Way Down:
Storing Secrets in the Cloud
and in the Data Center
behind Closed Doors
VCS based encryption
74
http://danielsomerfield.github.io/turtles
https://www.youtube.com/watch?v=OUSvv2maMYI
78. @petersouter 78
Hiera will plug into any secret service
app with a little bit of Ruby glue
Hiera is just key/value lookup
A hiera backend to basically any secret server setup is possible
81. @petersouter
If you want to know more about
Puppet + Vault, Seth Vargo from
Hashicorp is presenting tomorrow
81
82. @petersouter
Behind Closed Doors - Managing Passwords
in a Dangerous World by Noah Kantrowitz
● Really great in-depth presentation
● https://coderanger.net/talks/secrets/
● https://www.youtube.com/watch?v=TVEfY
O-5-RA
● Great breakdown of secret management,
advantages and disadvantages of approaches
and tooling
Want to know more about secrets?
82
94. @petersouter 94
“To keep up with the pace of Continuous Delivery, security must “shift
left,” earlier into design and coding and into the automated test cycles,
instead of waiting until the system is designed and built and then trying to
fit some security checks just before release. In DevOps, security must fit
into the way that engineers think and work: more iterative and
incremental, and automated in ways that are efficient, repeatable, and
easy to use.”
- DevOpsSec: Delivering Secure Software Through Continuous Delivery, Jim Bird
Shifting left!
95. @petersouter 95
How do we check things aren’t getting worse?
● Game days and internal evil attempt teams
● Continuous security integration (Gittyleaks/code-review)
● Dedicated security stories for sprints
○ Evil users or (mis)use cases
● Embedded security team members
● Dedicated audits on sensitive apps/stacks from external firms
96. @petersouter 96
Game Day example: Agent spoofing
Let's say someone gets access to an agent.
What’s the worst they can do?
97. @petersouter
As soon as security becomes a blocker,
you’ve lost!
Security has to be automated where possible, otherwise
we’re back to the throw-over-the-wall problems of
pre-DevOps!
97
100. @petersouter
Use the roles and profiles pattern for
original defaults
Separate secrets, reduce the surface area for problematic setups and always
read the docs before using Puppet modules
100