"Safely Storing Secrets and Credentials in Git for use by Puppet: The BlackBox Project" presented by Thomas A. Limoncelli, Stack Exchange at Puppet Camp NYC 2014

1. The BlackBox project
Safely storing secrets and credentials in Git
for use by Puppet
Tom Limoncelli, SRE, StackExchange.com
Blog: EverythingSysadmin.com

2. 125+ Q&A Communities
ServerFault.com
StackOverflow.com
(We <3 Puppet!)
StackExchange.com

3. What are secrets?
Anything you don’t want exposed externally.
● SSL Certificates (the private bits)
● Passwords
● API keys

4. Puppet manages secrets

6. If you store
secrets in git,
you’re gonna
have a bad
time.

7. ● Laptops get stolen.
● Workstations have guest accounts
● “Circle of Trust” now includes:
○ Everyone with admin access to workstations.
■ Your desktop support people?
○ Everyone with admin access to your git server:
■ Server team, storage team, backup team
○ Everyone you collaborate with that wants read-only
access to Puppet manifests.

8. You have 3 bad options:
1. Deny git access. (Hurts collaboration)
2. Permit git access. (Hurts security)
3. Email individual files. (Hurts… just hurts)

9. Option 4: Encrypt secret parts
● If a file contains secrets, encrypt before
checking into Git.
● Need to edit a secret?
○ Decrypt - Edit - Encrypt

10. What about Puppet master?
● After “git pull”, decrypt all files.
○ Automate this as part of CI.
● Files are unencrypted “at rest”.
● This does not decrease security:
○ No worse than what we were doing before.
○ If you can break into root or puppet on the master,
you’ve already won.

11. Easy, right?
Decrypt:
gpg -q --decrypt -o secret.crt secret.crt.gpg
Encrypt:
gpg --yes --trust-model=always --encrypt
-o secret.crt.gpg $(<keynames) secret.crt

12. Easy, right?
Decrypt:
gpg -q --decrypt -o secret.crt secret.crt.gpg
Encrypt:
gpg --yes --trust-model=always --encrypt
-o secret.crt.gpg $(<keynames) secret.crt
● ...and don’t make any typos when entering the command
● ...and don't accidentally check in the unencrypted version

13. Security is 1% technology plus 99% following
the procedures correctly.
Any process with more than 1 step probably
won't be followed consistently most of the time.
Related reading: "Why Johnny Can't Encrypt: A Usability Evaluation of PGP
5.0”, Alma Whitten", Usenix Security 1999

14. Therefore…. we automate
Introducing: Blackbox
Scripts for keeping Puppet secrets in git/hg.

15. User commands:
Decrypt for editing:
blackbox_edit_start.sh file
Encrypt when done:
blackbox_edit_end.sh file

16. First time a file is encrypted:
Enroll a file into the system:
blackbox_register_new_file.sh file

17. Commands that act on all GPG files:
Decrypt all files: (for use on puppet master)
blackbox_postdeploy.sh
Re-encrypt all files: (after new users added)
blackbox_update_all_files.sh

18. Everyone has their own key
This doesn’t use “symmetric encryption” where
there is one passphrase to decrypt/encrypt all
files.
We maintain a keyring of:
● Each person that should have access.
● A key for the Puppet master.

19. Indoctrinate a new user:
1. New user does this:
● Create GPG key.
● Add their username@host to blackbox-admins.
txt
● git commit -a
(Currently a doc, not a script. Patches gladly accepted.)

20. Indoctrinate a new user:
2. Existing admin does this:
$ gpg --import keyrings/live/pubring.gpg
$ blackbox_update_all_files.sh
$ git commit -a

21. Demo: Edit a file

22. Demo: Edit a file

23. Demo: Edit a file

24. Demo: Edit a file

25. Demo: Edit a file

26. Demo: Edit a file

27. Demo: Edit a file

30. Code is open source as of TODAY
● Entirely written in bash.
● MIT License.
● Download it now:
○ https://github.com/StackExchange/blackbox

31. In the project’s first 9 months:
StackExchange/ServerFault has eliminated
plaintext secrets in our Puppet git repo.
● 7 SREs+Devs sharing the repo securely.
● 50+ files now stored encrypted.
○ Mostly SSL certs and SSH private keys.
● 40+ individual passwords/API keys:
○ Everything from SNMP communities, SaaS API
keys, and many many passwords.

32. Future plans
❏ Open source scripts.
❏ More usability enhancements.
❏ Better setup documentation.

33. Join the open source project
http://github.com/StackExchange/blackbox

34. Q&A
URLs from this talk:
https://github.com/StackExchange/blackbox
EverythingSysadmin.com

35. Shameless plug
Pre-order now! Save 35%
Ships in September.
informit.com/TPOSA
Discount code TPOSA35
Read “rough cuts” today:
safaribooksonline.com

36. Q&A
URLs from this talk:
https://github.com/StackExchange/blackbox
EverythingSysadmin.com
informit.com/TPOSA (code TPOSA35)

37. ● Easier transition. No Puppet code changes
for big files like SSL certs.
● Faster. Zero run-time performance impact
on master.
● eyaml didn’t exist when we started.
Why didn’t we use eyaml?