Puppet Camp America Central, Sep 24 2020

1. The Dynamic Duo of Puppet and Vault
tame SSL Certificates

2. Nick Maludy
@NickMaludy
github.com/nmaludy Encore Technologies
https://encore.tech
@EncoreTechCincy
github.com/EncoreTechnologies
encoretechnologies.github.io
Director of Development, Husband, Dad
Managed Services Provider

3. Terminology
• Public Key Infrastructure (PKI)
• Certificate Signing Request (CSR)
• Certificate Authority (CA)
• Gate Keeper
• Public Certificate
• Share this with others
• Private Key
• Keep this to yourself
• Signed Certificate
• Public Cert generated from a CSR using crypto magic by a CA

4. Proper SSL Verification Flow
ServerClient
App/Browser
CA
public
Web Server
privpub2. Pub Key
1. Hello
3. Verify PUB KEY
MATCHES
ONE OF
THE CAs
TRUSTED!

5. Self Signed Verification Flow
ServerClient
App/Browser
CA
public
Web Server
privpub2. Pub Key
1. Hello
3. Verify PUB KEY
DOESN’T
MATCH
ANY CAs
NO TRUST!

6. PKI Old School
Root CA
Linux Windows
Root RootPublic Private Public Private
Apache / Nginx IIS
CSR CSR
Public Public
CSR CSR
Manually
Copy
Manually
Copy
Sign Sign
Manually
Copy
Manually
Copy
Manually
Copy
Client Client
Root Root
Root

7. Villains
•Painful signed certs
•Oprah – self signed certs for everyone
•No trust
•Disable validation
•MITM Attacks
•Renewal and Expiration
•Security tickets

8. Call For Help
•Security
• Centrally signed with CA
• Validation enabled
• Strong ciphers
•DevOps
• Auto renewal
• Cross-platform
• Integrated with services

9. encore/vault
•vault_cert resource
•Fork of jsok/vault
•Plan to upstream
•github.com/EncoreTechnologies/puppet-vault
Justice

10. PKI with Puppet + Vault (vault_cert)
Root CA
Vault CAPuppet Master
Root Vault
Sign Intermediate CA
Copy
Copy
Copy
Linux Windows
Root Vault Root VaultPublic Private Public Private
Apache / Nginx IIS
Client
Root Vault
Client
Root Vault

11. Check
Expiration
Check
Revocation
Revoke old Create New
Write to
filesystem
Bounce
service
vault_cert run

12. vault_cert { ‘blah’: }
Linux Simple
vault_cert { ‘nexus’:
regenerate_ttl => 3,
notify => Service['nginx’],
}
nginx::resource::server { 'nexus’:
ssl_port => 443,
ssl => true,
ssl_cert => '/etc/pki/tls/certs/nexus.crt',
ssl_key => '/etc/pki/tls/private/nexus.key’,
Linux Nginx
Linux
Public Private
Nginx
Vault CA

13. Vault::cert (linux)
•Management of file owner and permissions
•Contains:
• vault_cert {}
• file {}
vault::cert { ‘nginx’:
cert_group => ‘nginx’,
cert_owner => ‘nginx’,
cert_mode => ‘0640’,
priv_key_mode => ‘0600’,
notify => Service[‘nginx’],
}

14. Windows problem
• Certs in cert store have a path
• Cert:LocalMachineMy<UNIQUE-THUMBPRINT>
• Cert:LocalMachineMyABC1234
• Thumbprints are unique
• Thumbprints = hash of cert content
• Services bind to cert path
• relies on Thumbprint

15. Windows – Desired solution
THIS DOESN’T WORK
vault_cert { ‘host.domain.tld’: }
iis_binding { ‘chocolatey’:
binding_info => {
certificatehash => Vault_cert[‘host.domain.tld’]['thumbprint'],
},
}

16. Puppet problems
• Facts run before everything else
• Functions run on the server during compilation
• Deferred functions run on the client after facts, but before
the catalog is applied
• Can’t use these to “tie” things together that are created
during a catalog run

17. Windows solution – double run
• Facter
• Looks for existing certs in Cert:
• returns thumbprints
• If cert doesn’t exist, then no thumbprint
• Vault_cert Puppet Resource
• Cert doesn’t exist
• Create the cert
Run #1
• Facter
• Finds the cert create in Run #1
• Returns cert + thumbprint
• Iis Puppet Resource
• Use thumbprint from facts to bind
services (IIS)
Run #2

18. Windows solution – single run
• Facter
• Looks for existing certs in Cert:, returns thumbprints
• vault::cert() Puppet Function
• Checks if cert exists in facts
• Only create if cert does NOT exist
• Calls Vault API, creates a cert
• Returns the cert public key, private key and thumbprint
• Puppet Resources
• vault_cert – Given public private keys , writes the cert to Cert:
• iis_binding – Uses thumbprint from function call for binding
Run #1

19. $cert_details = vault::cert(...args...)
vault_cert { ‘blah’:
cert => $cert_details['cert’],
priv_key => $cert_details['priv_key’],
}
iis_binding { ‘chocolatey’:
binding_info => {
certificatehash => $cert_details['thumbprint'],
},
}
Single run solution Vault CA
Windows
Public Private
IIS

20. Windows “machine cert”
profile
class profile::machine_cert {
$cert_details = vault::cert(args)
vault_cert { $trusted['certname’]:
common_name => $trusted['certname’],
cert => $cert_details['cert’],
priv_key => $cert_details['priv_key’],
}
}
class { ‘winrm’:
certificate_hash => $profile::machine_cert::cert_details['thumbprint'],
}

21. CA Certs on Linux
class profile::ca (Hash $certs) {
class { 'trusted_ca': }
create_resources('trusted_ca::ca’, $certs)
}
profile::ca::certs:
myca.domain.tld:
content: |
-----BEGIN CERTIFICATE-----
xxx
-----END CERTIFICATE-----
Hiera
Puppet Master
Root Vault
Linux
Root Vault
puppet/trusted_ca

22. CA Certs on Windows
file { 'C:/ProgramData/Puppetlabs/ca_certs':
ensure => directory,
}
# root certs go into Cert:/LocalMachine/Root
$certs.each |$name, $data| {
file { "C:/ProgramData/Puppetlabs/ca_certs/${name}.crt":
ensure => file,
content => $data['content'],
}
$cert_details = vault::cert_details($data['content'])
sslcertificate { "${name}.crt":
location => 'C:ProgramDataPuppetlabsca_certs',
thumbprint => $cert_details['thumbprint'],
store_dir => 'Root',
interstore => true,
}
Puppet Master
Root Vault
Windows
Root Vault
puppet/sslcertificate

23. Puppet + Vault = Dynamic Duo
•Every server has a cert
•CA distributed
•Services bound to certs
•Certs auto-renew
•Services auto-refreshed
•Validation enabled

24. Future
•Monitoring-as-Code
•CMDB Auto-Update

25. Thanks!
@NickMaludy
github.com/nmaludy
@EncoreTechCincy
github.com/EncoreTechnologies
github.com/EncoreTechnologies/puppet-vault
slack.puppet.com
#puppet-camps
@nmaludy