ContainerDays Boston 2016: "Hiding in Plain Sight: Managing Secrets in a Container Environment" (Jeff Mitchell)

Managing Secrets in
Container Environments
Jeff Mitchell
HashiCorp
@jefferai

About Me
● Project Lead on Vault
○ A HashiCorp tool for managing secrets
○ https://vaultproject.io
● Previously, Architect at Akamai
○ Heavy focus on containerization and managing container security

What this talk is...
● A discussion of paradigms and considerations necessary
for managing secrets in container environments
● Applied security, not crypto theory
○ Goal: “Know what you need to know” when figuring out your
organization’s security policy

...and what this talk isn’t
● One True Answer™
○ Support requirements impact available technology/workflows
○ Each organization must set their own security policy and
implement practices that satisfy it
● A discussion of container/hypervisor/OS security
○ Let’s talk Linux containerization

...and what this talk isn’t
● Not a deep dive into security rabbit holes
(3-4 turtles max)

Scheduler
Secret Management Tool
Scheduler Agent
Container
Problem Space

Bear with me for four slides while I wax
philosophical…

What is “Security”?
● Security is the practice of risk management
○ Accepting some risks, guarding against violations of norms
● Anything that can elevate risk is a threat
○ Modeling threats informs security policy
● Risk increases with system complexity
○ More points of failure, confusion, and ingress = harder to secure

What is a “Secret”?
● A secret is something that will elevate your risk if exposed
to unauthorized entities
○ Undesired consequences are harm
○ Harm may come in many forms
■ Unauthorized data access
■ Identity spoofing
■ Private data egress
■ Regulatory fines
● An exposed secret is a threat

Secrets vs. Identifiers
● Not all things that can be disclosed are secrets, some are
identifiers
○ Username (identifier) vs. Password (secret)
○ TLS certificate (identifier) vs. TLS Key (secret)
○ GitHub user name (identifier) vs. API key (secret)
● Identifiers aren’t completely risk-free
○ Have chosen to ignore that risk

What is “Trust”?
● A trusted entity is one that will not divulge the secrets it
has access to
● Modeling trusted entities is companion to modeling threats
● Two concepts:
○ Circle of trust
○ Chain of trust

Circle of Trust
Entities we trust with any secret. For this talk:
RAM
root
Secret
Management
ToolEmployees?
CPU
Cloud?

Circle of Trust
Only allowed long-term storage is in circle of trust
RAM
root
Secret
Management
Tool
Persistent
Storage
Random
Users
General
apps/services
NSA
Random Wi-
Fi Hotspot
Your Mother’s
Notepad.txt Employees?
CPU
Cloud?

Chain of Trust
● The set of links (e.g. network hops) that any particular
secret travels through from entity A to entity B
○ Source/destination must be in circle of trust
● Any link is an access point/interception point
○ Accidental logging
○ Exploitation by attacker
○ Lookup by operator
○ Employee Post-It™
○ Conspiracy of One/Compromised employee

Problem Space
● We can now establish the problem space:
○ Managing secrets in an environment means establishing trust
chains in the environment
○ Links in the chains have associated risk
■ Minimize hops and minimize risk-per-link
■ Risk cannot be fully mitigated; must assume any given secret will
eventually be divulged
● Ultimate goal: zero trust
○ Don’t give the opportunity for risks to occur in the first place

Scheduler
Scheduler Agent
Container
Goal: Securely
move secret from
originator to new
container’s RAM

Scheduler
Scheduler Agent
Container
?

Secret Protection
● Establishing a train of trust requires defining the
requirements it must fulfill to keep secrets protected
● Good news: we only need to do this for one secret!

Secure Introduction
● First secret authenticates us to allow direct access to
more
● If you can protect this secret, you can protect any secret
○ (Generally)
● This concept is secure introduction (SI)

Secret Protection
● How do you protect secrets (perform SI)?
○ First establish success criteria based on acceptable risk
● For this talk:
○ Don’t let them live forever (rotate/expire)
○ Distribute them securely
○ Limit exposure if disclosed
○ Have a break-glass procedure
○ Detect unauthorized access

Rotation
● As lifetime increases, chance for exposure → ∞
○ Caches/logs
○ Cracked over time/enough packets
○ Debugging
● Secrets should be rotated “frequently”
○ User passwords vs. machine secrets
■ xkcd’s ______ ______ ______ ______ → bad policies + frequent
rotation = written down user passwords
■ Less frequently/more used = more likely overseen
■ Machines: it’s just data - plan/build for rotation and rotate often

Distribution
● The literal movement along the chain of trust
○ To/From
■ People
■ Machines
● Base level: never plaintext, always covered
○ Encryption
○ Wrapping
○ Etc.

Limit Exposure
● Principle of least privilege
○ DB credentials: only specific tables/operations
○ Login credentials: not root
○ API credentials: minimal function set

Access Detection
● Things have a way of being leaky
● Env vars: common way to pass in secrets. Also:
○ Often logged, sometimes multiple places
○ Easily discoverable by operator
■ Both Docker commands and non-Docker commands can spill the
beans

Access Detection
● Equally as important as protecting a secret is knowing if
an unintended party has intercepted it
● Audit logs are great…
○ ...but do you look at them?
● Active detection (when possible) is even better

Break-Glass
● Compromised?
○ Stop all further access to protected resources
○ Perform forensics
○ Rotate all secrets after re-establishing identities of legitimate
secret-holders
○ …
● Figure this out during the planning process -- not after!

● So...this is complicated…
● ...but there’s good news!

● Recent explosion of FOSS Secret Management (SM) tools
○ Vault
○ KeyWhiz
○ Knox
○ Conjur (Docker only)
○ Many more (apologies if I missed your favorite)...
Secret Management Tools

● If you only take away two things from this talk, make sure
they are the following:

1. Write your own crypto

2. Use it in a Secret Management tool

2. Use it in a Secret Management tool
NO!
NO!

1. Use a Secret Management tool
2. Don’t roll your own

Secret Management Tools
● Why?
○ Central, secure storage (avoid sprawl)
○ Central management
○ Codified, secure access mechanisms
○ Centralized auditing
■ Ensure knowledge of what secrets have been seen by what
users/services
○ Secret rotation/revocation/expiration

Big Caveat
● This isn’t a vendor talk, but I have a bias
● I don’t have exhaustive knowledge of non-Vault SMs
● Anything I say a SM can do can be done with Vault, and
may or may not be possible with other solutions
○ They may absolutely do things Vault can’t

SM + SI
● Explicit focus on the secure introduction problem
○ Core competency from ground up
○ Necessary tools/capabilities/primitives
● Containers can support existing SI paradigms
○ If you can securely get credentials to it, but imagine…
■ Managing 100k LDAP/AD users churning at high rate
■ Generating/dropping 100k Kerberos keytabs
○ ...and you’ve solved SI anyways

SM + SI
● Platform agnostic (with platform-based enhancements)
○ Change authentication mechanism “front-end”
■ Containers
■ EC2
■ Bare metal
■ Users

SMs <3 Schedulers
● Anyone working with containers at scale uses a scheduler
○ Nomad
○ Mesos
○ Fleet
○ Swarm
○ Kubernetes (All-In-One)

SMs <3 Schedulers
● Schedulers are sources of truth and provide hooks
● SM tools and schedulers can be a magical combination
○ May not be direct integration, glue code needed
○ OTOH avoiding tight coupling means better reuse across
environments

Scheduler
Scheduler Agent
Container

Scheduler
Scheduler Agent
Container
SI #1 (Traditional)

Scheduler
Scheduler Agent
Container
SI #1 (Traditional)
Uses: ∞
TTL: 8h (renewable)
Policy: create_app_tokens

Scheduler
Scheduler Agent
Container
SI #2 (New)
Uses: ∞
TTL: 8h (renewable)

Preconditions
● Suppose a SM authentication token has:
○ Unlimited or limited use-count
○ Limited time-to-live (TTL), possibly renewable
○ Set of authorization policies
■ e.g. use first secret (auth token) to get application secrets
○ Consistent ID in audit logs
○ Token-scoped secure storage tied to token lifetime
● Claim: These primitives allow excellent SI for containers

Scheduler
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)
Uses: ∞
TTL: 8h (renewable)
Container

Scheduler
App: db_writer
Security Policy: app_db_rw_pol
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)
Uses: ∞
TTL: 8h (renewable)
Container

Scheduler
App: db_writer
Create Token:
Policy: app_db_rw_pol
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)
Uses: ∞
TTL: 8h (renewable)
Container

Scheduler
Create Token:
Uses: 1
TTL: 30s (non-renewable)
Policy: none
Private storage:
Uses: ∞
TTL: 1h (renewable)
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)
App: db_writer
Uses: ∞
TTL: 8h (renewable)
Container

Scheduler
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)
ContainerImage: db_writer
TOKEN: (outer token)
Uses: 1
Policy: none
Private storage:
Uses: ∞
TTL: 1h (renewable)
Uses: ∞
TTL: 8h (renewable)

Scheduler
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)
Container
Uses: 1
Policy: none
Private storage:
Uses: ∞
TTL: 1h (renewable)
Uses: ∞
TTL: 8h (renewable)
Unwrap

Scheduler
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)
Uses: ∞
TTL: 8h (renewable)
Container
Uses: ∞
TTL: 1h (renewable)

Scheduler
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)
Container
Uses: ∞
TTL: 1h (renewable)
Uses: ∞
TTL: 8h (renewable)
Get DB
Creds

Scheduler
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)
Uses: ∞
TTL: 8h (renewable)
User: db_user
Password: db_pass
Expiration: 24h
Container

Scheduler
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)
Container
Uses: ∞
TTL: 1h (renewable)
Uses: ∞
TTL: 8h (renewable)
Get S3
Creds

Scheduler
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)
Uses: ∞
TTL: 8h (renewable)
ACCESS_KEY: ...
SECRET_KEY: ...
Expiration: 24h
Container

Scheduler
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)
Container
Uses: ∞
TTL: 1h (renewable)
Uses: ∞
TTL: 8h (renewable)
Get TLS
Cert

Scheduler
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)
Uses: ∞
TTL: 8h (renewable)
Certificate: ...
Private Key: ...
Issuing CA: ...
Container

How’d we do?
● Look at previous list:
○ Don’t let them live forever (rotate/expire)
✓ Outer token expires (use limit) with only copy of inner token value
○ Distribute them securely
✓ Inner token covered the entire way
○ Limit exposure if disclosed
✓ Only specific policies granted, can only access specific secrets
○ Have a break-glass procedure
✓ Can lock down access at SM tool
✓ Audit logs ensure we know area of exposure
○ Detect unauthorized access

How’d we do?
● Can detect unauthorized access of “real” authentication token due to
time and use limit
● Application:
○ Reads inner token: success
○ Reads storage but no inner token found: log error and recover (e.
g. fail job)
○ Denied access from invalid token: raise high-priority security alert
and check audit logs:
■ Access of outer token storage in audit logs? Unauthorized use
■ No access? Probably timing issue, but investigate carefully

How’d we do?
● Mechanism not container-specific
○ CM tool drops wrapping/outer token on EC2 instance
○ File injected into chroot
○ Etc.

Other Paradigms
● Remember, security is risk management
● Depending on your tech stack:
○ Different primitives might be available
○ May be constrained
● Never forget your success criteria

Wrap-Up
● Plan your risk tolerance/security policy/success criteria in
advance
○ Do it now!
● Use a secret management tool
○ You’re smart, but it’s hard
● Schedulers provide perfect integration points

ContainerDays Boston 2016: "Hiding in Plain Sight: Managing Secrets in a Container Environment" (Jeff Mitchell)

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Destacado

Destacado (6)

Similar a ContainerDays Boston 2016: "Hiding in Plain Sight: Managing Secrets in a Container Environment" (Jeff Mitchell)

Similar a ContainerDays Boston 2016: "Hiding in Plain Sight: Managing Secrets in a Container Environment" (Jeff Mitchell) (20)

Más de DynamicInfraDays

Más de DynamicInfraDays (16)

Último

Último (20)

ContainerDays Boston 2016: "Hiding in Plain Sight: Managing Secrets in a Container Environment" (Jeff Mitchell)