Presented at Open Source 101 2023 - Charlotte
Presented by Marc Boorshtein, Tremolo Security, Inc.
Title: Introduction to Kubernetes Security
Abstract: This 90 minute session will be an overview of Kubernetes security from both an operations perspective and an application development perspective. It won't give you everything you need to lock down a cluster, but will give you enough information to know what to look for and what questions to ask. The first half of the session will focus on cluster security:
* How users access the cluster
* How pipelines access the cluster
* Dashboards
* NetworkPolicies
* Secrets
* Node Security
* Defense-in-Depth
* Multi-tenancy
The second half will focus on building secure containers that run on your clusters:
* How your application can be be a security risk
* Basics of a secure image
* SBOMs
* Farm-to-Table Supply Chain Security
2. https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
Who Am I?
● CTO Tremolo Security
● Identity Management expert for 20+ years
● Experience in commercial and federal agencies
● Kubernetes since 2015
● Co-Author Kubernetes an Enterprise Guide: 2nd Ed
3. Agenda
● Part I - Cluster Security
○ What is Kubernetes?
○ How users access the cluster
○ How pipelines access the cluster
○ Dashboards
○ NetworkPolicies
○ Secrets
○ Node Security
○ Defense-in-Depth
○ Multi-tenancy
https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
4. Agenda
● Part II - Container Security
○ How your application can be be a security risk
○ Basics of a secure image
○ SBOMs
○ Farm-to-Table Supply Chain Security
https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
5. https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
What is Kubernetes?
● Scheduler - Determines what containers should run and where
● Kubelet - Works with the scheduler to run and manage containers
● API Server - API for interacting with the Kubelet and Scheduler
8. https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
How users access the cluster - Certificates
● “Break Glass” authentication
● Can’t be revoked
● Hardware certificates aren’t supported
● External CAs not supported
● Groups only supported through static subject mapping
● Requires point-to-point connectivity between the client and server
9. https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
How users access the cluster - OpenID Connect
● Use of a JSON Web Token (JWT)
● Bearer token
● Easily maps groups
● Should be short lived (1-2 minutes with clock skew)
● Allows for network segmentation
● Can not be revoked
● JWT Demo
10. https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
How users access the cluster - Impersonating Proxy
● Reverse Proxy authenticates users
● Reverse proxy sends headers with
the user’s request to tell the API
server who the user is
● Impersonation proxy is responsible
for authorizing inbound
impersonation
● ServiceAccount for Impersonating
Proxy is a privileged account
● Important to tie API access back to
original request
● Impersonation Demo
11. https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
How users access the cluster - TokenRequest API
● JWT issued by API Server
● Intended for identifying workloads to the API Server
● Intended for identifying workloads to external services
● Not meant for use from outside of the cluster
● Demo of a container talking to the API server
12. https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
How users access the cluster - Custom Webhooks
● If you’re not a cloud provider, don’t do it
13. https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
How users access the cluster - RBAC
● How the API server authorizes access
● Two Scopes
○ Cluster - Objects that impact everyone
○ Namespaced - Objects that are contained inside of a Namespace
■ Namespaces are Cluster scoped
● Roles/ClusterRoles → Define permissions
● RoleBindings/ClusterRoleBindings → Assign permissions
● Rights are enumerated
● No “negative” rights
● Don’t use “*”
● Bindings - Groups, not Users
14. https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
How users access the cluster - RBAC Aggregate
Roles
● Create large Roles & ClusterRoles without maintaining large objects
● Uses labels to assemble a Role/ClusterRole from smaller Roles/ClusterRoles
● Examples are admin, editor
● How to let an admin create a new instance of a CRD
15. https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
How pipelines access the cluster
● Don’t use long lived tokens
● Use a local identity to get short lived
token
● Spire (project) & SPIFFE (standards)
16. https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
Dashboards
● Centralized
○ Can be secured via reverse proxy and impersonation
○ Should never have a privileged identity
○ Rely on the user’s identity
● Local
○ Uses user’s kubectl configuration
○ Opens a local port on loop-back with no TLS or authentication
17. https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
NetworkPolicies
● “Firewalls” of the cluster
● Control both inbound and outbound traffic
● Dependent on CNI implementation
● Not on by default
● Important to add
● Demo of network policy
18. https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
Secrets
● Stored in etcd as base64 encoded string
○ Encoded to preserve binary data
● Threat model your Secrets
○ Plain Kubernetes Secrets are Fine - https://bit.ly/3K1nEcU
● Externalize Secrets
○ Mount directly to Pods
○ Synchronize Secrets
● DON’T EVER STORE SECRETS IN GIT OR HELM, EVEN WHEN
ENCRYPTED
19. https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
Node Security
● Don’t run containers as root
○ Unless you need to run a container as root
○ Init containers
● Limit, or drop all, capabilities
● You don’t need a port under 1024
● Admission Controller Webhooks - Enforce rules
● Mutating Webhooks - Enable sane defaults
● Common Tools
○ Pod Security Standards
○ OPA/GateKeeper
○ Kyverno
○ JSPolicy
20. https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
Defense-in-depth
● Don’t rely on a single layer
● Strong Identity+RBAC+Policy Enforcement
● Multi-factor Authentication
● Short lived tokens
● Self Service
21. https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
Multitenancy
● Better utilization of resources
● Manage sprawl
● “Something” needs to be multitenant
● Self Service
● Combination of all of the above
● Virtual clusters
23. https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
How Your Application Can Be A Security Risk
● ServiceAccount token mounted to your Pod
● RCE from an app can lead to access to your Pod’s tokens
● Lack of sane defaults and policies lead to a container breakout
● Leaking bearer tokens
● Debug tools
○ “Distroless” containers
○ Ephemeral Containers
24. https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
Basics of a Secure Image
● Switch from root
● Assume no Linux userid
● Write only to volumes
○ emptyDir for “scratch space”
● Rebuild often
● Be mindful of where you get base images from
● Take contracts and policies into consideration