Security in a containerized world - Jessie Frazelle

#ContainerDayFR
Security in a Containerized World

#ContainerDayFRParis Container Day 2017
Jessie Frazelle
Software Engineer
Googler.
I have contributed to many open source projects
including Docker, Go, Kubernetes, Runc, & the
Linux kernel.
Focus on runtime security for containers.
2

A brief history of containers and Paris...
3

A brief history of containers and Paris...
Paris is the true home of containers.
4

Paris Container Day 2017 #ContainerDayFR
5

Paris Container Day 2017 #ContainerDayFR
6

#ContainerDayFR
7
Security Models

Security Model: Stages
8
Single tenant,
multi-identities
Multi-user, but no hard
enforcement. Still have to
treat as one trust boundary
Cooperative soft
multi-tenancy
Multi-user, fine grained
authorization. Possibly not
fully-hardened
Often good enough for
multi-tenancy inside single
company (can fire bad
actors)
Hard multi-tenancy
Multi-tenant security
boundaries more strongly
enforced
Better resource isolation
E.g. comfortable running
code from multiple
third-parties on the same
cluster
Kubernetes 1.6+ Long term goal

Project/Cluster as Boundary
9
Authorization permissions
granted at project/cluster level
All nodes have same
authenticated identity
All pods have same
authorization permissions
All pods have full network access
my-project/my-cluster
node-1 node-2 node-3
team1-fe team1-fe
team1-db team1-db
team2-fe team2-fe
team2-db team2-db

Namespace/Pod as a Boundary
10
my-cluster
Authorization permissions
granted at
namespace/resource level
Nodes have individual
identities w/ per-node
permissions
Pods have identity with
fine-grained permissions
Pods network access can be
limited to what’s necessary
team1-fe team1-fe
team1-db team1-db
team2-fe team2-fe
team2-db team2-db
team1-namespace
team2-namespace

11
Authorization permissions granted at
Nodes have individual identities w/
per-node permissions
Pods have identity with fine-grained
permissions
Pods network access can be limited to
what’s necessary
Per-namespace/per-resource permissions
Move from: Everything has root-in-cluster.
To: Users, system have least privilege.
Examples:
● Alice can list Eng services, but not HR
● Bob can create Pods in Test namespace, not Prod
● Scheduler can read Pods but not Secrets

12
permissions
what’s necessary
Per-node identity & permissions
Move from: All nodes have vast permissions.
To: Each node has least privilege.
Examples:
● Node can only get info about Pods scheduled on it
● Compromised node doesn’t allow additional
escalation through Kubernetes API

13
permissions
what’s necessary
Pod identity & permissions
Move from: Running workloads have full system
access by default.
To: Workloads must be granted permissions.
Example:
● Running workload can list other objects in its
namespace, but not outside of it

14
permissions
what’s necessary
Network policy
Move From: All workloads receive traffic from
anywhere on the network.
To: Network connectivity is controllable by policy.
Example:
● Frontend layer can only communicate with
application layer, not other frontends.

#ContainerDayFR
15
Runtime Security

What is a sandbox?
16

What is a sandbox?
Provides a net reduction in attack surface.
17

What is a sandbox?
Compare code execution from inside and outside the
sandbox...
18

What is a sandbox?
Compare code execution from inside and outside the
sandbox…
Are more or less things possible now?
19

Chrome Sandbox
20

Chrome Sandbox
Each tab gets its own pid namespace.
21

Chrome Sandbox
What’s a pid namespace?
22

Chrome Sandbox
What’s a pid namespace?
Used to isolate the process ID number space.
23

Chrome Sandbox
Uses
unprivileged user namespaces
and
network namespaces.
24

Chrome Sandbox
Uses Seccomp-BPF
25

What is Seccomp?
SECure COMPuting with filters.
26

What is Seccomp?
SECure COMPuting with filters.
Allows developers to write BPF programs that
determine whether a given system call will be
allowed or not.
27

What is Seccomp?
What is BPF?
28

What is Seccomp?
What is BPF?
Berkeley Packet Filter
In-kernel bytecode machine that is used for tracing,
virtual networks, seccomp… and more.
29

Let’s apply these same principles
30
to containers...

What is a container?
Control what a process can see.
● PID
● Mount
● Network
● UTS
● IPC
● User
● Cgroup
31
Namespaces

What is a container?
Control what a process can use.
● Memory
● CPU
● Blkio
● Cpuacct
● Cpuset
● Devices
● Net_prio
● Freezer
32
Cgroups

Linux Security Module can control and audit various
process actions such as file (read, write, execute, etc)
and system functions
(mount, network tcp, etc)
AppArmor
33
AppArmor

AppArmor
Sane defaults
- Preventing writing to /proc/{num}, /proc/sys,
/sys
- Preventing mount
34
in Docker

AppArmor
Getting towards sane defaults
- Preventing writing to
/proc/{num}, /proc/sys, /sys
- Preventing mount
35
in Kubernetes
apiVersion: v1
kind: Pod
metadata:
name: hello-nginx
annotations:
container.apparmor.security.beta.kuberne
tes.io/nginx: runtime/default
spec:
containers:
- name: nginx
image: nginx
command: ["nginx", "-g", "daemon off;"]

Syscall filters allow an application to define what
syscalls it allows or denies.
Seccomp
36
AppArmor
Seccomp

Seccomp
Docker's default seccomp profile is a whitelist
which specifies the calls that are allowed.
37
in Docker

Seccomp
Docker's default seccomp profile is a whitelist
which specifies the calls that are allowed.
It blocks a bunch of bad stuff… not limited to the
following...
38
in Docker

Seccomp
add_key, keyctl, request_key
Prevent containers from using the kernel keyring,
which is not namespaced.
39
in Docker

Seccomp
clone, unshare
Deny cloning/unsharing new namespaces.
Also gated by CAP_SYS_ADMIN for CLONE_*
flags,
except CLONE_USERNS, which has a history of
vulns
40
in Docker

Seccomp
Pushing towards sane defaults
- Whitelist
- Prevent cloning new unprivileged
user namespaces (has a high rate
of past CVEs)
- Prevent ~150 other syscalls
which are uncommon or
dangerous
41
in Kubernetes
apiVersion: v1
kind: Pod
metadata:
name: hello-nginx
annotations:
container.seccomp.security.alpha.kuberne
tes.io/nginx: docker/default
spec:
containers:
- name: nginx
image: nginx

Provides a mechanism for supporting access control
security policies, including mandatory access
controls.
Controls over file systems, directories, files, and open
file descriptors.
Controls individual labels and controls for kernel
objects and services.
SELinux
42
AppArmor
Seccomp
SELinux

SELinux
Labeling systems like SELinux require that proper
labels are placed on volume content mounted into
a container.
Without a label, the security system might prevent
the processes running inside the container from
using the content.
Allows relabeling file objects on Docker volumes.
43
in Docker

SELinux
SELinux Policy for docker daemon.
44
in Docker

SELinux
Apply SELinux labels to volumes.
45
in Kubernetes
apiVersion: v1
kind: Pod
metadata:
name: hello-nginx
spec:
containers:
- name: nginx
image: nginx
securityContext:
capabilities:
drop:
- NET_RAW
volumeMounts:
...
securityContext:
seLinuxOptions:
level: "s0:c123,c456"
readOnlyRootFilesystem: true
runAsNonRoot: true
volumes: ...

A Linux flag (no_new_privs) that’s carried over `fork`,
`clone`, and `execve` to prevent new privileges from
being added to a process.
No New Privileges
46
AppArmor
Seccomp
SELinux
No new
privs

No New Privileges
Not applied by default unless set by the Docker
daemon.
47
in Docker

No New Privileges
On by default for all containers
without breaking setuid binaries.
48
in Kubernetes

RunAsNonRoot: Set containers to only run as a
non-root user
Security Context
49
in Kubernetes

non-root user
ReadOnlyRootFilesystem: Set container
filesystem as read-only
Security Context
50
in Kubernetes

non-root user
ReadOnlyRootFilesystem: Set container
filesystem as read-only
Capabilities: Set containers to run with specific
Capabilities
Security Context
51
in Kubernetes

RunAsNonRoot
Set containers to only run as a
non-root user
52
Security Context in Kubernetes
apiVersion: v1
kind: Pod
metadata:
name: hello-nginx
spec:
containers:
- name: nginx
image: nginx
securityContext:
capabilities:
drop:
- NET_RAW
volumeMounts:
...
securityContext:
seLinuxOptions:
level: "s0:c123,c456"
runAsNonRoot: true
volumes: ...

ReadOnlyRootFilesystem
Set container filesystem as read-only
53
apiVersion: v1
kind: Pod
metadata:
name: hello-nginx
spec:
containers:
- name: nginx
image: nginx
securityContext:
capabilities:
drop:
- NET_RAW
volumeMounts:
...
securityContext:
seLinuxOptions:
level: "s0:c123,c456"
runAsNonRoot: true
volumes: ...

Capabilities
Set containers to run with specific
Capabilities
54
apiVersion: v1
kind: Pod
metadata:
name: hello-nginx
spec:
containers:
- name: nginx
image: nginx
securityContext:
capabilities:
drop:
- NET_RAW
volumeMounts:
...
securityContext:
seLinuxOptions:
level: "s0:c123,c456"
runAsNonRoot: true
volumes: ...

A Pod Security Policy is a cluster-level resource
that controls the actions that a pod can perform
and what it has the ability to access.
The PodSecurityPolicy objects define a set of
conditions that a pod must run with in order to be
accepted into the system.
Pod Security Policy
55
in Kubernetes

Privileged
Allows or denies running of privileged
containers.
Default deny.
56
Pod Security Policy in Kubernetes
apiVersion: extensions/v1beta1
kind: PodSecurityPolicy
metadata:
name: restrictive
spec:
privileged: false
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
runAsUser:
rule: RunAsAny
fsGroup:
rule: RunAsAny
hostPorts:
min: 8000
max: 8080
volumes:
- '*'

defaultAddCapabilities
Default set of capabilities that will be
added to a container.
57
metadata:
name: restrictive
spec:
privileged: false
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
defaultAddCapabilities:
- ‘CAP_AUDIT_WRITE’
- ‘CAP_KILL’
- ‘CAP_NET_BIND_SERVICE’
fsGroup:
rule: RunAsAny
hostPorts:
min: 8000
max: 8080
volumes:
- '*'

requiredDropCapabilities
Capabilities that will be dropped from
a container.
58
metadata:
name: restrictive
spec:
privileged: false
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
requiredDropCapabilities:
- ‘CAP_NEW_RAW’
runAsUser:
rule: RunAsAny
fsGroup:
rule: RunAsAny
hostPorts:
min: 8000
max: 8080
volumes:
- '*'

allowedCapabilities
Capabilities a container can request
to be added.
59
metadata:
name: restrictive
spec:
privileged: false
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
allowedCapabilities:
- ‘CAP_NEW_ADMIN’
runAsUser:
rule: RunAsAny
fsGroup:
rule: RunAsAny
hostPorts:
min: 8000
max: 8080
volumes:
- '*'

volumes
Controls the usage of volume types,
defines which ones are allowed.
60
metadata:
name: restrictive
spec:
privileged: false
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
runAsUser:
rule: RunAsAny
fsGroup:
rule: RunAsAny
hostPorts:
min: 8000
max: 8080
volumes:
- 'hostPath'
- ‘gcePersistentDisk’

hostPorts
Controls the use of host ports, defines
an allowed range.
Default empty.
List of HostPortRange, defined by
min (inclusive) and max (inclusive),
which define the allowed host ports.
61
metadata:
name: restrictive
spec:
privileged: false
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
runAsUser:
rule: RunAsAny
fsGroup:
rule: RunAsAny
hostPorts:
min: 8000
max: 8080
volumes:
- '*'

hostPID
Allows or denies the use of host’s PID
namespace.
Default deny.
62
metadata:
name: restrictive
spec:
privileged: false
hostPID: false
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
runAsUser:
rule: RunAsAny
fsGroup:
rule: RunAsAny
hostPorts:
- min: 8000
max: 8080
volumes:
- '*'

hostIPC
Allows or denies the use of host’s IPC
namespace.
Default deny.
63
metadata:
name: restrictive
spec:
privileged: false
hostPID: false
hostIPC: false
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
runAsUser:
rule: RunAsAny
fsGroup:
rule: RunAsAny
hostPorts:
- min: 8000
max: 8080
volumes:
- '*'

seLinux
MustRunAs: Requires
seLinuxOptions to be configured if
not using pre-allocated values. Uses
seLinuxOptions as the default.
Validates against seLinuxOptions.
RunAsAny: No default provided.
Allows any seLinuxOptions to be
specified.
64
metadata:
name: restrictive
spec:
privileged: false
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
runAsUser:
rule: RunAsAny
fsGroup:
rule: RunAsAny
hostPorts:
- min: 8000
max: 8080
volumes:
- '*'

runAsUser
MustRunAs: Requires a range to be
configured. Uses the first value of the
range as the default. Validates against the
range.
MustRunAsNonRoot: Requires that the
pod be submitted with a non-zero
runAsUseror have the USER directive
defined in the image.
RunAsAny: No default provided. Allows
any runAsUserto be specified.
65
metadata:
name: restrictive
spec:
privileged: false
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
runAsUser:
rule: MustRunAsNonRoot
fsGroup:
rule: RunAsAny
hostPorts:
- min: 8000
max: 8080
volumes:
- '*'

supplementalGroups
MustRunAs: Requires at least one
range to be specified. Uses the
minimum value of the first range as
the default. Validates against all
ranges.
Allows any supplementalGroups
to be specified.
66
metadata:
name: restrictive
spec:
privileged: false
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
runAsUser:
rule: RunAsAny
fsGroup:
rule: RunAsAny
hostPorts:
- min: 8000
max: 8080
volumes:
- '*'

fsGroup
MustRunAs: Requires at least one
range to be specified. Uses the
minimum value of the first range as
the default. Validates against the first
ID in the first range.
Allows any fsGroup ID to be
specified.
67
metadata:
name: restrictive
spec:
privileged: false
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
runAsUser:
rule: RunAsAny
fsGroup:
rule: RunAsAny
hostPorts:
- min: 8000
max: 8080
volumes:
- '*'

readOnlyRootFilesystem
Requiring the use of a read only root
file system.
68
metadata:
name: restrictive
spec:
privileged: false
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
runAsUser:
rule: RunAsAny
fsGroup:
rule: RunAsAny
hostPorts:
- min: 8000
max: 8080
volumes:
- '*'

#ContainerDayFR
69
Network Policies

70
Network Policies
my-cluster
pod pod
pod pod
pod pod
pod pod
team1-namespace
team2-namespace
Behavior without a network
policy.
Everything can talk to
everything.

71
Network Policies
my-cluster
pod pod
pod pod
pod pod
pod pod
team1-namespace
team2-namespace
Network Isolation with
DefaultDeny
Explicitly define
communication between
pods as a whitelist

72
Network Policies
Network Isolation with
DefaultDeny
Explicitly define
communication between
pods as a whitelist
my-cluster
pod pod
pod pod
pod pod
pod pod
team1-namespace
team2-namespace

Network Policy
Setting DefaultDeny for a
namespace.
73
in Kubernetes
kind: Namespace
apiVersion: v1
metadata:
name: my-namespace
metadata:
annotations:
net.beta.kubernetes.io/network-policy: |
{
"ingress": {
"isolation": "DefaultDeny"
}
}

Network Policy
Explicitly define communication
between pods.
74
in Kubernetes
kind: NetworkPolicy
metadata:
name: my-network-policy
namespace: my-namespace
spec:
podSelector:
matchLabels:
role: db
ingress:
- from:
- namespaceSelector:
matchLabels:
project: myproject
- podSelector:
matchLabels:
role: frontend
ports:
- protocol: tcp
port: 6379

Thank you!
@jessfraz
75

Security in a containerized world - Jessie Frazelle

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Security in a containerized world - Jessie Frazelle

Similar a Security in a containerized world - Jessie Frazelle (20)

Último

Último (20)

Security in a containerized world - Jessie Frazelle