While there have been many improvements around securing containers, there is still a large gap in monitoring the behaviour of containers in production. Sysdig Falco is an open source behavioural activity monitor for containerized environments. Sysdig Falco can detect and alert on anomalous behaviour at the application, file, system, and network level. In this session get a deep dive into Falco: How does behavioural security differ from existing security solutions like image scanning, seccomp, SELinux or AppArmor? What can Sysdig Falco detect? Building and customizing rules for your Docker and Kubernetes apps. Forensics analysis with Sysdig Inspect even when the container doesn't exist anymore! Read more on: https://sysdig.com/blog/docker-runtime-security/ https://sysdig.com/blog/runtime-security-kubernetes-sysdig-falco/

1. Docker Runtime Security
looking inside your containers
Jorge Salamero - @bencerillo - Sysdig
Docker Meetup London
Docker Meetup CGN

2. Docker Security
– Security is hard
– containers are an opportunity for better security
– DevSecOps
– You should use this new awesome Cloud Native tool!
https://sysdig.com/blog/7-docker-security-vulnerabilities/

3. Come on...

4. Jorge Salamero
Tech Marketing aka container gamer @ Sysdig
github.com/bencer
@bencerillo
OSS fan
Monitoring, security, containers, IoT/home-automation,
motorsport
About me

5. Docker Security Tools process
– CI/CD pipeline to rebuild your containers
– An images registry
– Static scanning for known vulnerabilities
• CoreOS Red Hat Clair
• Anchore
• Red Hat OpenScap
• Vuls.io
• Other commercial vendors
https://sysdig.com/blog/20-docker-security-tools/

6. No, really, all the new cool stuff
– RBAC
– docker-bench, kube-bench
– Augeas
– Kubernetes ValidatingAdmissionWebhook
– Kubernetes PodSecurity Policy
– Kubernetes Network Policy
https://sysdig.com/blog/kubernetes-security-guide/

8. Even in Mordor you can find...
– 0-day vulnerabilities
– Wrong configuration
– Weak/leaked credentials
– Internal (malicious) activity
– how knows?

10. Runtime security
– Network, file system, all the things inspection
– Threat detection
– Privilege escalation
– Post-mortem analysis and forensics
https://sysdig.com/blog/docker-runtime-security/

11. Docker Runtime tools
– capabilities
– seccomp
– AppArmor / SELinux
– Falco
– Sysdig Inspect

12. Seccomp
- Application syscall sandboxing
- Create filter (BPF) with allowed syscalls
- Failures-> log / error return / kill
- Docker runsprocess under a seccomp profile
- Notable disallowed syscalls

13. MAC: SELinux / AppArmor
- Kernel-level interception/filtering
- features++ && complexity++
- Higher level:
- Actors (process)
- Actions (read/write on files/sockets)
- Targets (files, IPs, ports)
https://sysdig.com/blog/selinux-seccomp-falco-technical-discussion/

14. Sysdig Falco
A behavioral activity monitor
•Detects suspicious activity defined by a set of rules
•Uses Sysdig’s flexible and powerful filtering expressions
With full support for containers/orchestration
•Utilizes Sysdig’s container & orchestrator support
And flexible notification methods
•Alert to files, standard output, syslog, programs
Open Source
•Anyone can contribute rules or improvements

15. System call tracing

16. System calls for observability?
– clone() and execve() give you insight into process creation and
command execution.
– open(), close(), and the FD read and write functions offer visibility on
disk I/O.
– socket(), connect(), and accept() give insight into network activity.

18. A shell is run in a container container.id != host and proc.name = bash
Overwrite system binaries
fd.directory in (/bin, /sbin, /usr/bin, /usr/sbin)
and write
Container namespace change
evt.type = setns and not proc.name in
(docker, sysdig)
Non-device files written in /dev
(evt.type = create or evt.arg.flags contains O_CREAT)
and proc.name != blkid and fd.directory = /dev and
fd.name != /dev/null
Process tries to access camera
evt.type = open and fd.name = /dev/video0
and not proc.name in (skype, webex)

19. Falco architecture
falco_probe
kernel module
eBPF magic!
Kernel
User
syscalls
Sysdig libraries
events
alerting
Falco Rules
suspicious
events file
syslog
stdout
filter expression
shell
process

20. Falco Rules
25 common rules available OOTB
Focused on common container best practices:
■ Writing files in bin or etc directories
■ Reading sensitive files
■ Binaries being executed other than CMD/ENTRYPOINT
Default rules for most popular images (next week!)

21. Falco rules
.yaml file containing Macros, Lists, and Rules
- macro: bin_dir
condition: fd.directory in (/bin, /sbin, /usr/bin, /usr/sbin)
- list: shell_binaries
items: [bash, csh, ksh, sh, tcsh, zsh, dash]
- rule: write_binary_dir
desc: an attempt to write to any file below a set of binary directories
condition: bin_dir and evt.dir = < and open_write and not package_mgmt_procs
output: "File below a known binary directory opened for writing
(user=%user.name command=%proc.cmdline file=%fd.name)"
priority: WARNING

22. Falco rules
Rules
• name: used to identify rule
• desc: description of rule
• condition: filter expression, can contain macro
references
• output: message to emit when rule triggers,
can contain formatted info from event
• priority: severity of rule (WARNING, INFO, etc.)
Macros
• name: text to use in later rules
• condition: filter expression
snippet
Lists
• name: text to use later
• items: list of items

23. Falco rules
Filtering Expressions
• Use the same format as sysdig
• Full container, Kubernetes,
Mesos, Docker Swarm support
Rule Execution Order
• Falco rules are combined into
one giant filtering expression,
joined by ors
• Each rule must contain at least
one evt.type expression
• i.e. evt.type=open and …
• Allows for very fast filtering of
events

24. Sysdig Filter Expressions
Based on Field Classes:
fd - File Descriptors
process - Processes
evt - System Events
user - Users
group - Groups
syslog - Syslog messages
container - Container info
fdlist - FD poll events
k8s - Kubernetes events
mesos - Mesos events
span - Start/Stop markers
evtin - Filter based on Spans

25. Alerts and outputs
Sending Alerts
• Events matching filter
expression result in alerts
• Rule’s output field used to
format event into alert
message
• Falco configuration used to
control where alert message is
sent
Any combination of..
• Syslog
• File
• Standard Output
• Shell (e.g. mail -s "Falco
Notification"
someone@example.com)

26. A Custom Falco Rule
- rule: Node Container Runs Node
desc: Detect a process that’s not node started in a Node container.
condition: evt.type = execve and container.image startswith node and
proc.name != node
output: Node container started other process (user=%user.name
command=%proc.cmdline %container.info)
priority: INFO
tags: [container, apps]

27. A Custom Falco Rule
- rule: Node Container Runs Node
desc: Detect a process that’s not node started in a Node container.
condition: evt.type = execve and container.image startswith node and
proc.name != node
output: Node container started other process (user=%user.name
command=%proc.cmdline %container.info)
priority: INFO
tags: [container, apps]
Something is
executing a
program
In a container
based on the
Node image
And the
process name
isn’t node

28. - macro: proc_is_new
condition: proc.duration <= 5000000000
- rule: Read secret file after startup
desc: >
an attempt to read any secret file (e.g. files containing user/password/authentication
information) Processes might read these files at startup, but not afterwards.
condition: fd.name startswith /etc/secrets and open_read and not proc_is_new
output: >
Sensitive file opened for reading after startup (user=%user.name
command=%proc.cmdline file=%fd.name)
priority: WARNING
Falco real cool rule example.

29. Active Security
Falco, NATS, and kubeless
Falco NATS kubeless
Detects abnormal event,
Publishes alert to NATS
Subscribers receive
Falco alert through
NATS server
kubeless receives
Falco alert, firing a
function to delete the
offending Kubernetes
Pod

30. Functions for security operations
- Easily write simple functions to react to monitoring events
- Multiple subscribers can take multiple actions
- One function to kill/stop/pause/delete a pod
- One function to setup a Kubernetes network policy
- One function to notify teams
- One function to log events
- One function to trigger a Sysdig capture
- Small, reusable components

31. Installing Falco
• Debian Package
• apt-get -y install falco
• Redhat Package
• yum -y install falco
• Installation Script
• curl -s https://s3.amazonaws.com/download.draios.com/stable/install-falco | sudo bash
• Docker container
• docker pull sysdig/falco
• Full instructions
• https://github.com/draios/falco/wiki/How-to-Install-Falco-for-Linux

32. Installing Falco on Kubernetes
• Install Falco as Kubernetes daemonSet
• https://github.com/draios/falco/tree/dev/examples/k8s-using-daemonset
• Configuration stored in Kubernetes ConfigMaps
• Conditions in a Falco rule can leverage Kubernetes metadata to trigger events
• Falco events can include Kubernetes metadata to give notification context:
• name, id, labels for pods, replicationController, replicaSet, deployment, service, and
namespace
• Helm Chart WIP here: https://github.com/nestorsalceda/charts/tree/falco/stable/falco

33. Running Falco
• As a service
• $ service falco start
• alerts to syslog
• By hand
• $ sudo falco -r <rules file> -c <config file>
• alerts to syslog, stdout
• Using Docker
• docker run -i -t --name falco --privileged -v /var/run/docker.sock:/host/var/run/docker.sock -v /dev:/host/dev
-v /proc:/host/proc:ro -v /boot:/host/boot:ro -v /lib/modules:/host/lib/modules:ro -v /usr:/host/usr:ro
sysdig/falco
• Full Instructions
• https://github.com/draios/falco/wiki/Running-Falco

34. Join the community
• Website
•http://www.sysdig.org/falco
• Public Slack
•https://sysdig.slack.com/messages/falco
• Blog
•https://sysdig.com/blog/tag/falco/
• Sysdig Secure
•http://sysdig.com/product/secure

35. Learn more
Github
• https://github.com/draios/falco
• Pull Requests welcome!
Wiki
• https://github.com/draios/falco/wiki
Docker Hub
• https://hub.docker.com/r/sysdig/falco/

36. Post-mortem and forensics.
What?
Where?
Who?
Why?
logs? SSH into prod? and start messing around?

37. How we did this in the past?.

39. Sysdig Inspect
https://github.com/draios/sysdig-inspect

40. Thank you!
Jorg Salamero - @bencerillo - Sysdig
Thank You