This was a talk given internally at BloomReach as well as a guest lecture to a grad level Data Structures and Algorithms class at the University of Texas at Arlington. A Deep dive on the history of containers, and how they work under the cover utilizing Linux Kernel features such as Process Namespaces and Control Groups. I also go over a bit of the history of Container technology, going from Chroot and Jails and Zones, to LXC and Docker

1. CORWIN ONCORWIN ON
CONTAINERSCONTAINERS

2. WHO AM I?WHO AM I?
Corwin Brown
But you can call me Kory.
Traditionally come from a DevOps background.
Worked at places like:
Rackspace
Fitbit
Plus a bunch of places you haven't heard of.
Currently a Full-Stack Developer at BloomReach here in
Dallas.

3. WHAT ARE WE TALKING ABOUTWHAT ARE WE TALKING ABOUT
TODAY?TODAY?

4. CONTAINERS!CONTAINERS!
What is Containerization?
The History fo Containers.
How containers work.
Process Namespaces.
Control Groups.
How we can use containers here at BloomReach.
Questions at marked sections.

5. QUICK ASIDE ON DEVOPSQUICK ASIDE ON DEVOPS

6. "DevOps is about recognizing that the
backing infrastructure is not separate from
your application, but rather a vital part of
it."

7. CONTAINERSCONTAINERS

8. LINUX CRASH COURSELINUX CRASH COURSE
Everything in Linux is a file.
Network sockets.
Hardware devices.
etc
User space:
The Section of system memory that User Processes run
in.
Contrasted with Kernel space, where the Kernel
executes processes.

9. CONTAINERSCONTAINERS

10. WHAT IS CONTAINERIZATION?WHAT IS CONTAINERIZATION?
OS feature that allows for the existance of multiple
isolated User Spaces.
These user-space slices are known as containers.
Each container is made aware of only specific
resources.
A container can contain anything from a single running
process to an entire operating system.

11. VERY SIMILAR TO VIRTUALVERY SIMILAR TO VIRTUAL
MACHINESMACHINES
But no need to maintain a bunch of different OS installs.
No backups.
No policies around backups.
No upgrades.
No patches.
etc
Ship your application's context alongside your
application.

12. CONTAINERS TEND TO BE LIGHTERCONTAINERS TEND TO BE LIGHTER
THAN VMSTHAN VMS
Each VM would require significant overhead that comes
with running an OS.
Both in compute resources, and human time.
Allows you to efficiently bin-pack resources onto a
machine.

13. HISTORY OFHISTORY OF
CONTAINERSCONTAINERS

14. CHROOTCHROOT
"Change-Root"
Provided file system isolation.
Convinces a process its root directory is something
other than "/".

15. FREEBSD JAILS AND SOLARIS ZONESFREEBSD JAILS AND SOLARIS ZONES
File system and network isolation
Zones also provided also snapshots courtesy of ZFS.

16. LXCLXC
Originally the backend for Docker.
Generally works closer to a traditional Virtual Machine.
Generally runs an entire OS.
Produced containers tend to be less portable than Docker.
Designed to be a lightweight system, as opposed to a
lightweight service.

17. DOCKERDOCKER
Currently the most popular container engine.
Succeeded largely because of it's focus on ecosystem.

18. RKTRKT
Potentially the future.
Takes a lot of Docker concepts, and improves upon them.
Follows a more Linux-like process model.
Built to strongly adhere to the Open Container
Standard, but also be backwards compatible with
Docker.
As a drop in replacement, has a lot of potential promise
for the future.

19. HOW DO CONTAINERSHOW DO CONTAINERS
WORK?WORK?

20. PROCESS NAMESPACESPROCESS NAMESPACES
CONTROL GROUPS (CGROUPS)CONTROL GROUPS (CGROUPS)

21. PROCESS NAMESPACESPROCESS NAMESPACES
Allows the Kernel to isolate and virtualize system
resources.
Seven Namespaces:
mnt
pid
net
ipc
UTS
user
cgroup

22. EXAMPLEEXAMPLE
Instead of communicating with eth0:
Create a virtual interface.
Present that to a process instead of the hardware.
Allows for each container to be walled off from the
hadware.
Allows for each container to have it's own networking
information.

23. CONTROL GROUPSCONTROL GROUPS
Developed at Google in 2006 and merged into the Linux
Kernel in 2007.
Feature that limits, accounts for, and isolates system
resources.
Provides:
Resource limiting.
Prioritization.
Accounting.
Control.

24. CGROUP IMPLEMENTATIONCGROUP IMPLEMENTATION
Like most things with Linux, CGroups are represented on
the file system.
In Ubuntu 16.04, they exist at "/sys/fs/cgroup".

25. CGROUP FILESYSTEMCGROUP FILESYSTEM

26. CGROUP MEMORY LIMITINGCGROUP MEMORY LIMITING

27. CREATE NEW CGROUPCREATE NEW CGROUP
cgcreate -g memory:test_cgroup

28. SEE THE NEW GROUP!SEE THE NEW GROUP!

29. INSIDE THE NEW CGROUPINSIDE THE NEW CGROUP

30. IN SUMMARYIN SUMMARY
Process Namespaces: Control what you can see.
Control Groups: Control what you can do.

31. FURTHER READINGFURTHER READING
Mocker
https://github.com/tonybaloney/mocker

32. WHAT CAN YOU DOWHAT CAN YOU DO
WITH CONTAINERS?WITH CONTAINERS?

33. TESTING NEW TECHNOLOGIESTESTING NEW TECHNOLOGIES
Wanted to try out a monitoring backend for an alerting
service.
I'd used Prometheus in the past, and was interested in
testing it out.
... but I really didn't want to set it all up locally or maintain
a VM for ten minutes of vetting.

34. IN CASE OF DEMO FAILUREIN CASE OF DEMO FAILURE
$ docker run --rm -p9090:9090
prom/prometheus

35. DEVELOPING WITH SPECIFICDEVELOPING WITH SPECIFIC
DEPEDENCIESDEPEDENCIES
Project requires Postgres 9.6.
Not interested in installing Postgres locally.
What if I need a different version later?
Maintaining different installs is a pain.
I don't want my laptop chugging to run a database
when I don't need it.

36. IN CASE OF DEMO FAILUREIN CASE OF DEMO FAILURE
$ docker run --rm postgres:9.6.6

37. POSTGRESQL (CONTD)POSTGRESQL (CONTD)
Still need psql to interact with PostgreSQL.
Still have to install junk on my laptop.
...Or do you?

38. POSTGRESQL (CONTD)POSTGRESQL (CONTD)
I can spin up a container with PostgreSQL...
I can spin up a container with PSQL...
... and I can link them with a "Docker network".

39. IN CASE OF DEMO FAILUREIN CASE OF DEMO FAILURE
$ docker network create dbnet
$ docker run --rm --net dbnet --name db postgres:9.6.6
$ docker run --rm --net dbnet -it postgres:9.6.6 psql -h db -U postgr

40. POSTGRESQL (CONTD)POSTGRESQL (CONTD)
Actual workflow we use:.
To get a development environment setup, do the
following:
1. Open the src directory in a terminal.
2. Run some Make commands.

41. IN CASE OF DEMO FAILUREIN CASE OF DEMO FAILURE
$ make postgres-start
$ make populate-postgres
$ make postgres-shell

42. TESTINGTESTING
Can be difficult to test data oriented so ware.
Tends to lead to lots of Mocks...
... at which point you o en end up just checking if the
"return" keyword works.
Uness of course your tests involve spinning up a database
and checking against real data.

43. IN CASE OF DEMO FAILUREIN CASE OF DEMO FAILURE
$ make test

44. TESTING (CONTD)TESTING (CONTD)
This test process sets up a local PostgreSQL instance.
Including populating it with data.
Runs tests inside a container linked to that database.
Bam, testing against a production-ish database.
More over, a good end to end test you can run on Jenkins.

45. FRONTEND TESTINGFRONTEND TESTING
These workflows can benefit frontend developers too!
Spin up headless Chrome in a container and run end to
end tests!
Spin up a framework like Zalenium!
This one actually records and outputs video of test
failures!

46. BUILDINGBUILDING
Let's say you're running into problems with the bulid
machine.
You don't have access to make changes on that box.
Just build your application inside a container you DO have
control over.
DevStudio builds it's Javascript bits in a container.

47. THE FUTURETHE FUTURE
Imagine we get out of the business of running EC2
instances.
Batch jobs are just scheduled to run in our
Kubernetes/Mesos cluster.
Autoscale the number of container's deployed for any
service.
Deployments are less "Run this fabric script", more "Tell
Kubernetes to ensure X instances are running".
Get out of the business of maintaining servers, and into
the business of running services.
Ensure we get the box utilzation we pay for.
Datacenter as an Operating System

48. QUESTIONS?QUESTIONS?