Introducing containers into your infrastructure brings new capabilities, but also new challenges, in particular around configuration. This talk will take a look under the hood at some of those operational challenges including: * The difference between runtime and build-time configuration, and the importance of relating the two together. * Configuration drift, immutable mental models and mutable container file systems. * Who configures the orchestrators? * Emergent vs. model driven configuration. In the process we will identify some common problems and talk about potential solutions. Talk from PuppetConf 2016

2. The challenges of
container configuration
David Lutterkort
@lutterkort
lutter@puppet.com

3. Overview
● What is configuration ?
● Immutability
● Build vs Run
● Who configures the scheduler ?
● Conclusions
3

4. What is configuration ?

5. package/file/service
is only one instance of a more general problem
5

6. Configuration is any input into infrastructure
It needs to be managed
over time and at scale
6

7. Core configuration management features:
❏ describe system aspects in isolation
❏ combine aspects into whole
❏ common format for querying
❏ bridge across entire infrastructure
7

8. $ docker run -d
-e MYSQL_HOST=mysql.example.com
-e MYSQL_PORT=3306
--health-cmd /usr/bin/check
webapp

9. Immutability

10. $ docker run
--name example fedora:24
/bin/sh -c ‘while true; do
cat /etc/system-release;
sleep 1;
done’

11. $ docker run …
Fedora release 24 (Twenty Four)
Fedora release 24 (Twenty Four)
Fedora release 24 (Twenty Four)
Fedora release 24 (Twenty Four)
Fedora release 24 (Twenty Four)
Fedora release 24 (Twenty Four)
Fedora release 24 (Twenty Four)
Fedora release 24 (Twenty Four)

12. $ docker exec example /bin/sh -c
‘sed -i -e s/24/25/ /etc/system-release’

13. Fedora release 24 (Twenty Four)
Fedora release 24 (Twenty Four)
Fedora release 25 (Twenty Four)
Fedora release 25 (Twenty Four)
Fedora release 25 (Twenty Four)
Fedora release 25 (Twenty Four)
Fedora release 25 (Twenty Four)
Fedora release 25 (Twenty Four)
$ docker exec …

14. $ docker diff example
C /run
A /run/secrets
C /etc
C /etc/system-release

15. Containers are not immutable by default
Only as immutable as packages
15

16. $ docker run --read-only
--name example fedora:24
/bin/sh -c ‘while true; do
cat /etc/system-release;
sleep 1;
done’

17. $ docker exec example /bin/sh -c
‘sed -i -e s/24/25/ /etc/system-release’
sed: couldn't open temporary file
/etc/sed5OCs5t: Read-only file system

18. $ docker diff example
C /run
A /run/secrets

19. Suggestion
Enable --read-only whenever possible
19

20. require 'rubygems'
require 'sinatra'
require 'haml'
# Handle GET-request (Show the upload form)
get "/upload" do
haml :upload
end
# Handle POST-request (Receive and save the uploaded file)
post "/upload" do
File.open('uploads/' + params['myfile'][:filename], "w") do |f|
f.write(params['myfile'][:tempfile].read)
end
return "The file was successfully uploaded!"
end

21. $ docker run -d --read-only lutter/lolcat

22. require 'rubygems'
require 'sinatra'
require 'haml'
# Handle GET-request (Show the upload form)
get "/upload" do
haml :upload
end
# Handle POST-request (Receive and save the uploaded file)
post "/upload" do
File.open('uploads/' + params['myfile'][:filename], "w") do |f|
f.write(params['myfile'][:tempfile].read)
end
return "The file was successfully uploaded!"
end

23. $ docker run -d --read-only
-v /srv/lolcat/uploads:/app/uploads
lutter/lolcat

24. require 'rubygems'
require 'sinatra'
require 'haml'
# Handle GET-request (Show the upload form)
get "/upload" do
haml :upload
end
# Handle POST-request (Receive and save the uploaded file)
post "/upload" do
File.open('uploads/' + params['myfile'][:filename], "w") do |f|
f.write(params['myfile'][:tempfile].read)
end
return "The file was successfully uploaded!"
end

25. $ docker run -d --read-only
-v /srv/lolcat/uploads:/app/uploads
--tmpfs /tmp
lutter/lolcat

26. Suggestion
Use --tmpfs where needed
26

27. Without technical controls you only have
social guarantees of immutability
27

28. How do you know the correct
invocation for an image ?
28

29. Build vs Run

30. Given an image
❏ What machine built this image ?
❏ How do you run this image ?
❏ Who supports this image ?
❏ Does the image contain malware ?
30

31. Given a container
❏ Who built it ?
❏ How was it built ?
❏ What software does it contain ?
❏ Is the software up-to-date ?
31

32. FROM fedora:24
RUN dnf update -y &&
dnf install -y ruby rubygem-bundler &&
dnf clean all
COPY . /app
RUN cd /app && bundle install --path vendor/bundle
WORKDIR /app
VOLUME /app/uploads
EXPOSE 9292
CMD ["/usr/bin/bundle", "exec", "rackup"]

33. FROM fedora:24
RUN dnf update -y &&
dnf install -y ruby rubygem-bundler &&
dnf clean all
COPY . /app
RUN cd /app && bundle install --path vendor/bundle
WORKDIR /app
VOLUME /app/uploads
EXPOSE 9292
CMD ["/usr/bin/bundle", "exec", "rackup"]
Where did the base image come from ?

34. FROM fedora:24
RUN dnf update -y &&
dnf install -y ruby rubygem-bundler &&
dnf clean all
COPY . /app
RUN cd /app && bundle install --path vendor/bundle
WORKDIR /app
VOLUME /app/uploads
EXPOSE 9292
CMD ["/usr/bin/bundle", "exec", "rackup"]
What repositories and what package versions ?

35. FROM fedora:24
RUN dnf update -y &&
dnf install -y ruby rubygem-bundler &&
dnf clean all
COPY . /app
RUN cd /app && bundle install --path vendor/bundle
WORKDIR /app
VOLUME /app/uploads
EXPOSE 9292
CMD ["/usr/bin/bundle", "exec", "rackup"]
What was in this directory at build time ?

36. Time is your enemy
36

37. When do you rebuild images ?
37

38. Code changes and external factors
should trigger rebuild
38

39. Explain yourself with metadata
Docker labels are a great way to do that
39

40. Name : glibc
Version : 2.23.1
Release : 10.fc24
Architecture: x86_64
License : LGPLv2+ and LGPLv2+ with exceptions and GPLv2+
Signature : RSA/SHA256, Thu 18 Aug 2016 09:27:43 AM PDT,
Key ID 73bde98381b46521
Source RPM : glibc-2.23.1-10.fc24.src.rpm
Build Date : Thu 18 Aug 2016 06:37:42 AM PDT
Build Host : buildvm-16.phx2.fedoraproject.org
Packager : Fedora Project
Vendor : Fedora Project
Summary : The GNU libc libraries

41. $ docker inspect
-f "{{json .Config.Volumes}}" lutter/lolcat
{
"/app/uploads": {}
}

42. $ docker inspect
-f "{{json .Config.ExposedPorts}}" lutter/lolcat
{
"9292/tcp": {}
}

43. LABEL vendor=”ACME Incorporated”
com.acme.release-status=”beta”
com.acme.version=”0.1.0-beta”
com.acme.git.sha=”f260653a”

44. $ docker inspect
-f "{{json .Config.Labels}}" lutter/lolcat | jq
{
"com.acme.git.sha": "f260653a",
"com.acme.release-status": "beta",
"com.acme.version": "0.1.0-beta",
"vendor": "ACME Incorporated"
}

45. Suggestion
Decide upon and enforce
metadata standards
45

46. LABEL com.acme.dockerfile=”/Dockerfile”

47. $ docker inspect
-f "{{json .Config.Labels}}" lutter/alpine | jq
{
"com.example.dockerfile": "/Dockerfile"
}

48. $ docker run -it lutter/alpine cat /Dockerfile
FROM alpine
RUN apk add --update bash && rm -rf /var/cache/apk/*
COPY Dockerfile /
LABEL com.example.dockerfile="/Dockerfile"

49. Suggestion
Embed your Dockerfile in the image
49

50. LABEL com.acme.cmd.packages=”apk info -vv”

51. $ docker run -it lutter/alpine apk info -vv
musl-1.1.14-r12 - the musl c library (libc)
busybox-1.24.2-r11 - Size optimized toolbox of ...
alpine-baselayout-3.0.3-r0 - Alpine base dir ...
alpine-keys-1.1-r0 - Public keys for Alpine Linux ...
zlib-1.2.8-r2 - A compression/decompression Library
bash-4.3.42-r3 - The GNU Bourne Again shell
...

52. Suggestion
Make your images discoverable
52

53. puppetlabs/puppetlabs-image_build

54. class { 'nginx': }
nginx::resource::vhost { 'default':
www_root => '/var/www/html',
}
file { '/var/www/html/index.html':
ensure => present,
content => 'Hello Puppet and Docker',
}
exec { 'Disable Nginx daemon mode':
path => '/bin',
command => 'echo "daemon off;" >> /etc/nginx/nginx.conf',
unless => 'grep "daemon off" /etc/nginx/nginx.conf',
}

55. # metadata.yaml
cmd: nginx
expose: 80
image_name: puppet/nginx

56. $ puppet docker build
...
$ docker run -d -p 8080:80 acme/nginx-test
83d5fbe370e84d424c71c1c038ad1f5892fec579d28b...
$ curl http://127.0.0.1:8080
Hello Puppet and Docker

57. Who configures the scheduler ?

58. Schedulers/orchestrators isolate you from
❏ where individual containers run
❏ balancing due to new resources
❏ respawning due to failed resources
58

59. Schedulers operate on constraints
59

60. Decisions depend on accurate resource
information
60

61. $ docker daemon
--label environment=production
--label storage=ssd

62. $ docker run -d -P
--label com.example.environment=production
-e constraint:storage==ssd --name db mysql

63. template:
metadata:
labels:
app: guestbook
tier: frontend
spec:
containers:
- name: php-redis
image: gcr.io/google-samples/gb-frontend:v4
resources:
requests:
cpu: 100m
memory: 100Mi
env:
- name: GET_HOSTS_FROM
value: dns
# If your cluster config does not include a dns service, then to
# instead access environment variables to find service host
# info, comment out the 'value: dns' line above, and uncomment the
# line below.
# value: env
ports:
- containerPort: 80

64. How do you manage properties
for all your hosts ?
64

65. Suggestion
Compute host properties dynamically
65

66. $ facter -y | head -n 20
aio_agent_version: 1.7.0
augeas:
version: 1.4.0
disks:
sda:
model: SanDisk SDSSDA24
size: 223.57 GiB
size_bytes: 240057409536
vendor: ATA
...
dmi:
bios:
...
memory:
...

67. $ docker daemon
--label os=$(facter os.family)
--label kernel=$(facter kernelversion)
--label memory=$(facter memory.system.total_bytes)

68. https://forge.puppet.com/puppetlabs/docker_platform

69. class { 'docker':
labels => [
"os=${facts[os][family]",
"kernel=${facts[kernelversion]}",
"memory=${facts[memory][system][total_bytes]}"
],
}

70. Schedulers introduce higher-level primitives
70

71. Docker networks
Kubernetes services and replication controllers
Chronos jobs
71

72. Many interfaces imperative not declarative
72

73. $ kubectl get pod mypod -o yaml
| sed -e ‘s/(image:myimage):.*$/1:v4/’
| kubectl replace -f -

74. $ docker network create bob
ca7b185775966003d38ccbd9bba822fb570766e4bb
$ docker network create bob
Error response from daemon: network with name bob ...

75. docker_network { 'bob':
ensure => present,
driver => 'overlay',
subnet => '192.168.1.0/24',
gateway => '192.168.1.1',
ip_range => '192.168.1.4/32',
}

76. And everything is in YAML
76

77. “
The language to represent the data should be a simple, data-only
format such as JSON or YAML, and programmatic modification of
this data should be done in a real programming language, where
there are well-understood semantics, as well as good tooling.
Borg, Omega, and Kubernetes, ACM Queue, Volume 14 Issue 1 | http://queue.acm.org/detail.cfm?id=2898444
77

78. Code plus data has advantages
over data alone
78

79. https://forge.puppet.com/garethr/kubernete
s

80. kubernetes_pod { 'sample-pod':
ensure => present,
metadata => {
namespace => 'default',
},
spec => {
containers => [{
name => 'container-name',
image => 'nginx',
}]
},
}

81. controller_service_pair { 'redis-master':
app => 'redis',
role => 'master',
tier => 'backend',
port => 6379,
}

82. Conclusions

83. The difference between how you think a
system behaves and how it actually behaves
risks hard-to-debug production issues
83

84. Container use at scale and over time
requires meaningful abstraction
84

85. Configuration management as a discipline
provides tools to build those abstractions and
thereby minimize risk
85

86. 86
Project Blueshift booth
Exhibition Hall
Docker, Mesos, Kubernetes and Puppet? Don't Panic !
Deepak Giridharagopal, Thur, 4:45pm
Pulling the strings to containerize your life
Scott Coulton, Fri, 9:50am
Running Puppet software in Docker containers
Gareth Rushgrove, Fri, 1:30pm