Docker + jenkins in the enterprise (3)

© 2017 CloudBees, Inc. All Rights Reserved. 1
+
DevOps

Docker + Jenkins Use Cases
• Run Jenkins with Docker (or other container)
• Run Jenkins Masters as Docker Containers
• Run Jenkins Agents as Docker Containers
• Use Jenkins to Manage Your Docker SDLC
• Use Jenkins for End-to-End Container Orchestration
▸ Build Docker Images with Jenkins
▸ Run Docker Containers with Jenkins for:
- build
- test
- deploy
▸ Push Docker Images to registries with Jenkins
▸ Deploy Docker based applications with Jenkins
• Not Deploying Apps with Docker? You should still use Docker for build and test

Run Jenkins Masters with Docker
• High-Availability via Container Orchestration and Cluster
Fault-tolerance
• Use or Extend Existing Jenkins Docker Images
• Both OSS and CloudBees master images are available on Docker Hub
• Baked-In extension points provide easy customization
• Easier and Faster Provisioning of Jenkins Masters
• Containers spin up fast and can make better utilization of underlying
resources
• Easier Upgrades
• A Dockerfile lends itself to config-as-code
▸ Add/update plugins
▸ Add/update Jenkins configuration

• Add Additional Plugins
• Skip the Setup Wizard
• Requires security configuration to skip initial authentication via install token
• Otherwise, just pass in an additional JAVA_OPT
• Optimize JVM Settings
• Keep it Small and more Secure, Use Alpine
• Jenkins Alpine is 170MB vs Jenkins Debian 360MB*
• 19 vs 36 vulnerable components*
RUN /usr/local/bin/install-plugins.sh blueocean:1.5.0 slack:2.3
ENV JAVA_OPTS -Djenkins.install.runSetupWizard=false
Custom Master Images: FROM jenkins:2.107.2-alpine
ENV JAVA_OPTS -server -XX:+AlwaysPreTouch XX:+UseConcMarkSweepGC

Run Jenkins Agents with Docker
• Easier Management of Jenkins Tools
• maven:3.3.1-jdk8 OR maven:3.3.9-jdk9 OR …
• Or a Custom Agent Dockerfile right next to your Jenkinsfile
▸ Managed as code in source control
▸ Use Jenkins to manage these images
• Easy Enablement of Ephemeral Agents
• Spin up and down on demand in seconds instead of minutes (or longer)
• Config-as-code for Agents with Dockerfile
• Manage your agent configuration as Dockerfiles in source control
▸ Offload tool management to individual teams/users
• Use Jenkins to actually build and test the Docker image you use to build and
test applications

Custom Docker Images for Jenkins Agents with lots of tools
• https://hub.docker.com/r/cloudbees/jnlp-slave-with-java-build-tools/
• The ‘Kitchen Sink’ of Docker Jenkins Agents
▸ Common tools: openssh-client, unzip, wget, curl, git
▸ AWS CLI: aws-cli/1.11.41
▸ Azure CLI: 0.10.8
▸ Bower: 1.8.0
▸ Cloud Foundry CLI (latest) at /usr/local/bin/cf: 6.23.1
▸ Firefox at /usr/bin/firefox: 50.1.0
▸ Firefox Geckodriver at /usr/bin/geckodriver: v0.13.0
▸ gcc (latest): 5.4.0
▸ Grunt CLI: 1.2.0
▸ Gulp: 3.9.1
▸ Java: OpenJDK 8 (latest): 1.8.0_111
▸ JMeter (3.1) located in /opt/jmeter/
▸ Kubernetes CLI at /usr/local/bin/kubectl: 1.5.2
▸ Make (latest): 4.1
▸ Maven located in /usr/share/maven/: 3.3.9
▸ MySQL Client: 5.7.17
▸ Node.js at /usr/bin/nodejs: 6.9.4
▸ Npm at /usr/bin/npm: 3.10.10
▸ Open Shift V3 CLI at /usr/local/bin/oc: 1.3.0
▸ Python/2.7.12
▸ Selenium at /opt/selenium/selenium-server-standalone.jar: 2.53
▸ XVFB: 2:1.18.4

Better Yet, Don’t Use Monolithic Custom Images
Remember: A container a day keeps the monolith away
• Get Out of the CI/CD Tool Management Business
• Let Teams Manage Their Own Tools
• Allows use of same tools on laptops as are used by Jenkins
• Just about every tool imaginable is available, including a number of Docker
Hub Official Repositories
▸ Just a small sampling: fsharp, gcc, golang, gradle, groovy, haskell, java, maven, node,
perl, php, python, ruby, swift, ...
• Don’t/Can’t Use Docker Hub - No Problem
• Use a private Docker registry
• Let engineers build their own images - use Dockerfiles from Official images as
templates, have Dockerfiles in source code repos
• Use Jenkins to automate the process of building scanning and testing CI/CD
Docker images

Three Basic Kinds of Docker Enabled Agents
• Static Docker Hosts - an agent (VM, EC2
instance, etc) that is running the Docker
daemon
• DIND (Docker in Docker) Agents - a
Docker container running a Docker daemon
• DOD (Docker on Docker) - a Docker
container that maps the parent host’s Docker
socket

Static Docker Hosts as Agents
• Pros
• May perform better than other approaches as this approach does not
rely on spinning up additional Job specific containers
• Possibly easier to manage secure Docker access
• Could use a cluster with Jenkins Swarm plugin or CloudBees JNLP
Cloud to easily provision more capacity
• Cons
• More Jenkins administrative overhead than other approaches
▸ Must spin up a new physical Node when additional capacity is needed
▸ Requires a dedicated Node for a finite number of executors
• If not using cluster, could be inefficient use of resources
• Without some type of sharing mechanism, must be dedicated to
individual Masters

DIND (Docker in Docker) Containers as Agents
• Pros
• Dynamic and ephemeral DIND based agents would allow for very
secure ephemeral workspaces
• Easier to dynamically utilize/test with multiple Docker versions
• Easier to contain cluster shared resources
• Cons
• DIND agents must be run in –privileged mode
▸ Security implications
• Images for Job specific containers will not be shared across DIND
agents, even on same host1
• Ephemeral workspaces will not be shared across job runs
▸ Mount external volume from host or network mount to DIND agent
and through to any images run inside DIND agent2

DOD (Docker on Docker) - Mapping the Docker Socket
• Pros
• Docker images/layers used within agent will be shared across DOD
agents on same host
• Easier to share data between jobs by mapping volumes to host -
dependency cache, job workspace, etc.
• Cons
• Requires mounting the Docker socket of the host - security
implications
• Docker client on the agent should match host version
• Running Docker process from these agents in a clustered
environment (Mesos, Kubernetes, Docker Swarm) will result in
un-tracked/un-managed resources
• Must explictly map agent workspace to host or else Docker Pipeline
Plugin won’t work

There Is Another Way
1. Use a Container Orchestrations Service - such as Kubernetes (k8s) - to
manage your Jenkins agents
2. The Jenkins Kubernetes Plugin allows running agents in k8s pods that
offer a number of features to manage containerized CI/CD:
i. limit compute resources for containers
ii. control security sensitive aspects for pod creation
3. For build and push of Docker images there are some interesting
possibilities on the horizon for standalone, daemon-less, unprivileged
Dockerfile and OCI compatible container image builder:
i. img - can’t run in a container without some major work-arounds - yet
ii. buildah - must use privileged volumes
iii. kaniko - must run as root in the container

Restrictive k8s
PodSecurityPolicy
for Jenkins Agents
apiVersion: extensions/v1beta1
kind: PodSecurityPolicy
metadata:
name: restricted
spec:
privileged: false
fsGroup:
rule: RunAsAny
runAsUser:
rule: MustRunAsNonRoot
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
volumes:
- 'emptyDir'
- 'secret'
- 'downwardAPI'
- 'configMap'
- 'persistentVolumeClaim'
- 'projected'
hostPID: false
hostIPC: false
hostNetwork: false
allowPrivilegeEscalation: false

Docker as Agents Dos and Don’ts
• DON’T - USE DOCKER IN DOCKER (DIND)
• EXCEPT for specific use cases - like testing different versions of Docker
• DIND agents must be run in –privileged mode
• Caching is difficult to implement
• DO - Use Official Images
• Docker Hub provides hundreds of official images that are automatically scanned
for vulnerabilities
• Don’t pull directly from Docker Hub, proxy Docker Hub using a private registry
• Scan images for vulnerabilities
• DO - Use Docker (containers), even if it is just a tool for CI
• DO - Use a container orchestration service to manage CI/CD
infrastructure

Docker in Pipeline
• Docker Pipeline Plugin
• Provides several utility Pipeline Variables to simplify the use of Docker in
Pipeline jobs
▸ docker.image(‘golang:1.7’).inside() { sh ‘go build -v -o
go-demo’ }
• Pipeline Shared Libraries
• Wrap complex steps in a reusable parameterized component
• Declarative Pipeline
• Embraces Docker Pipeline and makes Docker a top level feature
▸ agent { docker label ‘golang:1.7’ } }
• Use Docker Compose with sh step
▸ sh ‘docker-compose run --rm unit’

Jenkins Kubernetes Agents
• Kurbernetes Plugin
• Provides the ability to run job specific containers in k8s Pods
podTemplate(label: 'kubernetes',
containers: [
containerTemplate(name: 'maven', image:
'maven:3.5.2-jdk-8-alpine', ttyEnabled: true, command: 'cat')
]) {
stage('Dummy') {
node('kubernetes') {
container('maven') {
sh 'mvn --version'
}
}
}
}

An Opinionated View for Using Docker Agents
• Use One Agent Type: a Docker Enabled Agent (Think DOND or DIND)
• Use Plain Vanilla Docker Images in your Pipeline Jobs
• A developer should use the same Docker image to build on their laptop as is used by
Jenkins - not some specialized Jenkins only agent image
• Why have Java in a Docker image being used for a Go build?
• Use the Docker Pipeline Plugin docker.image().inside(){} Step with Care
• Attempts to make it easier to use Docker from a Jenkins agent by mapping volumes
and user/group ids - many times results in issues that are hard to diagnose
• Also, be careful with Declarative syntax as agent { docker { image ‘maven’} } } is
generating the Docker Pipeline inside syntax behind the scenes
• Technically, you don’t need Jenkins Plugins to use Jenkins & Docker

Use Docker LABELS with your Jenkins Builds
Pass in things like commit SHAs as a label so you know exactly what
application commit triggered a specific Docker image build
Jenkinsfile:
sh "COMMIT_SHA=$(git rev-parse HEAD | tr -d 'n') docker-compose run --name
go-demo-unit unit-cache"
Dockerfile:
ARG COMMIT_SHA
LABEL beedemo.commit.sha=$COMMIT_SHA
Take a look at:
https://github.com/vfarcic/docker-jenkins-slave-dind/blob/master/Dockerfile

SDLC with Docker and Jenkins
1. Start with Dockerfiles in source control
2. Create your own base images
a. Docker only needs to load the common layers once, and they are cached
3. Use your own DIND image to build your images
a. isolates Docker version from CI/CD platform
4. Store application data outside of the application container - container
5. Run a security scan on newly built images (and of course any external
images you may be using)
6. Provide pre-prod Docker registries for pre-prod testing
a. Tools like Artifactory support multiple registries
b. Only allow secure, tested Docker images in production Docker registry
7. Use additional Docker images to test your image
8. I forget what 8 was for

Jenkins X
• Cloud Native Jenkins
• Commit to Deploy managed by Jenkins X on
Kubernetes
•

Ephemeral Jenkins Masters
Pros
• The Jenkins home directory becomes throw-away
• Fault-tolerance of individual Jenkins masters is a thing of the past
• Many of the ‘Cons’ force best practices
Cons
• Even a light-weight Docker based Jenkins instance takes several minutes to start-up
• Build history/statistics must be managed externally (same goes for artifacts)
• More complex to set-up and manage
• config-as-code for master plugins and configuration
• ensuring individual teams/users have masters when they need them

Docker + Jenkins Resources
• https://github.com/jenkinsci/docker
• https://github.com/jenkinsci/kubernetes-plugin
• Using Docker-in-Docker for your CI or testing environment? Think twice.
• https://jenkins.io/solutions/docker/
• https://jenkins.io/doc/book/pipeline/syntax/#agent see docker section
• https://jenkins.io/doc/pipeline/steps/docker-workflow/
• Some useful Docker Agents:
• Jenkins Swarm Agent
• DIND Docker Compose Agent

Docker + jenkins in the enterprise (3)

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Docker + jenkins in the enterprise (3)

Similar a Docker + jenkins in the enterprise (3) (20)

Último

Último (20)

Docker + jenkins in the enterprise (3)