Talk given at floss uk 2017 about how to fully automate your Jenkins setup, from the service to the jobs.

1. State of the Jenkins Automation
Julien Pivotto (@roidelapluie)
Floss Spring 2017
Manchester - March 15, 2017

2. whoami
Julien "roidelapluie" Pivotto
@roidelapluie
Sysadmin at Inuits
Automation, monitoring, HA
Jenkins user for 5+ years

3. inuits

4. Jenkins
Open Souce CI Server
Written in Java
First release in 2011
Fork of Hudson (2005)

5. What can you do with Jenkins?
Testing, building, deploying software
Testing, building, deploying services
Testing, building, deploying infra (IaC)

6. Mission Critical
Public Domain https://commons.wikimedia.org/wiki/File:Roaddamage59quake.JPG

7. Automating Jenkins
Creative Commons Attribution-ShareAlike 2.0 https://www.flickr.com/photos/machu/3131057286

8. Why is it hard?
For long, Jenkins has been UI-Driven
It is "easy" to deploy (.war file)
It has lots of plugins (and you need them)

9. Why is it needed?
Security (high value target)
Bugfixes
Plugins

10. XML everywhere!
Creative Commons Attribution 2.0 https://www.flickr.com/photos/generated/6035308729

11. Lot of things to automate
Jenkins Service
Jenkins configuration: security, plugins
Jenkins "data": Jobs/Views
Inside the jobs
Jenkins nodes

12. Automating the service
CC0 Public Domain https://pixabay.com/en/butler-tray-beverages-wine-964007/

13. Automating the Jenkins Service
Chef: Jenkins in the supermarket
Puppet: module rtyler/jenkins
Playbooks/etc for other tools as well
Docker

14. Jenkins Master in Docker
Jenkins upstream images: alpine/ubuntu
Please no executors on master
How do you deploy the container?
Pipeline to build&upgrade it

15. Jenkins Plugins
Fetched from https://updates.jenkins-ci.org
Can be installed from the UI :(
Can be installed from the CLI

16. Packaging Jenkins Plugins
Plugins have dependencies (against plugins &
Jenkins core)
They have a fixed download path
They are listed in updates.jenkins.io/update-
center.json
https://github.com/roidelapluie/Jenkins-
Plugins-RPM

17. Mirroring Jenkins Plugins
Mirror http://updates.jenkins-ci.org
By default, "latest" will be fetched
Don't cache too much
github.com/jenkinsci/docker install-
plugins.sh

18. Global configuration
CC0 Public Domain https://commons.wikimedia.org/wiki/File:Waiting_room_bell.jpg

19. Modifying XML files
DON'T

20. system-config-dsl-plugin
Like Job DSL, but for the system
https://github.com/jenkinsci/system-config-
dsl-plugin
JENKINS-31094
Downside: it does not exist yet

21. Creative Commons Attribution-Share Alike 3.0
https://commons.wikimedia.org/wiki/File:Groovy-logo.svg

22. Groovy
Yet another language to learn? yes.
Programming language for the Java platform
Scripting language
Fully integrated in Jenkins
Used by automation tools (chef cookbook,
puppet module...)

23. The Jenkins Script Console
A groovy console is available at /script
http://jenkins.example.com/script
also with curl
Requires "Overall/Run Scripts" permission

24. Sample Groovy scripts
Creative Commons Attribution-Share Alike 2.0
https://www.flickr.com/photos/bjornb/88101376

25. Set number of executors
import jenkins.model.Jenkins;
Jenkins.instance.setNumExecutors(0)

26. Set HTML formatter
import jenkins.model.Jenkins;
import hudson.markup.RawHtmlMarkupFormatter;
Jenkins.instance.setMarkupFormatter(
new RawHtmlMarkupFormatter(false));

27. Set system message
import jenkins.model.Jenkins;
Jenkins.instance.setSystemMessage("""
Welcome to <em>Jenkins</em>.
""");

28. Configure simple-theme
import jenkins.model.Jenkins;
def simpleThemePlugin =
Jenkins.instance.getDescriptor(
"org.codefirst.SimpleThemeDecorator")
simpleThemePlugin.cssUrl =
"/userContent/example.css"
simpleThemePlugin.save()
Everything in $JENKINS_HOME/userContent
is served under /userContent

29. Email
import jenkins.model.Jenkins
def desc = Jenkins.instance.getDescriptor(
"hudson.tasks.Mailer")
desc.setSmtpHost("172.17.0.1")
desc.save()

30. Disable usage statistics
import hudson.model.UsageStatistics;
hudson.model.UsageStatistics.DISABLED = true;

31. But also...
Setup Authorization/Authentication
Setup prefix url
Setup slaves, clouds
Setup global libraries
Setup credentials
Setup first jobs

32. Bonus
import org.jenkinsci.plugins.pipeline.utility
.steps.shaded.org.yaml.snakeyaml.Yaml
Yaml reader = new Yaml()
Map config = reader.load(text)
Thanks to pipeline utility step (readYaml()
step), you can easily read yaml files

33. Scripts sources
https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+Script+Console
https://github.com/jenkinsci/jenkins-scripts/tree/master/scriptler
https://github.com/infOpen/ansible-role-jenkins/tree/master/files/groovy_scripts
https://github.com/samrocketman/jenkins-bootstrap-jervis/tree/master/scripts
https://github.com/jenkinsci/puppet-
jenkins/blob/master/files/puppet_helper.groovy
http://pghalliday.com/jenkins/groovy/sonar/chef/configuration/management/2014
/09/21/some-useful-jenkins-groovy-scripts.html

34. Jenkins Javadocs
https://jenkins.io/doc/developer/guides/
http://javadoc.jenkins.io/archive/jenkins-
2.32/
http://javadoc.jenkins.io/
http://javadoc.jenkins.io/plugin/gerrit-trigger/

35. Now what?
Creative Commons Attribution 2.0 https://www.flickr.com/photos/51029297@N00/5275403364

36. Jenkins init.groovy.d
Upon startup, Jenkins will run
$JENKINS_HOME/init.groovy.d/*.groovy
scripts
Meanwhile, "Jenkins is getting ready..."
message is displayed
Drop files, restart Jenkins
Allows you to preconfigure everything --
without the GUI

37. init.groovy.d directory
behaviour
Scripts executed sequentally (alphanum
order)
Scripts that throws exceptions make startup
fail

38. Inside Jenkins

39. The Multiple Approaches
GUI .. but this talk is about automation, right?
init.groovy.d: to create your seed job
Jenkins Job Builder: declarative, python, yaml
Jenkins Job DSL: imperative, groovy

40. Jenkins Job Builder
An Openstack Project
Python (not a Jenkins Plugin!)
Support templates
Extensible
Can do raw xml
Limited support for plugins and pipeline
Put Jobs config under SCM

41. init.groovy.d
def jobManagement = new JenkinsJobManagement(
System.out, [:], new File('.'))
new DslScriptLoader(jobManagement).runScript("""
folder('jenkins') {
displayName('Jenkins')
}
pipelineJob("jenkins/seed"){
definition {
cpsScm {
scm {
git {
remote {
credentials('jenkins­git­na')
url('git.example.com/seed.git')
}
branches('master')
}
}
scriptPath('Jenkinsfile')
}
}
}
""")

42. Jenkins Job DSL
A Jenkins Plugin
2012
Groovy DSL to create views & jobs
Put Jobs config under SCM

43. Creative Commons Attribution-Share Alike 3.0
https://commons.wikimedia.org/wiki/File:Groovy-logo.svg

44. Job DSL support
Large community of users
Lots of plugins supported

45. Create a job
job('test') {
scm {
git('git://example.com/foo.git' 'master')
}
steps {
shell('make test')
}
}

46. Create a Pipeline job
pipelineJob('test'){
definition {
cps {
script(buildScript)
}
}
}

47. Set job properties
pipelineJob('test'){
description('Test Build')
parameters {
booleanParam('verbose', false, 'Be Verbose')
}
logRotator {
numToKeep(10)
}
}

48. Read yaml
import org.jenkinsci.plugins.pipeline.utility
.steps.shaded.org.yaml.snakeyaml.Yaml
Yaml reader = new Yaml()
Map config = reader.load(
readFileFromWorkspace('cfg.yaml')
Just like in init.groovy.d scripts
readFileFromWorkspace()

49. Loops
config.each(){ jobConfig ­>
pipelineJob(jobConfig.name) {
triggers {
if (jobConfig.triggers?.scm) {
scm(jobConfig.triggers.scm)
}
}
}
}

50. Create a view
listView('Project A') {
recurse()
jobs {
regex('myprj/[^/]+')
}
columns {
status()
weather()
name()
lastSuccess()
lastFailure()
lastDuration()
lastBuildConsole()
buildButton()
progressBar()
}
}

51. Work with plugins
buildMonitorView("OnScreen Status") {
jobs {
name('WatchA')
}
}
${JENKINS_URL}/plugin/job-dsl/api-viewer/

52. In Pipeline
jobDsl removedJobAction: 'DELETE',
removedViewAction: 'DELETE',
targets: 'seeds/*.groovy'

53. Load extra libraries
jobDsl additionalClasspath: 'src/*.jar',
removedJobAction: 'DELETE',
removedViewAction: 'DELETE',
targets: 'seeds/*.groovy'

54. Jenkins Pipeline
Creative Commons Attribution 2.0 https://www.flickr.com/photos/amerune/9294639633

55. Jenkins Pipeline
Jenkins Jobs as Code
"Jenkins file"
Imperative (aka scripted) Pipeline
Declarative Pipeline (2017)

56. What is a Jenkinsfile (aka
Pipeline)
A file that contains the definition of a job
No need of Gui
Defines Steps, Reports, Environments,
Nodes,...
Plugins can provide steps
Generic "step"

57. How to write Pipelines?
Visual Pipeline editor (WIP)
"Pipeline Syntax" link in jobs

58. Scriped pipeline
node ("Ubuntu && amd64") {
stage('checkout') {
checkout scm
}
stage('build') {
sh 'make'
}
stage('test') {
sh 'make test'
}
}

59. Real World Scripted Pipeline
node ("Ubuntu && amd64") {
checkout scm
stage('build') {
try {
sh 'make'
} catch(err) {
currentBuild.result = 'UNSTABLE'
publishArtifacts('err.log')
throw err
}
}
stage('test') {
sh 'make test'
}
step([$class: 'JavadocArchiver',
javadocDir: "target/site", keepAll: true])
}

60. Declarative Pipeline (1/2)
pipeline {
agent {
label("Ubuntu && amd64")
}
options {
timeout(time: 235, unit: 'MINUTES')
timestamps()
ansiColor('xterm')
}
stages {
stage('build') {
steps {
sh('make')
}
}
}

61. Declarative Pipeline (2/2)
pipeline {
post {
always {
junit params.jUnitReports
}
}
}

62. Declarative vs scripted
Declarative checkout code by default
Declarative get a workspace "OOTB"
Declarative enforces everything in stages
Declarative allow Pipeline-wide wrappers (e.g
ansiColor, timestamps)

63. Going further with Pipeline
Global Libraries Plugins
Job DSL Plugin

64. Jenkins Nodes
CC0 Public Domain https://www.flickr.com/photos/133259498@N05/27077266322/

65. Docker Docker Docker
Run jobs inside containers
Clean, short lived containers
Easy to update
Docker Plugin

66. Docker nodes pattern
Build Container
Tag it with a tag "candidate"
Push it to your registry
Run normal testing
Run actual builds with that "candidate"
If success -> tag with "release" && push

67. In practice
Docker Plugin config automated with groovy
Candidate and Release tags are setup as
slaves
They get two labels: "image" "tag" (e.g. "build-
centos-7" "candidate")
Jobs get a parameter "tag"

68. In the build Jenkinsfile
pipeline {
agent {
label("build­centos­7 && ${params.tag}")
}
}

69. In the docker-build Jenkinsfile
dockerImage = docker.build("build­centos­7",
"­­no­cache")
dockerImage.tag('candidate')
docker.withRegistry(url, credentials) {
dockerImage.push('candidate')
}
node("build­centos­7 && candidate") {
sh('test­script')
}
build job: 'myjob', parameters:
[string(name:'tag', value:'candidate')]
dockerImage.tag('release')
docker.withRegistry(url, credentials) {
dockerImage.push('release')
}

70. Pros/Cons
Updated containers won't block the builds (e.g
on packages updates)
Containers stay up to date
Don't forget that builds release artifacts
(sometime you don't want that in nodes tests)

71. Conclusion
Public Domain https://commons.wikimedia.org/wiki/File:YellowOnions.jpg

72. Jenkins can be FULLY
automated
My recommendations:
init.groovy.d
Jenkins Job DSL
Pipeline

73. You can go further
I am happy to talk about:
Jenkins master in Docker container
BlueOcean
... I use that but did not fit in this timeslot

74. Julien Pivotto
roidelapluie
roidelapluie@inuits.eu
Inuits
https://inuits.eu
info@inuits.eu
Contact