CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)

CI and CD Across the Enterprise with Jenkins
Cyrille Le Clerc, CloudBees
©2014 CloudBees, Inc. All Rights Reserved

About Me
@cyrilleleclerc
Cyrille Le Clerc
Open
Source
CTO
Product
Management
DevOps,
Infra
as
Code,
Con9nuous
Delivery

Agenda
• Con9nuous
Delivery
&
Jenkins
Workflow
• Scaling
Jenkins
• Con9nuous
Delivery
with
Jenkins
and
Puppet
and
Chef

Continuous Delivery

Continuous Delivery
• “Reliable
So+ware
Releases
through
Build,
Test,
and
Deployment
Automa=on”
• DONE
=
Shippable
into
Produc9on

Continuous Integration Steps
Con$nuous
Integra$on
Source Code
Quality Analysis
Compile &
Unit Tests
Binaries

Continuous Delivery Steps
QA
Con$nuous
Delivery
DEV
DEV
OPS
Source Code
Quality Analysis
Compile &
Unit Tests
Binaries
Perfs Environment
Func Test Environment
UAT Environment
Production Environment
GO LIVE!
Release!
Staging Environment

Continuous Delivery Pipeline
Compile &
Unit Tests
Perfs Tests
Selenium Tests Deploy UAT
Sonar Analysis
Deploy Staging Deploy Production

Jenkins: Hub of Continuous Delivery
• Connect
the
dots
• It
is
the
process
Source Code
Perfs Environment
Func Test Environment
UAT Environment
Binaries
Quality Analysis
Compile &
Unit Tests

CD Pipeline è Workflow
Compile &
Unit Tests
Perfs Tests
Selenium Tests Deploy UAT
Sonar Analysis
Deploy Staging Deploy Production

What we need
• Complex
pipelines
involving
mul9ple
stages
• Non-‐sequen$al
logic
such
as
loops,
forks
…
• Long-‐running
builds
must
survive
outages
• Interac$ons
with
humans
including
pauses,
input
• Restartable
builds
in
case
of
transient
errors
• Reusable
defini$ons
to
avoid
duplica9on
• Comprehensible
scripts
with
one
clear
defini9on

Workflow until today
• Job
chaining
• Build
Flow
Plugin
• Build
Pipeline
Plugin
• Build
Delivery
Pipeline

Workflow until today
• Many
atomic
jobs
• Hard
to
share
variables/state
between
jobs
• Limited
logic
• Mix
build
triggers,
parameterized
build
…

Build Flow Plugin
• Did
have
scriptability
and
extensibility
• Did
not
address
configura9on
“sprawl”
• Disjointed
view
of
what
really
ran
• No
ability
to
survive
restarts
• Almost
good
enough
but
could
not
go
further

New Workflow Syntax

Demo time!
https://github.com/cyrille-leclerc/spring-petclinic/blob/petclinic-1.0.0-clc/src/main/jenkins/workflow.groovy

Key features
• En9re
flow
is
one
concise
Groovy
script
– For
loops,
try-‐finally,
fork-‐join
…
• Can
restart
Jenkins
while
flow
is
running
• Allocate
slave
nodes
and
workspaces
– As
many
as
you
want,
when
you
want
• Stages
throYle
concurrency
of
builds
• Human
input/approval
integrated
into
flow
• Standard
project
concepts:
SCM,
ar9facts,
plugins

Groovy DSL vs. Graphical Workflow
• Familiar
control
flow
construc9on
• Familiar
“tools”
for
building
abstrac9ons
Func9ons,
classes,
variables,
…
• Workflow
in
version
control
• As
opposed
to:
Graphical
workflow
designer

Project setup
• One
workflow
is
defined
as
a
job
• Single
script
for
all
steps
• Build
triggers
&
parameters
like
regular
projects
• Each
workflow
execu9on
is
a
regular
Jenkins
build
displayed
in
regular
Jenkins
views
• Graphical
visualiza9on
of
actual
build
possible
Not
of
visualiza9on
job
defini9on
but
of
build
execu9on

Non-sequential logic
try {} catch {}

Interaction with humans

Restartable build / checkpoint

Visualization

Stages
• Semaphore
• Visualiza9on

Comprehensible scripts

Scaling with Jenkins

Problem Statement:
Scale!
• How
do
I
scale
Jenkins
from
10
to
1000
jobs?
Projects?
Users?
• How
do
I
comply
with
corporate
rules?
• How
do
I
keep
Jenkins
flexible
for
project
teams?

Typical Jenkins installations
Let’s think about this for a few minutes

The top down “structured” and “lets grow
vertically” approach
5
projects
0
Slaves
25
projects
5
Slaves
75
projects
50
Slaves
150
projects
400
Slaves

Vertical Scaling issues
When
is
the
next
failure?
Single
point
of
failure

The “organic” agile growth –
“let each group do what they want to” approach
Horizontally…
virally

Horizontal Issues: I Would Like to…
• Share executors between
masters
– Not possible today
Master-‐a
Master-‐b
Master-‐c

Horizontal Issues: I Would Like to…
• Share security information
– Security realm and roles
• Push configurations & upgrades
Master-‐a
Master-‐b
Master-‐c

oc.cloudbees.com
Jenkins
Opera9ons
Center
Sample Topology
Master-‐a
Master-‐b
Master-‐c
Shared
Roles
,
Security
Realm
and
versions
AYached-‐master
client
AYached-‐master
client
AYached-‐master
client
pushed
to
client
master
pushed
to
client
master

Demo time!

Continuous Delivery traceability with
Jenkins and puppet and Chef
xxx Environment

Puppet environments
Prod
style
environment:
puppet apply
– Master
&
agents
petclinic.war
Dev Environment
petclinic.war
Puppet Agent
Application Server
war file + puppet manifests
& modules
“Production style” Environment
Puppet
Master
Puppet Agent
& modules
Application Server Dev
environment:
– Standalone
agents
+
“puppet apply”

Test environment with Puppet and Vagrant
Dev Environment
petclinic.war
Puppet Agent
Application Server
vagrant provision
puppet apply
& modules
Vagrant
to
create
“clean”
puppet
managed
servers

CI with Jenkins and Vagrant and Puppet
petclinic.war
vagrant provision
puppet apply
Jenkins
builds
war
file
and
puppet
modules
and
deploys
via
vagrant
puppet
deployment
module
java
code Dev Environment
Puppet Agent
Application Server
& modules

CI with Jenkins and Vagrant and Puppet
petclinic.war
notification
vagrant provision
puppet apply
Jenkins
builds
war
file
and
puppet
modules
and
deploys
via
vagrant
puppet
deployment
module
java
code Dev Environment
Puppet Agent
Application Server
& modules

Jenkins – Puppet Integration

Demo time!
https://github.com/CloudBees-community/vagrant-puppet-petclinic/tree/vagrant-puppet-petclinic-0.1.0
jenkins-workflow-with-vagrant-and-puppet.groovy: https://gist.github.com/cyrille-leclerc/7f621c341e193622c844

Jenkins & Puppet
• Jenkins
Deployment
No9fica9on
Plugin
– hYps://wiki.jenkins-‐ci.org/display/JENKINS/Deployment+No9fica9on+Plugin
• Puppet
module
for
Jenkins
tracking
– hYps://github.com/jenkinsci/puppet-‐jenkinstracking
• Puppet
Labs
plugin
for
Jenkins
– hYps://wiki.jenkins-‐ci.org/display/JENKINS/Puppet+Plugin
• CloudBees
DevOps
page
– hYps://developer.cloudbees.com/bin/view/Main/DevOps
• This
demo
– hYps://github.com/CloudBees-‐community/vagrant-‐puppet-‐petclinic
– Jenkins
Workflow
with
Vagrant
and
Puppet:
hYps://gist.github.com/cyrille-‐leclerc/7f621c341e193622c844

Jenkins & Chef?
• Similar
concepts
• Jenkins
Deployment
No9fica9on
Plugin
– hYps://wiki.jenkins-‐ci.org/display/JENKINS/Deployment+No9fica9on+Plugin
• Puppet
module
for
Jenkins
tracking
– hYps://github.com/jenkinsci/puppet-‐jenkinstracking
• Jenkins
Chef
Tracking
– hYps://wiki.jenkins-‐ci.org/display/JENKINS/Chef+Tracking+Plugin
• CloudBees
DevOps
page
– hYps://developer.cloudbees.com/bin/view/Main/DevOps

Conclusion
• Jenkins
workflow
to
build
Con9nuous
Delivery
pipelines
• Jenkins
Opera9ons
Center
by
CloudBees
to
scale
Jenkins
• Puppet
&
Chef
deployments
can
be
integrated
in
Jenkins

CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Destacado

Destacado (15)

Similar a CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)

Similar a CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014) (20)

Más de CloudBees

Más de CloudBees (20)

Último

Último (20)

CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)