This document discusses automating SLI/SLO based build validation with Keptn and Jenkins. It begins by outlining the challenges of lengthy manual approval processes for promoting builds. It then provides inspiration from Google's SRE practices of using Service Level Indicators (SLIs), Service Level Objectives (SLOs), and Service Level Agreements (SLAs). The document demonstrates how Keptn can automate SLI/SLO-based evaluation to integrate with Jenkins pipelines. It includes demos of using Keptn for self-service SLI validation, automating existing Jenkins tests, and enabling performance as a self-service. The document promotes starting resources on GitHub and joining the Keptn community slack channel

1. Automating SLI/SLO based build
validation with Keptn and Jenkins
Andreas Grabner
DevOps Activist at Dynatrace
@grabnerandi
https://www.linkedin.com/in/grabnerandi
https://www.meetup.com/Jenkins-online-meetup
Part of #jenkinsOnlineMeetup
Star us @ https://github.com/keptn/keptn
Follow us @keptnProject
Tutorials @ https://tutorials.keptn.sh
Thu, June 4th @ 3PM GMT+1
Hands-On @ https://github.com/keptn-sandbox/jenkins-tutorial

2. 2Confidential
Use Case: Automate Build Approvals
Through SLI/SLO-based Quality Gates

3. Confidential 3
Challenge #1:Lengthy manualapproval
Build Deploy to
„Test“
Run Test
In „Test“
Manual Approval
Promote to
„Staging“
Deploy to
„Test“
Functional: Test Result Trend Not Enough Performance: Manual Comparison Is Slow Monitoring: Too much unstructed data
Which build is better Which data comes from my testIs this a real problem
~30-60min

4. 4Confidential
Inspirations to solve this problem

5. Confidential 5
Inspiration #1 –Google‘s SREPractices
• Service Level Indicators (SLIs)
• Definition: Measurable Metrics as the base for evaluation
• Example: ErrorRate ofLogin Requests
• Service Level Objectives (SLOs)
• Definition: Binding targets forService Level Indicators
• Example: Login ErrorRate must be less than 2%
• Service Level Agreements (SLAs)
• Definition: Business Agreement between consumer andprovidertypically based on SLO
• Example: Logins must be reliable & fast (ErrorRate, Response Time, Throughput) 99% within a 30 day window
• Google Cloud YouTubeVideo
• SLIs, SLOs, SLAs, oh my! (class SRE implements DevOps): https://www.youtube.com/watch?v=tEylFyxbDLE
SLIs drive SLOs which inform SLAs

6. Confidential 6
Inspiration #2 –ThomasSteinmaurer‘s implementationinContinuousDelivery
“Performance Signature”
for Build Nov 16
“Performance Signature”
for Build Nov 17
“Performance Signature” (SLA)
aggregated for every Build
“Multiple Metrics” (SLIs)
e.g: Memory, Errors …
for specific timeframe
Simple Evaluation (SLOs)
Against a threshold or prev build

7. Confidential 7
SLI/SLO-basedevaluationimplementationinKeptn
SLIs definedperSLI Provider as YAML
SLIProviderspecificqueries,e.g:DynatraceMetricsQuery
QualityGates
...
Dynatrace Prometheus Neoload
Scores SLIs
Queries SLI
Providers with
SLI Definitions &
Timeframe
SLOs definedon Keptn ServiceLevelas YAML
Listofobjectiveswithfixedorrelativepass& warncriteria
indicators:
error_rate: "builtin:service.errors.total.count:merge(0):avg"
count_dbcalls: "calc:service.toptestdbcalls:merge(0):sum"
jvm_memory: "builtin:tech.jvm.memory.pool.committed:merge(0):sum"
objectives:
- sli: error_rate
pass:
- criteria:
- "<=1“ # We expect a max error rate of 1%
- sli: jvm_memory
- sli: count_dbcalls
pass:
- criteria:
- "=+2%" # We allow a 2% increase in DB Calls to previous runs
warning:
- criteria:
- "<=10" # We expect no more than 10 DB Calls per TX
total_score:
pass: "90%"
warning: "75%"
0.5 1.0 0.0 info
7/8
(87.5%)
4/8
(50%)
$ keptn start-evaluation 30m myservice sli.yaml slo.yaml
5 DB Calls 360MB 4.3% 123SLI Value:
SLI Score:
Total Score
2
3
4
Tool X
1

8. Confidential 8
Exampleexplainer onSLI/SLO-basedEvaluation
Overall Failure Rate
Source: Dynatrace
Query: builtin:service.errors.total
Response Time LOGIN Tx
Source: Dynatrace
Query: calc:service.responsetime:filter(tx, LOGIN)
# DB Calls LOGIN Tx
Source: Dynatrace
Query: calc:service.dbcalls:filter(tx, LOGIN)
<= 5%
<= 2%
<=150ms & <=+10%
<= 400ms
<= +0%
<= +1
Build 1
0%
80ms
100ms
SLO: Overall Score Goal 90% 75%
Response Time 95th Perc
Source: Dynatrace
Query: builtin:service.responsetime(p95)
<=100ms
<= 250ms
SLOSLIs (Service Level Indicators) warn
pass
3
100%
Build 2
4%
120ms
90ms
3
75%
Build 3
1%
90ms
120ms
6
62.5%
Build 4
0%
95ms
95ms
3
100%
Build 1 Build 2 Build 3 Build 4
$ keptn send event start-evaluation myproject myservice starttime=build1_deploy endtime=build1_testsdone$ keptn send event start-evaluation myproject myservice starttime=build2_deploy endtime=build2_testsdone$ keptn send event start-evaluation myproject myservice starttime=build3_deploy endtime=build3_testsdone$ keptn send event start-evaluation myproject myservice starttime=build4_teststart endtime=build4_testsend

9. Confidential 9
EvaluationResult intheKeptns Bridge
SLO Result per SLI
OneBuild/Eval
Total Score per Build/Evaluation
Values per SLI
Values across builds/evaluations
Heatmap
Chart

10. Confidential 10
Solution #1:AutomateApprovalthroughSLI/SLO-basedQuality Gates
Build Deploy to
„Test“
Run Test
In „Test“
Manual Approval
Promote to
„Staging“
Deploy to
„Test“
Trigger
Quality Gate
Wait for
Result
SLI & SLO
Result: success, Score: 85/100
Run Test In „Test“
w Tagging
Rt(p95) < 500ms
#ofSQLs <= 5
cpu(max)< 80%
Java GC < 2%
...
T
a
g
g
i
n
g
Pull SLIs for Testing time frame
Validate
SLOs
Build Deploy to
„Test“
Promote to
„Staging“
Deploy to
„Test“~1min
~30-60min

11. 11Confidential
Setting up Keptn to be used byJenkins
Mydemo will use Dynatrace
Using Prometheus? https://tutorials.keptn.sh/?cat=prometheus

12. Confidential 12
Step #1:InstallKeptnwithMonitoring Support for Dynatrace:tutorials.keptn.sh
api.keptn.*
bridge.keptn.*
Evalulation Testing
Deploy Notify
...
...
...
...
$ keptn install (full or quality gate)
$ keptn configure monitoring dynatrace
Pre-Requisits
k8s or Ubuntu Linux (for microk8s)
Dynatrace Tenant, API & PaaS Token
Installation on k8s/OpenShift
Installation on Ubuntu
https://tutorials.keptn.sh/?cat=microk8s
GitOps
Dev Staging Production
Project: perfproject (repo)
• Stage: performance (branch)
/Service: perfservice (folder)
slo.yaml
dynatrace/sli.yaml
jmeter/mytest.jmx
API
API
Push Events (Start Test, Evaluation Done ...)
Pull Metrics (SLI)
Configuration (Tagging Rules, Dashboards ...)
$ keptn add-resource sli.yaml, slo.yaml, ...
$ keptn create project perfproject
Setup a Keptn Project (will automate through Jenkins)
$ keptn create service perfservice

13. Confidential 13
Step #2:Instrument yourappenv andprepare yourtests
docker run -p 80:8080 yourregistry/yourcontainer
I
n
f
r
a
P
r
o
c
e
s
s
S
e
r
v
i
c
e
T
e
s
t
homepage
version
echo
invoke
homepage
version
echo
invoke
Monitor the enviornment your CI/CD Deploys to
deploy Kubectl apply –f yourapp.yaml
java –jar yourapp.jar
CI/CD Envs

14. Confidential 14
Step #3:Create SLIs & SLOs basedon e.g:yourdashboards
Take the metrics & tags that work for you
And list them in your SLI.yaml including filter for Tag and Test Step
And start with a simple SLO.yaml, e.g: within 10% of last build

15. Confidential 15
Nowwe areready to haveKeptn automatetheanalysis forusandintegrate it inJenkins
Instead of manually comparing timeframes
1
2
3
4
1 2 3 4 x
1 2 3 4 x
Keptn automates that process based on SLIs & SLOs
X

16. Confidential 16
Demo#1:based onhttps://github.com/keptn-sandbox/jenkins-tutorial
1 2
3

17. Confidential 17
Demo#1:SLI/SLO Evaluationas aSelf-Service aka„Validate yourSLIs are working“
Keptn Init
Trigger
Quality Gate
Wait for Result
init "qgproject" "qualitystage"
"evalservice" "dynatrace"
addResource "dynatrace/sli.yaml"
addResource "slo.yaml"
addResource "dynatrace/dynatrace.conf.yaml"
sendStartEvaluationEvent "600" waitForEvaluationDoneEvent
GitOps
Project: qgproject (repo)
• Stage: qualitystage (branch)
/Service: evalservice (folder)
slo.yaml
dynatrace/sli.yaml
dynatrace/dynatrace.conf.yaml
builtin:service.requestCount.total:merge(0):sum
&entitySelector=tag($SERVICE),type(SERVICE)
builtin:service.requestCount.total:merge(0):sum
&entitySelector=tag(evalservice),type(SERVICE)
&from=now-600s&to=now
Each SLI will be transformed & pulled Result: failure, Score: 60/100
dynatrace.com/api/v2/metrics/query/transformedSLI
Keptn
Jenkins
Library
$ keptn add-resource sli.yaml, slo.yaml, ...$ keptn create project qgproject (qualitystage)$ keptn create service evalservice
$ keptn configure monitoring dynatrace qgproject
Quality Gate Result Event via
dynatrace.com/api/v2/events

18. Confidential 18
Demo#2:based onhttps://github.com/keptn-sandbox/jenkins-tutorial
1 2
3

19. Confidential 19
Demo#2:AutomateSLI/SLO Evaluationfor existing Tests executed inyourJenkinsPipeline
Keptn Init
Trigger
Quality Gate
Wait for Result
init "testwithqgproject" "qualitystage"
"testservice" "dynatrace"
addResource "dynatrace/sli.yaml"
addResource "slo.yaml"
addResource "dynatrace/dynatrace.conf.yaml"
sendStartEvaluationEvent waitForEvaluationDoneEvent
GitOps
Project: testwithqgproject (repo)
• Stage: qualitystage (branch)
/Service: testservice (folder)
slo.yaml
dynatrace/sli.yaml
dynatrace/dynatrace.conf.yaml
builtin:service.requestCount.total:merge(0):sum
&entitySelector=tag($SERVICE),type(SERVICE)
builtin:service.requestCount.total:merge(0):sum
&entitySelector=tag(testservice),type(SERVICE)
&from=startoftest&to=now
Each SLI will be transformed & pulled Result: failure, Score: 60/100
dynatrace.com/api/v2/metrics/query/transformedSLI
Keptn
Jenkins
Library
Run Simple Load Test
markEvaluationStartTime
-> Trigger your tests
Quality Gate Result Event via
dynatrace.com/api/v2/events

20. 20Confidential
Use Case: Performance-Driven Culture
Provide Performance as a Self-Service

21. Confidential 21
Challenge #2:AutomatingPerformance with Jenkinsrequires managingtools &environments
Build
Deploy to
„Test“
Standup Test
Infrastructure
Gather
Performance Data
How to enable different workloads?
How much hardware is needed to run these tests?Where do we run these tests?
Execute Tests
Evaluate
Results
Where do we stream test metrics to? How to analyze the results?
Who manages this infrastructure?
DIY (Do It Yourself) approach: lots of online guides available!

22. Confidential 22
Challenge #2:AutomatingPerformance with Jenkinsrequires managingtools &environments
Build
Deploy to
„Test“
Standup Test
Infrastructure
Gather
Performance Data
Execute Tests
Evaluate
Results
Build Deploy to „Test“ +
Notify Keptn
Wait
for Result
SLI & SLO
Result: success, Score: 85/100
Rt(p95) < 500ms
#ofSQLs <= 5
cpu(max)< 80%
Java GC < 2%
...
Pull SLIs for Testing time frame
Validate
SLOs
Test Scripts + Workload

23. Confidential 23
Demo#3:based onhttps://github.com/keptn-sandbox/jenkins-tutorial
1 2
3

24. Confidential 24
Demo#3:Performance asa Self-Service
Keptn Init
Trigger
Quality Gate Wait for Result
init "perfaasproject" "performance"
"perfaasservice" "dynatrace"
addResource "dynatrace/sli.yaml"
addResource "slo.yaml"
addResource "dynatrace/dynatrace.conf.yaml"
sendDeploymentFinishedEvent "http://appundertest.local" waitForEvaluationDoneEvent
GitOps
Project: testwithqgproject (repo)
• Stage: qualitystage (branch)
/Service: testservice (folder)
slo.yaml
dynatrace/sli.yaml
dynatrace/dynatrace.conf.yaml
requestCount:tag($SERVICE)
,type(SERVICE)
requestCount:
tag(perfaasservice),
type(SERVICE)
&from=startoftest&to=now
SLI Magic (short) Result: failure, Score: 60/100
Keptn
Jenkins
Library
addResource "jmeter/load.jmx"
addResource "jmeter/jmeter.conf.yaml"
jmeter/load.jmx
jmeter/jmeter.conf.yaml
Testing
url: http://appundertest.local
Quality Gate Result Event via
dynatrace.com/api/v2/events
Deploy & Test Events via
dynatrace.com/api/v2/events

25. Confidential 25
Additionallyinstalled Keptn Services inmydemos
• JMeterExtended Version
• Supporting multiple workload configurations via jmeter.conf.yaml
• https://github.com/keptn-contrib/jmeter-extended-service

26. 26Confidential
Keptn Jenkins Shared Library
https://github.com/keptn-sandbox/keptn-jenkins-library

27. Confidential 27
Using theLibrary: fullreadme on https://github.com/keptn-sandbox/keptn-jenkins-library
• 1:Add it in yourJenkinsGlobal Settings
• 2:ConfigureKeptn Properties: KEPTN_API_TOKEN,KEPTN_ENDPOINT,KEPTN_BRIDGE
• 3:Add it to yourpipeline

28. Confidential 28
Current Plansfor the Keptn JenkinsLibrary
• Support Credentials vs EnvironmentVariables for Keptn Tokenand Endpoint
• Potentially createa JenkinsPlugin to visualizeHeatmap in Jenkins
• Provide callbacks to Keptn whenKeptn calls JenkinsPipelines as part of end-2-enddelivery orchestration
• Extension ofhttps://github.com/keptn-sandbox/jenkins-service

29. Automating SLI/SLO based build
validation with Keptn and Jenkins
Andreas Grabner
DevOps Activist at Dynatrace
@grabnerandi
https://www.linkedin.com/in/grabnerandi
https://www.meetup.com/Jenkins-online-meetup
Part of #jenkinsOnlineMeetup
Star us @ https://github.com/keptn/keptn
Follow us @keptnProject
Tutorials @ https://tutorials.keptn.sh
Thu, June 4th @ 3PM GMT+1
Hands-On @ https://github.com/keptn-sandbox/jenkins-tutorial

30. Confidential 30
Q&A &ActionItems
• Please STARour projects on GitHub
• https://github.com/keptn/keptn
• https://github.com/keptn-sandbox/jenkins-service
• https://github.com/keptn-sandbox/keptn-jenkins-library
• https://github.com/keptn-sandbox/jmeter-extended-service
• JoinourSlack Channel
• https://github.com/keptn/community
• Tryour tutorials:
• https://tutorials.keptn.sh
• https://github.com/keptn-sandbox/jenkins-tutorial
• Giveus feedback and feel freeto contribute 

31. 31Confidential
Bonus Use Case: Break the Monolithic Pipeline
Let Keptn orchestrate your „Jenkins Micro-Pipelines“
using https://github.com/keptn-sandbox/jenkins-service

32. Confidential 32
Most monoliths are NOT enabled for „Cloud Native“
Cloud Native

33. Confidential 33
Mixedinformationabout
• Process(build, deploy,test,evaluate,…)
• Targetplatform (k8s, …)
• Environments(dev,hardening,…)
• Tools(Terraform,Helm,hey,…)
No clear separationof concerns
• Developers
• Define which artifact to use
• Want fast feedback on their code
• DevOpsEngineers
• Define which tools to use
• Ensure tools areproperly configured
• SiteReliabilityEngineers
• Define delivery processes
• Define operations workflows
33
It starts with pipelines whoare built like the monolithic counterparts!

34. Confidential 34
The experience weare looking for?
Not this! But this!

35. Confidential 35
Removehard dependencies and integrations
Build
Prepare
Deploy
Test
Notify
Rollback
Config Mgmt.
Deploy
Test
Monitoring
ChatOps
Rollback

36. Confidential 36
Removehard dependencies and integrations
Build
Prepare
Deploy
Test
Notify
Rollback
Config Mgmt.
Deploy
Test
Monitoring
ChatOps
Rollback
Eventing
Event:Deploy
Artifact:container1
Stage:Dev
Strategy:Blue/Green

37. Confidential 37
Removehard dependencies and integrations
Build
Prepare
Deploy
Test
Notify
Rollback
Eventing
Event:Test
URL:myservice
Stage:Dev
Strategy:Performance
which events to generate who consumes events
Config Mgmt.
Deploy
Test
Monitoring
ChatOps
Rollback

38. 38Confidential
How Keptnaddresses this problem

39. Confidential 39
Eventing
Keptn is built on an architecture that separates concerns
Application Plane
Define overall process for delivery and operations (stages, deployment, tests, remediation)
Control Plane
Follow application logic and communicate/configure required services
Deploy
Service
Test
Service
Validation
Service
Monitoring
Service
Config
Service
Remediate
Service
Artifact /
Microservice
API & UI
Site Reliability
Engineer
DevOps
Developer
shipyard.yaml
uniform.yaml

40. Confidential 40
Use Jenkins-Service https://github.com/keptn-sandbox/jenkins-service
1. Follow the readme on GitHub
2. Wait for next Keptn Webinar  for a deep dive
jenkins/jenkins.conf.yaml