PyCon-IL 2017 lecture about running Python on Jenkins.

1. The Snake and theThe Snake and the
ButlerButler
Jenkins as a Python execution
platform

2. Who am I?Who am I?
Barak Korren
bkorren@redhat.com
http://ifireball.wordpress.com
https://twitter.com/BKorren

3. AgendaAgenda
● What is Jenkins / Why do it
● Running Python code
● Doing I/O
● Persisting state

4. Jenkins?Jenkins?
“CI/CD Automation tool"

5. Jenkins?Jenkins?

6. Jenkins?Jenkins?

7. Jenkins?Jenkins?

8. Why?Why?

9. Jenkins JobsJenkins Jobs
“Free Style" Pipeline
String code
String status_file = 'calculator_status.dat'
stage('Load code') {
dir('python-on-jenkins') {
git(
poll: false,
url: repo_url
)
code = readFile code_file
}
}
stage('Load data') {
step([
$class: 'CopyArtifact',
filter: status_file,
fingerprintArtifacts: true,
projectName: env.JOB_NAME,
optional: true,
])
}

10. RunningRunning
Python codePython code

11. Python cmd linePython cmd line
def main():
args = parse_args()
print("Hello {}".format(args.who))
def parse_args():
"""Parse command line arguments
"""
parser = argparse.ArgumentParser(
Description=
'Tool that says hello'
)
parser.add_argument(
'who',
default='world',
nargs='?'
)
return parser.parse_args()

12. Python pluginsPython plugins
● "python"
● "ShiningPanda"
● "jython"

13. Python for “free”Python for “free”
node {
stage('Say hello') {
sh '''
#!/usr/bin/env python
from __future__ import
print_function
print('Hello world!')
'''.stripIndent()
}
}

14. Doing I/ODoing I/O

15. InputInput
from os import environ
name = environ.get('YOUR_NAME', 'World')
print('Hello {0}!'.format(name))

16. OutputOutput
with open('job_params.properties', 'w') as f:
f.write('YOUR_NAME={0}n'.format(name))

17. Move ObjectsMove Objects
def object_to_param_str(obj):
return b64encode(compress(
cPickle.dumps(obj)
)).decode('utf8')
def param_str_to_object(param_str):
return cPickle.loads(decompress(
b64decode(param_str.encode('utf8'))
))

18. Output to GroovyOutput to Groovy
build_spec = dict(
job='hello-with-params',
parameters=[{
'$class':
'StringParameterValue',
'name': 'YOUR_NAME',
'value': name
}]
)
with open('bld_spec.json', 'w') as f:
json.dump(build_spec, f)

19. Output to GroovyOutput to Groovy
build_spec = dict(
job='hello-with-params',
parameters=[{
'$class':
'StringParameterValue',
'name': 'YOUR_NAME',
'value': name
}]
)
with open('bld_spec.json', 'w') as f:
json.dump(build_spec, f)
build readJSON(
file: 'bld_spec.json'
)

20. Persisting statePersisting state

21. Persisting statePersisting state
Build #8Build #8 Build #9Build #9
Artifact StorageArtifact Storage
Archive Load

22. Persisting statePersisting state
def object_from_artifact(artifact_file, fallback_cls):
try:
with open(artifact_file) as fd:
return cPickle.load(fd)
except IOError as e:
# errno 2 is 'No such file or directory'
if e.errno == 2:
return fallback_cls()
raise
def object_to_artifact(obj, artifact_file):
with open(artifact_file, 'w') as fd:
cPickle.dump(obj, fd)

23. Persisting statePersisting state
@contextmanager
def persist_in_artifacts(artifact_file, fallback_cls):
obj = object_from_artifact(artifact_file,
fallback_cls)
yield obj
object_to_artifact(obj, artifact_file)

24. Persisting statePersisting state
@contextmanager
def persist_in_artifacts(artifact_file, fallback_cls):
obj = object_from_artifact(artifact_file,
fallback_cls)
yield obj
object_to_artifact(obj, artifact_file)
class CalculatorStatus(object):
def __init__(self):
self.current_value = 0

25. Persisting statePersisting state
@contextmanager
def persist_in_artifacts(artifact_file, fallback_cls):
obj = object_from_artifact(artifact_file,
fallback_cls)
yield obj
object_to_artifact(obj, artifact_file)
class CalculatorStatus(object):
def __init__(self):
self.current_value = 0
with persist_in_artifacts(status_file,
CalculatorStatus) as status:
status.current_value = ...

26. Persisting statePersisting state
stage('Load data') {
dir('state') { deleteDir(); touch(file: '_dummy_') }
step([
$class: 'CopyArtifact',
filter: status_file,
projectName: env.JOB_NAME,
optional: true,
])
}
stage('Run python') {
// ...
}
stage('Save data') {
archive status_file
}

27. RecapRecap
●
Running code via simple "shell" build
steps
● Input via env vars
●
Output via files (properties, json,
Pickle)
●
Object persistence via archived Pickle
files

28. Sharing the JVMSharing the JVM
● Jython:
– http://www.jython.org/
● Java classes in Jython
– https://github.com/jythontools/clamp/
● Python (Jython) Plugins
– https://github.com/jenkinsci/jenkins.py

29. Jython from GroovyJython from Groovy
// load interpreter from Maven
@Grab(group='org.python', module='jython-standalone',
version='2.7.0')
import org.python.util.PythonInterpreter
import org.python.core.PyString

30. Jython from GroovyJython from Groovy
@NonCPS
def loadPython() {
// create interpreter object
PythonInterpreter interpreter = new PythonInterpreter()
// compile and run python
interpreter.exec """
def some_python_func(some_arg):
# python code here...
""".stripIndent()
// return some functions
return interpreter.get('some_python_func')
}

31. Jython from GroovyJython from Groovy
pyfunc = loadPython()
# Call Python function
pyfunc.__call__(new PyString("some str..."))

32. ApplicationsApplications
//
DemoDemo

33. Thank You!Thank You!
● Slides:
– http://wp.me/p7png-Ct
● Source code:
– https://github.com/ifireball/python-on-jenkins
● Contact me:
– bkorren@redhat.com

34. Jenkins Job BuilderJenkins Job Builder
https://docs.openstack.org/infra/jenkins-job-builder/
- job-template:
name: some_job_a
parameters:
- name-params
builders:
- shell: !include-raw-escape: code.py
publishers:
- trigger-parameterized-builds:
- project: some_job_b
property-file: job_params.properties
condition: SUCCESS

35. ApplicationsApplications
● oVirt "Standard-CI":
– oVirt has many
different sub-projects
– The CI system is
project-agnostic
– Each project tell the CI
what to do
– We have local tool to
emulate CI system

36. ApplicationsApplications
●
oVirt “change-gating" flow:
– Runs “heavy” tests on batches of changes
– Runs bisection on failure to find root cause
Project 1Project 1
Project 2Project 2
Project NProject N
Change
queue
Change
queue
TestsTests