Jenkins Pipeline @ Scale. Building Automation Frameworks for Systems Integration

1. Jenkins Pipeline @ Scale Building Automation Frameworks for Systems Integration CloudBees | Jenkins Automotive and Embedded Day Böblingen, April 05, 2017 Oleg Nenashev, CloudBees

6. © 2017 CloudBees, Inc. All Rights Reserved. 6 ØHow/why do we build frameworks for Embedded? ØWhat has changed in Pipeline since 2016? ØHow to effectively use and develop Pipeline? ØLessons learnt Agenda

7. © 2017 CloudBees, Inc. All Rights Reserved. 7 • Pipeline 101 • Pipeline Generic Features • Blue Ocean • Declarative Pipeline • Automation Infrastructure • Agents, Tools, Hardware Peripherals • Added-value Pipeline features in CloudBees Jenkins Enterprise What is NOT in the talk? Ask the Experts, Jenkins Online Meetup Birds of Feather, Ask the Experts

8. © 2017 CloudBees, Inc. All Rights Reserved. 8 What is a typical Embedded Product? ` Hardware External IPFirmware / Embedded SW External SW Development tools / EDAs Demo software Demo hardware Developer kits “Main product” Documentation

9. © 2017 CloudBees, Inc. All Rights Reserved. 9 Embedded area. Welcome to the automation hell! • Almost everything needs to be automated • Multiple technologies and tools • Integration testing is important • Unstable hardware & networks • Legacy stuff is everywhere • …

10. © 2017 CloudBees, Inc. All Rights Reserved. 10 OK, Automotive… Source: http://www.popsautoelectric.com/auto-electric/ • System of systems • High level of integration • Components by 3rd parties • Many standards • High quality requirements

11. © 2017 CloudBees, Inc. All Rights Reserved. 11 1. The most popular CI automation server in the world 2. It’s a framework => flexible • Swiss Army Knife of Automation 3. It’s open source => customizable 4. Commercial support is available 5. Useful plugins Why Jenkins?

12. © 2017 CloudBees, Inc. All Rights Reserved. 12 Система Docs Plugins Configs Infra- structure Samples Demos How do we build CI Systems? Infra- structure Infra- structure • Agents (FKA ”slaves”) • Tools • Hardware peripherals • Computing grids

13. © 2017 CloudBees, Inc. All Rights Reserved. 13 Approach: Using “classic” Job types: • Freestyle, Matrix, JobDSL, … Classic Approaches Keywords: • Script steps to run the logic • Custom Tools Plugin to manage tools • Throttle Concurrent Builds and Lockable Resources to dispatch access to hardware • Naginator Plugin to restart builds on-demand • Parameterized Trigger / Copy artifacts to parallelize tests and to collect results JUC2015/London https://jenkins.io/solutions/embedded/

14. © 2017 CloudBees, Inc. All Rights Reserved. 14 Classic Approaches Spaghetti Automation • Barely traceable dependencies • Complex build flow • Complex configurations • Duplication • Expensive maintenance

17. © 2017 CloudBees, Inc. All Rights Reserved. 17 Jenkins System Configuration… as Code External Tools Jenkins CLI and REST API python- jenkins jenkins-client (java) Configuration Management systems Ansible, Chef Docker Solutions within Jenkins project Groovy Boot Hooks Scriptler Plugin SCM Sync Configuration Unstable Just examples…

19. © 2017 CloudBees, Inc. All Rights Reserved. 19 Pipeline is a… Ø Groovy DSL for automatic flow description * • Alternatives – Job DSL, Jenkins Job Builder Ø Job type in Jenkins (Pipeline Job) • Job definitions can be stored in SCM • Automatic job creation from repositories and organizations Ø Ecosystem for automation as code • ~20 dedicated plugins • Integration in many other plugins • Suggested by default in Jenkins 2 // Run on a node having the “my_fpga” label node("linux && ml509") { git url:"http://github.com/myorg/myproject.git" sh "make all" } http://bit.ly/pipeline-tutorial 19

21. © 2017 CloudBees, Inc. All Rights Reserved. 21 Why Jenkins Pipeline? • Robustness - master restarts & network • Parallelization across multiple nodes: • parallel() • node() • Ability to store the shared logic in standalone Pipeline libraries • Security • Two modes: Scripted and Declarative • Visualization: Stage View, Blue Ocean

22. © 2017 CloudBees, Inc. All Rights Reserved. 22 Система Docs Plugins Configs Infra- structure Samples Demos Automation Framework Vision Infra- structure Infra- structure SCM Infrastructure as Code Configuration as Code Documentation as Code Pipeline Libraries Pipelines

23. © 2017 CloudBees, Inc. All Rights Reserved. 23 Developer Experience • Encapsulation • Simple UI • Most of the code is in GitHub • No need in classic UI for 90% of users • Notifications to email/GitHub

24. Jenkins Pipeline in Embedded 2016

26. © 2017 CloudBees, Inc. All Rights Reserved. 26 2016 – Summary • Jenkins can be used to run tests on hardware in embedded and automotive projects • Jenkins Pipeline is helpful for test runs • Pipeline lacks some critical features

27. © 2017 CloudBees, Inc. All Rights Reserved. 27 • Management of resources. • Integration with Throttle Concurrent Builds plugin • Use-case: limiting the license utilization in automation infrastructures (JENKINS-31801) • Better support of tools. • Common build flow – dozens of tools • Integration with Custom Tools Plugin seems to be the best option, especially in the case of multiple tool versions (JENKINS-30680) • Pipeline package manager (JENKINS-34186) • Lack of plugins for EDA tools in Jenkins, code duplications • A common approach is to keep the shared "functions" in libraries • Pipeline Global Library and Pipeline Remote Loader can be used 2016 – Pipeline improvement areas From my talk at CloudBees | Jenkins Automotive and Embedded Day 2016

28. © 2017 CloudBees, Inc. All Rights Reserved. 28 • Management of resources. • Integration with Throttle Concurrent Builds plugin • Use-case: limiting the license utilization in automation infrastructures (JENKINS-31801) • Better support of tools. • Common build flow – dozens of tools • Integration with Custom Tools Plugin seems to be the best option, especially in the case of multiple tool versions (JENKINS-30680) • Pipeline package manager (JENKINS-34186) • Lack of plugins for EDA tools in Jenkins, code duplications • A common approach is to keep the shared "functions" in libraries • Pipeline Global Library and Pipeline Remote Loader can be used 2016 – Pipeline improvement areas From my talk at CloudBees | Jenkins Automotive and Embedded Day 2016

30. © 2017 CloudBees, Inc. All Rights Reserved. 30 PARTIAL - Better support of tools • withEnv() for environment variables • tool() step in Pipeline was improved • Tool reference in the Job • Section in Declarative Pipeline • Custom tools is not fully integrated • Follow JENKINS-30680 • ETA – Q2, 2017

31. © 2017 CloudBees, Inc. All Rights Reserved. 31 DONE – Pipeline Libs (Pipeline Package Manager) • Early 2016 • Global Pipeline Library • Pipeline Remote Loader Plugin • Late 2016 • Pipeline Shared Libraries • Support of @Grab – Pipeline libs can pull common Groovy libs Library Structure:

36. © 2017 CloudBees, Inc. All Rights Reserved. 36 Using Pipeline Libraries com.mycorp.libs.make Jenkinsfile makeProject() makeProject() implementation - global variable com.mycorp.libs.coverity com.mycorp.libs.cppunit Globally-defined library, automatic update @Library() definition

37. © 2017 CloudBees, Inc. All Rights Reserved. 37 Early adoption of any framework http://devopsreactions.tumblr.c om/post/150674592321/pettin g-your-incomprehensible- piece-of-code • It is working! • In several pet projects…

38. Lessons Learnt. What does NOT work well?

39. Lessons Learnt. What does NOT work well? • Development Tools • Non-Success Path handling • Declarative Pipeline • Manual release management

41. © 2017 CloudBees, Inc. All Rights Reserved. 41 How do I work? Intellij IDEA Filesystem SCM Plugin (with patch) Disclaimer: There may be better ways • Syntax file • Built-in documentation Jenkins Test Instance Source Code (local .git repos) • Production System Snapshot • Custom Pipeline Library Settings • Repositories for libraries • Jenkinsfile repos

44. © 2017 CloudBees, Inc. All Rights Reserved. 44 Pipeline Testing • No standard test framework offered by the Jenkins project • External project: https://github.com/lesfurets/JenkinsPipelineUnit “mvn verify” has been invoked… And the execution passed… For the source Jenkinsfile…

49. © 2017 CloudBees, Inc. All Rights Reserved. 49 Error Handling. Hardware failover • Pipeline allows re-running unreliable stages on other nodes. System failure (script returns 253) Node #1 Node #2. . . . . . for (def board : boards) { echo "trying board " + board; try { node(board) { checkout scm sh ‘./bin/run.sh’ // Call passed => DONE break; } } catch (Exception ex) { if (ex.message.contains ("exit code 253")) { // Fatal error fail("Test run failed") } } } Try all possible boards:

50. © 2017 CloudBees, Inc. All Rights Reserved. 50 Error handling. Hardware failover for (def board : boards) { echo "trying board " + board; try { node(board) { checkout scm sh ‘./bin/run.sh’ // Call passed => DONE break; } } catch (Exception ex) { if (ex.message.contains ("exit code 255")) { // Fatal error fail("Test run failed") } } } ??? • Hard to extract error cause • Special handling is required for every case • Cannot just use existing parsers • Build Failure Analyzer • Log Parser • Warnings • Try-catch blocks pollute the code • Hard to test

52. © 2017 CloudBees, Inc. All Rights Reserved. 52 Error handling. Hardware failover Declarative Pipeline Encapsulation via Libraries Simple use-case Other • Exception try/catch • timeout() wrapper • Proper failover • Logic in @NonCPS

54. © 2017 CloudBees, Inc. All Rights Reserved. 54 Declarative Pipeline • It is great for simple use-cases • Publishers & Notifiers • We do not need a framework for simple use-cases • Lack of features • “How do I pass option for the Docker Agent” • “How do I declare a resource requirement for Throttling?” • … • Scripted Pipeline is a recommended fallback, but there is no automatic migration

55. © 2017 CloudBees, Inc. All Rights Reserved. 55 Declarative Pipeline. Recommendations • Use Declarative Pipeline with caution • Complex agent definitions – use Scripted Pipeline • Complex throttling – use Scripted Pipeline • Post-build steps – work well • Tool definitions – work well • …

59. © 2017 CloudBees, Inc. All Rights Reserved. 59 Solutions • Input() step + milestone() for synchronization • Multiple Pipelines • Release() step is coming soon • Release Plugin Pipeline 1 Pipeline 2 Run Selector Staging Artifact Storage External ALM tool CD for the Component 1 CI/CD for the Component 2

60. Lessons learnt • No area-specific issues so far • High barrier to entry • It worth investment

61. © 2017 CloudBees, Inc. All Rights Reserved. 61 What I need? 2017 • Development Tools for Pipeline • IDE, Debugger, Test Tools • Community Pipeline Libraries • Common Embedded cases without plugins • Promotion engine for Pipeline • Ideally: Merge of standard Promoted Builds and Release plugins • Plugins for area-specific tools

62. © 2017 CloudBees, Inc. All Rights Reserved. 62 Takeaways • Jenkins Pipeline is a powerful engine for highly-customized Jenkins instances • Jenkins Pipeline can be effectively used in Embedded/Automotive use-cases • Barrier to entry is still high

63. © 2017 CloudBees, Inc. All Rights Reserved. 63 More info • Embedded solutions page: • https://jenkins.io/solutions/embedded/ • Links to the articles and videos • Some categorization Embedded Solutions Page …

64. © 2017 CloudBees, Inc. All Rights Reserved. 64 • Jenkins project: • Website: https://jenkins.io • Embedded Solutions page: https://jenkins.io/solutions/embedded/ • Pipeline: • Tutorial: https://github.com/jenkinsci/workflow-plugin/blob/master/TUTORIAL.md • Compatibility: https://github.com/jenkinsci/pipeline-plugin/blob/master/COMPATIBILITY.md • Examples: https://github.com/jenkinsci/pipeline-examples • CloudBees website: • Website: https://www.cloudbees.com • CloudBees Jenkins Platform: https://www.cloudbees.com/products/cloudbees-jenkins-platform Useful Links

66. Software at the speed of ideas THANK YOU! www.cloudbees.com