Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Jenkins Pipeline @ Scale. Building Automation Frameworks for Systems Integration

1.353 visualizaciones

Publicado el

This is a follow-up presentation to my talk at CloudBees | Jenkins Automotive and Embedded Day 2016, where I was presenting Pipeline usage strategies for use-cases in the Embedded area. In this presentation I talk about Jenkins Pipeline features for automation frameworks and talk about lessons learned in several project.

Publicado en: Software
  • Inicia sesión para ver los comentarios

Jenkins Pipeline @ Scale. Building Automation Frameworks for Systems Integration

  1. 1. Jenkins Pipeline @ Scale Building Automation Frameworks for Systems Integration CloudBees | Jenkins Automotive and Embedded Day Böblingen, April 05, 2017 Oleg Nenashev, CloudBees
  2. 2. © 2017 CloudBees, Inc. All Rights Reserved. 2 About me. Hardware/Embedded @oleg_nenashev oleg-nenashev LibreCoresSt. Petersburg Polytechnic University
  3. 3. © 2017 CloudBees, Inc. All Rights Reserved. 3 About me. Jenkins project @oleg_nenashev oleg-nenashev
  4. 4. © 2017 CloudBees, Inc. All Rights Reserved. 4 Oleg’s “Hall of Shame”(c) • Jenkins Core • Remoting • Plugins
  5. 5. © 2017 CloudBees, Inc. All Rights Reserved. 5 2016 http://bit.ly/nenashev-automotive-day-2016
  6. 6. © 2017 CloudBees, Inc. All Rights Reserved. 6 ØHow/why do we build frameworks for Embedded? ØWhat has changed in Pipeline since 2016? ØHow to effectively use and develop Pipeline? ØLessons learnt Agenda
  7. 7. © 2017 CloudBees, Inc. All Rights Reserved. 7 • Pipeline 101 • Pipeline Generic Features • Blue Ocean • Declarative Pipeline • Automation Infrastructure • Agents, Tools, Hardware Peripherals • Added-value Pipeline features in CloudBees Jenkins Enterprise What is NOT in the talk? Ask the Experts, Jenkins Online Meetup Birds of Feather, Ask the Experts
  8. 8. © 2017 CloudBees, Inc. All Rights Reserved. 8 What is a typical Embedded Product? ` Hardware External IPFirmware / Embedded SW External SW Development tools / EDAs Demo software Demo hardware Developer kits “Main product” Documentation
  9. 9. © 2017 CloudBees, Inc. All Rights Reserved. 9 Embedded area. Welcome to the automation hell! • Almost everything needs to be automated • Multiple technologies and tools • Integration testing is important • Unstable hardware & networks • Legacy stuff is everywhere • …
  10. 10. © 2017 CloudBees, Inc. All Rights Reserved. 10 OK, Automotive… Source: http://www.popsautoelectric.com/auto-electric/ • System of systems • High level of integration • Components by 3rd parties • Many standards • High quality requirements
  11. 11. © 2017 CloudBees, Inc. All Rights Reserved. 11 1. The most popular CI automation server in the world 2. It’s a framework => flexible • Swiss Army Knife of Automation 3. It’s open source => customizable 4. Commercial support is available 5. Useful plugins Why Jenkins?
  12. 12. © 2017 CloudBees, Inc. All Rights Reserved. 12 Система Docs Plugins Configs Infra- structure Samples Demos How do we build CI Systems? Infra- structure Infra- structure • Agents (FKA ”slaves”) • Tools • Hardware peripherals • Computing grids
  13. 13. © 2017 CloudBees, Inc. All Rights Reserved. 13 Approach: Using “classic” Job types: • Freestyle, Matrix, JobDSL, … Classic Approaches Keywords: • Script steps to run the logic • Custom Tools Plugin to manage tools • Throttle Concurrent Builds and Lockable Resources to dispatch access to hardware • Naginator Plugin to restart builds on-demand • Parameterized Trigger / Copy artifacts to parallelize tests and to collect results JUC2015/London https://jenkins.io/solutions/embedded/
  14. 14. © 2017 CloudBees, Inc. All Rights Reserved. 14 Classic Approaches Spaghetti Automation • Barely traceable dependencies • Complex build flow • Complex configurations • Duplication • Expensive maintenance
  15. 15. © 2017 CloudBees, Inc. All Rights Reserved. 15 ü Store build definitions with the project ü Track changes together with the project ü Tests together with the project Configuration as Code
  16. 16. © 2017 CloudBees, Inc. All Rights Reserved. 16 Configuration as Code in Jenkins Job ConfigurationJenkins system configuration
  17. 17. © 2017 CloudBees, Inc. All Rights Reserved. 17 Jenkins System Configuration… as Code External Tools Jenkins CLI and REST API python- jenkins jenkins-client (java) Configuration Management systems Ansible, Chef Docker Solutions within Jenkins project Groovy Boot Hooks Scriptler Plugin SCM Sync Configuration Unstable Just examples…
  18. 18. © 2017 CloudBees, Inc. All Rights Reserved. 18 Configuration as Code for jobs Job DSL Job Builder Plugin Jenkins Pipeline Pipeline is offered by default in Jenkins 2.0 … Groovy YAML Groovy
  19. 19. © 2017 CloudBees, Inc. All Rights Reserved. 19 Pipeline is a… Ø Groovy DSL for automatic flow description * • Alternatives – Job DSL, Jenkins Job Builder Ø Job type in Jenkins (Pipeline Job) • Job definitions can be stored in SCM • Automatic job creation from repositories and organizations Ø Ecosystem for automation as code • ~20 dedicated plugins • Integration in many other plugins • Suggested by default in Jenkins 2 // Run on a node having the “my_fpga” label node("linux && ml509") { git url:"http://github.com/myorg/myproject.git" sh "make all" } http://bit.ly/pipeline-tutorial 19
  20. 20. © 2017 CloudBees, Inc. All Rights Reserved. 20 Why Jenkins Pipeline?
  21. 21. © 2017 CloudBees, Inc. All Rights Reserved. 21 Why Jenkins Pipeline? • Robustness - master restarts & network • Parallelization across multiple nodes: • parallel() • node() • Ability to store the shared logic in standalone Pipeline libraries • Security • Two modes: Scripted and Declarative • Visualization: Stage View, Blue Ocean
  22. 22. © 2017 CloudBees, Inc. All Rights Reserved. 22 Система Docs Plugins Configs Infra- structure Samples Demos Automation Framework Vision Infra- structure Infra- structure SCM Infrastructure as Code Configuration as Code Documentation as Code Pipeline Libraries Pipelines
  23. 23. © 2017 CloudBees, Inc. All Rights Reserved. 23 Developer Experience • Encapsulation • Simple UI • Most of the code is in GitHub • No need in classic UI for 90% of users • Notifications to email/GitHub
  24. 24. Jenkins Pipeline in Embedded 2016
  25. 25. © 2017 CloudBees, Inc. All Rights Reserved. 25 2016 http://bit.ly/nenashev-automotive-day-2016
  26. 26. © 2017 CloudBees, Inc. All Rights Reserved. 26 2016 – Summary • Jenkins can be used to run tests on hardware in embedded and automotive projects • Jenkins Pipeline is helpful for test runs • Pipeline lacks some critical features
  27. 27. © 2017 CloudBees, Inc. All Rights Reserved. 27 • Management of resources. • Integration with Throttle Concurrent Builds plugin • Use-case: limiting the license utilization in automation infrastructures (JENKINS-31801) • Better support of tools. • Common build flow – dozens of tools • Integration with Custom Tools Plugin seems to be the best option, especially in the case of multiple tool versions (JENKINS-30680) • Pipeline package manager (JENKINS-34186) • Lack of plugins for EDA tools in Jenkins, code duplications • A common approach is to keep the shared "functions" in libraries • Pipeline Global Library and Pipeline Remote Loader can be used 2016 – Pipeline improvement areas From my talk at CloudBees | Jenkins Automotive and Embedded Day 2016
  28. 28. © 2017 CloudBees, Inc. All Rights Reserved. 28 • Management of resources. • Integration with Throttle Concurrent Builds plugin • Use-case: limiting the license utilization in automation infrastructures (JENKINS-31801) • Better support of tools. • Common build flow – dozens of tools • Integration with Custom Tools Plugin seems to be the best option, especially in the case of multiple tool versions (JENKINS-30680) • Pipeline package manager (JENKINS-34186) • Lack of plugins for EDA tools in Jenkins, code duplications • A common approach is to keep the shared "functions" in libraries • Pipeline Global Library and Pipeline Remote Loader can be used 2016 – Pipeline improvement areas From my talk at CloudBees | Jenkins Automotive and Embedded Day 2016
  29. 29. © 2017 CloudBees, Inc. All Rights Reserved. 29 Management of resources • Lockable Resources Plugin – since 2016 • Throttle Concurrent Builds – coming soon • Pipeline Milestones
  30. 30. © 2017 CloudBees, Inc. All Rights Reserved. 30 PARTIAL - Better support of tools • withEnv() for environment variables • tool() step in Pipeline was improved • Tool reference in the Job • Section in Declarative Pipeline • Custom tools is not fully integrated • Follow JENKINS-30680 • ETA – Q2, 2017
  31. 31. © 2017 CloudBees, Inc. All Rights Reserved. 31 DONE – Pipeline Libs (Pipeline Package Manager) • Early 2016 • Global Pipeline Library • Pipeline Remote Loader Plugin • Late 2016 • Pipeline Shared Libraries • Support of @Grab – Pipeline libs can pull common Groovy libs Library Structure:
  32. 32. © 2017 CloudBees, Inc. All Rights Reserved. 32 • Pipeline Shared Libraries • Libs are located in SCM • Versioning • Definition for scopes: ▸ Jenkins system-wide ▸ Folder ▸ Job
  33. 33. © 2017 CloudBees, Inc. All Rights Reserved. 33 Pipeline Library Definition [1/2]
  34. 34. © 2017 CloudBees, Inc. All Rights Reserved. 34 Pipeline Library Definition [2/2]
  35. 35. © 2017 CloudBees, Inc. All Rights Reserved. 35 Using Pipeline Libraries • Implicit Loading • Via Jenkins Global or Folder configuration • Explicit Loading • @Library
  36. 36. © 2017 CloudBees, Inc. All Rights Reserved. 36 Using Pipeline Libraries com.mycorp.libs.make Jenkinsfile makeProject() makeProject() implementation - global variable com.mycorp.libs.coverity com.mycorp.libs.cppunit Globally-defined library, automatic update @Library() definition
  37. 37. © 2017 CloudBees, Inc. All Rights Reserved. 37 Early adoption of any framework http://devopsreactions.tumblr.c om/post/150674592321/pettin g-your-incomprehensible- piece-of-code • It is working! • In several pet projects…
  38. 38. Lessons Learnt. What does NOT work well?
  39. 39. Lessons Learnt. What does NOT work well? • Development Tools • Non-Success Path handling • Declarative Pipeline • Manual release management
  40. 40. © 2017 CloudBees, Inc. All Rights Reserved. 40 Pipeline Developer Experience • No Debugger • No IDE • No built-in test framework • Limited static analysis features •Pipeline DSL != Groovy
  41. 41. © 2017 CloudBees, Inc. All Rights Reserved. 41 How do I work? Intellij IDEA Filesystem SCM Plugin (with patch) Disclaimer: There may be better ways • Syntax file • Built-in documentation Jenkins Test Instance Source Code (local .git repos) • Production System Snapshot • Custom Pipeline Library Settings • Repositories for libraries • Jenkinsfile repos
  42. 42. © 2017 CloudBees, Inc. All Rights Reserved. 42 Getting the Syntax File
  43. 43. © 2017 CloudBees, Inc. All Rights Reserved. 43 Filesystem SCM Plugin • Pending pull request: • https://github.com/jenkinsci/filesystem_scm-plugin/pull/2 Library definition in the test instance !!!
  44. 44. © 2017 CloudBees, Inc. All Rights Reserved. 44 Pipeline Testing • No standard test framework offered by the Jenkins project • External project: https://github.com/lesfurets/JenkinsPipelineUnit “mvn verify” has been invoked… And the execution passed… For the source Jenkinsfile…
  45. 45. Lessons Learnt. What does NOT work well? • Development Tools • Non-Success Path handling • Declarative Pipeline • Manual release management
  46. 46. © 2017 CloudBees, Inc. All Rights Reserved. 46 Error Handling • IMHO: Pipeline is designed for Success Path scenarios When it “Just works” When it does not
  47. 47. © 2017 CloudBees, Inc. All Rights Reserved. 47 Error Handling. Top-level Scripted Pipeline Declarative Pipeline • Global try/catch() When it does not…
  48. 48. © 2017 CloudBees, Inc. All Rights Reserved. 48 Error Handling When it “Just works” When it does not The interesting part… Failover, repeat
  49. 49. © 2017 CloudBees, Inc. All Rights Reserved. 49 Error Handling. Hardware failover • Pipeline allows re-running unreliable stages on other nodes. System failure (script returns 253) Node #1 Node #2. . . . . . for (def board : boards) { echo "trying board " + board; try { node(board) { checkout scm sh ‘./bin/run.sh’ // Call passed => DONE break; } } catch (Exception ex) { if (ex.message.contains ("exit code 253")) { // Fatal error fail("Test run failed") } } } Try all possible boards:
  50. 50. © 2017 CloudBees, Inc. All Rights Reserved. 50 Error handling. Hardware failover for (def board : boards) { echo "trying board " + board; try { node(board) { checkout scm sh ‘./bin/run.sh’ // Call passed => DONE break; } } catch (Exception ex) { if (ex.message.contains ("exit code 255")) { // Fatal error fail("Test run failed") } } } ??? • Hard to extract error cause • Special handling is required for every case • Cannot just use existing parsers • Build Failure Analyzer • Log Parser • Warnings • Try-catch blocks pollute the code • Hard to test
  51. 51. © 2017 CloudBees, Inc. All Rights Reserved. 51 Context retention • No way to propagate any data • Solution - @NonCPS steps within the library steps
  52. 52. © 2017 CloudBees, Inc. All Rights Reserved. 52 Error handling. Hardware failover Declarative Pipeline Encapsulation via Libraries Simple use-case Other • Exception try/catch • timeout() wrapper • Proper failover • Logic in @NonCPS
  53. 53. Lessons Learnt. What does NOT work well? • Development Tools • Non-Success Path handling • Declarative Pipeline • Manual release management
  54. 54. © 2017 CloudBees, Inc. All Rights Reserved. 54 Declarative Pipeline • It is great for simple use-cases • Publishers & Notifiers • We do not need a framework for simple use-cases • Lack of features • “How do I pass option for the Docker Agent” • “How do I declare a resource requirement for Throttling?” • … • Scripted Pipeline is a recommended fallback, but there is no automatic migration
  55. 55. © 2017 CloudBees, Inc. All Rights Reserved. 55 Declarative Pipeline. Recommendations • Use Declarative Pipeline with caution • Complex agent definitions – use Scripted Pipeline • Complex throttling – use Scripted Pipeline • Post-build steps – work well • Tool definitions – work well • …
  56. 56. Lessons Learnt. What does NOT work well? • Development Tools • Non-Success Path handling • Declarative Pipeline • Manual release management
  57. 57. © 2017 CloudBees, Inc. All Rights Reserved. 57 Pipeline vs. Application Lifecycle
  58. 58. © 2017 CloudBees, Inc. All Rights Reserved. 58 Pipeline vs. Application Lifecycle BUT: • Non-CD process with manual promotion • Dependent products • Snowflake development stages
  59. 59. © 2017 CloudBees, Inc. All Rights Reserved. 59 Solutions • Input() step + milestone() for synchronization • Multiple Pipelines • Release() step is coming soon • Release Plugin Pipeline 1 Pipeline 2 Run Selector Staging Artifact Storage External ALM tool CD for the Component 1 CI/CD for the Component 2
  60. 60. Lessons learnt • No area-specific issues so far • High barrier to entry • It worth investment
  61. 61. © 2017 CloudBees, Inc. All Rights Reserved. 61 What I need? 2017 • Development Tools for Pipeline • IDE, Debugger, Test Tools • Community Pipeline Libraries • Common Embedded cases without plugins • Promotion engine for Pipeline • Ideally: Merge of standard Promoted Builds and Release plugins • Plugins for area-specific tools
  62. 62. © 2017 CloudBees, Inc. All Rights Reserved. 62 Takeaways • Jenkins Pipeline is a powerful engine for highly-customized Jenkins instances • Jenkins Pipeline can be effectively used in Embedded/Automotive use-cases • Barrier to entry is still high
  63. 63. © 2017 CloudBees, Inc. All Rights Reserved. 63 More info • Embedded solutions page: • https://jenkins.io/solutions/embedded/ • Links to the articles and videos • Some categorization Embedded Solutions Page …
  64. 64. © 2017 CloudBees, Inc. All Rights Reserved. 64 • Jenkins project: • Website: https://jenkins.io • Embedded Solutions page: https://jenkins.io/solutions/embedded/ • Pipeline: • Tutorial: https://github.com/jenkinsci/workflow-plugin/blob/master/TUTORIAL.md • Compatibility: https://github.com/jenkinsci/pipeline-plugin/blob/master/COMPATIBILITY.md • Examples: https://github.com/jenkinsci/pipeline-examples • CloudBees website: • Website: https://www.cloudbees.com • CloudBees Jenkins Platform: https://www.cloudbees.com/products/cloudbees-jenkins-platform Useful Links
  65. 65. © 2017 CloudBees, Inc. All Rights Reserved. 65 Thank you! Contacts: E-mail: onenashev@cloudbees.com GitHub: oleg-nenashev Twitter: @oleg_nenashev
  66. 66. Software at the speed of ideas THANK YOU! www.cloudbees.com

×