Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Keeping your CI/CD pipeline as fast as it needs to be

276 visualizaciones

Publicado el

Presentation at the Manchester Java Community, October 2016, about how to build and manage a scalable Continuous Integration / Continuous Deployment pipeline.

Publicado en: Software
  • Inicia sesión para ver los comentarios

  • Sé el primero en recomendar esto

Keeping your CI/CD pipeline as fast as it needs to be

  1. 1. Abraham Marin-Perez @AbrahamMarin fromfragiletoagile.com Keeping Your CI / CD Pipeline as Fast as It Needs to Be #MCRJava @AbrahamMarin @EqualExperts
  2. 2. About Me
  3. 3. About Me
  4. 4. About Me
  5. 5. About Me
  6. 6. About Me
  7. 7. About Me
  8. 8. About Me
  9. 9. About This Talk
  10. 10.  Continuous Integration: check everything is still working after every commit  Continuous Deployment: every successful commit turns into a release About This Talk
  11. 11. About This Talk
  12. 12. About This Talk
  13. 13. About This Talk
  14. 14. About This Talk
  15. 15. SUPER APP # Files: 75 # Tests: 800 Build Time: 4 min Output: superapp.war
  16. 16. SUPER APP # Files: 113 # Tests: 1200 Build Time: 6 min Output: superapp.war
  17. 17. SUPER APP # Files: 169 # Tests: 1800 Build Time: 9 min Output: superapp.war
  18. 18. Slow feedback Broken builds mask issues Development paralysis Impact on ability to meet our SLAs Pay per use Missed business opportunities The Problems Of Size
  19. 19. Live with it Partial CD: only quick tests Phased CD: split into components Test Deprecation Policy Microservices How Organisations Manage Size
  20. 20. Microservices
  21. 21. Microservices
  22. 22. Microservices
  23. 23. SUPER APP # Files: 169 # Tests: 1800 Build Time: 9 min Output: superapp.war APP BACKEN D SUPER APP # Files: 115 # Tests: 1200 Build Time: 6 min Output: superapp.war # Files: 72 # Tests: 800 Build Time: 4 min Output: appbackend.jar
  24. 24. Microservices?
  25. 25. Microservices
  26. 26. Microservices
  27. 27. Microservices
  28. 28. Microservices
  29. 29. Microservices
  30. 30. Microservices
  31. 31. Build Pipeline Becomes a Network
  32. 32. To Caesar what is Caesar’s Based on John Ferguson Smart’s “Real-World Strategies for Continuous Delivery with maven and Jenkins” http://youtu.be/McTZtyb9M38
  33. 33. John’s approach Maven wasn’t built for Continuous Deployment commit commit commit ... 0.0.1-SNAPSHOT Release! 0.0.1
  34. 34. Don’t use RELEASE plugin Use VERSIONS plugin Set version to <version scheme>.<build number> Run mvn deploy Commit pom file to repository John’s approach
  35. 35. Set version to <version scheme>.<build number> mvn versions:set –DnewVersion=**your version** John’s approach
  36. 36. Run mvn deploy mvn clean deploy John’s approach
  37. 37. Commit pom file to repository <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-scm-plugin</artifactId> <version>1.9.1</version> <executions> <execution> <id>commit</id> <phase>deploy</phase> <goals> <goal>checkin</goal> </goals> </execution> </executions> </plugin> John’s approach
  38. 38. John’s approach
  39. 39. 0.0.1.1 commit BUILD! 0.0.1.2 commit BUILD! 0.0.1.3 commit BUILD! John’s approach
  40. 40. How do you scale this?
  41. 41. <dependency> <groupId>com.superappfactory</groupId> <artifactId>appbackend</artifactId> <version>LATEST</version> </dependency> <dependency> <groupId>com.superappfactory</groupId> <artifactId>appbackend</artifactId> <version>??????</version> </dependency> APP BACKEN D SUPER APP appbackend.jar superapp.war Setting up dependencies
  42. 42. Setting up dependencies APP BACKEN D SUPER APP appbackend.jar superapp.war
  43. 43. Rebuilding old versions
  44. 44. Using “LATEST” makes it impossible to build old versions correctly Rebuilding old versions
  45. 45. <dependency> <groupId>com.superappfactory</groupId> <artifactId>appbackend</artifactId> <version>1.5.3.1</version> </dependency> <dependency> <groupId>com.superappfactory</groupId> <artifactId>appbackend</artifactId> <version>???????</version> </dependency> Rebuilding old versions APP BACKEN D SUPER APP appbackend.jar superapp.war
  46. 46. Update versions of dependencies mvn versions:use-latest-releases
  47. 47. APP BACKEN D SUPER APP APP BACKEND SUPER APP DATA MODEL SUPER APP DATA MODEL GUI APP BACKEND
  48. 48. commitbuild deploy commit Problem: Infinite Trigger
  49. 49. APP BACKEND SUPER APP commit Problem: Unnecessary Rebuilds
  50. 50. Problem: Unnecessary Rebuilds
  51. 51. Get last committer buildAgent? Proceed normally Don’t run build NO YES touch skip_build Problem: Unnecessary Rebuilds
  52. 52. <profiles> <!-- Plugins that need to be disabled when doing a no-run --> <profile> <id>do.nothing</id> <activation> <file> <exists>skip_build</exists> </file> </activation> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>3.1</version> <configuration> <skipMain>true</skipMain> <skip>true</skip> </configuration> </plugin> Problem: Unnecessary Rebuilds
  53. 53. <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-install-plugin</artifactId> <version>2.5.2</version> <configuration> <skip>true</skip> </configuration> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-deploy-plugin</artifactId> <version>2.8.2</version> <configuration> <skip>true</skip> </configuration> </plugin> Problem: Unnecessary Rebuilds
  54. 54. APP BACKEND SUPER APP commit Problem: Unnecessary Rebuilds
  55. 55. Problem: Unnecessary Rebuilds
  56. 56. APP BACKEND SUPER APP commit Problem: Necessary Rebuilds
  57. 57. Get last committer buildAgent? Proceed normally Don’t run build NO YES Problem: Necessary Rebuilds
  58. 58. Get last committer buildAgent? Proceed normally NO YES Check dependencies Up to date? NO Don’t run build YES touch skip_build
  59. 59. Problem: Necessary Rebuilds
  60. 60. build deploy commit commit commit pom.xml Problem: Doomed Build
  61. 61. Get last committer buildAgent? Proceed normally NO YES Check dependencies Up to date? NO Don’t run build YES touch skip_build
  62. 62. Get last committer buildAgent? Check pom.xml NO YES Check dependencies Up to date? NO Don’t run build YES touch skip_build Up to date? NO Proceed normally YES
  63. 63. Problem: Doomed Build
  64. 64. A real case scenario
  65. 65. Service Service Service Parent POM Logging
  66. 66. Service Service Service Parent POM Logging 28%
  67. 67. Service Service Service Parent POM Logging 28% 28% 28%
  68. 68. Service Service Service Parent POM Logging 28% 28% 28% 20%
  69. 69. Service Service Service Parent POM Logging 48% 28% 28% 20%
  70. 70. Service Service Service Parent POM Logging
  71. 71. Service Service Service
  72. 72.  Build Time (BT): time an individual build takes to run  Change Rate (CR): percentage of commits upon an individual build with respect to the whole system Useful Metrics
  73. 73. Service Service Servi ce
  74. 74. Service Service Service Parent POM Logging 28%
  75. 75.  Impact Time (IT): total time to run a build and all the builds that will be triggered as a result Useful Metrics
  76. 76. No dependants  IT(A) = BT(A) A Useful Metrics
  77. 77. Serial execution  IT(A) = BT(A) + IT(B) + IT(C) B A C Useful Metrics
  78. 78. Parallel execution  IT(A) = BT(A) + max(IT(B), IT(C)) B A C Useful Metrics
  79. 79. Service Service Service
  80. 80. Weighted Impact Time (WIT): impact time of a build weighted according to its change rage WIT(A) = IT(A) * CR(A) Useful Metrics
  81. 81. Average Impact Time (AIT): total time needed, on average, to execute all necessary builds after any given commit anywhere in the system AIT = WIT(A) + WIT(B) + ... + WIT(Z) Useful Metrics
  82. 82. Sample Thresholds
  83. 83. Average Impact Time Average Impact Time is what indicates how well you have scaled your system Sample Thresholds
  84. 84. Maximum Impact Time In a worst-case scenario, a build won’t take longer than this. Sample Thresholds
  85. 85. Maximum Impact Time for Critical Components The same, but only for your most sensitive modules (log-in, payment gateway, etc.) Beware of dependencies! Sample Thresholds
  86. 86. Service Service Service
  87. 87. Manual processing takes time...
  88. 88.  Most CI systems provide an API  Calculations aren’t complex  Multiple graphical tools available Automating Build Analysis
  89. 89. github.com/quiram/build-hotspots Build Hotspots
  90. 90. https://commons.wikimedia.org/wiki/File:2012_Italian_GP_-_Lotus_wheel.jpg
  91. 91. Thank You @EqualExper ts equal- experts equalexperts.com Thank You fromfragiletoagile.com @AbrahamMarin #FastCI #MCRJava

×