Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Scaling Continuous Integration in the Cloud

1.251 visualizaciones

Publicado el

Publicado en: Tecnología, Empresariales
  • Sé el primero en comentar

  • Sé el primero en recomendar esto

Scaling Continuous Integration in the Cloud

  1. 1. Scaling Continuous Integrationin the CloudChris MountfordSenior Software Engineer on JIRA, Atlassian@chromosundrift 2
  2. 2. Boredom Avoidance 3
  3. 3. Boredom Avoidance 3
  4. 4. Agenda• Automated Testing• Continuous Integration • Visibility, Reliability, Latency• Scalability to the Cloud • A Parallel Universe • Map Reduce Testing 4
  5. 5. Agenda• Automated Testing• Continuous Integration • Visibility, Reliability, Latency• Scalability to the Cloud ...and Boredom Avoidance • A Parallel Universe • Map Reduce Testing 4
  6. 6. Automated Testing 5
  7. 7. “ I find your lack of tests disturbing ” Darth Vader, The Dark Side 6
  8. 8. “ there is no single development, in either technology or management technique, which by itself promises even one order of magnitude improvement within a decade in productivity, in reliability, in simplicity. ” Fred Brooks (1986). "No Silver Bullet — Essence and Accident in Software Engineering". 7
  9. 9. “ there is no single development, in either technology or management technique, which by itself promises even one order of magnitude improvement within a decade in productivity, in reliability, in simplicity. ” Fred Brooks (1986). "No Silver Bullet — Essence and Accident in Software Engineering". 7
  10. 10. No Silver Bullet? 8
  11. 11. No Silver Bullet? 8
  12. 12. No Silver Bullet? Automated testing is the silver bullet 8
  13. 13. No Silver Bullet? Automated testing is the silver bullet 8
  14. 14. Development Velocity 9
  15. 15. Development Velocity• Tests are an investment• Testing can speed you up or slow you down• Continuous Integration maximizes return on investment 10
  16. 16. Good Automated Tests• Deterministic• Timeless• Independent • Unit Tests • Integration Tests • Acceptance Tests 11
  17. 17. Continuous Integration 12
  18. 18. Continuous Integration 12
  19. 19. Continuous Integration Visibility 12
  20. 20. Continuous Integration Visibility Reliability 12
  21. 21. Continuous Integration Visibility Reliability Latency 12
  22. 22. Visibility• Wallboards• Music• Metric: Time to Fix • Assumes a Rollback Timebox• Everyone knows green is good 13
  23. 23. 14
  24. 24. 15
  25. 25. 16
  26. 26. Reliability• False Negatives - Wrong Red Builds• Build Infrastructure Failure• Metric: Baseline Success Rate • Canary Builds 17
  27. 27. Reliability• False Negatives - Wrong Red Builds• Build Infrastructure Failure• Metric: Baseline Success Rate • Canary Builds 17
  28. 28. Canary Builds• Metric: Success Rate• Assumes Tests are: • Deterministic • Timeless • Independent• Not monitoring 18
  29. 29. Canary Builds• Metric: Success Rate• Assumes Tests are: • Deterministic • Timeless LOL That’s half of one nine • Independent• Not monitoring 18
  30. 30. Canary Builds• Metric: Success Rate• Assumes Tests are: • Deterministic • Timeless LOL That’s half of one nine • Independent Not funny• Not monitoring 18
  31. 31. Latency• Time from checkin to build result• 10 minute ideal• Primary metric for Continuous Integration • High latency kills everything 19
  32. 32. Recap 20
  33. 33. Recap Automated Testing 20
  34. 34. Recap Automated Testing 20
  35. 35. Recap Continuous Integration Automated Testing 20
  36. 36. Recap Continuous Integration $ Automated Testing 20
  37. 37. Recap Visibility Continuous Integration $ Automated Testing 20
  38. 38. Recap Visibility Reliability Continuous Integration $ Automated Testing 20
  39. 39. Recap Visibility Reliability Latency Continuous Integration $ Automated Testing 20
  40. 40. Recap Visibility Reliability Latency Continuous Integration $ Automated Testing 20
  41. 41. Recap Map Reduce Parallel Computation JIRA Build Scalability Visibility Reliability Latency Continuous Integration $ Automated Testing 20
  42. 42. Recap Map Reduce Parallel Computation JIRA Build Scalability Visibility Reliability Latency Continuous Integration $ Automated Testing 20
  43. 43. JIRA Builds 21
  44. 44. JIRA Builds• Shipping is #1• Test Proliferation • Unit / Integration • Functional - JWebUnit • QUnit • Selenium / Web Driver 21
  45. 45. JIRA Builds• Shipping is #1 • Multiple Distributions• Test Proliferation • Multiple OSs, Browsers • Unit / Integration • Multiple Branches • Functional - JWebUnit • Trunk, Bug Fixing • QUnit • Transient Feature • Selenium / Web Driver Branches 21
  46. 46. High Latency + High CheckinFrequency 22
  47. 47. High Latency + High CheckinFrequency 3 days to fail! 22
  48. 48. Scaling JIRA Builds• Dedicated Bamboo Server - 50-60 Agents • Open VZ Containers • EC2 Elastic Bamboo Agents (cloud overflow capacity)• 20 Jobs per Build• Dedicated Build Engineer 23
  49. 49. JIRA Build Latency 24
  50. 50. 20 jobs to administer! 25
  51. 51. Boring and20 jobs to administer! Repetitive Work! 25
  52. 52. JIRA Builds - The Eternal War• Numbers of Tests • Subversion contention• Test complexity • Disabled canary builds! • Did I break this? • DVCS migration window• Bamboo Agents • Build Engineer Flat Out • More!? • Be nice to Bryce • Configuration • Boring and Repetitive 26
  53. 53. Our Parallel Universe # Tests # CPU cores CPU speed 27
  54. 54. One Test at a Time 28
  55. 55. Parallel Test Execution 29
  56. 56. Core 0 Core 1 Core 2 Core 3 Intel Nehalem 30
  57. 57. Core 0 Core 1 Core 2 Core 3 Intel Nehalem 4 Cores Good! 30
  58. 58. Intel SCCSingle-chip CloudComputer 48 Cores 31
  59. 59. “ The architecture for the SCC processor is arbitrarily scalable. This is an architecture that could, in principle, scale to 1000 cores. ” Timothy Mattson Researcher, Intel 32
  60. 60. “ The architecture for the SCC processor is arbitrarily scalable. This is an architecture that could, in principle, scale to 1000 cores. ” Timothy Mattson Researcher, Intel 32
  61. 61. 1000cores! 33
  62. 62. Recap 2 Map Reduce Parallel Computation JIRA Build Scalability Visibility Reliability Latency Continuous Integration $ Automated Testing 34
  63. 63. Monster Builds - a year later• Scaling JIRA Builds 2010• Parallel Builds Hack • Did it work? • Was it elegant? • Dozens of Atlassian Teams• What if we need 10x? 35
  64. 64. Clouds, Clusters & Grids• The Cloud is the new hotness • High growth commodity marketplace• Already using EC2 • Cheap to experiment• Functional Programming • Tests are functions 36
  65. 65. Hadoop Compute Clusters 37
  66. 66. Hadoop Compute Clusters 37
  67. 67. Hadoop Compute Clusters 37
  68. 68. Hadoop Compute Clusters 37
  69. 69. Imaginary Test Runner 38
  70. 70. The Map and The Reduce 39
  71. 71. The Map and The Reduce Map 39
  72. 72. The Map and The Reduce Map Reduce 39
  73. 73. The Map and The Reduce MapFramework Reduce 39
  74. 74. Hadoop Map Reduce Mapper Reducer ... Hadoop Map Reduce HDFS 40
  75. 75. Hadoop at Yahoo! 41
  76. 76. Hadoop at Yahoo! 4000 Nodes 41
  77. 77. Hadoop Distributes Test Load control test test test test node 1 2 3 ... t t = ~4000 tests node node node ... node n = ~100 nodes 1 2 3 n 42
  78. 78. Map Reduce Testing• Focus on subproblems: tests• Map Reduce coordinates the distribution and recombination• Scales to 4000 nodes and beyond• Resistant to infrastructure failure 43
  79. 79. Map Reduce Tests (Mr. T)• Java • Bamboo Integration• JUnit • Hadoop Cluster Agent• Hadoop • Cluster Monitoring, Management • Map Reduce • Whirl • HDFS • Cluster requisition API• Maven (Ant, Gradle?) 44
  80. 80. Map Reduce Tests (Mr. T)• Java • Bamboo Integration• JUnit • Hadoop Cluster Agent• Hadoop • Cluster Monitoring, Management I pity the fool! • Map Reduce • Whirl • HDFS • Cluster requisition API• Maven (Ant, Gradle?) 44
  81. 81. 45
  82. 82. 10 Minute Latency 45
  83. 83. “ Fix visibility with wallboards, measure reliability with canary builds and fix latency with parallel test ” execution. #summit11 46
  84. 84. @chromosundrift
  85. 85. @chromosundrift

×