Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Why does my jenkins freeze sometimes and what can I do about it?

1.200 visualizaciones

Publicado el

After getting frustrated with Jenkins getting slow or even stuck occasionally, I decided to investigate the root cause for that bad performance. In this talk I will show my findings about that issue (hint: GC), how Jenkins performance can be easily improved by tuning the JVM GC and a few exciting tools I've discovered along the way.
Originally presented at Jenkins TLV meetup @ Taboola

Publicado en: Software
  • Sé el primero en comentar

Why does my jenkins freeze sometimes and what can I do about it?

  1. 1. Why does my Jenkins freeze sometimes and what can I do about it? Jenkins session you may like
  2. 2. Taboola - Numbers On Average Every American Sees Taboola 70 Times a Month (comScore) 500K+ Requests/sec 15B recommendation/day 1B Monthly unique users 17TB+ incoming raw data/day 8 Data-Centers globally, with over ~2500 Production servers
  3. 3. Taboola - Jenkins 4 Jenkins servers 48 slaves Hundreds of builds / day ~ 5 releases per day
  4. 4. Taboola - Technologies 6
  5. 5. Agenda ● The problem ● JVM architecture ● Garbage collectors ● The solution
  6. 6. The problem Problem: Jenkins pages get load very slowly
  7. 7. JVM architecture http://www.oracle.com/technetwork/tutorials/tutorials-1876574.html
  8. 8. JVM architecture http://coding-geek.com/jvm-memory-model/
  9. 9. JVM architecture http://javatutorialhere.blogspot.co.il/2014/08/how-objects-are-stored-in-heap-and.html
  10. 10. JVM architecture Choose >, < or = 32GB Heap 35GB Heap ?
  11. 11. JVM architecture -Xmx - heap maximum size -Xms - initial heap size jinfo -flag UseCompressedOops <pid>
  12. 12. JVM architecture http://java-latte.blogspot.co.il/2014/03/metaspace-in-java-8.html
  13. 13. Garbage Collector
  14. 14. Garbage Collector
  15. 15. Garbage Collector https://medium.com/@petloverooka/heres-how-garbage-collection-in-java-really-work-2acacc171c1
  16. 16. Garbage Collector http://www.oracle.com/technetwork/tutorials/tutorials-1876574.html
  17. 17. The solution - G1GC and logs Enable G1Gc: -XX:+UseG1GC Logs: -Xloggc:/myJenkinsDir/gc-%t.log -XX:NumberOfGCLogFiles=5 -XX:+UseGCLogFileRotation - XX:GCLogFileSize=20m -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps - XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCCause - XX:+PrintTenuringDistribution -XX:+PrintReferenceGC -XX:+PrintAdaptiveSizePolicy - verbose:gc -XX:+UnlockDiagnosticVMOptions -XX:+PrintFlagsFinal
  18. 18. The solution - GC logs
  19. 19. The solution - Tools http://gceasy.io/
  20. 20. The solution - Tools
  21. 21. The solution - More GC Flags -XX:MaxGCPauseMillis=<n> (default 200ms) -XX:InitiatingHeapOccupancyPercent=45 -XX:ParallelGCThreads=15 -XX:ConcGCThreads=4 -XX:+PrintStringDeduplicationStatistics -XX:+UseStringDeduplication -XX:+DisableExplicitGC G1GC Tuning
  22. 22. The solution - Jconsole
  23. 23. The solution - Jconsole -Djava.rmi.server.hostname=<your Jenkins host> - Dcom.sun.management.jmxremote.port=<choose a port> - Dcom.sun.management.jmxremote.authenticate=false - Dcom.sun.management.jmxremote.ssl=false
  24. 24. The solution - Jenkins Old data Jenkins Old data: http://yourJenkinsHost/administrativeMonitor/OldData/manage
  25. 25. The solution - Summary ● Switch to G1GC ● Enable and analyze logs ● Add relevant flags ● Analyze logs again ● Remove old Jenkins data ● Monitor
  26. 26. Thank You! Tidhar Klein Orbach @tizkiko tidharko Tidhar Klein Orbach
  27. 27. Resources ● Joining the Big Leagues: Tuning Jenkins GC For Responsiveness and Stability ● JVM memory model ● Getting Started with the G1 Garbage Collector ● Everything I Ever Learned about JVM Performance Tuning @twitter ● Here’s How Garbage Collection in JAVA Really Work ● Java HotSpot VM Options ● Why 35GB Heap is Less Than 32GB – Java JVM Memory Oddities

×