Expecto Performa! The Magic and Reality of Performance Tuning
•
1 recomendación•11,345 vistas
In the enterprise there are rarely simple solutions to highly nuanced problems that satisfy all needs. Several customers might each ask "How do I make Jira/Confluence faster?" and each require a different answer. Using this example, this talk will pick apart the inputs, outputs, concerns, and realities of answering a short question with a long answer. We'll then discuss real-world examples from our own internal instances, to give you a taste of the process we've gone through to solve our own performance problems, and to show why there is no simple playbook; "it depends" on a lot! The key takeaways are: * The importance of having a shared definition of performance * The importance of having agreed-upon priorities, including what isn't important * The importance of measuring (allthethings) and understanding them * The thing you think is the problem might not be the problem, and vice versa. * The real world and the ideal world tend to look nothing alike!
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Destacado
Destacado (20)
Similar a Expecto Performa! The Magic and Reality of Performance Tuning
Similar a Expecto Performa! The Magic and Reality of Performance Tuning (20)
Más de Atlassian
Más de Atlassian (20)
Último
Último (20)
Expecto Performa! The Magic and Reality of Performance Tuning
- 1. Expecto Performa! The magic and reality of performance tuning MATT SHELTON | TECHNICAL ACCOUNT MANAGER | ATLASSIAN | @MATTSHELTON DENISE UNTERWURZACHER | SITE RELIABILITY ENGINEER | ATLASSIAN
- 2. Agenda Understanding the problem Let’s agree… to agree Measure (allthethings) Benchmarking The art of tuning
- 3. Agenda Understanding the problem Let’s agree… to agree Measure (allthethings) Benchmarking The art of tuning
- 4. Hi, I’m a TAM.
- 5. Hi, I’m a TAM.
- 6. © 2016 Walt Disney Pictures
- 7. © 2004 Warner Brothers
- 8. Any sufficiently advanced technology is indistinguishable from magic. ARTHUR C. CLARKE
- 10. After our maintenance window, Confluence is really slow! CONFLUENCE ADMIN
- 11. My users say that JIRA takes too long to create an issue, and that opening their boards takes forever! JIRA ADMIN
- 12. JIRA feels slower than it did last month, and way slower than it was earlier in the year. JIRA USER
- 13. My developers say JIRA is too slow, but it seems fine to me… PROJECT MANAGER
- 14. We just acquired this other company, they have Confluence as well and we want to merge them together. We’ll end up with 25,000 users and it’s not the fastest now, and then we want to start using Collaborative Editing, and open it up to outside contractors, and … CONFLUENCE ADMIN
- 15. MOVING TARGETS
- 16. Is it speed you want?!?
- 18. Toyota Camry 2016’s most popular mid-size car!
- 19. Tesla P100D World's fastest consumer sedan! Toyota Camry 2016’s most popular mid-size car!
- 20. Agenda Understanding the problem Let’s agree… to agree Measure (allthethings) Benchmarking The art of tuning
- 22. Expectations
- 23. Be reasonable… Don’t compare your internal Jira instance to a supercomputer! Expectations
- 24. Status Quo Is it ok today? It’s only a few seconds… Be reasonable… Don’t compare your internal Jira instance to a supercomputer! Expectations
- 25. Status Quo Is it ok today? It’s only a few seconds… Be reasonable… Don’t compare your internal Jira instance to a supercomputer… Expectations Latency A little is ok, but a lot can be a big problem.
- 26. Scalability I’ll see your 250 users, and raise you 2500, then 25,000… Expectations
- 27. User Behavior temet nosce (know thyself) Scalability I’ll see your 250 users, and raise you 2500, then 25,000… Expectations
- 28. Priorities!
- 29. Urgency Trying to fix what’s broken, or make it better?Priorities!
- 30. Urgency Trying to fix what’s broken, or make it better? Who cares? Discover, discern, and prioritize! Priorities!
- 31. Urgency Trying to fix what’s broken, or make it better? Who cares? Discover, discern, and prioritize! Now vs Later What’s most beneficial now? What would be helpful down the road? Priorities!
- 32. How much is it worth to you? VALUE
- 33. Staffing Levels
- 34. Staffing Levels Customer 1 Customer 2 Profile - 10,000 Users - Jira Data Center, Confluence Data Center - 1 Manager - 2 FT Sys Admin - 3 FT App Admins - 1 FT Dev - 1 FT SRE - 1 Architect Assessment Well-staffed. Team runs all of our applications as well as other developer tools.
- 35. Staffing Levels Customer 1 Customer 2 Profile - 20,000 Users - 2 Jira Data Center, 2 Confluence Data Center, 1 BB Data Center, FeCru, 3 Bamboo - 1 Team Lead - 2 FT/1 PT App Admin - 1 FT/1PT Sys Admins Assessment Under-staffed. Team runs all of our applications as well as at least 5 other tool across different geographies
- 37. I WANT TO GO TO THERE
- 38. Agenda Understanding the problem Let’s agree… to agree Measure (allthethings) Benchmarking The art of tuning
- 39. MEASURE (ALLTHETHING) It feels like it’s taking forever…
- 40. MEASURE (ALLTHETHING) It feels like it’s taking forever…
- 42. It depends… Bitbucket Make sure you monitor the CPU! (But don’t forget about DB load, SCM jobs, or disk speed, or…) Bamboo Make sure you monitor the network! (But don’t forget about CPU load, or page load time, or…) Jira Make sure you monitor disk I/O! (But don’t forget about heap use, or CPU load, or page load time, or…) Confluence Make sure you monitor Memory! (But don’t forget about the db connection pool, or disk I/O, or…)
- 44. Agenda Understanding the problem Benchmarking The art of tuning Let’s agree… to agree Measure (allthethings)
- 46. Extranet Long pauses Garbage collection pauses of 10-20s Nodes removed Lack of response means nodes are aggressively removed from the pool Garbage Collection Network latency Load balancers
- 47. Extranet Photo heavy blogs Latency causing slow image downloads Garbage Collection Network latency Load balancers
- 48. Extranet 10s health check Nodes are removed from the pool if there is no response for 10s Idempotency Failed requests replay across other nodes No draining period Requests fail immediately Garbage Collection Network latency Load balancers
- 49. Load balancing Node 1 Node 2 Node 3
- 50. Load balancing Node 1 Node 2 Node 3
- 51. Load balancing Node 1 Node 2 Node 3
- 52. Load balancing Node 1 Node 2 Node 3
- 53. Stop cascading failures Turn off idempotency Lower sensitivity of health check Allow draining
- 54. Buffers Node Load balancer 400k buffer Client
- 55. Buffers Node Load balancer ClientLoad balancer
- 56. Buffers Node Load balancer ClientLoad balancer
- 57. Buffers Node Load balancer ClientLoad balancer 400k 400k
- 58. Buffers Node Load balancerLoad balancer 400k Client Client 400k
- 59. Buffers Node Load balancer ClientLoad balancer 12mb
- 60. Buffers Node Load balancer ClientLoad balancer 12mb
- 61. Stop artificially throttling Increase the load balancer buffer size
- 62. What does ‘slow’ really mean? BENCHMARKING
- 63. You need data
- 65. Apdex Satisfactory vs unsatisfactory response times Score between 0 - 1
- 68. Know your infrastructure Network Database App servers Requests Largest potential for problems Work with your networking teams Access logging Pipe load balancer or Tomcat access logs into Splunk
- 69. Know your infrastructure Network Database App servers Requests Latency Check in System Information, or ping Slow queries Enable logging on the database
- 70. Know your infrastructure Network Database App servers Requests CPU Datadog or Logic Monitor I/O Datadog or Logic Monitor Memory Enable GC logging, use GCViewer
- 71. CPU and I/O (Datadog)
- 72. CPU and I/O (Datadog)
- 73. Know your infrastructure Network Database App servers Requests CPU Datadog or Logic Monitor I/O Datadog or Logic Monitor Memory Enable GC logging, use GCViewer
- 75. Know your infrastructure Network Database App servers Requests HTTP threads New Relic Load balancer Access logging Database connections Datadog
- 77. Know your infrastructure Network Database App servers Requests HTTP threads New Relic Load balancer Access logging Database connections Datadog
- 79. Know your infrastructure Network Database App servers Requests HTTP threads New Relic Load balancer Access logging Database connections Datadog
- 81. Peak times Know your peak and low load times Find your benchmarks Percentiles Averages mean nothing Identify Establish baselines for each area
- 82. Percentiles
- 83. Percentiles
- 84. Percentiles
- 85. Percentiles
- 86. Agenda Understanding the problem Benchmarking The art of tuning Let’s agree… to agree Measure (allthethings)
- 87. Before you start tuning Hold your horses
- 88. Be patient Wait for peak and low load times Go slow Know when to stop Is that last 100ms really worth it? Isolate Make one change at a time and benchmark
- 90. Agree on metrics
- 91. Start simple
- 92. Track everything
- 93. Adding too much capacity can cause other problems. TUNING
- 94. Spell book
- 95. Magic tricks Database App servers Requests Check your indexes
- 96. Magic tricks CPU Add more cores, limit concurrency Garbage Collection Adding more memory != success Database App servers Requests
- 97. Magic tricks Threading More database connections than HTTP threads Load balancer • Increase buffers • Turn off idempotency • Allow draining • ‘Least connections’ over ‘round robin’ Database App servers Requests
- 98. Limit complexity • Limit or combine custom fields • Clean up unused plugins • Keep general complexity of workflows low Magic tricks Jira Confluence Bitbucket Server
- 99. Tune for fault tolerance • Extend the cluster safety interval • Turn off idempotency at your load balancer • Know your garbage collection behaviour Magic tricks Jira Confluence Bitbucket Server
- 100. Load comes from git • Optimise for git over the JVM • Scale vertically over horizontally • Use docker and mirrors Magic tricks Jira Confluence Bitbucket Server
- 101. Upgrade
- 103. Planning for the future Capacity planning Alerting HTTP threads # of requests in highest load minute / 60 * average time to complete = threads in use/s Requests in highest load minute = 8400 Time to complete = 0.82s 8400 / 60 * 0.82 = 115 threads, or 29 per node
- 104. Planning for the future Capacity planning Alerting Our alerts • More than 500x 500 errors in a minute • More than 300 timeouts at the load balancer in an hour • Garbage collection pauses > 10s • Nodes being removed/readded at the load balancer • Cluster panics • Out of memory errors • Long running space exports
- 107. Accept Your Reality There are limits to performance tuning. Be ok with what’s fast enough.
- 108. Accept Your Reality There are limits to performance tuning. Be ok with what’s fast enough. Data is Your Friend …but having good data takes time. Move slowly and methodically.
- 109. Accept Your Reality There are limits to performance tuning. Be ok with what’s fast enough. Data is Your Friend …but having good data takes time. Move slowly and methodically. Chill Go slowly. Track Everything. Lather. Rinse. Repeat. (Always repeat.)
- 110. The Four Principles of Atlassian Performance Tuning Dan Hardiker CTO, Adaptavist SUMMIT EUROPE 2017
- 112. © 2007 Warner Brothers
- 113. © 2002 1492 Pictures
- 114. Thank you! MATT SHELTON | TECHNICAL ACCOUNT MANAGER | ATLASSIAN | @MATTSHELTON DENISE UNTERWURZACHER | SITE RELIABILITY ENGINEER | ATLASSIAN