Recomendados
Recomendados
Más contenido relacionado
Similar a Production Profiling: What, Why and How
Similar a Production Profiling: What, Why and How (20)
Más de RichardWarburton
Más de RichardWarburton (20)
Último
Último (20)
Production Profiling: What, Why and How
- 1. Opsian Production Profiling: What, Why and How Richard Warburton (@richardwarburto) Sadiq Jaffer (@sadiqj)
- 2. Opsian Why Performance Matters Development isn’t Production Profiling vs Monitoring Production Profiling Conclusion
- 5. Opsian Amazon: 100ms of latency costs 1% of sales Google: 500ms seconds in search page generation time drops traffic by 20% Responsive Applications make more Money
- 6. Opsian Improve Developer Productivity Source: ZT RebelLabs Developer Productivity Report 2017
- 8. Opsian Why Performance Matters Development isn’t Production Profiling vs Monitoring Production Profiling Conclusion
- 9. Opsian Development isn’t Production Testing in development can be easier Tooling often desktop-based Not representative of production
- 12. Opsian Virtualised environments can perform very differently “Two frequently used system calls are ~77% slower on AWS EC2” https://blog.packagecloud.io/eng/2017/03/08/system-calls-are-much-sl ower-on-ec2/
- 14. Opsian The JVM may have very different behaviour in production Hotspot does adaptive optimisation Production may optimise differently
- 15. Opsian Why Performance Matters Development isn’t Production Profiling vs Monitoring Production Profiling Conclusion
- 16. Opsian
- 17. Opsian Ambient/Passive/System Metrics Static numerical measure about the system CPU Time Usage / Page-load Times Cheap and sometimes effective
- 18. Opsian Coarse Grained Instrumentation Measures time within some instrumented section of the code Time spent inside the controller layer of your web-app or performing SQL queries More detailed and actionable though expensive
- 19. Opsian Logging Records arbitrary events emitted by the system being monitored log4j/slf4j/logback Logs of GC events Often manual, aids system understanding, expensive
- 20. Opsian Production Profiling Measures what part of your code is eating up some resource What methods take up CPU or wallclock time? What lines of code allocate the most objects? Where are your CPU Cache misses coming from? Automatic, can be cheap but often isn’t
- 21. Opsian Understand don’t Overwhelm Too Little You can’t understand production problems Too Much Needle in a Haystack You are the problem (overhead)
- 22. Opsian Unactionable Metrics Many metrics provide pretty graphs but don’t progress treatment
- 23. Opsian Normalisation of Deviance “Some of the tests always fail, so we just ignore them.” “Some of the alerts get triggered regularly, so we just ignore them.” Alert false positives have a cost
- 24. Opsian Instrumentation Cycle Too much overhead Reduce instrumentation granularity Lack of instrumentation Increase instrumentation granularity
- 25. Opsian Surely a better way? Diagnostics not Diagnosis Are there alternatives to the instrumentation dance? Can’t just rely on metrics - you need actionable insights for your app What about profiling?
- 26. Opsian Why Performance Matters Development isn’t Production Profiling vs Monitoring Production Profiling Conclusion
- 27. Opsian Profiling tells you what your application was doing
- 28. Opsian Instrumenting Profilers Add instructions to collect timings (Eg: JVisualVM Profiler) Inaccurate - modifies the behaviour of the program High Overhead - > 2x slower
- 29. Opsian Sampling/Statistical Profilers Interrupt application periodically to sample the stack Overhead dependent upon sampling mechanism Build up a statistical picture over time
- 30. Opsian Statistical Profiling in Java Periodic GetAllStackTraces Safepoint bias Latency and throughput impact
- 31. Opsian Advanced Statistical Profiling in Java OS Signals to interrupt threads on resource consumption threshold JVM’s signal handler-safe AsyncGetCallTrace to walk the stack
- 32. Opsian Open Source Java Profilers High Overhead VisualVM hprof Twitter’s CPUProfile Anything GetAllStackTraces based Low Overhead Async Profiler Honest Profiler Java Mission Control
- 33. Opsian Profiling Support in the Linux Kernel perf and eBPF perf-map-agent for the JVM Hardware events (L1/L2/L3 cache misses, branch mispredictions, etc.)
- 34. Opsian People are put off by practical as well as technical issues
- 35. Opsian Barriers to Ad-Hoc Production Profiling Low overhead unbiased tools are relatively new Generally requires access to production Process involves manual work - hard to automate
- 36. Opsian What if we profiled all the time?
- 37. Opsian Historical Data Allows for post-hoc incident analysis Enables correlation with other data/metrics Performance regression analysis
- 39. Opsian Putting Samples in Context Application version Environment parameters (machine type, CPU, location, etc.) Desktop profilers don’t do this
- 40. Opsian Summary Production Profiling is possible - all the time with low overhead Production Profiling is desirable - can give deep insights Problems can be overcome
- 41. Opsian Why Performance Matters Development isn’t Production Profiling vs Monitoring Prod Profiling Problems Conclusion
- 42. Opsian Performance Matters Development isn’t Production Metrics can be unactionable Instrumentation has high overhead Production Profiling provides insight
- 43. Opsian We need an attitude shift on profiling + monitoring
- 45. Opsian Please do Production Profiling. All the time.
- 47. Opsian The End