The document summarizes a performance analysis study of a large enterprise application called Zanzibar. The researchers initially struggled to get the application to scale even on modest hardware. Through analyzing javacores, they identified and addressed multiple bottlenecks, including lock contention, disk throughput issues, and inefficient Java code. These changes resulted in performance improvements of 1.4-5x. The researchers also developed the WAIT performance tool to help identify primary bottlenecks in deployed applications with low overhead. The tool uses standard OS and JVM data and has seen widespread adoption due to its ease of use.

1. Performance Analysis of Idle Programs Erik Altman Matthew Arnold Stephen Fink Nick Mitchell Peter Sweeney IBM T.J. Watson Research Center “ WAIT Performance Tool”

16. Results: 56 core POWER p595 Scaling with -checkin (first minute) 0 1000 2000 3000 4000 5000 6000 7000 2 4 8 16 32 56 Num Cores Docs/second Orig Modified

17. Results: 56 core POWER p595 Without -checkin flag 0 1000 2000 3000 4000 5000 6000 7000 2 4 8 16 32 56 Num cores Docs/sec Orig Modified

27. Upload Java Cores to WAIT Website

28. What is the CPU doing? What Java work is running? What Java work cannot run? View WAIT Report in a Browser What is memory consumption? Not directly available from profiling tools

30. Filesystem Bottleneck

31. Z/OS: Lock Contention

32. Lock Contention: Waiting vs Owning

33. Deadlock

35. Example Report: Memory Leak Disclaimer: Appearance and function of any offering may differ from this depiction.

36. Example Report: Memory Analysis Disclaimer: Appearance and function of any offering may differ from this depiction.

37. WAIT Implementation Details

42. WAIT State: RUNNABLE

43. WAIT State: WAITING

47. Rule Statistics (as of Mar 2010) Number of Rules DB2, MySql, Oracle, Apache, SqlServer Rule Coverage 6 LDAP 12 Logging 13 Classloader 22 JEE 30 Marshalling 30 Waiting for Work 46 Disk, Network I/O 41 Client Communication 59 Administrative 72 Database # Rules Category 23% Package Fallback 77% Category Rule 1,391,033 # Thread Stacks 830 # Reports

49. Who is using WAIT?

50. Tool in Software Lifecycle Entry Point Entry Point Exit Point Performance Tuning The tool applies everywhere in cycle. – Key: Lightweight and simple Build Use latest compiler Turn on optimization Enable parallelization* Analyze Static code analysis Find “hot spots” Identify performance bottlenecks Identify scalability bottlenecks* Code & Tune Refine compiler options/directives Use optimized libraries Recode part of application Introduce/increase parallelism* Test & Debug Run Application Check correctness Check concurrency issues* Monitor Measure performance Collect execution stats Validate performance gains Gather stats on scalability*

60. The End