Impact2014: Practical Performance Troubleshooting

© 2014 IBM Corporation
AAI-2419 Practical Performance
Chris Bailey
Todd Kindsfather

Please Note
IBM’s statements regarding its plans, directions, and intent are subject to change or
withdrawal without notice at IBM’s sole discretion.
Information regarding potential future products is intended to outline our general product
direction and it should not be relied on in making a purchasing decision.
The information mentioned regarding potential future products is not a commitment,
promise, or legal obligation to deliver any material, code or functionality. Information about
potential future products may not be incorporated into any contract. The development,
release, and timing of any future features or functionality described for our products remains
at our sole discretion.
Performance is based on measurements and projections using standard IBM benchmarks
in a controlled environment. The actual throughput or performance that any user will
experience will vary depending upon many factors, including considerations such as the
amount of multiprogramming in the user’s job stream, the I/O configuration, the storage
configuration, and the workload processed. Therefore, no assurance can be given that an
individual user will achieve results similar to those stated here.

Introduction to the Speakers
Chris Bailey
Java Serviceability and Cloud Integration Architect
• 13 years experience developing and deploying Java SDKs
• Recent work focus:
‒ Java integration into the cloud
‒ Java monitoring, diagnostics and troubleshooting
‒ Requirements gathering
‒ Highly resilient and scalable deployments
• Contact Details:
‒ baileyc@uk.ibm.com
‒ http://www.linkedin.com/in/chrisbaileyibm
‒ http://www.slideshare.net/cnbailey/

Introduction to the Speakers
Todd Kindsfather
Product Manager, IBM Application Performance Management
• 14 years at Tivoli (now called Cloud & Smarter Infrastructure), with 5 years as a
Services consultant and 9 as a product manager
• Recent work focus:
‒ Release of Application Performance Diagnostics on premise and SaaS diagnostics
offering
‒ Diagnostics in PaaS environments
‒ Requirements gathering and future release planning
• Contact Details:
‒ tkindsfa@us.ibm.com
‒ http://www.linkedin.com/in/toddkindsfather
‒ Twitter: @ToddKindsfather

5 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application
Goals of the Talk
■ Introduce a simple general methodology for performance analysis
■ Discuss the common performance bottlenecks
■ Show how to analyse a simple application

Agenda
■ Approaches to performance
■ Layers of the application
■ Identifying and resolving performance issues

Approaches to performance
■ Layered approach
– “Bottom up” or “Top down”
– Analyze and eliminate layers of the application
– Simplify the problem as you go
– Ideal for application health check
■ Outside in approach
– Start from where performance can be measured
– Work along the activity path
– Ideal for identified performance problems
■ A hybrid of both approaches can often be useful

Performance baseline
■ Important to have a repeatable and representative performance test
■ Measure baseline performance
– Internal measurements affect the performance of what your measuring
– External measurements have less impact on system performance
■ Where possible, measure multiple times
– Variation will occur between test runs
■ Where possible, ensure consistency
– Not just the load test that's run
– State of machine and network can have interesting effects

Page response performance benchmark: baseline
PlantsByWebSphere
servlet_ShoppingServlet
servlet_ShoppingServlet{2}
Shopping{1}
Shopping{4}
Shopping_1_1
Shopping_2_2
Shopping_2_3
Shopping_2_4
Shopping_2_5
0
5000
10000
15000
20000
Page Performance
PlantsbyWebSphere
Baseline
Page URL
AvgLatency(microseconds)

A Layered Approach
■ Three layers of a deployment:
– Infrastructure: Machine hardware and Operating System
– Java Runtime: Garbage collection
– Java Application: Java application code
■ Each can suffer from resource constraints, typically:
– Memory
– CPU
– Synchronization
– I/O

Infrastructure
■ Typical resource constraints:
– Memory: insufficient physical memory results in paging/swapping
– CPU: insufficient CPU time limits throughput of the application
– I/O: insufficient I/O limits throughput of the application
– Synchronization driven by Java runtime/Java application
■ Easy to diagnose
■ Easy to resolve (relatively)
■ Note that each can also be caused by deficiencies higher up the stack!

Infrastructure: Memory usage
■ Infrastructure uses memory for:
– Backing the process data: OS runtime, Java runtime, Java application
– Caching of IO: filesystem and network buffers
■ Lack of physical memory causes:
– Reduction and removal of IO caching
– Paging/swapping of process memory to disk
■ Paging/swapping is costly for a Java process
– Particularly affects Garbage Collection performance
• Paging usually occurs on Least Recently Used basis
• All of Java heap is traversed during mark and sweep phases
• Least Recently Used does not work well for the Java heap

Infrastructure: CPU usage
■ Insufficient CPU time availability will reduce performance
■ Can occur periodically:
– Cron Jobs running batch applications
– Database backups
■ Or during periods of high load:
– System becomes CPU bound, limiting performance

Detecting infrastructure issues
■ Operating System Tools (eg. “perfmon on Windows)
– Paging: using “perfmon” with “Process” counter for “Page Faults/sec”
– File Cache: using “perfmon” with “Memory” counter for “System Cache Resident Bytes”
– Per process: using “perfmon” with “Process” counter for “% Processor Time”
– Per machine: using “perfmon” with “Processor” counter for “% Processor Time”
– Network: using “perfmon” with “Network Interface” counter for “Output Queue Length”
– Disk: using “perfmon” with “Physical Disk” counter for “Current Disk Queue Length”
■ Health Center
Machine and Process CPU Usage
• Visualizes heap usage and gc pause times over time
• Identifies memory leaks
• Suggests command-line and tuning parameters
• Same recommendation logic as GCMV
Native Memory
•Detect native memory leaks in application
•Determine if external forces are using more memory
•Viewcomponents using the most native memory

Paging in perfmon

Resolving infrastructure issues
■ Add more physical resources to the process
– Assign more to the: Machine, Guest OS, LPAR, Zone, etc
■ Reduce the physical resource requirements
– Reduce the application footprint
– Reduce the application CPU usage
– Reduce the IO

Page response performance benchmark: Baseline
PlantsByWebSphere
Shopping{1}
Shopping{4}
Shopping_1_1
Shopping_2_2
Shopping_2_3
Shopping_2_4
Shopping_2_5
0
5000
10000
15000
20000
Page Performance
PlantsbyWebSphere
Baseline
Page URL

Page response performance benchmark: Paging removed
PlantsByWebSphere
Shopping{1}
Shopping{4}
Shopping_1_1
Shopping_2_2
Shopping_2_3
Shopping_2_4
Shopping_2_5
0
5000
10000
15000
20000
Page Performance
PlantsbyWebSphere
Baseline
Page URL
PlantsByWebSphere
Shopping{1}
Shopping{4}
Shopping_1_1
Shopping_2_2
Shopping_2_3
Shopping_2_4
Shopping_2_5
0
5000
10000
15000
20000
Page Performance
PlantsbyWebSphere
Baseline
No Paging
Page URL

Effect on page performance
– PlantsByWebSphere 1.7%
– servlet_ShoppingServlet{2} 22.8%
– servlet_ShoppingServlet 0.3%
– Shopping{1} 0.4%
– Shopping_1_1 17.9%
■ Improvement in page performance of 0.3% to 24.1%
– Biggest gains were for those pages that were already fast
■ Biggest gains are on most performant pages. Total gain is only ~4%
■ Small effect on overall page performance

Garbage Collection Pause Times

Effect on Garbage Collection Pause Times
■ Reduction in:
– Maximum pause time: 38%
– Average pause time: 13%
– Time spent in GC 11%
■ Large effect on GC performance, particularly pause times

Swapping/Paging and Java Applications
■ Swapping/Paging usually occurs on Least Recently Used basis
– Live but non-recently used data is written out to disk to free RAM for used data
■ Java however “uses” non-recently used data during GC
– In order to determine if the data is “live” and still required by the application.
– Any Java heap memory that has been swapped out, must be swapped in during GC
Java Heap
Recently used objects
Non-recently used objects
■ 10 objects are not recently used

Java Heap
Live objects according to GC mark phase
– 6 objects needed to be swapped back in

Java Heap
Live objects according to GC mark phase
– Further 4 objects locations needed to be swapped back in

Java runtime
– Memory: insufficient Java heap results in OutOfMemory or high GC overhead
– CPU garbage collection overhead, or driven by Java application
– Synchronization driven by Java application
– IO driven by Java application
■ Easy to diagnose
■ Easy to resolve (relatively)

Detecting Java runtime problems
■ Log and trace analysis:
– “Native” heap: OS level logs (ps, svmon, perfmon)
– Java heap: verbose:gc output
■ Health Center
Native Memory
•Detect native memory leaks in application
•Determine if external forces are using more memory
•Viewcomponents using the most native memory
Garbage Collection visualization
• Visualizes heap usage and gc pause times over time
• Identifies memory leaks
• Suggests command-line and tuning parameters
• Same recommendation logic as GCMV

Too Frequent Garbage Collection

© 2011 IBM Corporation31 JavaOne Session 23401 – The Hidden World of Your Java Application and What its Really Doing
Too Frequent Garbage Collection
Long Garbage Collection Cycles
Garbage collection performanceMemory
Time
Heap Size
Heap Occupancy

Resolving Java runtime problems
■ Add more resources to the Java runtime
– Java heap: Increase Java heap size
– Native heap: Move to 64bit or reduce Java heap size
■ Reduce the memory requirements
– Reduce the Java application footprint

Increased Java heap size

Effect on Garbage Collection Pause Times
■ Reduction in:
– Time spent in GC 59%
■ However this is only 4.84% of total time

PlantsByWebSphere
Shopping{1}
Shopping{4}
Shopping_1_1
Shopping_2_2
Shopping_2_3
Shopping_2_4
Shopping_2_5
0
5000
10000
15000
20000
Page Performance
PlantsbyWebSphere
Baseline
Page URL

PlantsByWebSphere
Shopping{1}
Shopping{4}
Shopping_1_1
Shopping_2_2
Shopping_2_3
Shopping_2_4
Shopping_2_5
0
5000
10000
15000
20000
Page Performance
PlantsbyWebSphere
Baseline
Page URL
PlantsByWebSphere
Shopping{1}
Shopping{4}
Shopping_1_1
Shopping_2_2
Shopping_2_3
Shopping_2_4
Shopping_2_5
0
5000
10000
15000
20000
Page Performance
PlantsbyWebSphere
Baseline
No Paging
Page URL

Page response performance benchmark: Java Heap size increased
PlantsByWebSphere
Shopping{1}
Shopping{4}
Shopping_1_1
Shopping_2_2
Shopping_2_3
Shopping_2_4
Shopping_2_5
0
5000
10000
15000
20000
Page Performance
PlantsbyWebSphere
Baseline
No Paging
Bigger Heap
Page URL

Page performance improvements
– PlantsByWebSphere 0.4%
– servlet_ShoppingServlet 0.0%
– servlet_ShoppingServlet{2} 33.2%
– Shopping{1} +0.6%
■ Improvement in page performance of -0.6% to 33.2%
– Without outliers: 0.0% to 14.2%
■ Total gain is only ~4%
– Relatively small effect on overall page performance

Java application
– Memory: insufficient caching affects application throughput and responsiveness
– CPU: insufficient threading causes limits on scalability
– Synchronsation: synchronized resources limits scalability and throughput of the application
– I/O: blocking on I/O limits throughput and responsiveness
■ Hard to diagnose
■ Can be expensive (or impossible!) to resolve

Java application CPU usage
■ High CPU usage by Java methods highlight areas of potential optimization
– Code is being invoked more than it needs to be
• Easily done with event driven models
– An algorithm is not the most efficient
• Easily done if performance is not the focus at development time
■ Fixing CPU bound applications requires knowledge of what code is being run
– Identify methods which are suitable for optimisation
• Optimising methods which the application doesn’t spend time in is a waste of your time
– Identify methods where more time is being spent that you expect
• “Why is so much of time being spent in this trivial method?”

Java application synchronization
■ Throughput does not increase linearly with load
■ At limit of throughput the CPU is still low
– Inability to scale
– Not all CPU can be utilized
– Limit on throughput and responsiveness
■ Bottleneck where threads need to synchronize with each other for application correctness
– Caused by large numbers of threads requiring synchronized resource at the same time
– Caused by long hold time by thread that owns resource
– Or a mixture of both

© 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application
1. Performance Overview
• Request summary highlights slowest
requests
• Plots response times / CPU over time
• Breakdown to show hot spots
• Distribution chart shows percentage of
requests that are slow
2. Drill down to view Request Instances
• Breakdown of response time or CPU for
individual requests
3. Determine the root cause
• Select a request instance to view a method
tree that visually identifies bottlenecks
• Stack trace is viewable in context
• Aggregated method calls also available
• Link to application source in IDE
Java Application (using APD Lite)

1
APD Lite Performance Overview

Right click the highlighted row
and select “Show Context/Stack
Data”
Drill down to Tree View

Right click the highlighted row
and select “Show Context/Stack
Data” to see SQL call details
Drill down to tree view for more complex case

Health Center: application method CPU usage

Health Center: application synchronization

ShoppingServlet.deliberateSlowMethod()
private void deliberateSlowMethod() {
// ----------------------------------------------------------------
// User clicked on the Tulips, let's tip toe through a
// slow method
// ----------------------------------------------------------------
System.out.println("==> STARTING SLOW METHOD");
long timestamp = System.currentTimeMillis();
long target = timestamp + SLOWTIME;
System.out.println("timestamp="+timestamp);
System.out.println("resume at="+target);
while(timestamp < target) {
timestamp = System.currentTimeMillis();
}
System.out.println("==> ENDING SLOW METHOD");
}

PlantsByWebSphere
Shopping{1}
Shopping{4}
Shopping_1_1
Shopping_2_2
Shopping_2_3
Shopping_2_4
Shopping_2_5
0
5000
10000
15000
20000
Page Performance
PlantsbyWebSphere
Baseline
Page URL

PlantsByWebSphere
Shopping{1}
Shopping{4}
Shopping_1_1
Shopping_2_2
Shopping_2_3
Shopping_2_4
Shopping_2_5
0
5000
10000
15000
20000
Page Performance
PlantsbyWebSphere
Baseline
Page URL
PlantsByWebSphere
Shopping{1}
Shopping{4}
Shopping_1_1
Shopping_2_2
Shopping_2_3
Shopping_2_4
Shopping_2_5
0
5000
10000
15000
20000
Page Performance
PlantsbyWebSphere
Baseline
No Paging
Page URL

Page response performance benchmark: Java Heap size increased
PlantsByWebSphere
Shopping{1}
Shopping{4}
Shopping_1_1
Shopping_2_2
Shopping_2_3
Shopping_2_4
Shopping_2_5
0
5000
10000
15000
20000
Page Performance
PlantsbyWebSphere
Baseline
No Paging
Bigger Heap
Page URL

Page response performance benchmark: deliberateSlowMethod() changed
PlantsByWebSphere
Shopping{1}
Shopping{4}
Shopping_1_1
Shopping_2_2
Shopping_2_3
Shopping_2_4
Shopping_2_5
0
5000
10000
15000
20000
Page Performance
PlantsbyWebSphere
Baseline
No Paging
Bigger Heap
App Fix
Page URL

Page performance improvements
■ servlet_ShoppingServlet 80.8% 5x improvement
■ Shopping{1} 94.3% 20x improvement
■ Improvement in page performance of 80 and 95%
– 5x and 20x improvement for affected pages
■ Total gain of 48%!

Health Center: application method CPU usage

Health Center: application synchronization

Page response performance benchmark: Summary
PlantsByWebSphere
Shopping{1}
Shopping{4}
Shopping_1_1
Shopping_2_2
Shopping_2_3
Shopping_2_4
Shopping_2_5
0
5000
10000
15000
20000
Page Performance
PlantsbyWebSphere
Baseline
No Paging
Bigger Heap
App Fix
Page URL

Summary
■ Increasing performance of applications is possible using:
– Repeatable benchmark
– Incremental measurements as changes are made
– Repeated testing and verification
■ Infrastructure resources affect performance, but vast majority of performance gains are in the
application!
■ Free tools are available to help you see what's going on:
– Operating System tools: perfmon, vmstat, ps, svmon, etc
– verbose:gc
– APD Lite
– HealthCenter
■ But you need to use multiple tools to get a complete picture.....
■ Surely there's an easier way?

1. APD Lite
– Pre-Production only
– Collects real time data
– Value in 15 minutes
– Key features
• On demand collection of code
level data in pre-production
environments
• Plugs into Eclipse IDE for
launch to application source
2. APD Standard
– Available as SaaS or on
premises
– Suited for all environments
– Collects real time and short-
term trend data
– Value in 15 minutes
– Key features
• Identify app server health at a
glance and view KPIs
• Automatic collection of code
level data in pre-production or
production for diagnosis
• Leverages lightweight
infrastructure for seamless
monitoring across
applications
APD Lite
Development Test Production
APD Standard
IBM Application Performance Diagnostics Overview

1. Identify there is a slowdown in response time/throughput for transactions for a JVM over
time
2. Identify, Isolate and Diagnose if an application response issue is caused by a WebSphere
resource issue:
• Heap / GC
• Thread Pool
• DB Connection Pool etc
3. Diagnose if an application response issue is caused by application code
4. Report slow transactions, SQLs, etc. with method tracing turned on dynamically
5. Alternative to Traps
Typical User Scenarios

Diagnose if an application response issue is caused by
application code
(Slow transaction – Bad SQL etc)

Transaction with High
Response Time – All
other metrics look ok

Transaction with High
Response Time – We
should look at the
Sequence Data

Problem
SQL

Summary
■ Infrastructure resources affect performance
– Paging and Garbage Collection much less than you might expect
– However, beware of CPU “starvation” from other processes!
■ Vast majority of performance gains are in the application!

References
■ Get Products and Technologies:
– IBM Monitoring and Diagnostic Tools for Java:
• https://www.ibm.com/developerworks/java/jdk/tools/
■ Learn:
– Health Center InfoCenter:
• http://publib.boulder.ibm.com/infocenter/hctool/v1r0/index.jsp
■ Discuss:
– IBM on Troubleshooting Java Applications Blog:
• https://www.ibm.com/developerworks/mydeveloperworks/blogs/troubleshootingjava/
– Health Center Forum:
• http://www.ibm.com/developerworks/forums/forum.jspa?forumID=1461
– IBM Java Runtimes and SDKs Forum:
• http://www.ibm.com/developerworks/forums/forum.jspa?forumID=367&start=0

We Value Your Feedback
Don’t forget to submit your Impact session and speaker feedback!
Your feedback is very important to us – we use it to continually
improve the conference.
Use the Conference Mobile App or the online Agenda Builder to quickly
submit your survey
• Navigate to “Surveys” to see a view of surveys for sessions
you’ve attended
71

Legal Disclaimer
• © IBM Corporation 2014. All Rights Reserved.
• The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained in this publication,
it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are subject to change by IBM without
notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor
shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of
IBM software.
• References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced
in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in
any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or
other results.
• If the text contains performance statistics or references to benchmarks, insert the following language; otherwise delete:
Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary
depending upon many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload
processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.
• If the text includes any customer examples, please confirm we have prior written approval from such customer and insert the following language; otherwise delete:
All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and
performance characteristics may vary by customer.
• Please review text for proper trademark attribution of IBM products. At first use, each product name must be the full name and include appropriate trademark symbols (e.g., IBM Lotus® Sametime®
Unyte™). Subsequent references can drop “IBM” but should include the proper branding (e.g., Lotus Sametime Gateway, or WebSphere Application Server). Please refer to
http://www.ibm.com/legal/copytrade.shtml for guidance on which trademarks require the ® or ™ symbol. Do not use abbreviations for IBM product names in your presentation. All product names
must be used as adjectives rather than nouns. Please list all of the trademarks that you use in your presentation as follows; delete any not included in your presentation. IBM, the IBM logo, Lotus,
Lotus Notes, Notes, Domino, Quickr, Sametime, WebSphere, UC2, PartnerWorld and Lotusphere are trademarks of International Business Machines Corporation in the United States, other
countries, or both. Unyte is a trademark of WebDialogs, Inc., in the United States, other countries, or both.
• If you reference Adobe® in the text, please mark the first use and include the following; otherwise delete:
Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.
• If you reference Java™ in the text, please mark the first use and include the following; otherwise delete:
Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.
• If you reference Microsoft® and/or Windows® in the text, please mark the first use and include the following, as applicable; otherwise delete:
Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both.
• If you reference Intel® and/or any of the following Intel products in the text, please mark the first use and include those that you use as follows; otherwise delete:
Intel, Intel Centrino, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other
countries.
• If you reference UNIX® in the text, please mark the first use and include the following; otherwise delete:
UNIX is a registered trademark of The Open Group in the United States and other countries.
• If you reference Linux® in your presentation, please mark the first use and include the following; otherwise delete:
Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others.
• If the text/graphics include screenshots, no actual IBM employee names may be used (even your own), if your screenshots include fictitious company names (e.g., Renovations, Zeta Bank, Acme)
please update and insert the following; otherwise delete: All references to [insert fictitious company name] refer to a fictitious company and are used for illustration purposes only.

Impact2014: Practical Performance Troubleshooting

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Impact2014: Practical Performance Troubleshooting

Similar a Impact2014: Practical Performance Troubleshooting (20)

Más de Chris Bailey

Más de Chris Bailey (20)

Último

Último (20)

Impact2014: Practical Performance Troubleshooting