Monitoring and Diagnostic Tools for Java

© 2013 IBM Corporation© 2013 IBM Corporation
Introduction to the IBM Monitoring
and Diagnostic Tools for Java
Chris Bailey, Peter Whitehead, and Guests!
IBM
TAW-1560

© 2013 IBM Corporation
Please Note
IBM’s statements regarding its plans, directions, and intent are subject to change
or withdrawal without notice at IBM’s sole discretion.
Information regarding potential future products is intended to outline our general
product direction and it should not be relied on in making a purchasing decision.
The information mentioned regarding potential future products is not a
commitment, promise, or legal obligation to deliver any material, code or
functionality. Information about potential future products may not be incorporated
into any contract. The development, release, and timing of any future features or
functionality described for our products remains at our sole discretion.
Performance is based on measurements and projections using standard IBM
benchmarks in a controlled environment. The actual throughput or performance
that any user will experience will vary depending upon many factors, including
considerations such as the amount of multiprogramming in the user’s job stream,
the I/O configuration, the storage configuration, and the workload processed.
Therefore, no assurance can be given that an individual user will achieve results
similar to those stated here.

Introduction to the Speakers
Chris Bailey
Java Serviceability and Cloud Integration Architect
• 13 years experience developing and deploying Java SDKs
• Recent work focus:
‒ Java integration into the cloud
‒ Java monitoring, diagnostics and troubleshooting
‒ Requirements gathering
‒ Highly resilient and scalable deployments
• Contact Details:
‒ baileyc@uk.ibm.com
‒ http://www.linkedin.com/in/chrisbaileyibm
‒ http://www.slideshare.net/cnbailey/

Peter Whitehead
Program Manager, Java Technology Center and ASPN Development
• 23 years experience developing storage subsystems, program
management and deploying IBM's Java SDKs
• Recent work focus:
‒ Improving Critical Patch Update concurrency
‒ Delivery of Java runtimes for IBM platforms and products
‒ Creation of serviceability tooling suite for IBM Java runtimes
‒ Delivery of distributed cloud operating environment for mobile networks
• Contact Details:
‒ peter_whitehead@uk.ibm.com

Plus Guests!
• Dan Julin: STSM, AIM Serviceability, eSupport and Tools
• Jason Edmeades: Messaging Service Architect
• Andrew Smithson: CICS Transaction Gateway Technical Lead
• Chuck Bridgham: Software Architect: Rational Analysis, Design and
Construction

Goals of the talk
• Introduce the IBM tools for monitoring, debugging and diagnosing problems in
the deployment life cycle
• Understand the major problems scenarios, and how to select the best tools to
diagnose the problem
• Learn how to extend the tools with knowledge of your application or to build
custom tools
• See how IBM is building on top of the Java tools platform

Agenda
• Tools for Java
‒ Introduction to the IBM Monitoring and Diagnostic Tools for Java
• Extending the Tools
‒ Overview of the APIs and Use Cases
• IBM Extensions for the Java Tools
‒ IBM Extensions for Memory Analyzer (IEMA)
‒ Rational Application Developer v9.0

Tools for Java

IBM Monitoring and Diagnostic Tools for Java
• A free suite of tools for understanding different aspects of Java
applications
− Garbage Collection and Memory Visualizer (GCMV)
− Health Center
− Memory Analyzer (and IBM Extensions for Memory Analyzer)
− Interactive Diagnostic Data Explorer (IDDE)
• Provide visualisation, analysis and recommendations
• Fully supported
• Available in the IBM Support Assistant
− Provides the “toolbox” in which analysis tools reside
− Assist with opening PMRs and working with IBM Support
− Search feature to query IBM and non-IBM knowledge banks

Garbage Collection and Memory Visualizer (GCMV)
• Tool to analyze Java and “Native” Heap memory
− Using verbose GC logs and OS level process memory logs
• Graphs to show garbage collection and Java heap statistics over time
• Not only for memory errors, very good for performance tuning
• Recommendations use heuristics to guide you towards issues that may
be limiting performance

GCMV Highlights
• Graphical Display of Data
− Allows graphing of all available
data: pause times, heap size etc
− Allows zoom, cropping and
change of axes value and units
− Allows comparison of multiple
files
• Analysis and Recommendations
− Provides tuning
recommendations based on data
and flags errors.
− Analysis can be limited using
cropping.
− Values and units used in analysis
can be changed by changing
axes values and units

Heath Center
• Live monitoring tool with very low overhead
• Pre-installed in JDK and WebSphere Application Server
− Enable at startup: -Xhealthcenter
− Enable at runtime:java -jar healthcenter.jar ID=<pid>
• Understand how your application is behaving
− Monitor Class loading, File I/O, Environment settings, Garbage
Collection, Method Profiling, Locking, Native memory use
• Diagnose potential problems, with recommendations
• Works at the JVM level – no domain-specific (e.g. J2EE) information
• Suitable for all Java applications

Health Center Highlights
Environment reporting
• Detects invalid Java options
• Detects options which may hurt performance or
serviceability
• Useful for remote diagnosis of configuration-
related problems
Class loading visualization
• Shows all loaded classes
• Shows load time
• Identifies shared classes
File I/O
• Monitor application file open/close events as
they occur
• Lists currently open files

Garbage Collection visualization
• Visualizes heap usage and gc pause times over
time
• Identifies memory leaks
• Suggests command-line and tuning parameters
• Same recommendation logic as GCMV
Native Memory
• Detect native memory leaks in application
• Determine if external forces are using more
memory
• Memory counters showing which parts of the
JVM are using the most native memory
Threads view
• List of current threads and states
• Number of threads over time
• See contended monitors

Method Profiling
• Always-on profiling shows application activity
• Identifies the hottest methods in an application
• Full call stacks to identify where methods are
being called from and what methods they call
• No byte code instrumentation, no recompiling
Java Lock Profiling
• Always-on lock monitoring
• Quickly allows the usage of all locks to be
profiled
• Helps to identify points of contention in the
application that are preventing scaling
Live control of application
• Trigger dumps
• Enable verbosegc collection

Memory Analyzer
• Tool for analyzing heap dumps and identifying memory leaks from JVMs
• Works with IBM system dumps, heapdumps and Sun HPROF binary
dumps
• Provides memory leak detection and footprint analysis
− Objects by Class, Dominator Tree Analysis, Path to GC Roots,
Dominator Tree by Class Loader
• Shows areas of memory wastage
− Collections, duplicate strings, substring/char arrays, constant value
primitives
• Displays Stack trace with object references
• Provides SQL like object query language (OQL)
• Provides extension points to write analysis plugins

Memory Analyzer Highlights
Overview:
• Overview of the dump including Java heap size
and total number of objects.
• Provides links to continued analysis
Path to GC Roots:
• Provides the reference chain that prevents an
object being garbage collected.
Dominator Tree grouped by Class Loader:
• Lists the biggest objects using a “keep alive
tree”. Grouping by Class
• Loader limits the analysis to a single application
in a JEE environment

Interactive Diagnostic Data Explorer
• Provides a visual representation of your set of dump files
• An investigation log editor in which you run commands. This editor
provides the following features:
− Command assistance
− Auto-completion of text
− The ability to save commands and output, which you can then
send to other people
− Highlighted text and flagging of issues
− The ability to add your own comments
• An outline view to help you navigate the commands and output in the log
editor
• The ability to create your own commands, as plug-ins, and import plug-
ins created by others

IDDE Highlights
• Interactive Editor:
− Command assistance
− Auto-completion of text
− Ability to add your own comments
• Multiple format support:
− All operating system
− Multiple file formats
− Multiple address spaces on z/OS
• Multiple plug-in support:
− DTFJ
− Dump Analyzer
− Extensions for Memory Analyzer

Problem Scenarios and Tools
GCMV Health Center Memory Analyzer IDDE
Performance  Garbage
Collection
performance
only
 Method
Profiling
 Lock Analysis
 Garbage
Collection
 Garbage
analysis
 Collection
efficiency
Memory  Garbage
Collection
memory
monitoring
 Native
(process)
memory
monitoring
 Garbage
Collection
memory
monitoring
 Native
(process)
memory
monitoring
 Large object
allocations
 Java heap
memory
analysis
Runtime  Process
settings
 Class Loading
 Thread
execution
analysis
 Application
state reports
 Thread
execution
analysis
 Application
state reports

Extending the Tools

Extending the Tools
• Each of the tools has extensions and/or integrations points
• Memory Analyzer and IDDE can be easily extended
• Health Center data can be integrated into other tools

Extending Health Center
• Health Center provides an API to access the data and recommendations
shown in the Health Center Client
• Allows custom tools to receive, analyze and interpret the data
• Health Center is a monitoring platform

import org.eclipse.equinox.app.IApplication;
import org.eclipse.equinox.app.IApplicationContext;
import org.eclipse.swt.widgets.Display;
import org.eclipse.swt.widgets.MessageBox;
import org.eclipse.swt.widgets.Shell;
import com.ibm.java.diagnostics.healthcenter.api.ConnectionProperties;
import com.ibm.java.diagnostics.healthcenter.api.HealthCenter;
import com.ibm.java.diagnostics.healthcenter.api.factory.HealthCenterFactory;
import com.ibm.java.diagnostics.healthcenter.api.threads.ThreadsData;
/**
* This class controls all aspects of the application's execution
*/
public class Application implements IApplication {
HealthCenter hcMon;
public Object start(IApplicationContext context) throws Exception {
ConnectionProperties hcConn = new ConnectionProperties();
hcMon = HealthCenterFactory.connect(hcConn, true);
try {
System.out.println("hcMonWaiting for 10 seconds to allow initial data to be parsed
from the connection");
Thread.sleep(10000);
} catch (InterruptedException e) {
e.printStackTrace();
}
checkForDeadlock();
return IApplication.EXIT_OK;
}
Setup connection properties
Create a Health Center
connection

public void checkForDeadlock() {
while (!detectDeadlock()) {
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
}
}
}
private boolean detectDeadlock() {
ThreadsData hcthreadsData = hcMon.getThreadsData();
if (hcthreadsData == null) {
System.out.println("No threads yet");
} else {
if (hcthreadsData.deadlockDetected()) {
Display display = new Display();
Shell shell = new Shell(display);
MessageBox mb = new MessageBox(shell);
String deadlockMessage = new String();
String[] hcThreadsRec = hcthreadsData
.getCriticalRecommendations();
for (String rec : hcThreadsRec) {
deadlockMessage = deadlockMessage + rec + "n";
}
mb.setMessage(deadlockMessage);
mb.setText("Deadlock detected");
mb.open();
display.dispose();
return true;
}
}
return false;
}
Request Threads data
Check for deadlock in threads
Access the threads
recommendations
Display deadlock detected message

Extending Memory Analyzer and IDDE
• Memory Analyzer provides public extension points to build in additional
“expert” knowledge
• org.eclipse.mat.api.nameResolver
− Provides readable description of an object in some of MAT's view
(like toString())
− Annotates object types with detail about an object
− eg. hostname and port for a Socket object
• org.eclipse.mat.report.query
− Adds a menu item that executes your plug-in code
− Creates report style output in any form you want: Tree, Text,
HTML, Pie Charts, etc.

import org.eclipse.mat.SnapshotException;
import org.eclipse.mat.snapshot.extension.IClassSpecificNameResolver;
import org.eclipse.mat.snapshot.extension.Subject;
import org.eclipse.mat.snapshot.model.IObject;
@Subject("com.ibm.ws.webcontainer.webapp.WebAppImpl")
public class WebApp implements IClassSpecificNameResolver {
public String resolve(IObject object){
String result = null;
try {
IObject appName = (IObject)object.resolveValue("applicationName");
IObject contextPath = (IObject)object.resolveValue("contextPath");
if (appName != null && contextPath != null) {
result = appName.getClassSpecificName() + " at " +
contextPath.getClassSpecificName();
}
} catch (SnapshotException e) {
}
return result;
}
}

NameResolvers “annotate” objects with a line of text:
Select object type to “annotate”
Return String with text to annotate
Lookup field value
Get the String name of the object field

import org.eclipse.mat.SnapshotException;
import org.eclipse.mat.snapshot.extension.IClassSpecificNameResolver;
import org.eclipse.mat.snapshot.extension.Subject;
import org.eclipse.mat.snapshot.model.IObject;
@Name("Hung Thread Stacks")
@Category(WASHelper.WAS_CATEGORY + WASHelper.HUNGTHREADS_SUBCATEGORY)
@Help("List any threads that may be hung.nn")
public class HungThreadsList extends BasePlugin {
public IResult execute(IProgressListener listener) throws Exception {
int[] hungThreadIds = HungThreads.getHungThreadIds(snapshot);
return new HungThreadsListOutgoingReferencesTree(
hungThreadIds, listener);
}
Set name of query in menu
Set category in menu
Return “IResult” report
Find the Hung Threads

public static int[] getHungThreadIds(ISnapshot snapshot)
throws SnapshotException {
int[] threadIds = findObjects(snapshot, "java.lang.Thread");
List<Integer> hungThreads = new ArrayList<Integer>();
for (int thr : threadIds) {
IObject thrObj = snapshot.getObject(thr);
if (isThreadHung(thrObj)) {
hungThreads.add(thr);
}
}
int[] hungThreadObjectIds = getArrayIntegers(hungThreads);
return hungThreadObjectIds;
}
Find all Threads
Check if the Thread is hung
Return the hung threads
public static boolean isThreadHung(IObject thread) throws SnapshotException {
boolean result = false;
boolean isHung = MATHelper.resolveValueBool(thread, "isHung");
if (isHung != null && isHung.booleanValue()) {
result = true;
}
return result;
}
Get isHung field
Check value of field
Return field value

public class HungThreadsListOutgoingReferencesTree extends OutgoingReferencesTree {
private static final int sortIndex = COLUMN_INDEX_MAX_BUILTIN + 2;
public HungThreadsListOutgoingReferencesTree(ISnapshot snapshot,
int[] roots, IProgressListener listener)
throws SnapshotException {
super(snapshot, roots, listener);
}
Customized
OutgoingReferencesTree
• Hung Threads and references:

IBM Produced Extensions

IBM Extensions for Memory Analyzer
• Brings IBM product domain knowledge to Memory Analyzer and IDDE
− WebSphere Application Server
− WebSphere eXtreme Scale
− WebSphere ESB
− WebSphere Portal Server
− CICS Transaction Gateway
...plus others to follow

WebSphere AppServer Extensions for Memory Analyzer
• Enable visualization and analysis of the Application Server and
deployed applications
• Shows configuration, status and detailed content information:
• Software install levels and startup time
• Installed applications and status
• Thread pool configuration and utilization
• HTTP session cache size and session content
• Application Class Loader leak detection

WebSphere AppServer Extensions for Memory Analyzer
App Server Overview
• Installed Levels
• Server, Node and Cell names
• Installed applications and status
• Breakdown of memory usage by component
Thread Pool Analysis
• Thread Pool configuration
• Current size of threads in pools
• Hung Thread detection
Web Application Analysis
• Application Status
• Deployed Context Root and URL
• Number of user sessions
Plus many more:
• HTTP Session Contents, EJB Container, Class Loader leak
analysis etc...

WebSphere ESB Extensions for Memory Analyzer
• Extends Memory Analyzer with product specifics for WebSphere ESB
− Augments existing Memory Analyzer data structures
− Provides detailed product specific reports
• Breakdown of installed ESB applications for improved understanding of
the environment
− Improved efficiency and effectiveness of problem determination
and resolution
− WebSphere ESB L3 able to improve time to resolution of memory
related PMRs
− Customers have greater understanding of memory usage and run
time performance
• Design recommendations report identifies inefficient mediation flow logic
patterns and recommends alternatives for improved performance

WebSphere ESB Extensions for Memory Analyzer
Mediation Flows
• List of Mediation Flow Components
− Number of instances and memory usage
• Breakdown of Mediation Flow Component
− Mediation flow logic, display names, etc.
Application Overview
• Breakdown of each WebSphere ESB
application
− Name, parsing mode, dependencies,
endpoints
− Binding and component information, etc.
Plus a number of other reports:
• WebSphere ESB Overview, Mediation
Primitives, Design Recommendations and
Statistics

CICS TG Extensions for Memory Analyzer
Improving First Failure Data Capture (FFDC)
• CICS customers comfortable with taking dumps
• Aim to provide similar function to IPCS
• Gain useful information about state of the product

CICS TG Extensions for Memory Analyzer
Available reports
• Current Workload
• Statistics
• Configuration
• Connection details
Aims:
• Get as much information as possible
• Avoid recreates and trace
• Allow support to fully investigate

Rational Application Developer 9.0 and Health Center
• In addition to supporting trace-based profiling, Rational Application
Developer v9.0 Beta now supports sample-based profiling for
diagnosing execution time problems.
− Provided by Health Center Agent, the same as in IBM Support
Assistant.
− Using the Health Center API, introduced in Health Center v2.1
− Very low overhead compare with trace-based profiling.
− Available on all supported profiling launch types, including profiling
on WAS servers.

Fully qualified method name
Number of samples taken
while a particular method
was being run at the top of
the stack
The % of samples taken while a particular method
was being run at the top of the stack
The % of samples taken while a particular method
was anywhere in the call stack

Summary
• IBM is building a set of integrated monitoring and diagnostic tools
• Taking a ground up approach
• Working across teams, products and brands
• Providing a consistent user experience

Questions?

Other Sessions
• TAW-2713: Roundtable: WAS Serviceability and Problem Determination Tools
− Wednesday 13:00 and Thursday 13:00, Zeno 4703
• TAD-2709: Roundtable: IBM Support Assistant: Future Strategies for PD
− Wednesday 10:15 and Thursday 14:15, Zeno 4702
• TAW-1217: Lab: Diagnostic and Performance Tools for WAS
− Thursday 13:00, Murano 3301B
• AVP-3225: Lab: Java Troubleshooting with ISA 5 and TMDA
− Wednesday 10:15, Zeno 4601
• ICF-4622: Roadmap for WebSphere Liberty Performance Monitoring
− Thursday 15:45, Titian 2303

References
• IBM Monitoring and Diagnostic Tools for Java™
− http://www.ibm.com/developerworks/java/jdk/tools/
• IBM Extensions for Memory Analyzer
− http://www.alphaworks.ibm.com/tech/iema
• Debugging WebSphere Application Server from Dumps (MA)
− http://www.ibm.com/developerworks/websphere/techjournal/1103_
supauth/1103_supauth.html
• IBM Support Assistant (ISA)
− http://www.ibm.com/software/support/isa
• Email: javatool@uk.ibm.com

We love your Feedback!
Don’t forget to submit your Impact session and speaker feedback!
• Your feedback is very important to us – we use it to improve next year’s
conference
• Go to the Impact 2013 SmartSite (http://impactsmartsite/com):
‒ Use the session ID number to locate the session
‒ Click the “Take Survey” link
‒ Submit your feedback

Legal Disclaimer
• © IBM Corporation 2013. All Rights Reserved.
• The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained
in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are
subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing
contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and
conditions of the applicable license agreement governing the use of IBM software.
• References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or
capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to
future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you
will result in any specific sales, revenue growth or other results.
• If the text contains performance statistics or references to benchmarks, insert the following language; otherwise delete:
Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will
experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage
configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.
• If the text includes any customer examples, please confirm we have prior written approval from such customer and insert the following language; otherwise delete:
All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs
and performance characteristics may vary by customer.
• Please review text for proper trademark attribution of IBM products. At first use, each product name must be the full name and include appropriate trademark symbols (e.g., IBM
Lotus® Sametime® Unyte™). Subsequent references can drop “IBM” but should include the proper branding (e.g., Lotus Sametime Gateway, or WebSphere Application Server).
Please refer to http://www.ibm.com/legal/copytrade.shtml for guidance on which trademarks require the ® or ™ symbol. Do not use abbreviations for IBM product names in your
presentation. All product names must be used as adjectives rather than nouns. Please list all of the trademarks that you use in your presentation as follows; delete any not included in
your presentation. IBM, the IBM logo, Lotus, Lotus Notes, Notes, Domino, Quickr, Sametime, WebSphere, UC2, PartnerWorld and Lotusphere are trademarks of International
Business Machines Corporation in the United States, other countries, or both. Unyte is a trademark of WebDialogs, Inc., in the United States, other countries, or both.
• If you reference Adobe® in the text, please mark the first use and include the following; otherwise delete:
Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.
• If you reference Java™ in the text, please mark the first use and include the following; otherwise delete:
Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.
• If you reference Microsoft® and/or Windows® in the text, please mark the first use and include the following, as applicable; otherwise delete:
Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both.
• If you reference Intel® and/or any of the following Intel products in the text, please mark the first use and include those that you use as follows; otherwise delete:
Intel, Intel Centrino, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and
other countries.
• If you reference UNIX® in the text, please mark the first use and include the following; otherwise delete:
UNIX is a registered trademark of The Open Group in the United States and other countries.
• If you reference Linux® in your presentation, please mark the first use and include the following; otherwise delete:
Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of
others.
• If the text/graphics include screenshots, no actual IBM employee names may be used (even your own), if your screenshots include fictitious company names (e.g., Renovations, Zeta
Bank, Acme) please update and insert the following; otherwise delete: All references to [insert fictitious company name] refer to a fictitious company and are used for illustration
purposes only.

Monitoring and Diagnostic Tools for Java

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Monitoring and Diagnostic Tools for Java

Similar a Monitoring and Diagnostic Tools for Java (20)

Más de Chris Bailey

Más de Chris Bailey (20)

Último

Último (20)

Monitoring and Diagnostic Tools for Java