Performance Tuning - Memory leaks, Thread deadlocks, JDK tools

Performance Tuning
Memory Leaks, Thread Contention,
JDK Tools for monitoring
Haribabu Nandyal
Performance Engineering Expert

Agenda
 Memory Leaks in Java applications
 Best Practices to avoid memory leaks
 Thread Contentions
 Thread dump and Analysis
 JDK tools for performance diagnosis

Finding Memory Leaks in Java
applications

Agenda
 What is memory leak?
 Implications of memory leak
 How to locate memory leak?
 Example Code for simple Memory Leak
 Analyze the dump logs
 Performance Checklists

 When an application dynamically allocates memory, and
does not free that memory after using it, that program
is said to be having memory leak.
 Memory Leak is the gradual loss of system available
memory when the application fails to return the memory
that it has consumed for temporary needs.
What is a memory leak?
 Memory can neither be used by the system nor any other program.
 Over a period of time; the system eventually runs out of memory
 slow response time resulting in performance issues
Implications of memory leak:

Memory Leak
Also known as unintentional object retention (the object that
is not needed anymore and should be unreachable is still
reachable, hence cannot be collected)

Basic Types of Memory Leaks
 A single large object that is unintentionally held in memory
 Small objects that are accumulated over time
Memory Leak Demo
 Run the project in demo/MemoryManagement/MemoryLeak
 Run Visual VM and open the project
 Monitor the project
Heap Snapshots Comparison
 Take memory snapshots in different points of time(advised to run garbage
collection before taking snapshot)
 Compare the snapshots (e.g. in Visual VM)
 Search for leaking objects within the objects that were identified as new

 Heap dump (non intrusive)
• Generate a dump file as large as the heap
memory
• Freezes application while dump is generated
 Profilers (intrusive)
• Need to reconfigure VM settings
• Attach a client profiler to introspect
 Monitor memory usage
• prstat, top, ps
 Monitor GC activity
• jstat, -verbose:gc, visualgc, jconsole
Track down where the problem is

 Overview -> overview on the running application
 Monitor -> view real-time, high-level data on the memory heap,
thread activity, and the classes loaded in the Java Virtual
Machine
 Threads -> displays a timeline of current thread activity
 Sampler -> provides a tool with performance and memory
profiler
 Profiler -> enables you to start and stop the profiling session of
a local application
To detect memory leak, use JvisualVM/Jconsole and monitor any application.

Navigate to Profiler->Memory option
Analyze the memory consumption of the live bytes against Class Names.

If you are suspecting Memory Leak,
collect Heap Dump

Heap Summary shows the different generations (areas of the heap),
with the size, the amount used, and the address range. The address
range is especially useful if you are also examining the process with
tools such as pmap.
Heap
def new generation total 1152K, used 435K [0x22960000, 0x22a90000, 0x22e40000)
eden space 1088K, 40% used [0x22960000, 0x229ccd40, 0x22a70000)
from space 64K, 0% used [0x22a70000, 0x22a70000, 0x22a80000)
to space 64K, 0% used [0x22a80000, 0x22a80000, 0x22a90000)
tenured generation total 13728K, used 6971K [0x22e40000, 0x23ba8000, 0x269600 00)
the space 13728K, 50% used [0x22e40000, 0x2350ecb0, 0x2350ee00, 0x23ba8000)
compacting perm gen total 12288K, used 1417K [0x26960000, 0x27560000, 0x2a9600 00)
the space 12288K, 11% used [0x26960000, 0x26ac24f8, 0x26ac2600, 0x27560000)
ro space 8192K, 62% used [0x2a960000, 0x2ae5ba98, 0x2ae5bc00, 0x2b160000)
rw space 12288K, 52% used [0x2b160000, 0x2b79e410, 0x2b79e600, 0x2bd60000)
If the Java VM flag -XX:+PrintClassHistogram is set, then the Ctrl-Break handler will produce
a heap histogram.
Heap Summary

Navigate to Classes -> select the suspected class -> Show in Instances View
Instances tab -> shows “out of memory”

Heap and lock contention issues:
• Avoid creating objects where primitives could be used (autoboxing
is not for free)
• Avoid String concatenations, use StringBuilder with append()
• Avoid StringBuilder overallocation, set initial size to reduce
resizing
• Avoid use of synchronized keyword, look for a java.util.concurrent
solution
• Use java.util.concurrent API instead of "synchronized" Collections
• Don't use exceptions for flow control (Exception objects
allocation), use flow control constructs (if/else, return, switch).
• Don't generate stack traces, they're expensive operations.
• Reduce number of system calls:
 Avoid accessing disk: use large buffers or direct buffering
 Avoid accessing underlying OS
 Avoid processing bytes and characters individually
Best Practices

Tools and Tips - Memory
1. Java heap space
Causes
• Low RAM memory
• Small heap
• Memory leak
Fix
• More physical memory
• Make heap larger
• Fix memory leaks
2. PermGen space
Causes
• Low memory for permanent generation area
• Classes on permanent generation are prevented to be
collected
 Classloader leak
Fix
• Make permanent generation area, larger
• Disable hot deployment or class reloading – for classloader
leak

3. Out of swap space
Cause
• Often is caused by JNI code
• No memory available for operating system
Fix
• Reserve more memory to operating system
• Lower the heap size
4. Unable to create new native thread
Cause
• Big stack size (-Xss)
• Small memory available
• Poor thread management
Fix
• Make stack size smaller
• Reserve more memory to operating system
• Lower heap size
• Better thread management
 Java 5 thread pool

5. Requested array size exceeds VM limit
Cause
• Attempt to allocate an array that is larger than the heap
size
Fix
• Lower the array size
• Make heap larger
6. GC overhead limit exceeded
Cause
• Too much time spent on GC (98%) and little heap (2%) to be
free
Fix
• Similar to “java heap space”
• Tune GC behavior

Agenda
 Process and Threads
 Thread States
 Thread Contentions
 Thread dump and Analysis

What is a Process?
 A program in execution
 Provides the resources needed to execute a
program
 One process can't access or modify another process
What is a Thread?
 A basic unit of CPU utilization
 Lightweight process (LWP)
 Multiple threads are executed within a process
 Multiple threads shares memory and resources of the
process in which it is created.
Threads

Threads in Java
Java language includes a powerful threading facility built into the
language. We can use the threading facility to:
 Increases the responsiveness of GUI applications
 Take advantage of multiprocessor systems
 Simplify program logic when there are multiple independent
entities
 Thread is cheap
 No need to worry about memory model in different
environments

Thread T1 – has a lock on object O1
Thread T2 – has a lock on object O2
T1 is waiting to access O2, while T2 is
waiting to access O1.
Deadlock
A situation in which a thread has got an object, while waiting
for another object which is locked by another thread, which in
turn is waiting for the object locked by the first thread.

synchronized(object) {
//
do something...
}
Race Condition and Synchronization
● Race conditions occur when multiple, asynchronously executing
threads access the same object (called a shared resource)
returning unexpected results.
● When two or more threads are trying to access the shared
object, the process of allowing only one thread access that
object is called Synchronization.
● Only one thread can execute the block of code protected by
the same lock at the same time.

Lock contention
• In the java application java thread waits for lock in synchronized method
• Occurs in multi threaded program
• Reduce frequency of synchronized method
• Synchronized block must be implemented to end as soon as possible
public void synchronized methodA(Object param)
{
//do something
while(1){}
}
Thread1 Thread3
Thread2
Thread4
Sychronized
MethodA
Lock contention : Thread lock example

"ExecuteThread: '12' for queue: 'weblogic.kernel.Default'" daemon prio=10 tid=0x0055ae20
nid=23 lwp_id=3722788 waiting for monitor entry [0x2fb6e000..0x2fb6d530]
:
at java.lang.ClassLoader.loadClass(ClassLoader.java:255)
:
at org.apache.xerces.jaxp.SAXParserFactoryImpl.newSAXParser(Unknown Source)
at org.apache.axis.utils.XMLUtils.getSAXParser(XMLUtils.java:252)
- locked <0x329fcf50> (a java.lang.Class)
"ExecuteThread: '13' for queue: 'weblogic.kernel.Default'" daemon prio=10 tid=0x0055bde0 nid=24
lwp_id=3722789 waiting for monitor entry [0x2faec000..0x2faec530]
at org.apache.axis.utils.XMLUtils.getSAXParser(XMLUtils.java:247)
- waiting to lock <0x329fcf50> (a java.lang.Class)
:
"ExecuteThread: '15' for queue: 'weblogic.kernel.Default'" daemon prio=10 tid=0x0061dc20 nid=26
lwp_id=3722791 waiting for monitor entry [0x2f9ea000..0x2f9ea530]
at org.apache.axis.utils.XMLUtils.releaseSAXParser(XMLUtils.java:283)
- waiting to lock <0x329fcf50> (a java.lang.Class)at
Lock contention : Thread dump example

Dead lock
• CWC “Circular Waiting Condition”
• Commonly it makes WAS hang up
• Remove CWC by changing lock direction
• Some JVMs detect dead lock automatically
DeadLock contention
sychronized methodA(){
call methodC();
}
sychronized methodB(){
call methodA();
}
sychronized methodC(){
call methodB();
}
Thread1
Sychronized
MethodA
Thread2
Sychronized
MethodB
Thread3
Sychronized
MethodC

FOUND A JAVA LEVEL DEADLOCK:
----------------------------
"ExecuteThread: '12' for queue: 'default'":
waiting to lock monitor 0xf96e0 (object 0xbd2e1a20, a weblogic.servlet.jsp.JspStub),
which is locked by "ExecuteThread: '5' for queue: 'default'"
"ExecuteThread: '5' for queue: 'default'":
waiting to lock monitor 0xf8c60 (object 0xbd9dc460, a weblogic.servlet.internal.WebAppServletContext),
which is locked by "ExecuteThread: '12' for queue: 'default'" JAVA STACK INFORMATION FOR
THREADS LISTED ABOVE:
------------------------------------------------
Java Stack for "ExecuteThread: '12' for queue: 'default'":
==========
at weblogic.servlet.internal.ServletStubImpl.destroyServlet(ServletStubImpl.java:434)
- waiting to lock <bd2e1a20> (a weblogic.servlet.jsp.JspStub)
at weblogic.servlet.internal.WebAppServletContext.removeServletStub(WebAppServletContext.java:2377)
at weblogic.servlet.jsp.JspStub.prepareServlet(JspStub.java:207)
:
Java Stack for "ExecuteThread: '5' for queue: 'default'":
==========
at weblogic.servlet.internal.WebAppServletContext.removeServletStub(WebAppServletContext.java:2370)
- locked <bd2e1a20> (a weblogic.servlet.jsp.JspStub)
at weblogic.servlet.internal.ServletStubImpl.getServlet(ServletStubImpl.java:368)
:
Found 1 deadlock.========================================================

What is "Thread dump“ ?
Thread dump" basically gives you information on what each
of the thread in the VM is doing at any given point of time.
Java's thread dumps are vital for server debugging. It helps us
to identify
 Performance Related issues
 Deadlock (System Locks)
 Timeout Issues
 Systems stops processing Traffic
Thread dumps

The thread dump consists of the thread stack, including thread state, for all Java
threads in the virtual machine.
The header line contains the following information about each thread:
Thread name
Indication if the thread is a daemon thread
Thread priority ( prio)
Thread ID ( tid), which is the address of a thread structure in memory
ID of the native thread ( nid)
Thread state: Indicates what the thread was doing at the time of the thread dump
Address range,: Estimate of the valid stack region for the thread
"DestroyJavaVM" prio=10 tid=0x00030400 nid=0x2 waiting on condition
[0x00000000..0xfe77fbf0] java.lang.Thread.State: RUNNABLE
"Thread2" prio=10 tid=0x000d7c00 nid=0xb waiting for monitor entry [0xf36ff000..0xf36ff8c0]
java.lang.Thread.State: BLOCKED (on object monitor) at
Deadlock$DeadlockMakerThread.run(Deadlock.java:32) - waiting to lock <0xf819a938> (a
java.lang.String) - locked <0xf819a970> (a java.lang.String)
"Low Memory Detector" daemon prio=10 tid=0x000c7800 nid=0x8 runnable
[0x00000000..0x00000000] java.lang.Thread.State: RUNNABLE
Thread Dumps

public class TestThread {
public static Object Lock1 = new Object();
public static Object Lock2 = new Object();
public static void main(String args[]) {
ThreadDemo1 T1 = new ThreadDemo1();
ThreadDemo2 T2 = new ThreadDemo2();
T1.start();
T2.start();
}
private static class ThreadDemo1 extends Thread
{
public void run() {
synchronized (Lock1) {
System.out.println("Thread 1: Holding lock
1...");
try { Thread.sleep(10); }
catch (InterruptedException e) {}
}
}
}
private static class ThreadDemo2 extends
Thread {
public void run() {
System.out.println("Thread 2:
Holding lock 2...");
try { Thread.sleep(10); }
catch (InterruptedException e) {}
Waiting for lock 1...");
Holding lock 1 & 2...");
}
}
}
}
}
Thread DeadLock: example code

Monitor the threads while application is running and if you notice
few threads in Monitor state, take a thread dump.
Thread DeadLock

JDK tools for performance
diagnosis
• VisualVM
• Btrace
• Visualgc Plugin
• jhat
• jmap
• jconsole
• jstat
• jps
• jstack
• Jstatd Daemon

Java Standard Edition 7 Documentation

Java VisualVM (swiss army knife for JVM process monitoring and tracing)
Java VisualVM is an utility that can generate and analyze heap dumps, track down
memory leaks, perform and monitor garbage collection, and perform lightweight
memory and CPU profiling.
There are host of standard Java VisualVM plug-ins available which can expand its
existing functionality. E.g: Jconsole, VisualGC etc
For example, most of the functionality of the JConsole tool is available via the
MBeans tab and the JConsole plug-in wrapper tab

Java VisualVM features:
• View a list of local and remote Java applications.
• View application configuration and runtime environment data:
PID, host, main class, arguments passed to the process, JVM version, JDK home, JVM flags, JVM
arguments, system properties etc.
• Monitor application memory consumption, running threads, and loaded classes.
• Trigger Garbage Collection.
• Create a heap dump.
Multiple views: summary, by class, by instance and also save the heap dump.
• Profile application performance or analyze memory allocation (for local
applications only) and save the profiling data.
• Create thread dump (stack trace of the application's active threads).
• Analyze applications offline, by taking application snapshots.
• Get additional plug-ins contributed by the community.

Display local and remote Java applications.
VisualVM automatically detects and lists locally and remotely running Java applications (jstatd must
be running on the remote host). This way you can easily see what Java applications are running on
your system or check if a remote J2EE server process is alive.
Display application configuration and runtime environment
For each application VisualVM shows basic runtime information: PID, main class, arguments passed
to java process, JVM version, JDK home, JVM flags and arguments and system properties.
Monitor application performance and memory consumption.
VisualVM monitors application CPU usage, GC activity, heap and permanent generation memory,
number of loaded classes and running threads. You can easily detect suspicious memory consumption
and take an action - invoke garbage collection in the application or take a heap dump and browse the
contents of application heap.
Monitor application threads.
All threads running in a Java process are displayed in a timeline and table. You can track thread
activity and uncover inefficient patterns.
Profile application performance or analyze memory allocation. VisualVM includes a built-in
application profiler which can visualize where most of the time is being spent or which objects
consume most of the memory.
VisualVM features

Take and display thread dumps. Taking and displaying a thread dump is as easy as clicking a mouse
button. Moreover, simultaneous thread dumps of multiple applications can be taken at once to start
uncovering distributed deadlocks.
Take and browse heap dumps. When you need to browse contents of application memory or
uncover a memory leak in your application, you'll find the built-in HeapWalker tool really handy. It
can read files written in hprof format and is also able to browse heap dumps created by the JVM on
an OutOfMemoryException.
Analyze core dumps. When a Java process crashes, a core dump can be generated by the JVM
containing important information about application state at the time of the crash. VisualVM is able
to display application configuration and runtime environment and to extract thread and heap dumps
from the core dump.
Analyze applications offline. VisualVM is able to save application configuration and runtime
environment together with all taken thread dumps, heap dumps and profiler snapshots into a single
application snapshot which can be later processed offline. This is especially useful for bug reports
where users can provide a single file containing all the necessary information to identify runtime
environment and application state.
IDE Integration
VisualVM can be integrated with your IDE (Eclipse and NetBeans IDE) to monitor and analyze the
code iteratively during development.
VisualVM features

JConsole Utility
• JConsole is a monitoring tool that uses built-in JMX instrumentation in the JVM to
provide information on performance and resource consumption of the running applications.
• Useful in high-level diagnosis on problems such as memory leaks, excessive class loading,
and running threads. It can also be used for tuning and heap sizing.

• Overview
This pane displays graphs of heap memory usage, number of threads, number of
classes, and CPU usage.
• Memory
For a selected memory area (heap, non-heap, various memory pools) provides
 Graph of memory usage over time
 Current memory size
 Amount of committed memory
 Maximum memory size.
 Garbage collector information, including the number of collections performed, and
the total time spent performing GC.
 Graph showing percentage of heap and non-heap memory currently used.
 Link to request to perform GC.
• Threads
 Details about Graph of thread usage over time
 Live threads - Current number of live threads
 Peak - Highest number of live threads since the Java VM started
 For a selected thread, the name, state, and stack trace.
 Button for Deadlock Detection
Jconsole components

• Classes
 Graph in number of classes loaded over time.
 Number of classes currently loaded into memory.
 Total number of classes loaded into memory since starting of Java VM, including
those subsequently unloaded.
 Total number of classes unloaded from memory since start of Java VM.
• VM Summary
 General information, such as the JConsole connection data, uptime for Java VM,
CPU time consumed by Java VM, complier name and total compile time etc.
 Thread and class summary information.
 Memory and garbage collection information, including number of objects pending
finalization etc.
 Information about the OS, including physical characteristics, amount of virtual
memory for the running process, swap space etc.
• MBeans
This pane displays a tree structure showing all platform and application MBeans
that are registered in the connected JMX agent. When you select an MBean in the
tree, its attributes, operations, notifications, and other information are displayed.
Jconsole components

JMap utility can obtain memory map information, including a heap histogram, from a Java
process, a core file, or a remote debug server.
The jmap command-line utility prints memory related statistics for a running VM or core file.
If jmap is used with a process or core file without any command-line options, then it prints
the list of shared objects loaded.
For more specific information, we can use the options -heap, -histo, or -permstat. In addition,
we can use dump:format=b,file= filename option, which causes jmap to dump the Java heap in
binary HPROF format to a specified file. This file can then be analyzed with the jhat tool.
Jmap Utility
The jstack command-line utility attaches to the specified process or core file and
prints the stack traces of all threads that are attached to the virtual machine,
including Java threads and VM internal threads, and optionally native stack frames.
The utility also performs deadlock detection.
The stack trace of all threads can be useful in diagnosing a number of issues such as
deadlocks or hangs.
Jstack Utility

The jstat utility uses the built-in instrumentation in the HotSpot VM to
provide information on performance and resource consumption of running
applications. The tool can be used when diagnosing performance issues, and in
particular issues related to heap sizing and garbage collection.
Jstat Utility
Jstatd Daemon
Visualgc Plugin
The jstatd daemon is a Remote Method Invocation (RMI) server application
that monitors the creation and termination of instrumented Java HotSpot VMs
and provides an interface to allow remote monitoring tools to attach to Java
VMs running on the local host.
The visualgc tool attaches to an instrumented HotSpot JVM and
collects and graphically displays garbage collection, class loader,
and HotSpot compiler performance data. The target JVM is
identified by its virtual machine identifier, or vmid.

jhat
• Java Heap Analysis Tool
• Parses a java heap dump and launches a web server
• Basic web interface for browsing the heap dump
• Useful for analyzing memory leaks
jhat Features
• View all class instances
• View class classloader
• View all references to an object
• View reference chains from Rootset
• Use OQL (Object Query Language) to query heap dumps
• SQL-like query language to query Java heap
• Based on JavaScript expression language
• Built-in objects and functions
• OQL help is available from the jhat OQL screen
OQL

Performance Tuning - Memory leaks, Thread deadlocks, JDK tools

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Destacado

Destacado (20)

Similar a Performance Tuning - Memory leaks, Thread deadlocks, JDK tools

Similar a Performance Tuning - Memory leaks, Thread deadlocks, JDK tools (20)

Último

Último (20)

Performance Tuning - Memory leaks, Thread deadlocks, JDK tools