Memory Classloader Leak Detection in WebSphere Application Server

1. IBM WebSphere Application Server V8.5
Classloader Memory Leak Prevention, Detection &
Remediation
© 2011 IBM Corporation

2. Overview
2 © 2011 IBM Corporation

3. Memory Leak Detection, Prevention and Fixing
 Customers have discovered a number of Classloader & ThreadLocal leaks in
WebSphere Application Server and their own applications
 We don’t have application level i.e. top down memory leak detection and protection
 Customers want to increase application uptime without cycling the server
 Frequent redeployments of the application result in OOM errors
Existing (v7 & up) PM39870: Improved classloader leak detection.
NEW in v8.5 (Not in Liberty Profile)
 Prevention – Code to proactively fix suspect application classloader leak patterns
 Detection - Recognize Application triggered classloader leaks & provide diagnostics
 Fixing – Leverage existing JDK APIs and reflection to remedy classloader leaks.
 Enabled by setting the setting JVM custom properties
3 © 2011 IBM Corporation

4. Usage scenarios
4 © 2011 IBM Corporation

5. Types of Classloader memory leaks
JRE triggered leaks
The context class loader becomes the web application class loader.
A reference is created to the web application class loader. This reference is never garbage
collected.
Pins the class loader, and all the classes loaded by it, in memory.
Application triggered leaks
Custom ThreadLocal class
Web application class instance as ThreadLocal value
Web application class instance indirectly held through a ThreadLocal value
ThreadLocal pseudo-leak
ContextClassLoader and threads created by web applications
ContextClassLoader and threads created by classes loaded by the common class loader
Static class variables
JDBC driver registration: RMI targets
5 © 2011 IBM Corporation

6. Detection, Prevention and Action
 Detection: Issue warnings when a memory leak is detected
 Prevention is on by default and applies only to JRE triggered leaks.
 Action: Take proactive action to fix memory leaks.
 Actions have reasonable defaults and are configured on a case-by-case
 Pseudo code for clearing leaks
protected void com.ibm.ws.classloader.clearReferences()
{ if(ENABLE_CLEAR_REFERENCES_JDBC)
clearReferencesJdbc();
if(ENABLE_CLEAR_REFERENCES_THREADS)
clearReferencesThreads();
if(ENABLE_CLEAR_REFERENCES_THREADLOCALS)
clearReferencesThreadLocals();
if(ENABLE_CLEAR_REFERENCES_RMI_TARGETS)
clearReferencesRmiTargets();
if(ENABLE_CLEAR_REFERENCES_STATICS)
clearReferencesStaticFinal();
}
6 © 2011 IBM Corporation

7. Memory Leak Policy Configuration
Enabled via JVM custom properties on a per application server /JVM basis
Existing (v7 & up) PM39870: IMPROVED CLASSLOADER LEAK DETECTION.
com.ibm.ws.runtime.detectAppCLLeaks=true
NEW in v8.5 (traditional WAS only, Not in Liberty) Enabled by setting the following JVM
custom properties
com.ibm.ws.runtime.component.MemoryLeakConfig.detectAppCLLeaks
com.ibm.ws.runtime.component.MemoryLeakConfig.clearAppCLLeaks
com.ibm.ws.runtime.component.MemoryLeakConfig.preventJreMemoryLeaks
com.ibm.ws.runtime.component.MemoryLeakConfig.clearReferencesStatic
com.ibm.ws.runtime.component.MemoryLeakConfig.clearReferencesInterruptThreads
com.ibm.ws.runtime.component.MemoryLeakConfig.clearReferencesStopTimerThreads
com.ibm.ws.runtime.component.MemoryLeakConfig.clearReferencesThreadLocal
7 © 2011 IBM Corporation

8. Persisted Leak Policy configuration in server.xml
<jvmEntries xmi:id="JavaVirtualMachine_1183122130078" verboseModeClass="true"
verboseModeGarbageCollection="true" verboseModeJNI="false"
runHProf="false" hprofArguments="" debugMode="false" debugArgs="-
agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=7777"
genericJvmArguments="-agentlib:getClasses -Xquickstart -Xalwaysclassgc"
executableJarFileName="" disableJIT="false">
<systemProperties xmi:id="Property_1317048628648"
name="com.ibm.ws.runtime.component.MemoryLeakConfig.detectAppCLLeaks"
value="true"/>
<systemProperties xmi:id="Property_1318975518491"
name="com.ibm.ws.runtime.component.MemoryLeakConfig.clearAppCLLeaks"
value="true"/>
<systemProperties xmi:id="Property_1318955284241"
name="com.ibm.ws.runtime.component.MemoryLeakConfig.generateSystemDumps“
value="false"/>
<systemProperties xmi:id="Property_1319119976147"
name="com.ibm.ws.runtime.component.MemoryLeakConfig.generateHeapDumps“
value="true"/>
<systemProperties xmi:id="Property_1317048628649"
name="com.ibm.ws.runtime.component.MemoryLeakConfig.monitorSystemApps“
value="false"/>
</jvmEntries>
8 © 2011 IBM Corporation

9. Memory Leak Configuration mbean
All the attributes of the Type MemoryLeakConfig mbean
Attribute Type Access
JvmThreadGroupNames java.lang.String RW
FilterPrefixes java.lang.String RW
RenewThreadPoolNames java.lang.String RW
DetectAppCLLeaks boolean RW
ClearAppCLLeaks boolean RW
MonitorSystemApps boolean RW
NoDumps boolean RW
GenerateHeapDumps boolean RW
GenerateSystemDumps boolean RW
ClearReferencesStatic boolean RW
ClearReferencesInterruptThreads boolean RW
ClearReferencesStopTimerThreads boolean RW
ClearReferencesHttpClientKeepAliveThread boolean RW
ClearReferencesThreadLocal boolean RW
LeakSweeperDelay int RW
ThreadPoolRenewalDelayFactor int RW
PreventJreMemoryLeaks boolean RW
LeakConfiguration java.lang.String RO
9 © 2011 IBM Corporation

10. Memory Leak Runtime mbean
# Look at all the operations of the MemoryLeakAdmin mbean
wsadmin>$Help all $leakAdmin
Name:
WebSphere:cell=smitaNode03Cell,name=LeakAdmin,type=MemoryLeakAdmin,node=smitaNo
de03,process=server1
Description: Information on the management interface of the MBean
Class name: com.ibm.ws.runtime.component.MemoryLeakAdmin
Operation
java.lang.String findLeaks()
java.lang.String fixLeaks()
java.lang.String fixLeaks(java.lang.String)
10 © 2011 IBM Corporation

11. Leak Detection & Fixing Messages
 CWMML0015E: The web application [WasSwat#WasSwatWeb.war] created a
ThreadLocal with key of type [test.memleak.MyThreadLocal] (value
[test.memleak.MyThreadLocal@216c691]) and a value of type
[test.memleak.MyCounter] (value [test.memleak.MyCounter@21942ff])
but failed to remove it when the web application was stopped.
 CWMML0010E: The web application [LeakApp#leak.war] appears to have
started a thread named [Thread-73] but has failed to stop it.
 CWMML0011E: The web application [LeakApp#leak.war] appears to have
started a TimerThread named [leaked-thread] via the java.util.Timer
API but has failed to stop it.
 CWMML0024W: About to interrupt thread [leakingThread] which is
currently executing
 CWMML0026I: ClassLoader memory leak is fixed. Clearing leak
References succeeded for LeakApp#leak.war.
11 © 2011 IBM Corporation

12. Demonstration
12 © 2011 IBM Corporation

13. Summary
13 © 2011 IBM Corporation

14. Summary
 Improves resiliency and general robustness of the application server by fixing application
and WebSphere system memory leaks
 Simple configuration and runtime administration via Dynamic mbeans and JVM custom
properties
Feature exists in two parts
 First, it prevents memory leaks through a life-cycle listener that calls various parts of the
Java API. Its common that if the web application is the first code to call the Java APIs, the
web application class loader will be pinned in memory, causing leaks. The listener ensures
that WebSphere core runtime is the first to make a call, and therefore prevents the class
loader from being pinned in memory.
 Second, it handles detection by executing code when a web application is stopped,
undeployed or reloaded. It scans the code for standard causes of memory leaks, and where
it can, fixes the leaks. Implemented in the WebSphere Application Classloader, there are a
series of expandable, standard API calls and some reflection tricks that help this detection
feature do its job.
14 © 2011 IBM Corporation

15. References
 Memory leaks in Java Platform, Enterprise Edition applications
 http://thegreatgazoo.raleigh.ibm.com/infocenter85/topic/com.ibm.websphere.nd.multiplatfo
 Configuring the memory leak policy
 http://thegreatgazoo.raleigh.ibm.com/infocenter85/index.jsp?topic=/com.ibm.websphere.n
15 © 2011 IBM Corporation

16. Feedback
Your feedback is valuable
You can help improve the quality of IBM Education Assistant content to better meet your
needs by providing feedback.
 Did you find this module useful?
 Did it help you solve a problem or answer a question?
 Do you have suggestions for improvements?
Click to send email feedback:
mailto:iea@us.ibm.com?subject=Feedback_about_WAS_..........................................ppt
This module is also available in PDF format at: ../WAS_......................................................pdf
16 © 2011 IBM Corporation