Troubleshooting Native Memory Leaks in Java Applications

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Troubleshooting Native Memory
Leaks in Java Applications
Poonam Parhar
Consulting Member of Technical Staff
Java Platform Group
October, 2018

Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, timing, and pricing of any
features or functionality described for Oracle’s products may change and remains at the
sole discretion of Oracle Corporation.
Confidential – Oracle Internal/Restricted/Highly Restricted

About me
• Poonam Parhar
• JVM Sustaining Engineer at Oracle
• Co-author of the book ‘Java Performance Companion’
• JavaOne RockStar
• Speaker at Java Conferences
• Published articles
• Blog: https://blogs.oracle.com/poonam/
3

Agenda
• Layout of Memory From the Perspective of the JVM
• Detect Native Memory Leaks
• OutOfMemoryError: Native Memory
• Common Issues and their Solutions
• Approaches/Tools to Troubleshoot a Native Memory Leak

Virtual and Physical Memory
• Address Space or Virtual Memory determined by address size e.g. 32-bit or
64-bit
• Physical memory (RAM)
Image source: https://en.wikipedia.org

Memory allocated by a Java application
Java Heap
- Memory allocated for JVM’s internal
functioning such as Java Heap memory
management
- Memory used by loaded jars and native
libraries
- Thread stacks and thread-local storage
- …
Memory
Storage of classes and metadata, including
dynamically generated classes through reflection
(Metaspace)
JIT bytecode compilation and storage of compiled
code (codecache)
- JNI/JVMTI code allocating memory
- NIO allocations
- Direct Bytebuffers

Native leaks are hard to diagnose
• Native allocations can come from several different places
• Could be due to JNI allocations, JIT compiler allocations or creation of
threads, …

Detect Native Memory Leaks

Demo
A simple Java program with a native memory leak

Monitor Native Memory
• ps, top, pmap on Unix systems
– watch -n 1 ps v <pid>
– top –p <pid>
– Examine pmap output
• PerfMon or VMMap on Windows

bash-3.2$ ps v 27890
PID STAT TIME SL RE PAGEIN VSZ RSS LIM TSIZ %CPU %MEM COMMAND
27890 S+ 0:01.68 0 0 0 10068904 63652 - 0 3.4 0.4 java -Djava.library.pat
...

poonam@poonam-VirtualBox:/media/sf_shared/native_leak_tests$ diff pmap.1 pmap.9
12a13,14
> 00007f4060000000 32276K rw--- [ anon ]
> 00007f4061f85000 33260K ----- [ anon ]
56,57c58
< 00007f40a4000000 18952K rw--- [ anon ]
< 00007f40a5282000 46584K ----- [ anon ]
---
> 00007f40a4000000 65536K rw--- [ anon ]
146c147
< total 3222140K
---
> total 3287676K
poonam@poonam-VirtualBox:/media/sf_shared/native_leak_tests$ diff pmap.2 pmap.9
12a13,14
> 00007f4060000000 32276K rw--- [ anon ]
> 00007f4061f85000 33260K ----- [ anon ]
56,57c58
< 00007f40a4000000 25600K rw--- [ anon ]
< 00007f40a5900000 39936K ----- [ anon ]
---
> 00007f40a4000000 65536K rw--- [ anon ]
146c147
< total 3222140K
---
> total 3287676K

OutOfMemoryError: Native Memory
14

# A fatal error has been detected by the Java
Runtime Environment:
#
# java.lang.OutOfMemoryError : unable to
create new native Thread
# A fatal error has been detected by the Java
Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 32756
bytes for ChunkPool::allocate. Out of swap
space?
#
# Internal Error (allocation.cpp:166),
pid=2290, tid=27 # Error: ChunkPool::allocate
15
Native OutOfMemoryError

Native OutOfMemoryError
• JVM is not able to allocate from native memory
– Not managed by the JVM
• This process or other processes on the system are eating up native memory
• Can make more room for native allocations by:
– Reducing the Java Heap, PermGen/Metaspace, number of threads and their stack
sizes etc.
– Reducing the number of processes running on the system
• If the above don’t help, we might be facing a native memory leak
– Example: JNI code allocating native buffers
16

Common Issues: Native
OutOfMemoryError
17

Native Heap OutOfMemoryError with 64-bit JVM
• Running with 32-bit JVM puts a maximum limit of 4GB on the process size
– So more likely you’ll run out of native memory with 32-bit Java processes
• Running with a 64-bit JVM gets us access to unlimited memory, so we would
expect never to run out of native memory
• However, we might see OutOfMemoryErrors occurring in a 64-bit JVM too
• CompressedOops feature implementation determines where the Java heap
should be placed in the address space
Ø Below 4GB address boundary – no scaling, no offset addition
ØBelow 32GB address boundary – no offset addition
• The position of the Java heap can put a cap on the maximum capacity of the
native heap.
18

Memory Map
0000000100000000 8K r-x-- /sw/.es-base/sparc/pkg/jdk-1.7.0_60/bin/sparcv9/java
0000000100100000 8K rwx-- /sw/.es-base/sparc/pkg/jdk-1.7.0_60/bin/sparcv9/java
0000000100102000 56K rwx-- [ heap ]
0000000100110000 2624K rwx-- [ heap ] <--- native heap
00000001FB000000 24576K rw--- [ anon ] <--- Java Heap starts here
0000000200000000 1396736K rw--- [ anon ]
0000000600000000 700416K rw--- [ anon ]
19

Solution for OutOfMemoryError with 64-bit JVM
• Can be resolved by using option -XX:HeapBaseMinAddress=n to specify
the address the Java heap should be based at
• Setting it to a higher address would leave more room for native heap
20

JAXB Issues
• JAXB internally uses Inflater/Deflater to compress/uncompress files
– Inflater/Deflater use native memory to hold their data
– Depend on Finalizer to deallocate the java objects and the associated native memory
data
– Delay in running Finalizer can exhaust native memory
• JAXBContext.newInstance() called for every new request
– Classes from the context get reloaded again
– Increases the native memory usage
• Environment upgrade fails to upgrade all the JAXB jar files
– Classes linking errors leading to re-loading of classes
21

OutOfMemoryError: Direct buffer memory
• ByteBuffer.allocateDirect(SIZE_OF_BUFFER)
• DirectByteBuffers are garbage collected by using a phantom reference and
a reference queue
• Maximum direct memory is unbounded, can be limited by using the JVM
option -XX:MaxDirectMemorySize=n
• We can explicitly call: ((DirectBuffer)buffer).cleaner().clean();
• Make GCs more frequent allowing DirectByteBuffers to be GC’ed
22

NIO ByteBuffers
• Java NIO APIs use ByteBuffers as the source and destination of I/O calls
• Java Heap ByteBuffers and Native Heap ByteBuffers
• Java Heap ByteBuffer for I/O use a temporary direct ByteBuffer per thread
• If large native ByteBuffers from multiple threads are used for I/O calls
ØNative Memory Exhaustion
• -Djdk.nio.maxCachedBufferSize=m (1.8.0_102, 9 and above)
23

Native Memory: Diagnostic Data
24

Native Memory: Diagnostic Data
• Native memory leaks in the JVM
• Native Memory Tracker (NMT) output
• Native Memory Leaks outside the JVM
– Tools like jemalloc, valgrind, libumem, dbx, purify, UMDH
• Process map output with tools like pmap
• Core file
25

Native Memory Tracker
• Native Memory Tracker (NMT) that can be used to track native memory
that is used internally by the JVM
• It cannot track memory allocated outside the JVM or by native libraries
• https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/to
oldescr007.html
26

NMT
• Start the process with NMT enabled using NativeMemoryTracking
• The output level can be set to a ‘summary’ or ‘detail’ level:
• -XX:NativeMemoryTracking=summary
• -XX:NativeMemoryTracking=detail
• Use jcmd to get the native memory usage details:
• jcmd <pid> VM.native_memory
• jcmd <pid> VM.native_memory baseline
• jcmd <pid> VM.native_memory detail.diff/summary.diff
27

Demo
Classes space leak

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 30
• bash-3.2$ jcmd 39057 VM.native_memory summary.diff
• 39057:
• Native Memory Tracking:
• Total: reserved=5761678KB +52943KB, committed=472350KB +104143KB
• - Java Heap (reserved=4194304KB, committed=163328KB +7680KB)
• (mmap: reserved=4194304KB, committed=163328KB +7680KB)
•
• - Class (reserved=1118333KB +47579KB, committed=117949KB +89963KB)
• (classes #68532 +58527)
• (malloc=8317KB +2523KB #5461 +3371)
• (mmap: reserved=1110016KB +45056KB, committed=109632KB +87440KB)
•
• - Thread (reserved=21594KB -2057KB, committed=21594KB -2057KB)
• (thread #22 -2)
• (stack: reserved=21504KB -2048KB, committed=21504KB -2048KB)
• (malloc=65KB -6KB #111 -10)
• (arena=25KB -2 #42 -4)
•
• - Code (reserved=250400KB +244KB, committed=5612KB +1348KB)
• (malloc=800KB +244KB #1498 +234)
•
• - GC (reserved=159039KB +18KB, committed=145859KB +50KB)
• (malloc=5795KB +18KB #856 +590)
•
• - Compiler (reserved=153KB, committed=153KB)
• (malloc=22KB #72 -2)
• (arena=131KB #3)
•
• - Internal (reserved=13537KB +6949KB, committed=13537KB +6949KB)
• (malloc=13505KB +6949KB #70630 +59119)
• (mmap: reserved=32KB, committed=32KB)
•
• - Symbol (reserved=2715KB +9KB, committed=2715KB +9KB)
• (malloc=1461KB +9KB #702 +29)
• (arena=1255KB #1)
•
• - Native Memory Tracking (reserved=1416KB +1031KB, committed=1416KB +1031KB)
• (malloc=140KB +34KB #2197 +518)
• (tracking overhead=1275KB +997KB)
•
• - Arena Chunk (reserved=186KB -832KB, committed=186KB -832KB)
• (malloc=186KB -832KB)

Native Memory Leaks Outside the JVM
• For the native memory leaks stemming from outside the JVM, we need to
rely on the native memory leak tools for their detection and
troubleshooting
• Native Memory Leak Detection Tools
– jemalloc
– valgrind
– libumem
– dbx
– Purify
– User-Mode Dump Heap (UMDH)
31

jemalloc
• Memory allocator and profiler
• FreeBSD
– Scalability in terms of multithreading by avoiding lock contentions
– Heap profiling capabilities
• Very little performance impact (heavy malloc use, ~2%)
– Can be used in production systems
• Download it from https://github.com/jemalloc/jemalloc
• ./configure --prefix=/usr/local --enable-prof
• make

Demo
Memory Leak with Deflater

Run application with jemalloc
export JEMALLOC_PATH=/media/sf_shared/jemalloc-5.1.0
export MALLOC_CONF=prof:true,lg_prof_sample:0,prof_final:true
Or
export MALLOC_CONF=prof:true,lg_prof_interval:30,lg_prof_sample:21
LD_PRELOAD=${JEMALLOC_PATH}/lib/libjemalloc.so.2 java -Xmx1024m -Xms1024m -
XX:MaxNewSize=20m TestDeflater

jeprof --show_bytes `which java` jeprof.23299.0.f.heap
(jeprof) top
Total: 341324080 B
325433344 95.3% 95.3% 325433344 95.3% deflateInit2_
15292392 4.5% 99.8% 15292392 4.5% os::malloc@926750
368704 0.1% 99.9% 368704 0.1% readCEN
116448 0.0% 100.0% 325549792 95.4% Java_java_util_zip_Deflater_init
81920 0.0% 100.0% 81920 0.0% std::__once_callable
12016 0.0% 100.0% 12016 0.0% _dl_new_object
3840 0.0% 100.0% 3840 0.0% allocate_dtv
3616 0.0% 100.0% 3616 0.0% _nl_intern_locale_data
2048 0.0% 100.0% 2208 0.0% __tzfile_read
1984 0.0% 100.0% 1984 0.0% _dl_check_map_versions

• jeprof --show_bytes --gif `which java` jeprof.23299.0.f.heap > leak.gif

Valgrind
• valgrind --leak-check=yes --log-file=valgrind_full.log -v java <options and
class file>
• --leak-check=summary | yes | full
• May not be effective for Java applications
– Reports allocations done by the JVM for its internal functioning
• Impacts application performance significantly
– 20 to 30 times slower
• Can be useful in tracking down JVM memory leaks but we have NMT for
that

Windows: User Mode Dump Heap
(UMDH) tool

UMDH
• Install Debugging Tools for Windows
• Use ’gflags’ to set the ‘Create user mode stack trace database flag’ for the
process
– gflags /i java.exe +ust
• https://docs.microsoft.com/en-us/windows-
hardware/drivers/debugger/umdh
• umdh -p:<pid> -f:first.log , umdh -p:<pid> -f:second.log, …
• umdh -d -v -l first.log second.log > report.log

pmap and core file

$ diff pmap.15767.1 pmap.15767.3
69,70c69,70
< 00007f6d68000000 17036K rw--- [ anon ]
< 00007f6d690a3000 48500K ----- [ anon ]
---
> 00007f6d68000000 63816K rw--- [ anon ]
> 00007f6d6be52000 1720K ----- [ anon ]
-------------------------------------------------------------
$ gdb `which java` core.15767
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
...
(gdb) x/100s 0x00007f6d690a3000
0x7f6d690a3000: "mory Leak "
0x7f6d690a300c: "Alert: JNI Memory Leak "
0x7f6d690a3025: "Alert: JNI Memory Leak "
0x7f6d690a303e: "Alert: JNI Memory Leak "

#include "JNINativeLeak.h"
#include <stdlib.h>
#include <memory.h>
JNIEXPORT void JNICALL Java_JNINativeLeak_allocateMemory
(JNIEnv *env, jobject obj, jint size) {
char* bytes = (char*) malloc(size);
printf("Allocated %d bytes at %p n", size, (void*)bytes);
for (int i=0; i<40; i++) {
strcpy(bytes+i*25, "Alert: JNI Memory Leak ");
}
}

Summary
• Identify native memory leak
• Monitor using j* tools and native tools such as ps, top, pmap
• Rule out common issues
• Leaks from the JVM
– NMT
• Leaks outside the JVM
– Native tools: jemalloc, valgrind, libumem, purify, UMDH, pmap and core file
• https://github.com/poonamparhar/native_leaks

Troubleshooting Native Memory Leaks in Java Applications

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Troubleshooting Native Memory Leaks in Java Applications

Similar a Troubleshooting Native Memory Leaks in Java Applications (20)

Más de Poonam Bajaj Parhar

Más de Poonam Bajaj Parhar (6)

Último

Último (20)

Troubleshooting Native Memory Leaks in Java Applications