Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Performance Tuning - Memory leaks, Thread deadlocks, JDK tools

How to identify memory leaks in java applications, thread contentions, using jdk in built tools to identify performance bottlenecks

  • Inicia sesión para ver los comentarios

Performance Tuning - Memory leaks, Thread deadlocks, JDK tools

  1. 1. Performance Tuning Memory Leaks, Thread Contention, JDK Tools for monitoring Haribabu Nandyal Performance Engineering Expert
  2. 2. Agenda  Memory Leaks in Java applications  Best Practices to avoid memory leaks  Thread Contentions  Thread dump and Analysis  JDK tools for performance diagnosis
  3. 3. Finding Memory Leaks in Java applications
  4. 4. Agenda  What is memory leak?  Implications of memory leak  How to locate memory leak?  Example Code for simple Memory Leak  Analyze the dump logs  Performance Checklists
  5. 5.  When an application dynamically allocates memory, and does not free that memory after using it, that program is said to be having memory leak.  Memory Leak is the gradual loss of system available memory when the application fails to return the memory that it has consumed for temporary needs. What is a memory leak?  Memory can neither be used by the system nor any other program.  Over a period of time; the system eventually runs out of memory  slow response time resulting in performance issues Implications of memory leak:
  6. 6. Memory Leak Also known as unintentional object retention (the object that is not needed anymore and should be unreachable is still reachable, hence cannot be collected)
  7. 7. Basic Types of Memory Leaks  A single large object that is unintentionally held in memory  Small objects that are accumulated over time Memory Leak Demo  Run the project in demo/MemoryManagement/MemoryLeak  Run Visual VM and open the project  Monitor the project Heap Snapshots Comparison  Take memory snapshots in different points of time(advised to run garbage collection before taking snapshot)  Compare the snapshots (e.g. in Visual VM)  Search for leaking objects within the objects that were identified as new
  8. 8.  Heap dump (non intrusive) • Generate a dump file as large as the heap memory • Freezes application while dump is generated  Profilers (intrusive) • Need to reconfigure VM settings • Attach a client profiler to introspect  Monitor memory usage • prstat, top, ps  Monitor GC activity • jstat, -verbose:gc, visualgc, jconsole Track down where the problem is
  9. 9.  Overview -> overview on the running application  Monitor -> view real-time, high-level data on the memory heap, thread activity, and the classes loaded in the Java Virtual Machine  Threads -> displays a timeline of current thread activity  Sampler -> provides a tool with performance and memory profiler  Profiler -> enables you to start and stop the profiling session of a local application To detect memory leak, use JvisualVM/Jconsole and monitor any application.
  10. 10. Classes Threads Monitor tab
  11. 11. Navigate to Profiler->Memory option Analyze the memory consumption of the live bytes against Class Names.
  12. 12. If you are suspecting Memory Leak, collect Heap Dump
  13. 13. Heap Summary shows the different generations (areas of the heap), with the size, the amount used, and the address range. The address range is especially useful if you are also examining the process with tools such as pmap. Heap def new generation total 1152K, used 435K [0x22960000, 0x22a90000, 0x22e40000) eden space 1088K, 40% used [0x22960000, 0x229ccd40, 0x22a70000) from space 64K, 0% used [0x22a70000, 0x22a70000, 0x22a80000) to space 64K, 0% used [0x22a80000, 0x22a80000, 0x22a90000) tenured generation total 13728K, used 6971K [0x22e40000, 0x23ba8000, 0x269600 00) the space 13728K, 50% used [0x22e40000, 0x2350ecb0, 0x2350ee00, 0x23ba8000) compacting perm gen total 12288K, used 1417K [0x26960000, 0x27560000, 0x2a9600 00) the space 12288K, 11% used [0x26960000, 0x26ac24f8, 0x26ac2600, 0x27560000) ro space 8192K, 62% used [0x2a960000, 0x2ae5ba98, 0x2ae5bc00, 0x2b160000) rw space 12288K, 52% used [0x2b160000, 0x2b79e410, 0x2b79e600, 0x2bd60000) If the Java VM flag -XX:+PrintClassHistogram is set, then the Ctrl-Break handler will produce a heap histogram. Heap Summary
  14. 14. Navigate to Classes -> select the suspected class -> Show in Instances View Instances tab -> shows “out of memory”
  15. 15. Heap and lock contention issues: • Avoid creating objects where primitives could be used (autoboxing is not for free) • Avoid String concatenations, use StringBuilder with append() • Avoid StringBuilder overallocation, set initial size to reduce resizing • Avoid use of synchronized keyword, look for a java.util.concurrent solution • Use java.util.concurrent API instead of "synchronized" Collections • Don't use exceptions for flow control (Exception objects allocation), use flow control constructs (if/else, return, switch). • Don't generate stack traces, they're expensive operations. • Reduce number of system calls:  Avoid accessing disk: use large buffers or direct buffering  Avoid accessing underlying OS  Avoid processing bytes and characters individually Best Practices
  16. 16. Tools and Tips - Memory 1. Java heap space Causes • Low RAM memory • Small heap • Memory leak Fix • More physical memory • Make heap larger • Fix memory leaks 2. PermGen space Causes • Low memory for permanent generation area • Classes on permanent generation are prevented to be collected  Classloader leak Fix • Make permanent generation area, larger • Disable hot deployment or class reloading – for classloader leak
  17. 17. Tools and Tips - Memory 3. Out of swap space Cause • Often is caused by JNI code • No memory available for operating system Fix • Reserve more memory to operating system • Lower the heap size 4. Unable to create new native thread Cause • Big stack size (-Xss) • Small memory available • Poor thread management Fix • Make stack size smaller • Reserve more memory to operating system • Lower heap size • Better thread management  Java 5 thread pool
  18. 18. Tools and Tips - Memory 5. Requested array size exceeds VM limit Cause • Attempt to allocate an array that is larger than the heap size Fix • Lower the array size • Make heap larger 6. GC overhead limit exceeded Cause • Too much time spent on GC (98%) and little heap (2%) to be free Fix • Similar to “java heap space” • Tune GC behavior
  19. 19. Threads
  20. 20. Agenda  Process and Threads  Thread States  Thread Contentions  Thread dump and Analysis
  21. 21. What is a Process?  A program in execution  Provides the resources needed to execute a program  One process can't access or modify another process What is a Thread?  A basic unit of CPU utilization  Lightweight process (LWP)  Multiple threads are executed within a process  Multiple threads shares memory and resources of the process in which it is created. Threads
  22. 22. Threads in Java Java language includes a powerful threading facility built into the language. We can use the threading facility to:  Increases the responsiveness of GUI applications  Take advantage of multiprocessor systems  Simplify program logic when there are multiple independent entities  Thread is cheap  No need to worry about memory model in different environments
  23. 23. Thread States
  24. 24. Thread T1 – has a lock on object O1 Thread T2 – has a lock on object O2 T1 is waiting to access O2, while T2 is waiting to access O1. Deadlock A situation in which a thread has got an object, while waiting for another object which is locked by another thread, which in turn is waiting for the object locked by the first thread.
  25. 25. synchronized(object) { // do something... } Race Condition and Synchronization ● Race conditions occur when multiple, asynchronously executing threads access the same object (called a shared resource) returning unexpected results. ● When two or more threads are trying to access the shared object, the process of allowing only one thread access that object is called Synchronization. ● Only one thread can execute the block of code protected by the same lock at the same time.
  26. 26. Lock contention • In the java application java thread waits for lock in synchronized method • Occurs in multi threaded program • Reduce frequency of synchronized method • Synchronized block must be implemented to end as soon as possible public void synchronized methodA(Object param) { //do something while(1){} } Thread1 Thread3 Thread2 Thread4 Sychronized MethodA Lock contention : Thread lock example
  27. 27. "ExecuteThread: '12' for queue: 'weblogic.kernel.Default'" daemon prio=10 tid=0x0055ae20 nid=23 lwp_id=3722788 waiting for monitor entry [0x2fb6e000..0x2fb6d530] : at java.lang.ClassLoader.loadClass( : at org.apache.xerces.jaxp.SAXParserFactoryImpl.newSAXParser(Unknown Source) at org.apache.axis.utils.XMLUtils.getSAXParser( - locked <0x329fcf50> (a java.lang.Class) "ExecuteThread: '13' for queue: 'weblogic.kernel.Default'" daemon prio=10 tid=0x0055bde0 nid=24 lwp_id=3722789 waiting for monitor entry [0x2faec000..0x2faec530] at org.apache.axis.utils.XMLUtils.getSAXParser( - waiting to lock <0x329fcf50> (a java.lang.Class) : "ExecuteThread: '15' for queue: 'weblogic.kernel.Default'" daemon prio=10 tid=0x0061dc20 nid=26 lwp_id=3722791 waiting for monitor entry [0x2f9ea000..0x2f9ea530] at org.apache.axis.utils.XMLUtils.releaseSAXParser( - waiting to lock <0x329fcf50> (a java.lang.Class)at Lock contention : Thread dump example
  28. 28. Dead lock • CWC “Circular Waiting Condition” • Commonly it makes WAS hang up • Remove CWC by changing lock direction • Some JVMs detect dead lock automatically DeadLock contention sychronized methodA(){ call methodC(); } sychronized methodB(){ call methodA(); } sychronized methodC(){ call methodB(); } Thread1 Sychronized MethodA Thread2 Sychronized MethodB Thread3 Sychronized MethodC
  29. 29. FOUND A JAVA LEVEL DEADLOCK: ---------------------------- "ExecuteThread: '12' for queue: 'default'": waiting to lock monitor 0xf96e0 (object 0xbd2e1a20, a weblogic.servlet.jsp.JspStub), which is locked by "ExecuteThread: '5' for queue: 'default'" "ExecuteThread: '5' for queue: 'default'": waiting to lock monitor 0xf8c60 (object 0xbd9dc460, a weblogic.servlet.internal.WebAppServletContext), which is locked by "ExecuteThread: '12' for queue: 'default'" JAVA STACK INFORMATION FOR THREADS LISTED ABOVE: ------------------------------------------------ Java Stack for "ExecuteThread: '12' for queue: 'default'": ========== at weblogic.servlet.internal.ServletStubImpl.destroyServlet( - waiting to lock <bd2e1a20> (a weblogic.servlet.jsp.JspStub) at weblogic.servlet.internal.WebAppServletContext.removeServletStub( at weblogic.servlet.jsp.JspStub.prepareServlet( : Java Stack for "ExecuteThread: '5' for queue: 'default'": ========== at weblogic.servlet.internal.WebAppServletContext.removeServletStub( at weblogic.servlet.jsp.JspStub.prepareServlet( at weblogic.servlet.jsp.JspStub.prepareServlet( - locked <bd2e1a20> (a weblogic.servlet.jsp.JspStub) at weblogic.servlet.internal.ServletStubImpl.getServlet( : Found 1 deadlock.========================================================
  30. 30. Thread Contention
  31. 31. What is "Thread dump“ ? Thread dump" basically gives you information on what each of the thread in the VM is doing at any given point of time. Java's thread dumps are vital for server debugging. It helps us to identify  Performance Related issues  Deadlock (System Locks)  Timeout Issues  Systems stops processing Traffic Thread dumps
  32. 32. The thread dump consists of the thread stack, including thread state, for all Java threads in the virtual machine. The header line contains the following information about each thread: Thread name Indication if the thread is a daemon thread Thread priority ( prio) Thread ID ( tid), which is the address of a thread structure in memory ID of the native thread ( nid) Thread state: Indicates what the thread was doing at the time of the thread dump Address range,: Estimate of the valid stack region for the thread "DestroyJavaVM" prio=10 tid=0x00030400 nid=0x2 waiting on condition [0x00000000..0xfe77fbf0] java.lang.Thread.State: RUNNABLE "Thread2" prio=10 tid=0x000d7c00 nid=0xb waiting for monitor entry [0xf36ff000..0xf36ff8c0] java.lang.Thread.State: BLOCKED (on object monitor) at Deadlock$ - waiting to lock <0xf819a938> (a java.lang.String) - locked <0xf819a970> (a java.lang.String) "Low Memory Detector" daemon prio=10 tid=0x000c7800 nid=0x8 runnable [0x00000000..0x00000000] java.lang.Thread.State: RUNNABLE Thread Dumps
  33. 33. Thread Dumps
  34. 34. Thread Contention Issues
  35. 35. public class TestThread { public static Object Lock1 = new Object(); public static Object Lock2 = new Object(); public static void main(String args[]) { ThreadDemo1 T1 = new ThreadDemo1(); ThreadDemo2 T2 = new ThreadDemo2(); T1.start(); T2.start(); } private static class ThreadDemo1 extends Thread { public void run() { synchronized (Lock1) { System.out.println("Thread 1: Holding lock 1..."); try { Thread.sleep(10); } catch (InterruptedException e) {} } } } private static class ThreadDemo2 extends Thread { public void run() { synchronized (Lock2) { System.out.println("Thread 2: Holding lock 2..."); try { Thread.sleep(10); } catch (InterruptedException e) {} System.out.println("Thread 2: Waiting for lock 1..."); synchronized (Lock1) { System.out.println("Thread 2: Holding lock 1 & 2..."); } } } } } Thread DeadLock: example code
  36. 36. Monitor the threads while application is running and if you notice few threads in Monitor state, take a thread dump. Thread DeadLock
  37. 37. Thread DeadLock
  38. 38. JDK tools for performance diagnosis • VisualVM • Btrace • Visualgc Plugin • jhat • jmap • jconsole • jstat • jps • jstack • Jstatd Daemon
  39. 39. Java Standard Edition 7 Documentation
  40. 40. Java VisualVM (swiss army knife for JVM process monitoring and tracing) Java VisualVM is an utility that can generate and analyze heap dumps, track down memory leaks, perform and monitor garbage collection, and perform lightweight memory and CPU profiling. There are host of standard Java VisualVM plug-ins available which can expand its existing functionality. E.g: Jconsole, VisualGC etc For example, most of the functionality of the JConsole tool is available via the MBeans tab and the JConsole plug-in wrapper tab
  41. 41. Java VisualVM features: • View a list of local and remote Java applications. • View application configuration and runtime environment data: PID, host, main class, arguments passed to the process, JVM version, JDK home, JVM flags, JVM arguments, system properties etc. • Monitor application memory consumption, running threads, and loaded classes. • Trigger Garbage Collection. • Create a heap dump. Multiple views: summary, by class, by instance and also save the heap dump. • Profile application performance or analyze memory allocation (for local applications only) and save the profiling data. • Create thread dump (stack trace of the application's active threads). • Analyze applications offline, by taking application snapshots. • Get additional plug-ins contributed by the community.
  42. 42. Display local and remote Java applications. VisualVM automatically detects and lists locally and remotely running Java applications (jstatd must be running on the remote host). This way you can easily see what Java applications are running on your system or check if a remote J2EE server process is alive. Display application configuration and runtime environment For each application VisualVM shows basic runtime information: PID, main class, arguments passed to java process, JVM version, JDK home, JVM flags and arguments and system properties. Monitor application performance and memory consumption. VisualVM monitors application CPU usage, GC activity, heap and permanent generation memory, number of loaded classes and running threads. You can easily detect suspicious memory consumption and take an action - invoke garbage collection in the application or take a heap dump and browse the contents of application heap. Monitor application threads. All threads running in a Java process are displayed in a timeline and table. You can track thread activity and uncover inefficient patterns. Profile application performance or analyze memory allocation. VisualVM includes a built-in application profiler which can visualize where most of the time is being spent or which objects consume most of the memory. VisualVM features
  43. 43. Take and display thread dumps. Taking and displaying a thread dump is as easy as clicking a mouse button. Moreover, simultaneous thread dumps of multiple applications can be taken at once to start uncovering distributed deadlocks. Take and browse heap dumps. When you need to browse contents of application memory or uncover a memory leak in your application, you'll find the built-in HeapWalker tool really handy. It can read files written in hprof format and is also able to browse heap dumps created by the JVM on an OutOfMemoryException. Analyze core dumps. When a Java process crashes, a core dump can be generated by the JVM containing important information about application state at the time of the crash. VisualVM is able to display application configuration and runtime environment and to extract thread and heap dumps from the core dump. Analyze applications offline. VisualVM is able to save application configuration and runtime environment together with all taken thread dumps, heap dumps and profiler snapshots into a single application snapshot which can be later processed offline. This is especially useful for bug reports where users can provide a single file containing all the necessary information to identify runtime environment and application state. IDE Integration VisualVM can be integrated with your IDE (Eclipse and NetBeans IDE) to monitor and analyze the code iteratively during development. VisualVM features
  44. 44. JVisualVM
  45. 45. JVisualVM
  46. 46. JVisualVM
  47. 47. JVisualVM
  48. 48. JConsole Utility • JConsole is a monitoring tool that uses built-in JMX instrumentation in the JVM to provide information on performance and resource consumption of the running applications. • Useful in high-level diagnosis on problems such as memory leaks, excessive class loading, and running threads. It can also be used for tuning and heap sizing.
  49. 49. Jconsole components
  50. 50. • Overview This pane displays graphs of heap memory usage, number of threads, number of classes, and CPU usage. • Memory For a selected memory area (heap, non-heap, various memory pools) provides  Graph of memory usage over time  Current memory size  Amount of committed memory  Maximum memory size.  Garbage collector information, including the number of collections performed, and the total time spent performing GC.  Graph showing percentage of heap and non-heap memory currently used.  Link to request to perform GC. • Threads  Details about Graph of thread usage over time  Live threads - Current number of live threads  Peak - Highest number of live threads since the Java VM started  For a selected thread, the name, state, and stack trace.  Button for Deadlock Detection Jconsole components
  51. 51. • Classes  Graph in number of classes loaded over time.  Number of classes currently loaded into memory.  Total number of classes loaded into memory since starting of Java VM, including those subsequently unloaded.  Total number of classes unloaded from memory since start of Java VM. • VM Summary  General information, such as the JConsole connection data, uptime for Java VM, CPU time consumed by Java VM, complier name and total compile time etc.  Thread and class summary information.  Memory and garbage collection information, including number of objects pending finalization etc.  Information about the OS, including physical characteristics, amount of virtual memory for the running process, swap space etc. • MBeans This pane displays a tree structure showing all platform and application MBeans that are registered in the connected JMX agent. When you select an MBean in the tree, its attributes, operations, notifications, and other information are displayed. Jconsole components
  52. 52. JMap utility can obtain memory map information, including a heap histogram, from a Java process, a core file, or a remote debug server. The jmap command-line utility prints memory related statistics for a running VM or core file. If jmap is used with a process or core file without any command-line options, then it prints the list of shared objects loaded. For more specific information, we can use the options -heap, -histo, or -permstat. In addition, we can use dump:format=b,file= filename option, which causes jmap to dump the Java heap in binary HPROF format to a specified file. This file can then be analyzed with the jhat tool. Jmap Utility The jstack command-line utility attaches to the specified process or core file and prints the stack traces of all threads that are attached to the virtual machine, including Java threads and VM internal threads, and optionally native stack frames. The utility also performs deadlock detection. The stack trace of all threads can be useful in diagnosing a number of issues such as deadlocks or hangs. Jstack Utility
  53. 53. The jstat utility uses the built-in instrumentation in the HotSpot VM to provide information on performance and resource consumption of running applications. The tool can be used when diagnosing performance issues, and in particular issues related to heap sizing and garbage collection. Jstat Utility Jstatd Daemon Visualgc Plugin The jstatd daemon is a Remote Method Invocation (RMI) server application that monitors the creation and termination of instrumented Java HotSpot VMs and provides an interface to allow remote monitoring tools to attach to Java VMs running on the local host. The visualgc tool attaches to an instrumented HotSpot JVM and collects and graphically displays garbage collection, class loader, and HotSpot compiler performance data. The target JVM is identified by its virtual machine identifier, or vmid.
  54. 54. jhat • Java Heap Analysis Tool • Parses a java heap dump and launches a web server • Basic web interface for browsing the heap dump • Useful for analyzing memory leaks jhat Features • View all class instances • View class classloader • View all references to an object • View reference chains from Rootset • Use OQL (Object Query Language) to query heap dumps • SQL-like query language to query Java heap • Based on JavaScript expression language • Built-in objects and functions • OQL help is available from the jhat OQL screen OQL