java performance tuning session: full tutorial
This post is a summary of the Java performance tuning session given at on April 27th 2010.
Maybe we should start with the basic fact the when allocation memory for the JVM and in order to avoid OutOfMemoryError you should take into consideration that the total size of the memory allocated is composed of: 1-The heap (controlled by the Xms and Xmx parameters) 2-The permanent generation (controlled by the –XX:MaxPermSize parameter)
3-The memory allotted for each thread (controlled by the –Xss parameter). Of course, more memory is allocated in case you are using JNI and memory is also allocated for the native threads on the specific OS you are running your JVM on.
The important point being total memory= Heap+ PermGen + xss * (number of threads).
Methodology to find the causes of a memory and thread leaks:
1-Once a suspicion arises with respect to a memory leak either on the heap, the permanent generation or on the stack, you should generate a memory heap dump and a thread dump.
Option 1-The JVM should be configured with the following JVM argumentXX:+HeapDumpOnOutOfMemoryError which will trigger a heap dump on the event of an out of memory error.
Option 2-On the command line (\java1.6\bin\) invoke jps ( http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jps.html ) to get the PID of the java process you intend monitoring:
Once you have the PID (3272) use it to generate a Heap dump. On the command line (\java1.6\bin\) invoke jmap (http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jmap.html):
jmap -heap:format=b <pid>
Note: generating a full class histogram and a heap dump can last a substantial amount of time, for the duration of which the execution of code running inside the JVM is delayed.
Option 3-using VisualVM (https://visualvm.dev.java.net/) attach to the JVM process and click the “Heap dump” button to generate a heap dump on the fly:
Option a: once you have the PID of the process (see above using jps), you can use jstack (http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jstack.html) to generate a thread dump:
jstack [-l] <pid>
Note that I “piped” the result into a file for postmortem analysis. Also note that in case of a deadlock the following text will be printed out: “Found <n> Java-level deadlock: “ where <n> is the number of deadlock detected.
Option b: using VisualVM (see below) attach to the JVM process and click the “Thread dump” button to generate a thread dump on the fly:
2-Now that we have a thread and a memory dump we can analyze them offline. We shall see that later. Now let’s see how we can monitor the JVM in real time. One tool that was already mentioned in VisualVM. It now ships as part of JDK 1.6 and can be invoked from the JDK’s bin directory. The second tool is jstat (http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jstat.html).
Jstat is an excellent command line tool for getting the summary of garbage collection statistics and for detecting issues such as wrong ratio between the “Eden” and “Tenured” sections of the JVM heap memory. (see http://www.coderfriendly.com/wp-content/uploads/2009/06/javagc.png for a memory map layout). It can be invoked as follows:
jstat -gcutil -t <pid> 1s 30
To get a better understanding of what is being printed out I suggest you read the documentation. Usually you would trace down the changes through, for instance, the S0 memory section (or any other section for that matter) and if you see unusual behavior (such as too frequent GC) then you can change the JVM settings and re-test.
VisualVM’s UI has several different tabs for watching in real-time several JVM parameters. For instance, there is the “Monitor” tab which provides an insight to the size of the Heap and PermGen sizes, likewise there is the “Threads” tab which can be utilized to inspect each and every thread running under the JVM. The last tab is the “Profiler” tab used to profile CPU and memory in real-time. I would use a commercial tool for that (e.g. profiling) such as YourKit or Jprofiler rather than using VisaulVM for real-time profiling since the aforementioned tools can provide more detailed information with respect to Java methods CPU consumption.
3-Analysing thread dumps: a very useful tool for thread dump analysis is no other the TDA (thread dump analyzer http://www.ohloh.net/p/tda ) . Once you have a thread dump ready, you can invoke it for offline analysis:
4-Performing heap dump analysis. The best open source tool for heap dump analysis is no other than MAT (memory analysis tool). One you install and invoke it you can open a heap dump (see a complete example here (A Leaking Class Loader demo http://dev.eclipse.org/blogs/memoryanalyzer/2008/05/17/the-unknown-generation-perm/ ).
I will provide two examples of real heap dumps which will provide an insight into the types of memory links that you are likely to encounter.
On a production server the JVM crashed and generated a Heap dump. VisualVM was quick to detect the largest objects and it was easy to see that “a” and”b” objects are very large. What was puzzling was, how come we have such a large object in the JVM? And how come two objects seem very very similar in size.
Further analysis revealed that the cause of the leak was a StringBuilder was grew and grew over a period of 3 months until it reached 565MB, so this answers the first question, it was a programmer who erroneously appended strings to the StringBuilder:
The second issue was: why are there two objects of almost the exact same size? Again the answer was a programmer who erroneously held to object twice cloned.
As can be seen, JBOSS message cache was accumulating messages without purging them which resulted in a hooping 1.2GB cache.