Configure WSO2 products for Continuous JFR Recordings

Production systems are daily exposing to huge traffic and sometimes these production systems are not serving requests as expected. In those kind of scenarios, We need a way to figure out what went wrong in the past few hours.

With Java Flight Recorder, If we have already enabled it, It is very easy to figure out what went wrong with information on

• Memory Usage
• CPU Usage 
• Thread Usage 
• Etc.

In order to configure a WSO2 server to continuously record this information, You can change the startup script of wso2 server as follows.

For that, you need to open wso2server.sh file which is located in the bin directory of the WSO2 server and add following lines to it. 

 -XX:+UnlockCommercialFeatures \  
 -XX:+FlightRecorder \  
 -XX:FlightRecorderOptions=defaultrecording=true,disk=true,maxage=60m,repository=./tmp,dumponexit=true,dumponexitpath=./ \  

Once you add them, It will look like follows.

  -Dfile.encoding=UTF8 \   
  -XX:+UnlockCommercialFeatures \   
  -XX:+FlightRecorder \   
  -XX:FlightRecorderOptions=defaultrecording=true,disk=true,maxage=60m,repository=./tmp,dumponexit=true,dumponexitpath=./ \   
 org.wso2.carbon.bootstrap.Bootstrap $  

Then you can save it and restart the server. It will automatically, Dump records to the tmp directory in  your WSO2 server. 

Acquire Heap Dump of Java Process

When it comes to production troubleshooting with Java Servers, It is a main fact that we need to analyze the memory consumption.  Normally if we have configured following properties , It will automatically create the heap dump when there is an OutOfMemory exception.

 -XX:+HeapDumpOnOutOfMemoryError  
 -XX:HeapDumpPath=<path>  

There are some OOM exceptions which is not creating heap dump since it is not related to memory. You can understand on various OutOfMemory Exceptions in the article [1].


But, If you need to get a heap dump at a time which server is consuming memory but not yet throwing OOM Exception, You can use following command to do that.

Assuming that you have set JAVA_HOME and PID is the Process ID.

In a Linux based system.

 jmap -dump:format=b,file=./heap.hprof <PID>  

In a windows based system

 <JAVA_HOME>/bin/jmap -dump:format=b,file=c:\temp\heap.hprof <PID>  

Once you obtain the heap dump, You can analyze it using tools like MAT, VisualVM.



[1] https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/memleaks002.html