Java, GlassFish v3, High CPU and Memory Usage, Locked Threads, Death

We’ve been successfully running a core Java system at work to handle student assignment submissions for approximately 2 years. It has been running on an old version of GlassFish (and by old, I mean old. The Oracle consultants didn’t recognise it as being an official version). It was recommended we moved versions at the earliest possible opportunity.

And so we did. We tested fairly thoroughly on GlassFish v3.1.1 and everything went through fine.

We deployed to the live architecture, with some minor architecture changes to the code base to keep GlassFish v3 happy – the old version of GlassFish seemed to be a bit more forgiving of the way we packaged our EJBs. We also made some changes to break out separate functionality to avoid downtime for key users during upgrades and planned maintenance.

And everything went fine. Although over the next 2 days the CPU usage on the Solaris boxes kept rising. The stats revealed by the command ‘prstat -a’ according to our Sys Admin were particularly unhealthy, and indicated imminent meltdown. And sure enough not long later the boxes fell over and died. Upon restart we saw this behaviour repeat – the CPU usage slowly rising, until eventually the box would give up the ghost and keel over.

We went back to basics and eliminated the load balancer, we reduced the functionality being provided, but alas zero difference.

After taking countless stack traces (note this important command to get a dump of the active threads from GlassFish):

./asadmin generate-jvm-report --type=thread --user admin --passwordfile passfile

And looking through the thread list, nothing was terribly worrying. There were no deadlocks, and no locked threads. There was a valid line that was being repeated, but seemed to be the inner realms of Java that we were calling. We would normally expect to see a maximum of 5-6 of these track staces at any one time, but on one occasion when the box keeled over, there were 47 threads in this state. The line of importance was:

Thread "http-thread-pool-8080(76)" thread-id: 27,866 thread-state: RUNNABLE
   at: java.io.FileInputStream.readBytes(Native Method)
   at: java.io.FileInputStream.read(FileInputStream.java:177)

This led us to this post on Stack Overflow where similar symptoms were being experienced:

http://stackoverflow.com/questions/10178162/infinite-100-cpu-usage-at-java-io-fileinputstream-readbytesnative-method

Having put in place the temporary fix described in the SO post:

if (in.available() == 0) { 
   Thread.sleep(1000); 
   if in.available() == 0) return;
}

Latest Status

With this code fix in place we appear to have been relatively stable for the past 4 days. Its still early days, but this certainly appears at the moment to be the solution.

We have noted that we’re running Java 1.6.0_21 on the box, which is quite dated, and indeed the support matrix for GlassFish 3 suggests a minimum of 1.6_026. We’ve been testing our application with the latest version of Java 6 and all appears good. We’ll look to move to that version, and see about removing the temporary fix to see if that provides a more permanent solution.

Its worth noting that we have experienced this on both GlassFish 3.1.1 and GlassFish 3.1.2 (we upgraded one of our servers during a stage of panic in the hope GF would have resolved the issue).