Type: New Feature
Affects Version/s: None
Fix Version/s: None
Number of attachments :
We have had in the past reports for threads going 100% CPU and hogging the server, but never able to track down exactly what was causing that.
This issue is to write a tool that tracks down 100% CPU cases.
The idea is to poll the current threads via ThreadMXBean.getThreadCpuTime(long) (and/or related methods) to see how much a thread is busy.
This needs to be done using System.nanoTime(), and not with System.currentTimeMillis().
The percentage calculated from the value of ThreadMXBean.getThreadCpuTime(long) needs to be understood exactly in case of multi cores: if I have 4 cores and 1 thread at 100% of one core, does the percentage show 100% or 25% ?
Along with that calculation, which will tell us what thread is spinning, it will be useful to get few stack traces of that thread during the spinning period (and be sure they're taken during that spinning period) via ThreadMXBean.getThreadInfo(long).getStackTrace().
This should give more insights of where the thread is spinning.
Stack traces taken at different times can be compared to see if they relate, for example if they have the same stack frames up to a certain point, then it may mean the thread is spinning inside a method that is probably the last frame in common among the stack traces.
In other words, the tool should give us as much information as possible about the 100% CPU cases.
The tool itself should be installable as an MBean via an additional jetty-XXX.xml file so that the tool current information can be inspected via JConsole. This means it will run in-VM with Jetty and the application, and eventually started/stopped at will.
The code for this tool may probably live in its own module here at the Codehaus, for now, unless we want to make it available at eclipse as well, but I would start here.