100% CPU on RHEL5+ Kernels

Joakim, do you have a VM to reproduce this on?

Daniel,

What Java JVM are you using? (The output from $ java -version would be good)
This seems suspiciously similar to the JVM bugs in Java 1.6 (some with NIO and some with SSL), yet I can see from your stack trace that you are using BIO and non-SSL at that.
I'll be setting up a RHEL5 environment in virtualbox to test this, and I want as similar an environment to yours.

Also, is an upgrade to Jetty 7.1.x series as an option? (Jetty 6.x is no longer getting improvements, and is considered in maintenance mode)

Joakim

Yeah, I'm using JDK1.6.0u11 32-bit for the java environment, but we've used up to update 20 (32-bit and 64-bit) and still experienced the issue. I've researched the java issue and tried it on JVM versions that were suppose to be fixed for that particular issue, but the issue still occurs with them.

If I can get a concrete answer on why this is happening, and if upgrading to Jetty 7.1.x will fix this, then I can take that back to management and get programming involved to recompile the web application against that version of jetty. But, without some solid information, I can't do anything.

Thanks

Another comment about this – this issue only occurs when the service is under load. The process can run forever with no issues on RHEL5 if nobody is using it.

Not sure how helpful that information is... I noticed that there's a loop in the run() method of the SocketConnector that checks the status of the socket... Is it possible there's a problem with this logic? Maybe something about RHEL5 networking that causes tcp sessions to not close in the way that the SocketConnect is expecting them to?

Can you comment about the style of load you have?

Interested in things like ...

1. number of requests in a given time span.
2. the average amount of data in / out per request.
3. timespan per request.
4. variety of incoming clients.
5. stateless or stateful.
6. and anything else you could provide.

This would help us tweak up a load generation tooling on our side in attempt to tickle the same issue you are having.

Different configurations of the above can stress different aspects of a high load system (low on memory, low on sockets, low on memory, low on IO, high on GC, etc ...)

1. We'll have about 250 requests going to/from the web application simultaneously. These requests are all supplied and returned over AJP.
2. A few kilobytes at most. We serve static content from the Apache side.
3. Not sure what you mean by this question. I would say though that we have had simultaneous requests occurring. Maybe a few thousand over a 10 minute or so period? Not really sure how to track this metric either...
4. The source of the data will always be from Apache using mod_ajp.
5. We use DWR within our webapp, so I'm going to guess stateful. I can get more information when I get in touch with a programmer.
6. The load and memory usage on the boxes seems ok. We have enterprise monitoring that monitors the servers, and within the JVM our web application tracks the memory usage. Also, we have a script to track the garbage collection and nothing really seems out of hand. These boxes are all configured with single-processor. Most of them are running on VMware ESX 3.5 – although, we have a dual proc physical server that experiences the same issue.

Thanks for your quick replies, and let me know what more information I can provide.

Thanks,
Dan

Stack Traces:
Thread 2903 "btpool0-5": (state = IN_JAVA)
at org.mortbay.io.bio.StreamEndPoint.isClosed(StreamEndPoint.java:74)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:213)
at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)

Thread 2903 "btpool0-5": (state = IN_JAVA)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:227)
at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)

Thread 2903 "btpool0-5": (state = IN_JAVA)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:380)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:227)
at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)

Thread 2903 "btpool0-5": (state = IN_JAVA)
at org.mortbay.io.bio.StreamEndPoint.isClosed(StreamEndPoint.java:74)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:213)
at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)

In the past I have wiresharked the server to see if there was anything excessive going on, and I haven't noticed anything outside of the norm. There's really no discernible pattern that I can see from this (except that it always seems to happen as I'm leaving for lunch...). I'm not sure I've tried using the SelectChannel Connector... I'll look into doing that.

Thanks

Joakim,

I just wanted to check in with you and see if you had any progress on reproducing the issue. I have managed to hack together a way of killing the errant threads when they appear using JConsole and JDB, but this is a manual band-aid. I'm also in the process of seeing what it will take to get our web application to properly deploy on Jetty7 – so far I'm not having much luck, but I will keep trying.

I can say for certain, at this point, that the looped threads are sitting inside of loop of the run() method, and I suspect that it has something to do with the AJP connector closing before sending all of the data that it promised. It seems that the logic is such that the connector will sit there waiting for more data, so the isClosed() is returning an incorrect result, but there is no data to actually handle() and therefore the loop continues perpetually. However, I am not talented enough to reproduce the conditions of this scenario, but I would guess that adding a data.length check into the loop condition would cause the loop to break appropriately. If I come up with a way to reproduce this, I will let you know.

Please let me know if there is any additional information I can provide.

Thanks,
Dan

We have so far been unable to replicate this scenario.

Which has us thinking that there must be something missing, behavior-wise, that might be triggering this.
Do you see this with certain client or network configurations?
We've seen strange situations arise from old / broken HTTP clients.
We've also seen strange situations crop up from bad/misconfigured proxies, routers, etc.

Is there any strong reason you are using Jetty behind Apache httpd with AJP?
We typically recommend running Jetty directly, and not behind Apache or nginx.
You get lower memory usage, faster performance, and an increase in number of simultaneous clients using Jetty directly.
That and you can start to use more advanced features Jetty provides such as Continuations, Async, Bayeux, Cometd, WebSockets, etc.

If you have to run behind Apache http, have you considered just straight mod_proxy?

Also, Have you tried the suggestions from Greg Wilkins? (previous comment on this issue)

Thanks for your reply. We don't see this problem with specific clients or network connections, however after I read your comment yesterday, I did some looking, and it turns out that on our RHEL4 servers we are using mod_jk 1.2.25, and on our RHEL5 servers we are using mod_jk 1.2.28. Errors in the mod_jk.log file have timestamps correlating directly to the manifestation of the CPU problem on our jetty servers. So, at this point, I'm thinking that the breakdown is somewhere with the mod_jk connector and jetty – perhaps there was a new feature or something to gracefully handle a certain type of network or client problem that was implemented between 1.2.25 and 1.2.28 – I don't know...

There's not a strong reason to continue to use AJP, except that it would require careful production changes to move to another configuration such as ProxyPass or forward-facing our jetty servers. Originally, we had used Apache as a buffer, so as to not externally expose our application servers.

Also, in looking at the Thread Dumps, it would appear as though our connectors are currently using the SelectChannelConnector. I see a bunch of the pooled threads "accept"'ing with this class in memory. Is there another way for me to tell for certain?

Thanks again for your help.

After a little bit more research here, I can confirm that this is a mod_jk + jetty issue. When making a request, if you modify the request headers to include an arbitrary content-length that is way bigger than the request could possibly be (I used 50000), the browser will sit until the mod_jk timeout is reached. After the mod_jk timeout is reached, jetty goes into a spin (presumably waiting for more data?) and consumes all available CPU resources.

I used the firefox plugin "Modify Headers" to reproduce this problem. Still, do you think that Jetty7 will solve this issue? I feel comfortable switching to ProxyPass due to the sheer volume of issues surrounding mod_jk and AJP13 in general.

Thanks

Daniel,

thanks for the help diagnosing this.

in general we definitely prefer mod_proxy_http, then mod_proxy_ajp and finally only if there is NO OTHER CHOICE mod_jk.

This does sound like a bad response from jetty, but only to an bad AJP request from apache. So we will try to fix soon, but I have to say that AJP is not a high priority for us.

regards