Using an http client with 10 sec timeout to hit jetty server - Some requests get queued and the response comes after 10 second time out - how to resolve this

Hi, checking first, but does this happen on the latest version of Jetty as well?

would test this on the latest version. can only test on jetty 6.1.x versions as cannot migrate to jetty7 (As the problem is on production env only).

can you try 6.1.19?

i will try on jetty 6.1.19 from tonight for a full day tomorrow.

By the ways even SocketConnector does not help

70 connectors is a really really large number. I would suggest 2 or 4 acceptors max.

The other thing to do is to get a thread dump of the server during one of the "queued" events.

I am also experiencing an issue that seems to be related. When a test client makes as many connections as maxThreads available in Jetty, I get this timeout issue. I was able to reproduce it using 50 simultaneous client threads going against the jetty server running with maxThreads of 50. I see the same problem with 200 threads. If I increase the number of maxThreads for Jetty to 51 or 201 respectively, then the read timeout does not occur on the client. It seems as though there is some issue when the maximum number of threads is reached.

I verified this issue with Jetty 6.1.18, 6.1.19 and 6.1.21.
I tried it using the SocketConnector, BlockingChannelConnector and the SelectChannelConnector. With the SelectChannelConnector the problem happens on occasion. With the other two, it always occurs.

Things get worse though when the client has no timeout specified. When I run the test again with no timeout on the client, it will indefinitely hang waiting for the response from the Jetty server. If I debug Jetty, I notice that all of the allocated threads are all waiting for input from HttpConnection. If I attempt to run a separate client to make a connection, that connection is also blocked. Essentially if the client does not have a timeout, it can hang any further requests to the server.

Just a note about my environment. I'm running on Windows XP with JRE 1.6.0_13 and using the CXF to start embedded Jetty. I'm going to try to narrow down the problem further.

Kanonymous,

Do you have any further information on this situation? Are you using HTTP1.0 or HTTP1.1 for these requests?

Why do you have the number of Acceptors set so high? Have you tried decreasing the number of Acceptors (rule of thumb: one Acceptor per core)? Note you can also set:

+ the thread priority on the Acceptors to increase the likelihood of new connection acceptance vs handling existing connections
+ the size of the accept queue

Note also that you should use the QueuedThreadPool rather than the BoundedThreadPool, as the QueuedThreadPool has less lock contention in some situations.

regards
Jan

Chris,

If the server-side takes a long time to service requests, such that all threads in the threadpool are busy, then no new connections can be created. The acceptors will start to queue the connection requests. You need to ensure that the serverside application processes requests as quickly as possible, or use Continuations, to ensure that threads are freed up as soon as possible.

regards
Jan

Kanonymous,

Did you try reproducing with the QueuedThreadPool?

regards
Jan

Jan,

Yes. This was a case where all the threads were busy. I did this on purpose to simulate a high load coming from customers (higher than the server could handle). The acceptors should start to queue new connections.

As threads in the pool completed, requests in the acceptor queue seemed to not ever be served. The client did not get a response even when there were threads available on the server.

Chris,

6.1.5 is now a very old release. The current release is 6.1.22. If possible, could you please retest your setup with that release? Also ensure that you are using the SelectChannelConnector (which is nio and thus not thread per connection), and have about as many acceptor threads configured as you have CPUs.

Also, you mention in a previous post that jetty threads are waiting on input, which makes me suspect that the client has not flushed out all of the output. If you do recreate this problem, 2 things would be ideal to get: 1) a thread dump from jetty 2) a wireshark (or other ip packet sniffer) trace of some of the hung connections.

Also, are you sure your client-side is able to handle the load? We've found for testing that we needed an async client http library, so we wrote one. You might want to check it out:

http://docs.codehaus.org/display/JETTY/Jetty+HTTP+Client

thanks,
Jan

Jan,

As indicated in the original post I was using version 6.1.21. The issue was more with connectors other than the SelectChannelConnector though it seemed to happen with that one too.

It may well have been an issue with the client. I am no longer able to run the same test as that environment is no longer available.

Greg,

Do you have any comment on the postings, particularly from Chris who has tested more recent versions of jetty? Does the fact that the jetty threads are all in read indicate that the client has not flushed all bytes yet (could be a problem with the test implementation too)?

regards
Jan

I'm closing this issue for now because we cannot reproduce.

There are a number of things that have improved in this area for the handling of closes and I suggest trying a the latest jetty-7 release.

If you still believe this is a problem, please reopen with either a reproducable test harness or the exact step you took to produce the problem.