Affects Version/s: 6.0.1
Fix Version/s: 6.1.0rc2
Number of attachments :
We are using Jetty in Hadoop (http://lucene.apache.org/hadoop/) in Embedded mode. It is a framework for doing parallel computations on large (distributed) data sets and the framework runs on clusters of 100s of nodes. The main requirement of having Jetty here is to serve JSP pages (for UI) and static files (for collecting outputs of computations from different nodes and aggregate). So we run Jetty on all nodes (something like 350 nodes) and every node asks for outputs from all other nodes (it is basically all-to-all http communication) and the amount of static data that any node might serve up is in the order of 10 GB.
Till a few days back we were using Jetty 5.1.4 and it was working fine for our requirements. We moved to Jetty6.0.1 (stable) recently and ran into weird problems with static-file-serving. What happens is that Jetty gets really very very slow after serving up a few files (in the order of 100s) and it does not come out of this state at all. When we do a thread dump, we notice that all the Jetty worker threads are trying to do a read from some socket as part of parsing http requests.
"btpool0-2" prio=1 tid=0x081bbd70 nid=0x147d runnable [0x4dbfa000..0x4dbfb130]
at java.net.SocketInputStream.socketRead0(Native Method)
All clients start timing out on connects to the http ports and things become really really slow. The weird thing is that no such problem happens with Jetty6 although the http client code is exactly the same and so on. I am also attaching the code that we use to start a Jetty server in embedded mode (the methods there are invoked from another class).