Jetty
  1. Jetty
  2. JETTY-177

Jetty's 6.0 threads become unresponsive after serving up a couple of static files

    Details

    • Type: Bug Bug
    • Status: Resolved Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 6.0.1
    • Fix Version/s: 6.1.0rc2
    • Component/s: None
    • Labels:
      None
    • Number of attachments :
      3

      Description

      We are using Jetty in Hadoop (http://lucene.apache.org/hadoop/) in Embedded mode. It is a framework for doing parallel computations on large (distributed) data sets and the framework runs on clusters of 100s of nodes. The main requirement of having Jetty here is to serve JSP pages (for UI) and static files (for collecting outputs of computations from different nodes and aggregate). So we run Jetty on all nodes (something like 350 nodes) and every node asks for outputs from all other nodes (it is basically all-to-all http communication) and the amount of static data that any node might serve up is in the order of 10 GB.
      Till a few days back we were using Jetty 5.1.4 and it was working fine for our requirements. We moved to Jetty6.0.1 (stable) recently and ran into weird problems with static-file-serving. What happens is that Jetty gets really very very slow after serving up a few files (in the order of 100s) and it does not come out of this state at all. When we do a thread dump, we notice that all the Jetty worker threads are trying to do a read from some socket as part of parsing http requests.

      "btpool0-2" prio=1 tid=0x081bbd70 nid=0x147d runnable [0x4dbfa000..0x4dbfb130]
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:129)
      at org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
      at org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
      at org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
      at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
      at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)

      All clients start timing out on connects to the http ports and things become really really slow. The weird thing is that no such problem happens with Jetty6 although the http client code is exactly the same and so on. I am also attaching the code that we use to start a Jetty server in embedded mode (the methods there are invoked from another class).

      1. StatusHttpServer.java
        8 kB
        Devaraj Das
      2. thread-dump-with-selectorconnector
        64 kB
        Devaraj Das

        Activity

        Hide
        Greg Wilkins added a comment -

        I have attached the jar. Please try it.

        Also here is the code that I am using to server files with:

        package org.mortbay.jetty.example;

        import java.io.File;

        public class RandomFileHandler extends AbstractHandler
        {
        public void handle(String target, HttpServletRequest request, HttpServletResponse response, int dispatch) throws IOException, ServletException
        {
        System.err.println("RandomFileHandler");

        Request base_request=(request instanceof Request)?(Request)request:HttpConnection.getCurrentConnection().getRequest();
        base_request.setHandled(true);

        String s=request.getParameter("size");
        int size=s==null?1024:Integer.parseInt(s);

        File file=File.createTempFile("test",".txt");
        FileOutputStream out=new FileOutputStream(file);
        // file.deleteOnExit();
        long len=size-2;
        byte[] data="0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ\r\n".getBytes("ISO_8859_1");
        while (len>0)

        { int chunk=data.length; if (chunk>len) chunk=(int)len; out.write(data,0,chunk); len-=chunk; }

        out.write("\r\n".getBytes());
        out.close();

        System.err.println("Created file "file" of size "+file.length());

        if (file.length()!=size)
        throw new IllegalStateException("WRONG SIZE? "+file.length());

        response.setContentType("text/plain");
        // response.setContentLength(size);
        response.setStatus(HttpServletResponse.SC_OK);

        NIOBuffer buffer=new NIOBuffer(file);
        ((HttpConnection.Output)response.getOutputStream()).sendContent(buffer);
        System.err.println("sent content "+file);
        }

        public static void main(String[] args) throws Exception
        {
        Server server=new Server();
        Connector connector=new SelectChannelConnector();
        connector.setPort(8080);
        server.setConnectors(new Connector[]

        { connector }

        );

        Handler handler=new RandomFileHandler();
        server.setHandler(handler);

        server.start();
        server.join();
        }

        }

        Show
        Greg Wilkins added a comment - I have attached the jar. Please try it. Also here is the code that I am using to server files with: package org.mortbay.jetty.example; import java.io.File; public class RandomFileHandler extends AbstractHandler { public void handle(String target, HttpServletRequest request, HttpServletResponse response, int dispatch) throws IOException, ServletException { System.err.println("RandomFileHandler"); Request base_request=(request instanceof Request)?(Request)request:HttpConnection.getCurrentConnection().getRequest(); base_request.setHandled(true); String s=request.getParameter("size"); int size=s==null?1024:Integer.parseInt(s); File file=File.createTempFile("test",".txt"); FileOutputStream out=new FileOutputStream(file); // file.deleteOnExit(); long len=size-2; byte[] data="0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ\r\n".getBytes("ISO_8859_1"); while (len>0) { int chunk=data.length; if (chunk>len) chunk=(int)len; out.write(data,0,chunk); len-=chunk; } out.write("\r\n".getBytes()); out.close(); System.err.println("Created file " file " of size "+file.length()); if (file.length()!=size) throw new IllegalStateException("WRONG SIZE? "+file.length()); response.setContentType("text/plain"); // response.setContentLength(size); response.setStatus(HttpServletResponse.SC_OK); NIOBuffer buffer=new NIOBuffer(file); ((HttpConnection.Output)response.getOutputStream()).sendContent(buffer); System.err.println("sent content "+file); } public static void main(String[] args) throws Exception { Server server=new Server(); Connector connector=new SelectChannelConnector(); connector.setPort(8080); server.setConnectors(new Connector[] { connector } ); Handler handler=new RandomFileHandler(); server.setHandler(handler); server.start(); server.join(); } }
        Hide
        Devaraj Das added a comment -

        Sorry, the new bugfixed jetty still seems to be having some bugs that shows up in our environment. So what's happening most likely is that the clients gets bad data from jetty servers and this leads to lot of failures in the Hadoop framework later on. I have not got the time to analyze it yet though.
        The short term decision of the Hadoop team is to roll back to the earlier version - Jetty5.1.4.

        Show
        Devaraj Das added a comment - Sorry, the new bugfixed jetty still seems to be having some bugs that shows up in our environment. So what's happening most likely is that the clients gets bad data from jetty servers and this leads to lot of failures in the Hadoop framework later on. I have not got the time to analyze it yet though. The short term decision of the Hadoop team is to roll back to the earlier version - Jetty5.1.4.
        Hide
        Greg Wilkins added a comment -

        From you comment, does that mean that the static content fix worked, but that there were other issues?
        Or is there still trouble with the static content

        thanks for helping with these issues and sorry to cause you extra development cycles.

        Show
        Greg Wilkins added a comment - From you comment, does that mean that the static content fix worked, but that there were other issues? Or is there still trouble with the static content thanks for helping with these issues and sorry to cause you extra development cycles.
        Hide
        Devaraj Das added a comment -

        Yeah the fix worked from the point of view that I stopped getting the exceptions. So I am assuming that the content gets delivered. But there are other problems later on (which I have not yet debugged) and these problems don't appear if we run Hadoop with Jetty5.1.4.

        Show
        Devaraj Das added a comment - Yeah the fix worked from the point of view that I stopped getting the exceptions. So I am assuming that the content gets delivered. But there are other problems later on (which I have not yet debugged) and these problems don't appear if we run Hadoop with Jetty5.1.4.
        Hide
        Greg Wilkins added a comment -

        I think the specific issue was solved.

        please open new JIRAs for any additional issues

        Show
        Greg Wilkins added a comment - I think the specific issue was solved. please open new JIRAs for any additional issues

          People

          • Assignee:
            Greg Wilkins
            Reporter:
            Devaraj Das
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: