Jetty
  1. Jetty
  2. JETTY-747

HttpClient enters an infinite loop (and leaks connections) if HttpGenerator.flush throws an exception

    Details

    • Type: Bug Bug
    • Status: Resolved Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 6.1.12.rc3
    • Fix Version/s: 7.0.0pre4, 6.1.12
    • Component/s: HTTP
    • Labels:
      None
    • Number of attachments :
      2

      Description

      When using Jetty HttpClient with NIO, if HttpGenerator.flush throws an exception (because the remote server has died, for example), the connection is leaked. Also, the connection remains selectable, and so the selector thread enters an infinite loop. Here's the stack trace (this repeats indefinitely):

      116 [HttpClient-2] WARN o.m.log : EXCEPTION on HttpExchange@1714575288=POST//localhost:3456/ping#9
      org.mortbay.jetty.EofException
      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:787)
      at org.mortbay.jetty.client.HttpConnection.handle(HttpConnection.java:233)
      at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
      at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:497)
      Caused by: java.io.IOException: Broken pipe
      at sun.nio.ch.FileDispatcher.write0(Native Method)
      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
      at sun.nio.ch.IOUtil.write(IOUtil.java:60)
      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
      at org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:169)
      at org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
      at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:721)
      ... 3 common frames omitted

      This happens because org.mortbay.jetty.client.HttpConnection.handle() eats the exception:

      public void handle() throws IOException
      {
      // ...
      try
      {
      // ...

      catch (IOException e)
      {
      synchronized(this)
      {
      if (_exchange!=null)

      { _exchange.getEventListener().onException(e); _exchange.setStatus(HttpExchange.STATUS_EXCEPTED); }

      }
      // XXX exception eaten, when it should be propagated upwards
      }
      // ...
      }

      so

      1. jetty-6.httpclient-leak.patch
        5 kB
        Tudor Bosman
      2. JettyClientTest.java
        9 kB
        Tudor Bosman

        Activity

        Hide
        Tudor Bosman added a comment -

        (sorry, incomplete description, saved too early)

        The exception should be propagated upstream to SelectChannelEndPoint.run(), which would then close the connection:

        public void run()
        {
        try

        { _connection.handle(); }

        catch (ClosedChannelException e)

        { Log.ignore(e); }

        catch (EofException e)
        {
        Log.debug("EOF", e);
        try

        {close();}
        catch(IOException e2){Log.ignore(e2);}
        }
        catch (HttpException e)
        {
        Log.debug("BAD", e);
        try{close();}

        catch(IOException e2)

        {Log.ignore(e2);}
        }
        catch (Throwable e)
        {
        Log.warn("handle failed", e);
        try{close();}
        catch(IOException e2){Log.ignore(e2);}

        }
        finally

        { undispatch(); }

        }

        Show
        Tudor Bosman added a comment - (sorry, incomplete description, saved too early) The exception should be propagated upstream to SelectChannelEndPoint.run(), which would then close the connection: public void run() { try { _connection.handle(); } catch (ClosedChannelException e) { Log.ignore(e); } catch (EofException e) { Log.debug("EOF", e); try {close();} catch(IOException e2){Log.ignore(e2);} } catch (HttpException e) { Log.debug("BAD", e); try{close();} catch(IOException e2) {Log.ignore(e2);} } catch (Throwable e) { Log.warn("handle failed", e); try{close();} catch(IOException e2){Log.ignore(e2);} } finally { undispatch(); } }
        Hide
        Tudor Bosman added a comment -

        Test case attached.

        Show
        Tudor Bosman added a comment - Test case attached.
        Hide
        Tudor Bosman added a comment -

        Propagating the exception ("throw e" at the place in the code indicated by "XXX exception eaten", above) appears to fix the problem.

        Show
        Tudor Bosman added a comment - Propagating the exception ("throw e" at the place in the code indicated by "XXX exception eaten", above) appears to fix the problem.
        Hide
        Tudor Bosman added a comment -

        The fix mentioned above (add "throw e") is not sufficient, as the connection doesn't get reclaimed. Here's a patch that resolves the problem and also passes all existing unit tests. (in unified diff format against the jetty-6 branch)

        Show
        Tudor Bosman added a comment - The fix mentioned above (add "throw e") is not sufficient, as the connection doesn't get reclaimed. Here's a patch that resolves the problem and also passes all existing unit tests. (in unified diff format against the jetty-6 branch)
        Hide
        Greg Wilkins added a comment -

        Tudor,

        thanks for the patch. I've cleaned it up just a little and applied it to both jetty-6 and jetty-7.

        cheers

        Show
        Greg Wilkins added a comment - Tudor, thanks for the patch. I've cleaned it up just a little and applied it to both jetty-6 and jetty-7. cheers
        Hide
        Lukas Zapletal added a comment - - edited

        I am running into the same problem described here with both 6.1.21 and also latest stable 6.1.23.

        I have tried to run the test case and it was successful but only because somebody commented out the test. Probably because it was failing from some version between 6.1.12 - 6.1.23 and it took some time to end (to fail).

        I will create new bugreport for this with link to this bug/patch/fix. I do not have access to the history of that file.

        This bug also causes the AsyncProxyServlet does not properly when connection is broken (Broken Pipe).

        Created http://jira.codehaus.org/browse/JETTY-1210

        Show
        Lukas Zapletal added a comment - - edited I am running into the same problem described here with both 6.1.21 and also latest stable 6.1.23. I have tried to run the test case and it was successful but only because somebody commented out the test. Probably because it was failing from some version between 6.1.12 - 6.1.23 and it took some time to end (to fail). I will create new bugreport for this with link to this bug/patch/fix. I do not have access to the history of that file. This bug also causes the AsyncProxyServlet does not properly when connection is broken (Broken Pipe). Created http://jira.codehaus.org/browse/JETTY-1210

          People

          • Assignee:
            Greg Wilkins
            Reporter:
            Tudor Bosman
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: