Jetty

HttpClient enters an infinite loop (and leaks connections) if HttpGenerator.flush throws an exception

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: 6.1.12.rc3
  • Fix Version/s: 7.0.0pre4, 6.1.12
  • Component/s: HTTP
  • Labels:
    None
  • Number of attachments :
    2

Description

When using Jetty HttpClient with NIO, if HttpGenerator.flush throws an exception (because the remote server has died, for example), the connection is leaked. Also, the connection remains selectable, and so the selector thread enters an infinite loop. Here's the stack trace (this repeats indefinitely):

116 [HttpClient-2] WARN o.m.log : EXCEPTION on HttpExchange@1714575288=POST//localhost:3456/ping#9
org.mortbay.jetty.EofException
at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:787)
at org.mortbay.jetty.client.HttpConnection.handle(HttpConnection.java:233)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:497)
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
at sun.nio.ch.IOUtil.write(IOUtil.java:60)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
at org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:169)
at org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:721)
... 3 common frames omitted

This happens because org.mortbay.jetty.client.HttpConnection.handle() eats the exception:

public void handle() throws IOException
{
// ...
try
{
// ...

catch (IOException e)
{
synchronized(this)
{
if (_exchange!=null)

{ _exchange.getEventListener().onException(e); _exchange.setStatus(HttpExchange.STATUS_EXCEPTED); }

}
// XXX exception eaten, when it should be propagated upwards
}
// ...
}

so

  1. Text File
    jetty-6.httpclient-leak.patch
    15/Oct/08 6:47 PM
    5 kB
    Tudor Bosman
  2. Java Source File
    JettyClientTest.java
    15/Oct/08 5:51 PM
    9 kB
    Tudor Bosman

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Tudor Bosman added a comment -

(sorry, incomplete description, saved too early)

The exception should be propagated upstream to SelectChannelEndPoint.run(), which would then close the connection:

public void run()
{
try

{ _connection.handle(); }

catch (ClosedChannelException e)

{ Log.ignore(e); }

catch (EofException e)
{
Log.debug("EOF", e);
try

{close();}
catch(IOException e2){Log.ignore(e2);}
}
catch (HttpException e)
{
Log.debug("BAD", e);
try{close();}

catch(IOException e2)

{Log.ignore(e2);}
}
catch (Throwable e)
{
Log.warn("handle failed", e);
try{close();}
catch(IOException e2){Log.ignore(e2);}

}
finally

{ undispatch(); }

}

Show
Tudor Bosman added a comment - (sorry, incomplete description, saved too early) The exception should be propagated upstream to SelectChannelEndPoint.run(), which would then close the connection: public void run() { try { _connection.handle(); } catch (ClosedChannelException e) { Log.ignore(e); } catch (EofException e) { Log.debug("EOF", e); try {close();} catch(IOException e2){Log.ignore(e2);} } catch (HttpException e) { Log.debug("BAD", e); try{close();} catch(IOException e2) {Log.ignore(e2);} } catch (Throwable e) { Log.warn("handle failed", e); try{close();} catch(IOException e2){Log.ignore(e2);} } finally { undispatch(); } }
Hide
Permalink
Tudor Bosman added a comment -

Test case attached.

Show
Tudor Bosman added a comment - Test case attached.
Hide
Permalink
Tudor Bosman added a comment -

Propagating the exception ("throw e" at the place in the code indicated by "XXX exception eaten", above) appears to fix the problem.

Show
Tudor Bosman added a comment - Propagating the exception ("throw e" at the place in the code indicated by "XXX exception eaten", above) appears to fix the problem.
Hide
Permalink
Tudor Bosman added a comment -

The fix mentioned above (add "throw e") is not sufficient, as the connection doesn't get reclaimed. Here's a patch that resolves the problem and also passes all existing unit tests. (in unified diff format against the jetty-6 branch)

Show
Tudor Bosman added a comment - The fix mentioned above (add "throw e") is not sufficient, as the connection doesn't get reclaimed. Here's a patch that resolves the problem and also passes all existing unit tests. (in unified diff format against the jetty-6 branch)
Hide
Permalink
Greg Wilkins added a comment -

Tudor,

thanks for the patch. I've cleaned it up just a little and applied it to both jetty-6 and jetty-7.

cheers

Show
Greg Wilkins added a comment - Tudor, thanks for the patch. I've cleaned it up just a little and applied it to both jetty-6 and jetty-7. cheers
Hide
Permalink
Lukas Zapletal added a comment - - edited

I am running into the same problem described here with both 6.1.21 and also latest stable 6.1.23.

I have tried to run the test case and it was successful but only because somebody commented out the test. Probably because it was failing from some version between 6.1.12 - 6.1.23 and it took some time to end (to fail).

I will create new bugreport for this with link to this bug/patch/fix. I do not have access to the history of that file.

This bug also causes the AsyncProxyServlet does not properly when connection is broken (Broken Pipe).

Created http://jira.codehaus.org/browse/JETTY-1210

Show
Lukas Zapletal added a comment - - edited I am running into the same problem described here with both 6.1.21 and also latest stable 6.1.23. I have tried to run the test case and it was successful but only because somebody commented out the test. Probably because it was failing from some version between 6.1.12 - 6.1.23 and it took some time to end (to fail). I will create new bugreport for this with link to this bug/patch/fix. I do not have access to the history of that file. This bug also causes the AsyncProxyServlet does not properly when connection is broken (Broken Pipe). Created http://jira.codehaus.org/browse/JETTY-1210

People

  • Assignee:
    Greg Wilkins
    Reporter:
    Tudor Bosman
Vote (0)
Watch (2)

Dates

  • Created:
    Updated:
    Resolved: