Jetty
  1. Jetty
  2. JETTY-1377

Weird IO loops with long-held zombie client connections

    Details

    • Type: Bug Bug
    • Status: Resolved Resolved
    • Priority: Major Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: 7.4.0
    • Fix Version/s: 7.6.0
    • Component/s: HTTP
    • Labels:
      None
    • Environment:
      Jetty 7.4.0
    • Number of attachments :
      0

      Description

      We recently had a client bug in one of our service consumers which involved the client sending malformed HTTP requests. In particular, the Content-Length header for PUT requests was off and the client forgot about the socket and never closed the connection.

      Jetty's BlockingChannelConnector and SocketConnector both had odd responses to this situation.

      SocketConnector goes into a hot loop of throwing SocketException instances and rescuing them:

      "qtp1213779113-151" - Thread t@151
         java.lang.Thread.State: RUNNABLE
      	at java.lang.Throwable.fillInStackTrace(Native Method)
      	- locked <7a96bbd0> (a java.net.SocketException)
      	at java.lang.Throwable.<init>(Throwable.java:196)
      	at java.lang.Exception.<init>(Exception.java:41)
      	at java.io.IOException.<init>(IOException.java:41)
      	at java.net.SocketException.<init>(SocketException.java:29)
      	at java.net.SocketInputStream.read(SocketInputStream.java:113)
      	at org.eclipse.jetty.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:388)
      	at org.eclipse.jetty.io.bio.StreamEndPoint.fill(StreamEndPoint.java:132)
      	at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.fill(SocketConnector.java:209)
      	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:289)
      	at org.eclipse.jetty.http.HttpParser.blockForContent(HttpParser.java:1139)
      	at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:57)
      

      And BlockingChannelConnector spins on reading from the connection:

      "qtp2084696615-1065" - Thread t@1065
         java.lang.Thread.State: RUNNABLE
      	at org.eclipse.jetty.io.nio.ChannelEndPoint.fill(ChannelEndPoint.java:173)
      	- locked <587fc800> (a java.nio.HeapByteBuffer)
      	at org.eclipse.jetty.server.nio.BlockingChannelConnector$BlockingChannelEndPoint.fill(BlockingChannelConnector.java:233)
      	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:289)
      	at org.eclipse.jetty.http.HttpParser.blockForContent(HttpParser.java:1139)
      	at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:57)
      

      They both peg that worker thread indefinitely; it seems like request parsing should respect the timeout settings for the connector. I suspect SelectChannelConnector would also behave badly under these circumstances, but I didn't get the chance to try.

        Activity

        Hide
        Coda Hale added a comment -

        Doesn't seem fixed in 7.4.5, either.

        Show
        Coda Hale added a comment - Doesn't seem fixed in 7.4.5, either.
        Hide
        Michael Gorovoy added a comment - - edited

        Greetings,

        Could you test your issue with the latest 7.5.0-SNAPSHOT available from http://goo.gl/eOQHm? Greg committed a series of changes to HttpParser code, including the area that you are referring to, that should have fixed this issue.

        Cheers,
        Michael

        Show
        Michael Gorovoy added a comment - - edited Greetings, Could you test your issue with the latest 7.5.0-SNAPSHOT available from http://goo.gl/eOQHm? Greg committed a series of changes to HttpParser code, including the area that you are referring to, that should have fixed this issue. Cheers, Michael
        Hide
        Michael Gorovoy added a comment -

        Going to mark this ticket as resolved pending confirmation that it has been fixed.

        Show
        Michael Gorovoy added a comment - Going to mark this ticket as resolved pending confirmation that it has been fixed.
        Hide
        Coda Hale added a comment -

        I can no longer duplicate this issue with BlockingChannelConnector or SelectChannelConnector, but this is still broken for SocketConnector.

        FWIW, this is a very easy issue to test. I have a fully packaged test case, complete with repro steps here: https://github.com/codahale/jetty-parser-bug-example

        Show
        Coda Hale added a comment - I can no longer duplicate this issue with BlockingChannelConnector or SelectChannelConnector , but this is still broken for SocketConnector . FWIW, this is a very easy issue to test. I have a fully packaged test case, complete with repro steps here: https://github.com/codahale/jetty-parser-bug-example
        Hide
        Greg Wilkins added a comment -

        thanks for creating a test case, but I've tried to reproduce with 7.2.2, 7.4.2 and 7.5.2-SNAPSHOT, using firefox and chrome.... no joy.

        So I'm going to reclose for now, as we certainly have the fix you suggest into the HttpParser.
        Also, when 7.5.2 comes out, it will have a bit more debugging in it to detect busy loops and dump info.

        If you can reproduce on 7.5.2, then please re-open.

        Show
        Greg Wilkins added a comment - thanks for creating a test case, but I've tried to reproduce with 7.2.2, 7.4.2 and 7.5.2-SNAPSHOT, using firefox and chrome.... no joy. So I'm going to reclose for now, as we certainly have the fix you suggest into the HttpParser. Also, when 7.5.2 comes out, it will have a bit more debugging in it to detect busy loops and dump info. If you can reproduce on 7.5.2, then please re-open.

          People

          • Assignee:
            Greg Wilkins
            Reporter:
            Coda Hale
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: