Jetty
  1. Jetty
  2. JETTY-380

SelectChannelConnector blocks for maxIdleTime ms

    Details

    • Type: Bug Bug
    • Status: Resolved Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 6.1.4
    • Fix Version/s: None
    • Component/s: NIO
    • Labels:
      None
    • Environment:
      Windows XP, Java 1.6.0_01-b06
    • Number of attachments :
      0

      Description

      When I press the Refresh button in Firefox, the browser sends 18 HTTP GET requests to my servlet application (for alls the GIFs, Stylesheets and Scripts that are references by the page). Firefox seems to use two TCP connections, because some requests are processed in parallel. (The Jetty server and the Firefox browser run on the same Windows-XP machine).
      Often, but not always, Jetty blocks for SelectChannelConnector.maxIdleTime milliseconds during these requests.

      I tried different Jetty versions.
      The effect does not occur for: 6.0.0 and 6.0.1.
      It occurs for 6.0.2, 6.1.0, 6.1.1 and 6.1.4.

      When I use SocketConnector instead of SelectChannelConnector, the effect does not occur.

      While the connection is blocked (until the timeout occurs after maxIdleTime ms), other parallel requests are possible, but it has strange effects on the i/o operations of other threads. Sometimes it blocks console i/o and even loading Java class code from disk seems to be affected.

      When I press Ctrl-Break during the blocking, the following stack trace is displayed for the SelectChannelConnector thread:
      "btpool0-9 - Acceptor0 SelectChannelConnector@0.0.0.0:8080" prio=6 tid=0x0e38ac00 nid=0x1b0 runnable [0x0e7cf000..0x0e7cfa14]
      java.lang.Thread.State: RUNNABLE
      at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
      at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:274)
      at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:256)
      at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:137)
      at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

      • locked <0x045c5568> (a sun.nio.ch.Util$1)
      • locked <0x045c5578> (a java.util.Collections$UnmodifiableSet)
      • locked <0x045c54f0> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
        at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:432)
        at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:169)
        at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
        at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:516)
        at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)

      Jetty does not print any error messages (even with -DDEBUG=1) and after the timeout everything continues normal.

        Activity

        Hide
        Greg Wilkins added a comment -

        Christian,

        that stack trace is totally normal and that is where I would expect the acceptor thread to be.

        the blocking on the other hand, is not normal.

        Can you capture one of these events with tcpdump or wireshark or similar.
        Can you also reproduce in a simply small webapp that you can attach to this issue?

        finally, can you go to the about:config page of your firefox and report the value of
        network.http.pipelining. You can try switching that value and see if that changes anything.

        regards

        Show
        Greg Wilkins added a comment - Christian, that stack trace is totally normal and that is where I would expect the acceptor thread to be. the blocking on the other hand, is not normal. Can you capture one of these events with tcpdump or wireshark or similar. Can you also reproduce in a simply small webapp that you can attach to this issue? finally, can you go to the about:config page of your firefox and report the value of network.http.pipelining. You can try switching that value and see if that changes anything. regards
        Hide
        Christian d'Heureuse added a comment -

        Greg, thanks for your quick reply.

        When I change network.http.pipelining from true to false in Firefox, the error does not occur any more.

        And when I have network.http.pipelining=true and change network.http.pipelining.maxrequests from 8 to 4 (which is the default), the error also does no longer occur.

        It occurs with the following settings:
        network.http.pipelining=true
        network.http.pipelining.maxrequests=8

        I will now try to capture the TCP packets. I cannot do it directly, because Wireshark cannot capture the localhost interface in Windows.

        Show
        Christian d'Heureuse added a comment - Greg, thanks for your quick reply. When I change network.http.pipelining from true to false in Firefox, the error does not occur any more. And when I have network.http.pipelining=true and change network.http.pipelining.maxrequests from 8 to 4 (which is the default), the error also does no longer occur. It occurs with the following settings: network.http.pipelining=true network.http.pipelining.maxrequests=8 I will now try to capture the TCP packets. I cannot do it directly, because Wireshark cannot capture the localhost interface in Windows.
        Hide
        Greg Wilkins added a comment -

        Don't worry about wireshark dump.

        I have managed to reproduce this by making a test harness that creates large pipelines

        working on diagnosis and fix

        Show
        Greg Wilkins added a comment - Don't worry about wireshark dump. I have managed to reproduce this by making a test harness that creates large pipelines working on diagnosis and fix
        Hide
        Greg Wilkins added a comment -

        DOH!

        Jetty has a handler loop that keeps trying to parse requests while progress is being made.
        If progress is not being made, it tries a few more times... just in case it is a NIO zero byte moment.

        Unfortunately, pipelined requests don't need IO to be handled! So with no IO, after 4 times around
        the loop, we just stop return with no more progress!

        The fix is to simply clear the no-progress count every time a request is handled!

        Show
        Greg Wilkins added a comment - DOH! Jetty has a handler loop that keeps trying to parse requests while progress is being made. If progress is not being made, it tries a few more times... just in case it is a NIO zero byte moment. Unfortunately, pipelined requests don't need IO to be handled! So with no IO, after 4 times around the loop, we just stop return with no more progress! The fix is to simply clear the no-progress count every time a request is handled!
        Hide
        Greg Wilkins added a comment -

        In SVN now, plus a test harness to check pipelines.

        Show
        Greg Wilkins added a comment - In SVN now, plus a test harness to check pipelines.
        Hide
        Christian d'Heureuse added a comment -

        I can confirm that the error is fixed with this code modification.
        Thank you very much for the quick solution.

        Show
        Christian d'Heureuse added a comment - I can confirm that the error is fixed with this code modification. Thank you very much for the quick solution.

          People

          • Assignee:
            Greg Wilkins
            Reporter:
            Christian d'Heureuse
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: