Jetty
  1. Jetty
  2. JETTY-256

continuation may cause 100%cpu occasionally!

    Details

    • Type: Bug Bug
    • Status: Resolved Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 6.0.0beta8, 6.0.0beta9, 6.0.0beta10, 6.0.0beta11, 6.0.0beta12, 6.0.0beta14, 6.0.0beta15, 6.0.0beta16, 6.0.0beta17, 6.0.0beta18, 6.0.0RC0, 6.0.0rc1, 6.0.0rc2, 6.0.0rc3, 6.0.0rc4, 6.0.0, 6.0.1, 6.0.2, 6.0.3, 6.1.0pre0, 6.1.0pre1, 6.1.0pre2, 6.1.0pre3, 6.1.0rc0, 6.1.0rc1, 6.1.0rc2, 6.1.0rc3, 6.1.0, 6.1.1rc0, 6.1.1rc1, 6.1.1, 6.1.2rc0
    • Fix Version/s: 6.1.2rc2
    • Component/s: Continuations
    • Labels:
      None
    • Environment:
      jdk1.5 gentoo-linux
    • Number of attachments :
      1

      Description

      continuation may cause 100%cpu occasionally.

      sun.nio.ch.SelectorImpl.select(long) invoked 314658 times in 16889ms ! (see the YourKit snapshot )

      may be the blocking Selector.select method stop blocking occasionally.
      CODE SelectorManager.doSelect [
      if (wait > 0)
      _selector.select(wait); <====================== may return immediately.
      else if (wait == 0)
      _selector.selectNow();
      else
      _selector.select();
      ]

      suggest code [
      if (wait > 0)

      { int ret = _selector.select(wait); if (ret == 0) Thread.sleep(1000); //do something }

      else if (wait == 0)
      _selector.selectNow();
      else
      _selector.select();

      ]

        Activity

        Hide
        floyd.yao added a comment -

        is it a bug caused by the new feature of jetty 6? recently i found it in redhat server also

        Show
        floyd.yao added a comment - is it a bug caused by the new feature of jetty 6? recently i found it in redhat server also
        Hide
        Greg Wilkins added a comment -

        Thanks all for the information on this one! It does appear to be a JVM bug that is fixed in java6

        I have finally been able to reproduce the problem and it is a real shocker!! It is like somebody designed a JVM
        bug specifically to make life REALLY hard for the way Jetty uses NIO.

        I can detect that the failure is occuring (select returns for no reason before timeout), and I can inspect every
        channel in the select set. But unless I write something, they all look perfectly fine (and open) - so I have no
        way of detecting which channel is the one that has been badly closed.

        So the only solution that I have found is to cancel all keys that have zero interested ops and to recreate
        them again when there are indeed needed again. This appears to be working, but the cancelled keys
        appears to provoke a couple of issues in Jetty that assumed key valid == channel open

        Anyway.... I am very close to a fix ( I think ) and currently I can't produce any failures. But I'm going to
        sleep on it and test again in the morning to be doubly certain.

        thanks again!

        Show
        Greg Wilkins added a comment - Thanks all for the information on this one! It does appear to be a JVM bug that is fixed in java6 I have finally been able to reproduce the problem and it is a real shocker!! It is like somebody designed a JVM bug specifically to make life REALLY hard for the way Jetty uses NIO. I can detect that the failure is occuring (select returns for no reason before timeout), and I can inspect every channel in the select set. But unless I write something, they all look perfectly fine (and open) - so I have no way of detecting which channel is the one that has been badly closed. So the only solution that I have found is to cancel all keys that have zero interested ops and to recreate them again when there are indeed needed again. This appears to be working, but the cancelled keys appears to provoke a couple of issues in Jetty that assumed key valid == channel open Anyway.... I am very close to a fix ( I think ) and currently I can't produce any failures. But I'm going to sleep on it and test again in the morning to be doubly certain. thanks again!
        Hide
        Greg Wilkins added a comment -

        I have checked in what I think is a resolution for this issue.

        However, some more stress testing of the server is not required as some additional
        instabilities may have been introduced as a result!

        This will put back 6.1.2rc2 to the end of the month

        Show
        Greg Wilkins added a comment - I have checked in what I think is a resolution for this issue. However, some more stress testing of the server is not required as some additional instabilities may have been introduced as a result! This will put back 6.1.2rc2 to the end of the month
        Hide
        Doug Daniels added a comment -

        I'm running JETTY 7.0.0pre5 Continuations and I'm running into this issue when running in a Redhat Linux VMWare environment. Is the 6.1.2rc2 resolution merged into the 7.0.0pre5 code?

        We believe we are seeing the RST, ACK issue on our network, and we're seeing 2 threads causing100% CPU utilization:

        "qtp0-9 - Acceptor0 SelectChannelConnector@0.0.0.0:8080" - Thread t@28
        java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

        • locked sun.nio.ch.Util$1@46f8d1d0
        • locked java.util.Collections$UnmodifiableSet@1cfc87b7
        • locked sun.nio.ch.EPollSelectorImpl@67454dc9
          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
          at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:403)
          at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:174)
          at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:123)
          at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:698)
          at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520)

        "qtp0-8 - Acceptor1 SelectChannelConnector@0.0.0.0:8080" - Thread t@27
        java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

        • locked sun.nio.ch.Util$1@f79455e
        • locked java.util.Collections$UnmodifiableSet@697ea809
        • locked sun.nio.ch.EPollSelectorImpl@17b49fcf
          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
          at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:403)
          at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:174)
          at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:123)
          at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:698)
          at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520)
        Show
        Doug Daniels added a comment - I'm running JETTY 7.0.0pre5 Continuations and I'm running into this issue when running in a Redhat Linux VMWare environment. Is the 6.1.2rc2 resolution merged into the 7.0.0pre5 code? We believe we are seeing the RST, ACK issue on our network, and we're seeing 2 threads causing100% CPU utilization: "qtp0-9 - Acceptor0 SelectChannelConnector@0.0.0.0:8080" - Thread t@28 java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) locked sun.nio.ch.Util$1@46f8d1d0 locked java.util.Collections$UnmodifiableSet@1cfc87b7 locked sun.nio.ch.EPollSelectorImpl@67454dc9 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:403) at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:174) at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:123) at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:698) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520) "qtp0-8 - Acceptor1 SelectChannelConnector@0.0.0.0:8080" - Thread t@27 java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) locked sun.nio.ch.Util$1@f79455e locked java.util.Collections$UnmodifiableSet@697ea809 locked sun.nio.ch.EPollSelectorImpl@17b49fcf at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:403) at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:174) at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:123) at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:698) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520)
        Hide
        Greg Wilkins added a comment -

        Doug,

        the jetty-7 development has moved to eclipse and the latest release is a 7.0.0.RC1, which has handling for these JVM bugs.

        Show
        Greg Wilkins added a comment - Doug, the jetty-7 development has moved to eclipse and the latest release is a 7.0.0.RC1, which has handling for these JVM bugs.

          People

          • Assignee:
            Greg Wilkins
            Reporter:
            Boyce Lee
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: