Jetty
  1. Jetty
  2. JETTY-937

SelectChannelConnector 100% CPU usage on Linux

    Details

    • Type: Bug Bug
    • Status: Closed Closed
    • Priority: Major Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: 6.1.15.rc5
    • Fix Version/s: 7.0.1, 6.1.22
    • Component/s: NIO
    • Labels:
      None
    • Environment:
      CentOS 5.x, 32/64-bit, 32/64-bit JDK 1.6.0_12 and older, Kernel 2.6.18-92.1.22.el5 and also older versions, Jetty 6.1.14, 6.1.15 trunk
    • Number of attachments :
      5

      Description

      The problem itself is described at http://markmail.org/message/zudwd2ve5l2qijbw . There has been no reply more than a month.
      We are having exactly the same problem in a similar environment.

      It can be caused by various JDK/NIO bugs which are still not fixed in the JDK 1.6.0_12, like: http://bugs.sun.com/view_bug.do?bug_id=6693490 .

      At some point Acceptor thread starts to use 100% CPU and doesn't exit this state until application is restarted. It appears that it's spinning in the tight loop. Some other applications provide workaround for this problem like recycling the acceptor and canceling the keys when it happens, so probably the same approach should be implemented in Jetty. I'm not a big expert in this area.

      I've contacted Chris directly and he told me that they are still having the same problem and found no solutions/workarounds. Interesting thing is that our applications have some common components: they both use Jetty and MINA framework in the same application, so it can be related as MINA has its own acceptors which may somehow affect Jetty NIO acceptors. MINA versions are 1.x and 2.x, so could be that the problem is not related to the MINA as well.

      In our application we had to switch Jetty to the old SocketConnector instead of the SelectChannelConnector. This fixed the problem, but it may be not an option for those using Continuations and lots of idle connections.

      MINA itself doesn't seem to be affected by this problem, so it could be Jetty/JDK/OS specific.

      Attached are the screenshots from the profiler before and after the problem.

      The most common traces from this thread using 100% CPU (50% on the screenshots as it's a dual core machine):

      Stacks at 1:27:10
      21684929@qtp-7760420-1 - Acceptor0 SelectChannelConnector@0.0.0.0:8000 [RUNNABLE] CPU time: 1:22
      sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int)
      sun.nio.ch.EPollArrayWrapper.poll(long)
      sun.nio.ch.EPollSelectorImpl.doSelect(long)
      sun.nio.ch.SelectorImpl.lockAndDoSelect(long)
      sun.nio.ch.SelectorImpl.select(long)
      org.mortbay.io.nio.SelectorManager$SelectSet.doSelect()
      org.mortbay.io.nio.SelectorManager.doSelect(int)
      org.mortbay.jetty.nio.SelectChannelConnector.accept(int)
      org.mortbay.jetty.AbstractConnector$Acceptor.run()
      org.mortbay.thread.QueuedThreadPool$PoolThread.run()

      Stacks at 1:28:39
      21684929@qtp-7760420-1 - Acceptor0 SelectChannelConnector@0.0.0.0:8000 [RUNNABLE] CPU time: 2:33
      sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int)
      sun.nio.ch.EPollArrayWrapper.poll(long)
      sun.nio.ch.EPollSelectorImpl.doSelect(long)
      sun.nio.ch.SelectorImpl.lockAndDoSelect(long)
      sun.nio.ch.SelectorImpl.selectNow()
      org.mortbay.io.nio.SelectorManager$SelectSet.doSelect()
      org.mortbay.io.nio.SelectorManager.doSelect(int)
      org.mortbay.jetty.nio.SelectChannelConnector.accept(int)
      org.mortbay.jetty.AbstractConnector$Acceptor.run()
      org.mortbay.thread.QueuedThreadPool$PoolThread.run()

      Stacks at 1:36:09
      21684929@qtp-7760420-1 - Acceptor0 SelectChannelConnector@0.0.0.0:8000 [RUNNABLE] CPU time: 9:00
      sun.misc.Unsafe.getInt(long)
      sun.nio.ch.NativeObject.getInt(int)
      sun.nio.ch.EPollArrayWrapper.getDescriptor(int)
      sun.nio.ch.EPollArrayWrapper.poll(long)
      sun.nio.ch.EPollSelectorImpl.doSelect(long)
      sun.nio.ch.SelectorImpl.lockAndDoSelect(long)
      sun.nio.ch.SelectorImpl.selectNow()
      org.mortbay.io.nio.SelectorManager$SelectSet.doSelect()
      org.mortbay.io.nio.SelectorManager.doSelect(int)
      org.mortbay.jetty.nio.SelectChannelConnector.accept(int)
      org.mortbay.jetty.AbstractConnector$Acceptor.run()
      org.mortbay.thread.QueuedThreadPool$PoolThread.run()

      1. jrockit-dump.txt
        120 kB
        Daniel Peters
      2. rso-stack.txt
        125 kB
        Damian Bradicich
      1. normal.png
        23 kB
      2. problem.png
        35 kB

        Activity

          People

          • Assignee:
            Greg Wilkins
            Reporter:
            Serge Baranov
          • Votes:
            5 Vote for this issue
            Watchers:
            15 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: