Jetty
  1. Jetty
  2. JETTY-937

SelectChannelConnector 100% CPU usage on Linux

    Details

    • Type: Bug Bug
    • Status: Closed Closed
    • Priority: Major Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: 6.1.15.rc5
    • Fix Version/s: 7.0.1, 6.1.22
    • Component/s: NIO
    • Labels:
      None
    • Environment:
      CentOS 5.x, 32/64-bit, 32/64-bit JDK 1.6.0_12 and older, Kernel 2.6.18-92.1.22.el5 and also older versions, Jetty 6.1.14, 6.1.15 trunk
    • Number of attachments :
      5

      Description

      The problem itself is described at http://markmail.org/message/zudwd2ve5l2qijbw . There has been no reply more than a month.
      We are having exactly the same problem in a similar environment.

      It can be caused by various JDK/NIO bugs which are still not fixed in the JDK 1.6.0_12, like: http://bugs.sun.com/view_bug.do?bug_id=6693490 .

      At some point Acceptor thread starts to use 100% CPU and doesn't exit this state until application is restarted. It appears that it's spinning in the tight loop. Some other applications provide workaround for this problem like recycling the acceptor and canceling the keys when it happens, so probably the same approach should be implemented in Jetty. I'm not a big expert in this area.

      I've contacted Chris directly and he told me that they are still having the same problem and found no solutions/workarounds. Interesting thing is that our applications have some common components: they both use Jetty and MINA framework in the same application, so it can be related as MINA has its own acceptors which may somehow affect Jetty NIO acceptors. MINA versions are 1.x and 2.x, so could be that the problem is not related to the MINA as well.

      In our application we had to switch Jetty to the old SocketConnector instead of the SelectChannelConnector. This fixed the problem, but it may be not an option for those using Continuations and lots of idle connections.

      MINA itself doesn't seem to be affected by this problem, so it could be Jetty/JDK/OS specific.

      Attached are the screenshots from the profiler before and after the problem.

      The most common traces from this thread using 100% CPU (50% on the screenshots as it's a dual core machine):

      Stacks at 1:27:10
      21684929@qtp-7760420-1 - Acceptor0 SelectChannelConnector@0.0.0.0:8000 [RUNNABLE] CPU time: 1:22
      sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int)
      sun.nio.ch.EPollArrayWrapper.poll(long)
      sun.nio.ch.EPollSelectorImpl.doSelect(long)
      sun.nio.ch.SelectorImpl.lockAndDoSelect(long)
      sun.nio.ch.SelectorImpl.select(long)
      org.mortbay.io.nio.SelectorManager$SelectSet.doSelect()
      org.mortbay.io.nio.SelectorManager.doSelect(int)
      org.mortbay.jetty.nio.SelectChannelConnector.accept(int)
      org.mortbay.jetty.AbstractConnector$Acceptor.run()
      org.mortbay.thread.QueuedThreadPool$PoolThread.run()

      Stacks at 1:28:39
      21684929@qtp-7760420-1 - Acceptor0 SelectChannelConnector@0.0.0.0:8000 [RUNNABLE] CPU time: 2:33
      sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int)
      sun.nio.ch.EPollArrayWrapper.poll(long)
      sun.nio.ch.EPollSelectorImpl.doSelect(long)
      sun.nio.ch.SelectorImpl.lockAndDoSelect(long)
      sun.nio.ch.SelectorImpl.selectNow()
      org.mortbay.io.nio.SelectorManager$SelectSet.doSelect()
      org.mortbay.io.nio.SelectorManager.doSelect(int)
      org.mortbay.jetty.nio.SelectChannelConnector.accept(int)
      org.mortbay.jetty.AbstractConnector$Acceptor.run()
      org.mortbay.thread.QueuedThreadPool$PoolThread.run()

      Stacks at 1:36:09
      21684929@qtp-7760420-1 - Acceptor0 SelectChannelConnector@0.0.0.0:8000 [RUNNABLE] CPU time: 9:00
      sun.misc.Unsafe.getInt(long)
      sun.nio.ch.NativeObject.getInt(int)
      sun.nio.ch.EPollArrayWrapper.getDescriptor(int)
      sun.nio.ch.EPollArrayWrapper.poll(long)
      sun.nio.ch.EPollSelectorImpl.doSelect(long)
      sun.nio.ch.SelectorImpl.lockAndDoSelect(long)
      sun.nio.ch.SelectorImpl.selectNow()
      org.mortbay.io.nio.SelectorManager$SelectSet.doSelect()
      org.mortbay.io.nio.SelectorManager.doSelect(int)
      org.mortbay.jetty.nio.SelectChannelConnector.accept(int)
      org.mortbay.jetty.AbstractConnector$Acceptor.run()
      org.mortbay.thread.QueuedThreadPool$PoolThread.run()

      1. jrockit-dump.txt
        120 kB
        Daniel Peters
      2. rso-stack.txt
        125 kB
        Damian Bradicich
      1. normal.png
        23 kB
      2. problem.png
        35 kB

        Activity

        Hide
        Charles Oliver Nutter added a comment -

        This is not going to make it into 1.6.3, but we'll need to investigate a bit more. Does it sound (to those of you who have done some investigation) like this is a problem in JRuby or a problem outside of JRuby?

        Show
        Charles Oliver Nutter added a comment - This is not going to make it into 1.6.3, but we'll need to investigate a bit more. Does it sound (to those of you who have done some investigation) like this is a problem in JRuby or a problem outside of JRuby?
        Hide
        Diego Plentz added a comment -

        Since this issue has more then 2 years and nobody could provide a reproducible way of hitting this bug(and it's probably jvm related), I think we should close it as "cannot reproduce".

        Show
        Diego Plentz added a comment - Since this issue has more then 2 years and nobody could provide a reproducible way of hitting this bug(and it's probably jvm related), I think we should close it as "cannot reproduce".
        Hide
        Christoph Rueger added a comment -

        Even though this issue is closed I would like to post an update here, as this bug hits me as well.
        When I start my application it just takes a few (<50) requests. I can just hit refresh in my browser for some minutes and baaam the app does not respond anymore and I have a java process with 100% CPU running.

        Environment:
        java version "1.6.0_26"
        Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
        Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

        Debian 6.0.2 Linux 2.6.32-5-xen-amd64 x86_64

        Embedded Jetty in Eclipse Equinox as part of the Eclipse Gyrex Project.
        I have the following jetty related bundles running:

        org.eclipse.equinox.http.jetty_2.0.100.v20110502
        org.eclipse.gyrex.admin.ui.http.jetty_1.0.0.v20110907-2000
        org.eclipse.gyrex.http.jetty_1.0.0.v20110923-1955
        org.eclipse.jetty.client_7.4.2.v20110526
        org.eclipse.jetty.continuation_7.4.2.v20110526
        org.eclipse.jetty.http_7.4.2.v20110526
        org.eclipse.jetty.io_7.4.2.v20110526
        org.eclipse.jetty.rewrite_7.4.2.v20110526
        org.eclipse.jetty.security_7.4.2.v20110526
        org.eclipse.jetty.server_7.4.2.v20110526
        org.eclipse.jetty.servlet_7.4.2.v20110526
        org.eclipse.jetty.servlets_7.4.2.v20110526
        org.eclipse.jetty.util_7.4.2.v20110526
        

        And I also see stack traces from a thread dump which look similar to the ones presented here:

        "jetty-server-admin-71 Acceptor0 SelectChannelConnector@0.0.0.0:3110 STARTED" prio=10 tid=0x0000000041c41000 nid=0x451c runnable [0x00007f84fe7e3000]
           java.lang.Thread.State: RUNNABLE
        	at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
        	- locked <0x0000000780a1ddf0> (a java.lang.Object)
        	at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:92)
        	at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:830)
        	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
        	at java.lang.Thread.run(Thread.java:662)
        
        "jetty-server-admin-70 Selector0 SelectChannelConnector@0.0.0.0:3110 STARTED" prio=10 tid=0x000000004156c800 nid=0x451b runnable [0x00007f84fe8e4000]
           java.lang.Thread.State: RUNNABLE
        	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
        	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
        	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
        	- locked <0x0000000780a1d948> (a sun.nio.ch.Util$2)
        	- locked <0x0000000780a1d938> (a java.util.Collections$UnmodifiableSet)
        	- locked <0x0000000780a1d710> (a sun.nio.ch.EPollSelectorImpl)
        	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
        	at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:504)
        	at org.eclipse.jetty.io.nio.SelectorManager.doSelect(SelectorManager.java:228)
        	at org.eclipse.jetty.server.nio.SelectChannelConnector$1.run(SelectChannelConnector.java:268)
        	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
        	at java.lang.Thread.run(Thread.java:662)
        
        "jetty-server-69" prio=10 tid=0x0000000041710000 nid=0x451a waiting on condition [0x00007f84fe9e5000]
           java.lang.Thread.State: TIMED_WAITING (parking)
        	at sun.misc.Unsafe.park(Native Method)
        	- parking to wait for  <0x000000078099ea28> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
        	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
        	at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
        	at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
        	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
        	at java.lang.Thread.run(Thread.java:662)
        
        "jetty-server-68" prio=10 tid=0x0000000042c18000 nid=0x4519 waiting on condition [0x00007f84feae6000]
           java.lang.Thread.State: TIMED_WAITING (parking)
        	at sun.misc.Unsafe.park(Native Method)
        	- parking to wait for  <0x000000078099ea28> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
        	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
        	at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
        	at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
        	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
        	at java.lang.Thread.run(Thread.java:662)
        
        "jetty-server-66" prio=10 tid=0x00007f850086b000 nid=0x4517 waiting on condition [0x00007f84fece8000]
           java.lang.Thread.State: TIMED_WAITING (parking)
        	at sun.misc.Unsafe.park(Native Method)
        	- parking to wait for  <0x000000078099ea28> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
        	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
        	at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
        	at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
        	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
        	at java.lang.Thread.run(Thread.java:662)
        
        "jetty-server-65 Selector1 SelectChannelConnector@0.0.0.0:8081 STARTED" prio=10 tid=0x00007f8500b69800 nid=0x4516 runnable [0x00007f84fede9000]
           java.lang.Thread.State: RUNNABLE
        	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
        	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
        	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
        	- locked <0x0000000780986678> (a sun.nio.ch.Util$2)
        	- locked <0x0000000780986668> (a java.util.Collections$UnmodifiableSet)
        	- locked <0x0000000780986000> (a sun.nio.ch.EPollSelectorImpl)
        	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
        	at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:504)
        	at org.eclipse.jetty.io.nio.SelectorManager.doSelect(SelectorManager.java:228)
        	at org.eclipse.jetty.server.nio.SelectChannelConnector$1.run(SelectChannelConnector.java:268)
        	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
        	at java.lang.Thread.run(Thread.java:662)
        
        "jetty-server-64 Selector0 SelectChannelConnector@0.0.0.0:8081 STARTED" prio=10 tid=0x00007f850028f800 nid=0x4515 runnable [0x00007f84feeea000]
           java.lang.Thread.State: RUNNABLE
        	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
        	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
        	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
        	- locked <0x000000078099e728> (a sun.nio.ch.Util$2)
        	- locked <0x000000078099e718> (a java.util.Collections$UnmodifiableSet)
        	- locked <0x000000078097fc08> (a sun.nio.ch.EPollSelectorImpl)
        	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
        	at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:504)
        	at org.eclipse.jetty.io.nio.SelectorManager.doSelect(SelectorManager.java:228)
        	at org.eclipse.jetty.server.nio.SelectChannelConnector$1.run(SelectChannelConnector.java:268)
        	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
        	at java.lang.Thread.run(Thread.java:662)
        
        "jetty-server-63 Acceptor1 SelectChannelConnector@0.0.0.0:8081 STARTED" prio=10 tid=0x00007f850093b000 nid=0x4514 runnable [0x00007f84fefeb000]
           java.lang.Thread.State: RUNNABLE
        	at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
        	- locked <0x000000078097f1a0> (a java.lang.Object)
        	at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:92)
        	at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:830)
        	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
        	at java.lang.Thread.run(Thread.java:662)
        
        "jetty-server-62 Acceptor0 SelectChannelConnector@0.0.0.0:8081 STARTED" prio=10 tid=0x00007f8500536000 nid=0x4513 waiting for monitor entry [0x00007f84ff0ec000]
           java.lang.Thread.State: BLOCKED (on object monitor)
        	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:136)
        	- waiting to lock <0x000000078097f1a0> (a java.lang.Object)
        	at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:92)
        	at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:830)
        	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
        	at java.lang.Thread.run(Thread.java:662)
        

        Any more thoughts ideas on this?

        Show
        Christoph Rueger added a comment - Even though this issue is closed I would like to post an update here, as this bug hits me as well. When I start my application it just takes a few (<50) requests. I can just hit refresh in my browser for some minutes and baaam the app does not respond anymore and I have a java process with 100% CPU running. Environment: java version "1.6.0_26" Java(TM) SE Runtime Environment (build 1.6.0_26-b03) Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode) Debian 6.0.2 Linux 2.6.32-5-xen-amd64 x86_64 Embedded Jetty in Eclipse Equinox as part of the Eclipse Gyrex Project. I have the following jetty related bundles running: org.eclipse.equinox.http.jetty_2.0.100.v20110502 org.eclipse.gyrex.admin.ui.http.jetty_1.0.0.v20110907-2000 org.eclipse.gyrex.http.jetty_1.0.0.v20110923-1955 org.eclipse.jetty.client_7.4.2.v20110526 org.eclipse.jetty.continuation_7.4.2.v20110526 org.eclipse.jetty.http_7.4.2.v20110526 org.eclipse.jetty.io_7.4.2.v20110526 org.eclipse.jetty.rewrite_7.4.2.v20110526 org.eclipse.jetty.security_7.4.2.v20110526 org.eclipse.jetty.server_7.4.2.v20110526 org.eclipse.jetty.servlet_7.4.2.v20110526 org.eclipse.jetty.servlets_7.4.2.v20110526 org.eclipse.jetty.util_7.4.2.v20110526 And I also see stack traces from a thread dump which look similar to the ones presented here: "jetty-server-admin-71 Acceptor0 SelectChannelConnector@0.0.0.0:3110 STARTED" prio=10 tid=0x0000000041c41000 nid=0x451c runnable [0x00007f84fe7e3000] java.lang. Thread .State: RUNNABLE at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152) - locked <0x0000000780a1ddf0> (a java.lang. Object ) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:92) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:830) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529) at java.lang. Thread .run( Thread .java:662) "jetty-server-admin-70 Selector0 SelectChannelConnector@0.0.0.0:3110 STARTED" prio=10 tid=0x000000004156c800 nid=0x451b runnable [0x00007f84fe8e4000] java.lang. Thread .State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x0000000780a1d948> (a sun.nio.ch.Util$2) - locked <0x0000000780a1d938> (a java.util.Collections$UnmodifiableSet) - locked <0x0000000780a1d710> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:504) at org.eclipse.jetty.io.nio.SelectorManager.doSelect(SelectorManager.java:228) at org.eclipse.jetty.server.nio.SelectChannelConnector$1.run(SelectChannelConnector.java:268) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529) at java.lang. Thread .run( Thread .java:662) "jetty-server-69" prio=10 tid=0x0000000041710000 nid=0x451a waiting on condition [0x00007f84fe9e5000] java.lang. Thread .State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x000000078099ea28> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558) at java.lang. Thread .run( Thread .java:662) "jetty-server-68" prio=10 tid=0x0000000042c18000 nid=0x4519 waiting on condition [0x00007f84feae6000] java.lang. Thread .State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x000000078099ea28> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558) at java.lang. Thread .run( Thread .java:662) "jetty-server-66" prio=10 tid=0x00007f850086b000 nid=0x4517 waiting on condition [0x00007f84fece8000] java.lang. Thread .State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x000000078099ea28> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558) at java.lang. Thread .run( Thread .java:662) "jetty-server-65 Selector1 SelectChannelConnector@0.0.0.0:8081 STARTED" prio=10 tid=0x00007f8500b69800 nid=0x4516 runnable [0x00007f84fede9000] java.lang. Thread .State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x0000000780986678> (a sun.nio.ch.Util$2) - locked <0x0000000780986668> (a java.util.Collections$UnmodifiableSet) - locked <0x0000000780986000> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:504) at org.eclipse.jetty.io.nio.SelectorManager.doSelect(SelectorManager.java:228) at org.eclipse.jetty.server.nio.SelectChannelConnector$1.run(SelectChannelConnector.java:268) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529) at java.lang. Thread .run( Thread .java:662) "jetty-server-64 Selector0 SelectChannelConnector@0.0.0.0:8081 STARTED" prio=10 tid=0x00007f850028f800 nid=0x4515 runnable [0x00007f84feeea000] java.lang. Thread .State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x000000078099e728> (a sun.nio.ch.Util$2) - locked <0x000000078099e718> (a java.util.Collections$UnmodifiableSet) - locked <0x000000078097fc08> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:504) at org.eclipse.jetty.io.nio.SelectorManager.doSelect(SelectorManager.java:228) at org.eclipse.jetty.server.nio.SelectChannelConnector$1.run(SelectChannelConnector.java:268) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529) at java.lang. Thread .run( Thread .java:662) "jetty-server-63 Acceptor1 SelectChannelConnector@0.0.0.0:8081 STARTED" prio=10 tid=0x00007f850093b000 nid=0x4514 runnable [0x00007f84fefeb000] java.lang. Thread .State: RUNNABLE at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152) - locked <0x000000078097f1a0> (a java.lang. Object ) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:92) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:830) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529) at java.lang. Thread .run( Thread .java:662) "jetty-server-62 Acceptor0 SelectChannelConnector@0.0.0.0:8081 STARTED" prio=10 tid=0x00007f8500536000 nid=0x4513 waiting for monitor entry [0x00007f84ff0ec000] java.lang. Thread .State: BLOCKED (on object monitor) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:136) - waiting to lock <0x000000078097f1a0> (a java.lang. Object ) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:92) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:830) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529) at java.lang. Thread .run( Thread .java:662) Any more thoughts ideas on this?
        Hide
        Diego Plentz added a comment -

        Just an update: jetty 7 has a lot of improvements handling closed connections. If you have mobile clients that drop connections, it's likely that you hit this bug. Updating to jetty 7.5.2(or the to-be-released 7.5.3) can help.

        Related: JETTY-1441

        Show
        Diego Plentz added a comment - Just an update: jetty 7 has a lot of improvements handling closed connections. If you have mobile clients that drop connections, it's likely that you hit this bug. Updating to jetty 7.5.2(or the to-be-released 7.5.3) can help. Related: JETTY-1441
        Show
        Diego Plentz added a comment - Commits that may helped to fix this: @7.5.2 https://github.com/eclipse/jetty.project/commit/633f3b15886664e116478145a3e0139fc8f89847 https://github.com/eclipse/jetty.project/commit/d0a2557527ed7d660ee22f10741ec8d245008ede @7.5.3 https://github.com/eclipse/jetty.project/commit/ea56eaff005e6b14c3f3464adf95e1e13742df68 https://github.com/eclipse/jetty.project/commit/06f4ada935de096869177cfd60e62a667a14f321 https://github.com/eclipse/jetty.project/commit/3476887f8a1e2281d9ec7ea85a0d3f4dd5017cc8 So, if you hit this bug, try updating to 7.5.3 and it will (probably) be solved.

          People

          • Assignee:
            Greg Wilkins
            Reporter:
            Serge Baranov
          • Votes:
            5 Vote for this issue
            Watchers:
            15 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: