IllegalStateException(SelectorManager line 352) cause CPU usage 100% !

The snippet of std output log (stdout.log):

2007-10-20 11:34:20.715::WARN: EXCEPTION
java.lang.IllegalStateException
at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:352)
at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:168)
at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:517)
at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
2007-10-20 11:34:20.715::WARN: EXCEPTION
java.lang.IllegalStateException
at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:352)
at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:168)
at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:517)
at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
2007-10-20 11:34:20.715::WARN: EXCEPTION
java.lang.IllegalStateException
at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:352)
at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:168)
at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:517)
at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
2007-10-20 11:34:20.715::WARN: EXCEPTION
java.lang.IllegalStateException
at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:352)
at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:168)
at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:517)
at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)

May be , ..., the BoundedThreadPool dispatching failed trigger the illegal state?

My configuration is (jetty.xml) :

<Configure id="Server" class="org.mortbay.jetty.Server">
<Set name="ThreadPool">
<New class="org.mortbay.thread.BoundedThreadPool">
<Set name="minThreads">50</Set>
<Set name="maxThreads">4000</Set>
<Set name="lowThreads">200</Set>
</New>
</Set>

<Call name="addConnector">
<Arg>
<New class="org.mortbay.jetty.nio.SelectChannelConnector">
<Set name="port"><SystemProperty name="jetty.port" default="80"/></Set>
<Set name="maxIdleTime">45000</Set>
<Set name="Acceptors">2</Set>
<Set name="statsOn">false</Set>
<Set name="lowResourcesConnections">20000</Set>
<Set name="lowResourcesMaxIdleTime">30000</Set>
</New>
</Arg>
</Call>

<Set name="handler">
<New id="context" class="org.mortbay.jetty.webapp.WebAppContext">
<Set name="contextPath">/</Set>
<Set name="resourceBase"><SystemProperty name="jetty.home" default="."/>/root</Set>
<Set name="defaultsDescriptor"><SystemProperty name="jetty.home" default="."/>/etc/webdefault.xml
</Set>
</New>
</Set>
<Set name="stopAtShutdown">true</Set>
<Set name="sendServerVersion">false</Set>
<Set name="sendDateHeader">true</Set>

</Configure>

Can you give me some more background information? ie what were you running, when did the exceptions start happening, what was the load? etc. etc.

Can you always reproduce?

Can you try to reproduce with java 1.5

Hopefully you can reproduce... because I think I found the cause.
There was was test code left in for a scheduling edge case - that I have now removed in svn head.
If you can reproduce, can you try checking out trunk and seeing if it is fixed.

i can only reproduce occasionally.

the client program(not browser) send a request per 20 seconds.
to save the system resources that connecting spent, i use the SelectChannelConnector and set maxIdletime to 45000 and lowResourcesMaxIdleTime to 30000.
the connection count is close to 20000.

the exception below also appears occasionally on the log file.

2007-10-22 11:05:09.672::WARN: dispatch failed!
2007-10-22 11:05:09.672::WARN: EXCEPTION
java.lang.IllegalStateException
at org.mortbay.thread.BoundedThreadPool$PoolThread.dispatch(BoundedThreadPool.java:524)
at org.mortbay.thread.BoundedThreadPool.dispatch(BoundedThreadPool.java:86)
at org.mortbay.jetty.nio.SelectChannelConnector$1.dispatch(SelectChannelConnector.java:86)
at org.mortbay.io.nio.SelectChannelEndPoint.dispatch(SelectChannelEndPoint.java:68)
at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:476)
at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:168)
at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:517)
at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)

Sometimes, cpu usage still get to 100% without logging "java.lang.IllegalStateException
at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect".

I change the underlying threadpool implementation to jdk ThreadPoolExecutor, adjust the max poolsize to 20000, It seems the jetty server run stably.

In addition, I found the doc of org.mortbay.thread.concurrent.ThreadPool has some issue.
When using an unbounded queue, the value of the maximumPoolSize doesn't have any effect. (See jdk doc)
So, my new configuration file look like this:
<New class="org.mortbay.thread.concurrent.ThreadPool">
<Arg>0</Arg>
<Set name="corePoolSize">200</Set>
<Set name="maximumPoolSize">20000</Set>
</New>

20000 is a seriously large number of threads! are you sure you have the RAM for that many stacks?

Are you still seeing this problem with the patched server?

The problem has been resolved since using the ThreadPoolExecutor.

I guess that the default threadpool(BoundedThreadPool) has some issues.

I've added a test harness for BoundedThreadPool and I can't induce any errors in that.

I don't know why the WARN appears occasionally on the log file when using BoundedThreadPool. May be synchronization caused?

After using the ThreadPoolExecutor, there is not any warn.

For the max pool size(20000) mentioned above, there are no relations with this problem, because the active thread count can't get to the max pool size on current circumstance .

The WARN IS:
2007-10-22 11:05:09.672::WARN: dispatch failed!
2007-10-22 11:05:09.672::WARN: EXCEPTION
java.lang.IllegalStateException
at org.mortbay.thread.BoundedThreadPool$PoolThread.dispatch(BoundedThreadPool.java:524)
at org.mortbay.thread.BoundedThreadPool.dispatch(BoundedThreadPool.java:86)
at org.mortbay.jetty.nio.SelectChannelConnector$1.dispatch(SelectChannelConnector.java:86)
at org.mortbay.io.nio.SelectChannelEndPoint.dispatch(SelectChannelEndPoint.java:68)
at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:476)
at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:168)
at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:517)
at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)