Hung threads consuming 100% CPU with SSL enabled and IE 8 as client

Problem:

Hung threads consume 100% of CPU (and cause load averages of 40-50 on my system) when SSL-enabled Jetty 7.4.5 gets hit by an IE 8 browser. Same problem does not occur with Chrome 14.0.835.186.

Thread dump of a sample hung thread:

"qtp52888755-307799" prio=10 tid=0x00002aaabb96b000 nid=0xe18 runnable [0x000000004f4db000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)

• locked <0x000000050cbd8580> (a java.lang.Object)
  at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:755)
  at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
• locked <0x000000050cbd8620> (a com.sun.net.ssl.internal.ssl.AppInputStream)
  at org.eclipse.jetty.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:388)
  at org.eclipse.jetty.io.bio.StreamEndPoint.fill(StreamEndPoint.java:132)
  at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.fill(SocketConnector.java:209)
  at org.eclipse.jetty.server.ssl.SslSocketConnector$SslConnectorEndPoint.fill(SslSocketConnector.java:612)
  at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:289)
  at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:214)
  at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:411)
  at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:241)
  at org.eclipse.jetty.server.ssl.SslSocketConnector$SslConnectorEndPoint.run(SslSocketConnector.java:664)
  at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
  at java.lang.Thread.run(Thread.java:662)

Steps I took to reproduce:

Setting up the test environment for Jetty 7.4.0

• Download and install the opennms-core and opennms-webapp-jetty packages for OpenNMS 1.9.91 (http://sourceforge.net/projects/opennms/files/OpenNMS/unstable-1.9.91/; a PostgreSQL DB backend is also required)

• Configure OpenNMS to use SSL (http://www.opennms.org/wiki/Standalone_HTTPS_with_Jetty)

• Start OpenNMS

Reproducing the bug:

• Hit either /opennms or /opennms-remoting from an IE 8 browser, and do nothing else (don't even need to log in)

• Wait and watch the load average start climbing from hung Jetty SSL threads. The hung threads seem to accumulate faster if more concurrent connections are opened.

Setting up the test environment for Jetty 7.4.5

• Follow above installation steps

• Relocate /opt/opennms/lib/jetty*7.4.0*.jar and /opt/opennms/lib/servlet*.jar somewhere else

• Download Jetty 7.4.5 (http://dist.codehaus.org/jetty/jetty-hightide-7.4.5/jetty-hightide-7.4.5.v20110725.zip), unzip, and copy jetty*7.4.5*.jar and servlet*.jar from the lib/ directory into /opt/opennms/lib

• (Re)start OpenNMS

• Perform the same reproduction test as above

FYI, I have a parallel bug open with the OpenNMS team (http://issues.opennms.org/browse/NMS-4940) with some more details and thread dumps, but this is smelling more and more like a Jetty bug, so the team pointed me here.

Let me know if I can provide any more information to help. Thanks!