Jetty
  1. Jetty
  2. JETTY-1435

Hung threads consuming 100% CPU with SSL enabled and IE 8 as client

    Details

    • Type: Bug Bug
    • Status: Resolved Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 7.4.0, 7.4.5
    • Fix Version/s: 7.5.1
    • Component/s: Security and SSL
    • Labels:
      None
    • Environment:
      CentOS 5.6 x64
      Oracle JDK 6u26 x64
      Webapp running on top of Jetty: OpenNMS 1.9.91 (bundled with Jetty 7.4.0, also tested with 7.4.5)
      Client browser - IE 8.0.7600.16385
    • Number of attachments :
      0

      Description

      Problem:

      Hung threads consume 100% of CPU (and cause load averages of 40-50 on my system) when SSL-enabled Jetty 7.4.5 gets hit by an IE 8 browser. Same problem does not occur with Chrome 14.0.835.186.

      Thread dump of a sample hung thread:

      "qtp52888755-307799" prio=10 tid=0x00002aaabb96b000 nid=0xe18 runnable [0x000000004f4db000]
      java.lang.Thread.State: RUNNABLE
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:129)
      at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
      at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
      at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)

      • locked <0x000000050cbd8580> (a java.lang.Object)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:755)
        at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
      • locked <0x000000050cbd8620> (a com.sun.net.ssl.internal.ssl.AppInputStream)
        at org.eclipse.jetty.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:388)
        at org.eclipse.jetty.io.bio.StreamEndPoint.fill(StreamEndPoint.java:132)
        at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.fill(SocketConnector.java:209)
        at org.eclipse.jetty.server.ssl.SslSocketConnector$SslConnectorEndPoint.fill(SslSocketConnector.java:612)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:289)
        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:214)
        at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:411)
        at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:241)
        at org.eclipse.jetty.server.ssl.SslSocketConnector$SslConnectorEndPoint.run(SslSocketConnector.java:664)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
        at java.lang.Thread.run(Thread.java:662)

      Steps I took to reproduce:

      Setting up the test environment for Jetty 7.4.0

      • Start OpenNMS

      Reproducing the bug:

      • Hit either /opennms or /opennms-remoting from an IE 8 browser, and do nothing else (don't even need to log in)
      • Wait and watch the load average start climbing from hung Jetty SSL threads. The hung threads seem to accumulate faster if more concurrent connections are opened.

      Setting up the test environment for Jetty 7.4.5

      • Follow above installation steps
      • Relocate /opt/opennms/lib/jetty*7.4.0*.jar and /opt/opennms/lib/servlet*.jar somewhere else
      • (Re)start OpenNMS
      • Perform the same reproduction test as above

      FYI, I have a parallel bug open with the OpenNMS team (http://issues.opennms.org/browse/NMS-4940) with some more details and thread dumps, but this is smelling more and more like a Jetty bug, so the team pointed me here.

      Let me know if I can provide any more information to help. Thanks!

        Activity

        Hide
        Joakim Erdfelt added a comment -

        btw, a very handy blog entry on identifying a thread with high cpu usage.
        http://www.onkarjoshi.com/blog/166/identifying-a-thread-with-high-cpu-usage-in-a-java-app/

        (added here for reference at a later date)

        Show
        Joakim Erdfelt added a comment - btw, a very handy blog entry on identifying a thread with high cpu usage. http://www.onkarjoshi.com/blog/166/identifying-a-thread-with-high-cpu-usage-in-a-java-app/ (added here for reference at a later date)
        Hide
        Andy Ellsworth added a comment -

        Confirmed - OpenNMS 1.9.91 + Jetty 7.5.1 = no bug. Thanks for your diligence.

        I'll suggest to the OpenNMS team that they migrate to 7.5.1 since it seems to work fine there. Out of curiosity, is the 7.4 line still being developed?

        Show
        Andy Ellsworth added a comment - Confirmed - OpenNMS 1.9.91 + Jetty 7.5.1 = no bug. Thanks for your diligence. I'll suggest to the OpenNMS team that they migrate to 7.5.1 since it seems to work fine there. Out of curiosity, is the 7.4 line still being developed?
        Hide
        Greg Wilkins added a comment -

        7.4 is not being developed and 7.5 is a pretty natural upgrade path, with only a few uncommonly used API changes.

        Show
        Greg Wilkins added a comment - 7.4 is not being developed and 7.5 is a pretty natural upgrade path, with only a few uncommonly used API changes.
        Hide
        Andy Ellsworth added a comment -

        Sounds reasonable. I'll let you gentlemen close the bug with whatever resolved-in version you see fit. Thanks for the assist.

        Show
        Andy Ellsworth added a comment - Sounds reasonable. I'll let you gentlemen close the bug with whatever resolved-in version you see fit. Thanks for the assist.
        Hide
        Joakim Erdfelt added a comment -

        Fixed at some point between jetty 7.4.0 and 7.5.1.
        Exact cause of bug in 7.4.0 still not known.

        Show
        Joakim Erdfelt added a comment - Fixed at some point between jetty 7.4.0 and 7.5.1. Exact cause of bug in 7.4.0 still not known.

          People

          • Assignee:
            Joakim Erdfelt
            Reporter:
            Andy Ellsworth
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: