Jetty
  1. Jetty
  2. JETTY-1438

Sockets are not getting closed (likely introduced in #JETTY-547)

    Details

    • Type: Bug Bug
    • Status: Resolved Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: 6.1.26
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:
      Tested on Mac OS X (10.6 and 10.7) and Linux (various Ubuntu systems).
    • Number of attachments :
      0

      Description

      Application crashes under load with "Too many open files" after several hours. Application is written in Clojure using Ring (ring-jetty-adapter). I managed to track it down to a jetty problem using binary search over jetty versions 6.1.14 to 6.1.26:

      6.1.26 - fails
      6.1.25 - OK
      6.1.23 - OK
      6.1.20 - OK
      6.1.14 - OK

      Reading the release logs the likely culprit is #JETTY-547 (Jetty should rely on socket.shutdownOutput() to close sockets).

      The symptoms are reproducible after 1-3 hours of stress testing (4-6 simultaneous requester scripts banging on the app) on my Mac OS X system and after 8-12 hours on a Linux box. Investigating with YourKit shows that sockets are NOT being closed and some remain in open state until file descriptors are exhausted. Interestingly enough, it isn't all sockets that remain open, just some, in batches, it seems. This is why even on my Mac system (limited to 256 fds per process) it takes hours of stress testing to discover the problem.

      Netstat shows that no sockets linger in TIME_WAIT.

      My current JVM on the Mac is

      java version "1.6.0_26"
      Java(TM) SE Runtime Environment (build 1.6.0_26-b03-383-11A511)
      Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02-383, mixed mode)

      but we first noticed the problem in April (after upgrading ring to a newer version, which bumped jetty version from 6.1.14 to 6.1.26) and the JVMs might have been different then.

      I marked the bug as MAJOR because it's a showstopper for us we can't use jetty 6.1.26 because of it.

        Issue Links

          Activity

          Hide
          Mark Triggs added a comment -

          Just a quick note that I hit this same issue in ring-jetty-adapter and debugged it separately My (duplicate) issue is JETTY-1458.

          Show
          Mark Triggs added a comment - Just a quick note that I hit this same issue in ring-jetty-adapter and debugged it separately My (duplicate) issue is JETTY-1458 .

            People

            • Assignee:
              Greg Wilkins
              Reporter:
              Jan Rychter
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: