Details

    • Type: Bug Bug
    • Status: Resolved Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 6.0.0beta8, 6.0.0beta9, 6.0.0beta10, 6.0.0beta11, 6.0.0beta12, 6.0.0beta14, 6.0.0beta15, 6.0.0RC0
    • Fix Version/s: 6.0.0rc1
    • Component/s: HTTP, NIO
    • Labels:
      None
    • Environment:
      FreeBSD 5.4-RELEASE-p3
      Java HotSpot(TM) Server VM (build diablo-1.5.0_06-b00, mixed mode)
    • Number of attachments :
      4

      Description

      New Jetty, apparently, leaks socket descriptors. When running unattended, Jetty would take all available OS descriptors, leading to the "Too many open files in system" (ENFILE?) error affecting the whole system. (There were no such problem with Jetty5).

      root@alf.bizlink.ru:~/# date; sockstat | grep 15379 | wc -l; netstat -p tcp -n | grep 80.83.180.18.80 | wc -l; fstat -p 15379 | grep tcp | wc -l
      Thu May 25 11:21:29 CEST 2006
      30
      111
      50
      root@alf.bizlink.ru:~/# date; sockstat | grep 15379 | wc -l; netstat -p tcp -n | grep 80.83.180.18.80 | wc -l; fstat -p 15379 | grep tcp | wc -l
      Thu May 25 11:27:42 CEST 2006
      90
      221
      180
      root@alf.bizlink.ru:~/# date; sockstat | grep 15379 | wc -l; netstat -p tcp -n | grep 80.83.180.18.80 | wc -l; fstat -p 15379 | grep tcp | wc -l
      Thu May 25 11:34:42 CEST 2006
      102
      203
      306
      root@alf.bizlink.ru:~/# date; sockstat | grep 15379 | wc -l; netstat -p tcp -n | grep 80.83.180.18.80 | wc -l; fstat -p 15379 | grep tcp | wc -l
      Thu May 25 11:43:03 CEST 2006
      109
      326
      446
      root@alf.bizlink.ru:~/# date; sockstat | grep 15379 | wc -l; netstat -p tcp -n | grep 80.83.180.18.80 | wc -l; fstat -p 15379 | grep tcp | wc -l
      Thu May 25 11:50:05 CEST 2006
      118
      299
      544
      root@alf.bizlink.ru:~/# date; sockstat | grep 15379 | wc -l; netstat -p tcp -n | grep 80.83.180.18.80 | wc -l; fstat -p 15379 | grep tcp | wc -l
      Thu May 25 12:00:57 CEST 2006
      103
      187
      677
      root@alf.bizlink.ru:~/# date; sockstat | grep 15379 | wc -l; netstat -p tcp -n | grep 80.83.180.18.80 | wc -l; fstat -p 15379 | grep tcp | wc -l
      Thu May 25 12:19:59 CEST 2006
      105
      305
      987

      Here we see, that amount of open sockets (sockstat) remain roughly the same, amount of TCP/IP stack entries (netstat) - too, but the amount of socket descriptors used by the process (fstat | grep tcp) - increases.

      The leak was present at the time of beta8 till today (last tested was the SVN version of Jetty from 2006-05-25).
      I can give JVMDI (java debugger) access to the machine to a specific IP.

        Activity

        Hide
        Greg Wilkins added a comment -

        Artem,

        The single 0 is OK - it is a zero chunk and indicates no content. Although generally speaking
        I prefer if jetty was to send back a content-length: 0 instead of an empty chunk.

        But the strange this is that there is a WWW-Authenticate challenge in a 302 response?
        This should be a 401 response?

        What is your security constraint configuration?
        this really should be a new issue! Can you open a new issue and attach your web.xml

        Show
        Greg Wilkins added a comment - Artem, The single 0 is OK - it is a zero chunk and indicates no content. Although generally speaking I prefer if jetty was to send back a content-length: 0 instead of an empty chunk. But the strange this is that there is a WWW-Authenticate challenge in a 302 response? This should be a 401 response? What is your security constraint configuration? this really should be a new issue! Can you open a new issue and attach your web.xml
        Hide
        Greg Wilkins added a comment -

        YIPEEEEE!!!!!!!!!!!!!!!!!! fixed!

        The issue was that ab (and I expect other clients) sends a strange ack for the last data packet... which causes
        the server to reset the connection rather than close it gracefully. This caused socket.shutdownOutput()
        to throw an alreadyShutdown exception, which caused the calls to socket.close() and channel.close() to be missed!

        The reason the socket.shutdownOutput and socket.close are there is because a pure channel.close() does
        not appear to allow the socket to close quickly and there is a big delay injected.

        So the fix is to protect all of these with finally calls, so that all three methods will always be called.
        Obvious with 20:20 hindsight!

        Thanks all for your perseverence in helping me find this one and thanks Christian for giving me access to
        his machine.

        Show
        Greg Wilkins added a comment - YIPEEEEE!!!!!!!!!!!!!!!!!! fixed! The issue was that ab (and I expect other clients) sends a strange ack for the last data packet... which causes the server to reset the connection rather than close it gracefully. This caused socket.shutdownOutput() to throw an alreadyShutdown exception, which caused the calls to socket.close() and channel.close() to be missed! The reason the socket.shutdownOutput and socket.close are there is because a pure channel.close() does not appear to allow the socket to close quickly and there is a big delay injected. So the fix is to protect all of these with finally calls, so that all three methods will always be called. Obvious with 20:20 hindsight! Thanks all for your perseverence in helping me find this one and thanks Christian for giving me access to his machine.
        Hide
        Artem Kozarezov added a comment -

        Cool.

        Show
        Artem Kozarezov added a comment - Cool.
        Hide
        Kirill Lastovirya added a comment -

        was the issue solved? I see very similar behavior on jetty versions starting from 6.1.16
        can I create an issue, I can provide detailed description

        Show
        Kirill Lastovirya added a comment - was the issue solved? I see very similar behavior on jetty versions starting from 6.1.16 can I create an issue, I can provide detailed description
        Hide
        Jesse McConnell added a comment -

        yes this issue was resolved ages ago

        if your having a similar problem then please do open a new issue and provide details

        Show
        Jesse McConnell added a comment - yes this issue was resolved ages ago if your having a similar problem then please do open a new issue and provide details

          People

          • Assignee:
            Greg Wilkins
            Reporter:
            Artem Kozarezov
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: