Jetty
  1. Jetty
  2. JETTY-1167

GZipFilter cause OutOfMemoryError

    Details

    • Type: Bug Bug
    • Status: Resolved Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 6.1.21
    • Fix Version/s: 6.1.23
    • Component/s: HTTP
    • Labels:
      None
    • Environment:
      Windows 2003 Server
      Java 1.5.0u17
    • Number of attachments :
      0

      Description

      An OutOfMemoryError within GzipFilter. It happens randomly on a production system, but I can't reproduce it in my development environment.

      java.lang.OutOfMemoryError
      	at java.util.zip.Deflater.init(Native Method)
      	at java.util.zip.Deflater.<init>(Deflater.java:121)
      	at java.util.zip.GZIPOutputStream.<init>(GZIPOutputStream.java:46)
      	at org.mortbay.servlet.GzipFilter$GzipStream.doGzip(GzipFilter.java:525)
      	at org.mortbay.servlet.GzipFilter$GzipStream.<init>(GzipFilter.java:417)
      	at org.mortbay.servlet.GzipFilter$GZIPResponseWrapper.newGzipStream(GzipFilter.java:392)
      	at org.mortbay.servlet.GzipFilter$GZIPResponseWrapper.getOutputStream(GzipFilter.java:342)
      

      My diagnosis:
      This is actually caused by the C heap running out of memory.

      The gzip filter creates GZipOutputStream objects, but doesn't call close() as that would close the underlying socket stream. GZipOutputStream calls out to C code which uses the C heap. If GZipOutputStream objects aren't cleaned up by GC fast enough (so finalized can be called), it's possible that the C heap can run out of memory.

      This is a known issue in the JVM:
      http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4797189
      http://markmail.org/message/zflsen5q2uchpas3

      A possible solution:

      • Extend GZipOutputStream to free up C memory when finish() is called.
      • Modify doGZip() to instantiate the extended class instead of GZipOutputStream
      GZOutput.java
      class GZOutput extends GZipOutputStream {
        GZOutput(OutputStream out, int size) throws IOException { super(out, size); }
        public void finish() {
          super.finish();
          def.end();  //protected java.util.zip.Deflater
        }
      }
      

      We did something similar to the above, but we did it in a more complicated way to avoid having to modify Jetty source.

      This doesn't entirely fix the issue; finish() is never called when an exception is caught in doFilter() and isCommitted() is false. Perhaps wrappedResponse.finish() should be called in the catch, but I'm not sure.

        Activity

        Hide
        Anatoliy Salistra added a comment -

        This is a very old and well known issue. Please fix it.

        Show
        Anatoliy Salistra added a comment - This is a very old and well known issue. Please fix it.
        Hide
        Kalle Korhonen added a comment -

        This is a JVM issue. As noted in the linked JVM bug report, running with the incremental garbage collector (-incgc) fixes the issue, I can verify that. This is pure speculation but my understanding is that the new G1 collector would also fix the issue (which is going to be the default in JDK1.7) which might be why they didn't bother fixing this particular issue.

        Show
        Kalle Korhonen added a comment - This is a JVM issue. As noted in the linked JVM bug report, running with the incremental garbage collector (-incgc) fixes the issue, I can verify that. This is pure speculation but my understanding is that the new G1 collector would also fix the issue (which is going to be the default in JDK1.7) which might be why they didn't bother fixing this particular issue.
        Hide
        Greg Wilkins added a comment -

        while you are looking at memory issues, can you check this one also.

        Show
        Greg Wilkins added a comment - while you are looking at memory issues, can you check this one also.
        Hide
        Simone Bordet added a comment -

        What suggested by the reporter is not needed.
        I checked in both JDK 5 and JDK 6 and Deflater.end() is called upon stream close.
        The stream is always closed as part of stream lifecycle, and it is done by Jetty.

        So what remains is the JVM issue related to running the finalizers before the native heap runs out of space, and not much we can do about.

        Show
        Simone Bordet added a comment - What suggested by the reporter is not needed. I checked in both JDK 5 and JDK 6 and Deflater.end() is called upon stream close. The stream is always closed as part of stream lifecycle, and it is done by Jetty. So what remains is the JVM issue related to running the finalizers before the native heap runs out of space, and not much we can do about.
        Hide
        Anatoliy Salistra added a comment -

        NO-NO-NO! This is Not TRUE. Deflater.end() IS called by the stream close(), but Jetty never calls this method! That would close the underlying stream, which should never happen if HTTP wants to maintain long-term connections, as it must as of HTTP1.1. What happens is, the code keeps creating gzip wrappers around an underlying stream, and then 'finishing' them, rather than closing. Just run a code in debugger and let me know when Deflater.end() is invoked.

        Show
        Anatoliy Salistra added a comment - NO-NO-NO! This is Not TRUE. Deflater.end() IS called by the stream close(), but Jetty never calls this method! That would close the underlying stream, which should never happen if HTTP wants to maintain long-term connections, as it must as of HTTP1.1. What happens is, the code keeps creating gzip wrappers around an underlying stream, and then 'finishing' them, rather than closing. Just run a code in debugger and let me know when Deflater.end() is invoked.
        Hide
        Simone Bordet added a comment -

        I just run the code in a debugger and I do see the close() method being called.
        Do you have a reproducible test case where Jetty does not close the stream ?

        Show
        Simone Bordet added a comment - I just run the code in a debugger and I do see the close() method being called. Do you have a reproducible test case where Jetty does not close the stream ?
        Hide
        Michael Slattery added a comment -

        If a downstream filter or servlet closes the outputstream/writer then GZipOutputStream.close() method will get called. However, that's rare and not the way to write a good Servlet. In all other cases, Jetty will eventually close the underlying stream, but it has no way of closing the GZipOutputStream.

        To test, put a breakpoint on GZipFilter.GzipStream.close() (line 466ish). That's the only place where GZipOutputStream.close() is ever called in GZipFilter. If the breakpoint fires, verify that it's not because of a downstream Servlet/Filter calling close(). In that case, use a different Servlet.

        Show
        Michael Slattery added a comment - If a downstream filter or servlet closes the outputstream/writer then GZipOutputStream.close() method will get called. However, that's rare and not the way to write a good Servlet. In all other cases, Jetty will eventually close the underlying stream, but it has no way of closing the GZipOutputStream. To test, put a breakpoint on GZipFilter.GzipStream.close() (line 466ish). That's the only place where GZipOutputStream.close() is ever called in GZipFilter. If the breakpoint fires, verify that it's not because of a downstream Servlet/Filter calling close(). In that case, use a different Servlet.
        Hide
        Simone Bordet added a comment -

        This is fixed by closing the gzip stream from finish().

        Show
        Simone Bordet added a comment - This is fixed by closing the gzip stream from finish().

          People

          • Assignee:
            Simone Bordet
            Reporter:
            Michael Slattery
          • Votes:
            2 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: