Jetty

GZipFilter cause OutOfMemoryError

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: 6.1.21
  • Fix Version/s: 6.1.23
  • Component/s: HTTP
  • Labels:
    None
  • Environment:
    Windows 2003 Server
    Java 1.5.0u17
  • Number of attachments :
    0

Description

An OutOfMemoryError within GzipFilter. It happens randomly on a production system, but I can't reproduce it in my development environment.

java.lang.OutOfMemoryError
	at java.util.zip.Deflater.init(Native Method)
	at java.util.zip.Deflater.<init>(Deflater.java:121)
	at java.util.zip.GZIPOutputStream.<init>(GZIPOutputStream.java:46)
	at org.mortbay.servlet.GzipFilter$GzipStream.doGzip(GzipFilter.java:525)
	at org.mortbay.servlet.GzipFilter$GzipStream.<init>(GzipFilter.java:417)
	at org.mortbay.servlet.GzipFilter$GZIPResponseWrapper.newGzipStream(GzipFilter.java:392)
	at org.mortbay.servlet.GzipFilter$GZIPResponseWrapper.getOutputStream(GzipFilter.java:342)

My diagnosis:
This is actually caused by the C heap running out of memory.

The gzip filter creates GZipOutputStream objects, but doesn't call close() as that would close the underlying socket stream. GZipOutputStream calls out to C code which uses the C heap. If GZipOutputStream objects aren't cleaned up by GC fast enough (so finalized can be called), it's possible that the C heap can run out of memory.

This is a known issue in the JVM:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4797189
http://markmail.org/message/zflsen5q2uchpas3

A possible solution:

  • Extend GZipOutputStream to free up C memory when finish() is called.
  • Modify doGZip() to instantiate the extended class instead of GZipOutputStream
GZOutput.java
class GZOutput extends GZipOutputStream {
  GZOutput(OutputStream out, int size) throws IOException { super(out, size); }
  public void finish() {
    super.finish();
    def.end();  //protected java.util.zip.Deflater
  }
}

We did something similar to the above, but we did it in a more complicated way to avoid having to modify Jetty source.

This doesn't entirely fix the issue; finish() is never called when an exception is caught in doFilter() and isCommitted() is false. Perhaps wrappedResponse.finish() should be called in the catch, but I'm not sure.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Anatoliy Salistra added a comment -

This is a very old and well known issue. Please fix it.

Show
Anatoliy Salistra added a comment - This is a very old and well known issue. Please fix it.
Hide
Permalink
Kalle Korhonen added a comment -

This is a JVM issue. As noted in the linked JVM bug report, running with the incremental garbage collector (-incgc) fixes the issue, I can verify that. This is pure speculation but my understanding is that the new G1 collector would also fix the issue (which is going to be the default in JDK1.7) which might be why they didn't bother fixing this particular issue.

Show
Kalle Korhonen added a comment - This is a JVM issue. As noted in the linked JVM bug report, running with the incremental garbage collector (-incgc) fixes the issue, I can verify that. This is pure speculation but my understanding is that the new G1 collector would also fix the issue (which is going to be the default in JDK1.7) which might be why they didn't bother fixing this particular issue.
Hide
Permalink
Greg Wilkins added a comment -

while you are looking at memory issues, can you check this one also.

Show
Greg Wilkins added a comment - while you are looking at memory issues, can you check this one also.
Hide
Permalink
Simone Bordet added a comment -

What suggested by the reporter is not needed.
I checked in both JDK 5 and JDK 6 and Deflater.end() is called upon stream close.
The stream is always closed as part of stream lifecycle, and it is done by Jetty.

So what remains is the JVM issue related to running the finalizers before the native heap runs out of space, and not much we can do about.

Show
Simone Bordet added a comment - What suggested by the reporter is not needed. I checked in both JDK 5 and JDK 6 and Deflater.end() is called upon stream close. The stream is always closed as part of stream lifecycle, and it is done by Jetty. So what remains is the JVM issue related to running the finalizers before the native heap runs out of space, and not much we can do about.
Hide
Permalink
Anatoliy Salistra added a comment -

NO-NO-NO! This is Not TRUE. Deflater.end() IS called by the stream close(), but Jetty never calls this method! That would close the underlying stream, which should never happen if HTTP wants to maintain long-term connections, as it must as of HTTP1.1. What happens is, the code keeps creating gzip wrappers around an underlying stream, and then 'finishing' them, rather than closing. Just run a code in debugger and let me know when Deflater.end() is invoked.

Show
Anatoliy Salistra added a comment - NO-NO-NO! This is Not TRUE. Deflater.end() IS called by the stream close(), but Jetty never calls this method! That would close the underlying stream, which should never happen if HTTP wants to maintain long-term connections, as it must as of HTTP1.1. What happens is, the code keeps creating gzip wrappers around an underlying stream, and then 'finishing' them, rather than closing. Just run a code in debugger and let me know when Deflater.end() is invoked.
Hide
Permalink
Simone Bordet added a comment -

I just run the code in a debugger and I do see the close() method being called.
Do you have a reproducible test case where Jetty does not close the stream ?

Show
Simone Bordet added a comment - I just run the code in a debugger and I do see the close() method being called. Do you have a reproducible test case where Jetty does not close the stream ?
Hide
Permalink
Michael Slattery added a comment -

If a downstream filter or servlet closes the outputstream/writer then GZipOutputStream.close() method will get called. However, that's rare and not the way to write a good Servlet. In all other cases, Jetty will eventually close the underlying stream, but it has no way of closing the GZipOutputStream.

To test, put a breakpoint on GZipFilter.GzipStream.close() (line 466ish). That's the only place where GZipOutputStream.close() is ever called in GZipFilter. If the breakpoint fires, verify that it's not because of a downstream Servlet/Filter calling close(). In that case, use a different Servlet.

Show
Michael Slattery added a comment - If a downstream filter or servlet closes the outputstream/writer then GZipOutputStream.close() method will get called. However, that's rare and not the way to write a good Servlet. In all other cases, Jetty will eventually close the underlying stream, but it has no way of closing the GZipOutputStream. To test, put a breakpoint on GZipFilter.GzipStream.close() (line 466ish). That's the only place where GZipOutputStream.close() is ever called in GZipFilter. If the breakpoint fires, verify that it's not because of a downstream Servlet/Filter calling close(). In that case, use a different Servlet.
Hide
Permalink
Simone Bordet added a comment -

This is fixed by closing the gzip stream from finish().

Show
Simone Bordet added a comment - This is fixed by closing the gzip stream from finish().

People

  • Assignee:
    Simone Bordet
    Reporter:
    Michael Slattery
Vote (2)
Watch (3)

Dates

  • Created:
    Updated:
    Resolved: