OOM

Another solution would be to have a pool of direct memory buffers. Summary: you can't just allocate direct memory buffers, becouse the GC will "forget" about them, causing OOM. But with small memory buffers the pool would be ok.

The following code prints
"OOM after 16357 iterations."
providing there is the "-Xms128m" option in the command line.

<Nice>
// Prove that there can be a GC problem with direct memory buffers.
import java.util.*;
import java.nio.*;
public void main(Template this, Cms cms) {
LinkedList<ByteBuffer> ll = new LinkedList();
int counter = 0;
try {
for (;counter < 99000

} catch (OutOfMemoryError err)

}
</Nice>

I've just discovered, that after the OOM the JVM had silently exited!
Can it be the catch block in org.mortbay.io.IO?

With out of memory errors, it is very important to realize that the code that throws the
exception is not necessarily the code that is using all the memory. It may just be the
unlucky bit of code that asked for more memory after something else has used up all
the heap.

So we have to determine what is using the memory before we can decide that it
is Jetty that is leaking. It may be the application or something else that is being
used.

It is not as simple as direct buffers, as I have loaded tested direct buffers to
high levels and observed no memory leaks

However, it may well be jetty. I am re-running my long duration tests now and so
far I cannot see any evidence of memory loss on 5 sites that I am running.

Is it possible for you to profile these apps and see what is eating the memory?
Can you catogorize the type of request that eats memory? eg static content
requests, applicaton requests, JSP requests? etc.

> With out of memory errors, it is very important to realize that the code that throws the
> exception is not necessarily the code that is using all the memory. It may just be the
> unlucky bit of code that asked for more memory after something else has used up all
> the heap.

That unlucky bit of code has been throwing OOM for about a month now.
There is plenty of RAM for GC to use. As i've already explained, it is the native memory that gets shortage. Of that native memory i know only three possible users, that's Berkley DB, and there NIO is currently turned off; that's ZLIB library we use to compress web pages, that one is explicitly closed when finished. And only source left is Jetty.

> So we have to determine what is using the memory before we can decide that it is Jetty that is leaking.

No we don't, because Jetty isn't leaking; it's using native memory buffers, and /these/ are leaking, under some GC conditions, when only eden collections occur. To see that they indeed leak, refer to the test posted earlier.

> However, it may well be jetty. I am re-running my long duration tests now and so
> far I cannot see any evidence of memory loss on 5 sites that I am running.

Does it perform a full garbage collection during the test? If so, you won't see the leak.

> Is it possible for you to profile these apps and see what is eating the memory?

I don't think i have the tools to see how /native/ memory is used in the JVM. However, i'm pretty sure it's native buffers, since our server runs without full collection, but with plenty of eden collections, for a few hours, and that's the perfect environment for the direct memory leak described earlier.
As i've said, there is no shortage of HEAP memory, which is obvious from just the -Xloggc log.

OK - things are not as simple as they seam....
Firstly, Jetty does pool it's buffers, so there should not be a high rate of churn - unless there is something else going on (I tend not to recycle during exceptional cases).

Also, the test program that you gave me does not OOM on my machine. I even increased the iterations by a factor of 10 and it still works perfectly!

java version "1.5.0_06"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
Java HotSpot(TM) Client VM (build 1.5.0_06-b05, mixed mode, sharing)

So something unusual is happening on your machine - to a) make it not
recycle buffers and b) for them to leak. It would be good to know what that is.

However, I have added the method setUseDirectBuffers to all NIOConnectors (defaults to true). So you can try switching it off. In svn very shortly.

> Also, the test program that you gave me does not OOM on my machine.
> I even increased the iterations by a factor of 10 and it still works perfectly!

Did you use the "-Xms128m" (and -Xmx256m or something, too)?
Does "-Xloggc" report full collections (it shouldn't)?
Do you have a multiprocessor machine (the parallel collector will then kick in)?

Note. OOM is an unrecoverable exception. You really want the JVM to die when it happens as you have no idea what threads have been hit by it and have failed.

I just did java -Xms64m TestAllocate and it worked fine.
java -Xloggc:gc.out -Xms64m TestAllocate is reporting full GCs (why should it
not?)

This is on a single CPU machine (a ibm thinkpad a30p running ubuntu linux).

jconsole (which is probably not reporting native memory) shows a typical sawtooth
pattern. There is no increase in unix process size and no reduction in free swap space (often an indication of mapped memory).

> java -Xloggc:gc.out -Xms64m TestAllocate is reporting full GCs (why should it not?)

Because the point of the test case is to show that native memory shortage can happen under some circumstances, becouse GC fails to detect the need to do the full collection. A good garbage collection pattern is when only eden generations occur for a long time - it means that most of the useful data is cached, and most of the temporary data is collected in time, and GC is working at top performance, so it's not some bad pattern.

> Note. OOM is an unrecoverable exception.
> You really want the JVM to die when it happens as you have
> no idea what threads have been hit by it and have failed.

Thanks for the notice, i'm a C++ programmer, with Brainbench master degree in system programming, so i should know what OOM is, nevertheless i would never agree that you should System.exit the application after an OOM; for starters, JVM native memory is a separate pool, see the -XX:MaxDirectMemorySize option; furthermore, even if the application should exit after OOM, it might do so in specific order, by closing some native databases first, for example, and by giving some time (a second would suffice) to active working threads to finish. As it seems, a spurious System.exit already caused logic data corruption, not critical, but annoying, in a non-transaction logic we have.
Note, also, that this has nothing to do with the problem, which is that direct memory buffers produce unexpectant OOM's.

Thanks for the option you've incorporated, if there is a leak, perhaps with this option we'll see it more clear, and if not, this option is very useful anyway.

Artem,

sorry - I didn't mean to cause offense. It is just that "handling" an OOM exception is somewhat of a FAQ.
I think we agree that a graceful-as-possible shutdown and restart is probably the best handling.

I think we are also in agreement about the GC of direct memory buffers: on some JVMs with some
usage patterns native memory shortages may occur. If this is frequently the case, then I would make
the default to not use direct buffers - if this is not frequent, then I think the current default to use direct
buffers is correct.

I will also investigate the situations in which direct buffers are currently allocated, as I would hope that most
would be recycled.

Oh - I found another place that I create direct buffers - some static resources.
Just checking in a fix so that this also respects the getUseDirectBuffers() of the connector

> I think we agree that a graceful-as-possible shutdown and restart is probably the best handling.

From the point of the application, which expects that its libraries are unable to handle OOM,
but from the point of a library or other reusable component, it depends.
You can't gracefully shutdown a space shuttle, and OOM can be a pretty expected case in some environments, and i knew programs which were designed and did perform fine under "no more free memory" conditions, and i wrote such programs myself, and i don't see how OOM is worse then Thread.interrupt, becouse Thread.interrupt never happens naturally, but OOM - does (in a multiuser environment on a server it is to be expected).

> Artem, sorry - I didn't mean to cause offense.
> It is just that "handling" an OOM exception is somewhat of a FAQ.

Yeah, and it sounded like you were lecturing me, which would be fine if up to the point. Otherwise i might have a bad temper about particulars, which is not to be taken personally, of course, so please forgive me.

> I think we are also in agreement about the GC of direct memory buffers:
> on some JVMs with some usage patterns native memory shortages may occur.

I've asked for two things: an option and a pool. You have given me both. Compete agreement here. I think you can close the bug report (now i think, perhaps it was more of a feature request). Is it the cause of my problems (frequent restarts of Jetty, which began on this particular server strictly after the move to Jetty 6 from Jetty 5, and that's with JETTY-63 bug already closed) or not - it is yet to be seen and might be a subject for future reports/requests.

> If this is frequently the case, then I would make the default to not use direct buffers
> - if this is not frequent, then I think the current default to use direct
> buffers is correct.

Your default, to use direct memory buffers, in my personal opinion is perfectly all right.

> I will also investigate the situations in which direct buffers are currently allocated,
> as I would hope that most would be recycled.

I will try to switch direct buffers off and see if it helps. Even if Jetty is not the source of the shortage (which is the case if all the buffers are pooled), making it independent of that particular limited resource will greatly help me to pinpoint the real cause of problems (otherwise i can't track it, becouse Jetty stops responding and i have to automatically restart the application in order to avoid the downtime)...
Thank you for your patience.

Greg, it looks like you forgot to /add/ AbstractNIOConnector into the last SVN commit.

java.lang.OutOfMemoryError
at sun.misc.Unsafe.allocateMemory(Native Method)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:99)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
at sun.nio.ch.IOUtil.write(IOUtil.java:134)
at sun.nio.ch.SocketChannelImpl.write0(SocketChannelImpl.java:331)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:354)
at java.nio.channels.SocketChannel.write(SocketChannel.java:360)
at org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:238)
at org.mortbay.jetty.nio.HttpChannelEndPoint.flush(HttpChannelEndPoint.java:140)
at org.mortbay.jetty.HttpGenerator.flushBuffers(HttpGenerator.java:769)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:322)
at org.mortbay.jetty.nio.HttpChannelEndPoint.run(HttpChannelEndPoint.java:269)
at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
(Jetty SVN revision 806)

From examining the source, it seems that sun.nio.ch.IOUtil will copy a non-direct buffer into a direct one, but if the buffer is already an instanceof DirectBuffer, sun.nio.ch.IOUtil will just use it. As it seems, the "setUseDirectBuffers" option is not a solution.

OK, so this brings us back to two questions:
1) why are the direct buffers not being pooled
2) why does this cause grief on some JVM/machines but not on others?

Could you try putting a println into AbstractNIOBuffer to get a feel for how often your
system is creating new buffers?

I've added a println into constructor in jetty\src\main\java\org\mortbay\io\nio\NIOBuffer.java
1712 allocations in twenty minutes.

> 2) why does this cause grief on some JVM/machines but not on others?

A combination of limited memory (either "-Xmx" or, if that one is high, "ulimit -d") with a lack of full collections (achieved with parralel collector default "-XX:GCTimeRatio", which is too low, or with "-Xms").
In my case, "-Xmx" was high (in recent versions of JVM, default value of "-XX:MaxDirectMemorySize" is calculated from "-Xmx", see SDN Bug ID 4879883), but "ulimit -d" was hardcoded by the operating system to be 500m; after i increased "ulimit -d" to 1000m, the ratio between "leaked" native memory and "-Xms" became bearable, that is, the leaked native memory is swapped away (about 500m of swap is created and lying there undisturbed), and the process survives till the -Xms is reached and full collection occurs. So, either decreasing -Xms (increasing -XX:GCTimeRatio), or increasing the memory limit (-Xmx and ulimit) is a workaround, although you can't increase the memory limit much unless you have a 64-bit memory space.

OK - I have found that I am not recylcing one of the response buffers!
So I am working on a patch that will improve this.... stay tuned.

in svn now... RC2 coming out tonight

The fix confirmed. Three days without a fault.