Details
Bug
Resolved
Major
Description
I have a CometD 1.1.1 application running on Jetty 7.1.4.
I am currently load testing this application using the CometD java client. The test setup is as follows:
1) 3500 clients (split over two machines)
2) Broadcasts to all clients every second, each broadcast message data size is approx. 1000 bytes unencrypted, not including the other CometD message data or HTTP request and response headers.
I am using the SslSelectChannelConnector, 4 acceptors, 50 threads.
The Jetty startup parameters are:
-Xms1g -Xmx2g -XX:NewRatio=2 -XX:+UseParallelGC -XX:+UseParallelOldGC -XX:MaxGCPauseMillis=25
With this setup, the Jetty server resident memory grows and grows very quickly until it starts producing errors like:
[2010-07-02 21:07:28,762][qtp2661678-76][WARN ][org.eclipse.jetty.util.log] handle failed
java.lang.OutOfMemoryError: null
at sun.misc.Unsafe.allocateMemory(Native Method) ~[na:1.6.0_20]
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:99) ~[na:1.6.0_20]
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288) ~[na:1.6.0_20]
at org.eclipse.jetty.io.nio.DirectNIOBuffer.<init>(DirectNIOBuffer.java:46) ~[jetty-io-7.1.4.v20100610.jar:7.1.4.v20100610]
at org.eclipse.jetty.server.nio.AbstractNIOConnector.newRequestBuffer(AbstractNIOConnector.java:53) ~[jetty-server-7.1.4.v20100610.jar:7.1.4.v20100610]
at org.eclipse.jetty.http.HttpBuffers$1.newBuffer(HttpBuffers.java:34) ~[jetty-http-7.1.4.v20100610.jar:7.1.4.v20100610]
at org.eclipse.jetty.io.ThreadLocalBuffers.getBuffer(ThreadLocalBuffers.java:60) ~[jetty-io-7.1.4.v20100610.jar:7.1.4.v20100610]
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:557) ~[jetty-http-7.1.4.v20100610.jar:7.1.4.v20100610]
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:211) ~[jetty-http-7.1.4.v20100610.jar:7.1.4.v20100610]
at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:424) ~[jetty-server-7.1.4.v20100610.jar:7.1.4.v20100610]
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:489) ~[jetty-io-7.1.4.v20100610.jar:7.1.4.v20100610]
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436) [jetty-util-7.1.4.v20100610.jar:7.1.4.v20100610]
at java.lang.Thread.run(Thread.java:619) [na:1.6.0_20]
and finally the JVM usually crashes with:
# # A fatal error has been detected by the Java Runtime Environment: # # java.lang.OutOfMemoryError: requested 32756 bytes for ChunkPool::allocate. Out of swap space? # # Internal Error (allocation.cpp:117), pid=5925, tid=1771887504 # Error: ChunkPool::allocate # # JRE version: 6.0_20-b02 # Java VM: Java HotSpot(TM) Server VM (16.3-b01 mixed mode linux-x86 ) # An error report file with more information is saved as:
I have even tried this with a 64-bit JVM, setting the -XX:MaxDirectMemorySize=6g and the error still happened once the JVM resident size had grown to approximately 6g plus the heap size.
I noted that if the maximum heap memory size (-Xmx) is set small enough that the system has to do Full GC's every few seconds (or Full GC's are forced to run every 30s or so via System.gc()), then the problem does not happen. The problem only happens if the system runs for about a minute without a Full GC (with my normal application setting of 2G, Full GC's only happen every 15-20 minutes or so).
With a smaller message size (between 100-130 bytes) this did not seem to be a consistent problem, though I did get the OOM error occasionally. However, with the larger message size I can consistently trigger the OOM.
This error is very similar to http://jira.codehaus.org/browse/JETTY-102, and I suspect a direct buffer leak somewhere.
Activity
| Field | Original Value | New Value |
|---|---|---|
| Assignee | Greg Wilkins [ gregw ] |
| Assignee | Greg Wilkins [ gregw ] | Simone Bordet [ sbordet ] |
| Status | Open [ 1 ] | Resolved [ 5 ] |
| Fix Version/s | 7.3.2 [ 17237 ] | |
| Resolution | Fixed [ 1 ] |
| Fix Version/s | 7.4.0 [ 17302 ] | |
| Fix Version/s | 7.3.2 [ 17237 ] |