Details
-
Type:
Bug
-
Status:
Resolved
-
Priority:
Major
-
Resolution: Duplicate
-
Affects Version/s: 6.1.26
-
Fix Version/s: None
-
Component/s: None
-
Labels:None
-
Environment:Tested on Mac OS X (10.6 and 10.7) and Linux (various Ubuntu systems).
-
Number of attachments :
Description
Application crashes under load with "Too many open files" after several hours. Application is written in Clojure using Ring (ring-jetty-adapter). I managed to track it down to a jetty problem using binary search over jetty versions 6.1.14 to 6.1.26:
6.1.26 - fails
6.1.25 - OK
6.1.23 - OK
6.1.20 - OK
6.1.14 - OK
Reading the release logs the likely culprit is #JETTY-547 (Jetty should rely on socket.shutdownOutput() to close sockets).
The symptoms are reproducible after 1-3 hours of stress testing (4-6 simultaneous requester scripts banging on the app) on my Mac OS X system and after 8-12 hours on a Linux box. Investigating with YourKit shows that sockets are NOT being closed and some remain in open state until file descriptors are exhausted. Interestingly enough, it isn't all sockets that remain open, just some, in batches, it seems. This is why even on my Mac system (limited to 256 fds per process) it takes hours of stress testing to discover the problem.
Netstat shows that no sockets linger in TIME_WAIT.
My current JVM on the Mac is
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03-383-11A511)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02-383, mixed mode)
but we first noticed the problem in April (after upgrading ring to a newer version, which bumped jetty version from 6.1.14 to 6.1.26) and the JVMs might have been different then.
I marked the bug as MAJOR because it's a showstopper for us — we can't use jetty 6.1.26 because of it.
Issue Links
- duplicates
-
JETTY-1458
BIO connector leaks socket file handles under certain conditions
-
Just a quick note that I hit this same issue in ring-jetty-adapter and debugged it separately
My (duplicate) issue is
JETTY-1458.