Weird IO loops with long-held zombie client connections

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Cannot Reproduce
Affects Version/s: 7.4.0
Fix Version/s: 7.6.0
Component/s: HTTP
Labels:
None
Environment:
Jetty 7.4.0

Number of attachments :
0

Description

We recently had a client bug in one of our service consumers which involved the client sending malformed HTTP requests. In particular, the Content-Length header for PUT requests was off and the client forgot about the socket and never closed the connection.

Jetty's BlockingChannelConnector and SocketConnector both had odd responses to this situation.

SocketConnector goes into a hot loop of throwing SocketException instances and rescuing them:

"qtp1213779113-151" - Thread t@151
   java.lang.Thread.State: RUNNABLE
	at java.lang.Throwable.fillInStackTrace(Native Method)
	- locked <7a96bbd0> (a java.net.SocketException)
	at java.lang.Throwable.<init>(Throwable.java:196)
	at java.lang.Exception.<init>(Exception.java:41)
	at java.io.IOException.<init>(IOException.java:41)
	at java.net.SocketException.<init>(SocketException.java:29)
	at java.net.SocketInputStream.read(SocketInputStream.java:113)
	at org.eclipse.jetty.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:388)
	at org.eclipse.jetty.io.bio.StreamEndPoint.fill(StreamEndPoint.java:132)
	at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.fill(SocketConnector.java:209)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:289)
	at org.eclipse.jetty.http.HttpParser.blockForContent(HttpParser.java:1139)
	at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:57)

And BlockingChannelConnector spins on reading from the connection:

"qtp2084696615-1065" - Thread t@1065
   java.lang.Thread.State: RUNNABLE
	at org.eclipse.jetty.io.nio.ChannelEndPoint.fill(ChannelEndPoint.java:173)
	- locked <587fc800> (a java.nio.HeapByteBuffer)
	at org.eclipse.jetty.server.nio.BlockingChannelConnector$BlockingChannelEndPoint.fill(BlockingChannelConnector.java:233)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:289)
	at org.eclipse.jetty.http.HttpParser.blockForContent(HttpParser.java:1139)
	at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:57)

They both peg that worker thread indefinitely; it seems like request parsing should respect the timeout settings for the connector. I suspect SelectChannelConnector would also behave badly under these circumstances, but I didn't get the chance to try.

Activity

Ascending order - Click to sort in descending order

Hide

Permalink

Greg Wilkins added a comment - 25/May/11 7:16 PM

I tried with 7.4.2-SNAPSHOT and could not reproduce:
gregw@Brick: ~
[575] telnet localhost 8082
Trying ::1...
Connected to localhost.localdomain.
Escape character is '^]'.
PUT /dump HTTP/1.1
Host: localhost:8082
Content-Length: 20

abcd
efgh
Connection closed by foreign host.

gregw@Brick: ~
[576] telnet localhost 8083
Trying ::1...
Connected to localhost.localdomain.
Escape character is '^]'.
PUT /dump HTTP/1.1
Host: localhost:8083
Content-Length: 20

abcd
efgh
Connection closed by foreign host.

both timed out correctly and did not cause any CPU looping (8082 is socket connector 8083 is blocking channel connector).

Please reopen if you can reproduce

Show

Greg Wilkins added a comment - 25/May/11 7:16 PM I tried with 7.4.2-SNAPSHOT and could not reproduce: gregw@Brick: ~ [575] telnet localhost 8082 Trying ::1... Connected to localhost.localdomain. Escape character is '^]'. PUT /dump HTTP/1.1 Host: localhost:8082 Content-Length: 20 abcd efgh Connection closed by foreign host. gregw@Brick: ~ [576] telnet localhost 8083 Trying ::1... Connected to localhost.localdomain. Escape character is '^]'. PUT /dump HTTP/1.1 Host: localhost:8083 Content-Length: 20 abcd efgh Connection closed by foreign host. both timed out correctly and did not cause any CPU looping (8082 is socket connector 8083 is blocking channel connector). Please reopen if you can reproduce

Hide

Permalink

Coda Hale added a comment - 26/May/11 6:49 PM

Here's a full test case which describes how to reproduce this:

https://github.com/codahale/jetty-parser-bug-example

Show

Coda Hale added a comment - 26/May/11 6:49 PM Here's a full test case which describes how to reproduce this: https://github.com/codahale/jetty-parser-bug-example

Hide

Permalink

Coda Hale added a comment - 26/May/11 8:00 PM

And I can duplicate it with 7.4.2:

2011-05-26 17:56:38.049:INFO::jetty-7.4.2.v20110526
2011-05-26 17:56:38.089:INFO::started o.e.j.s.ServletContextHandler{/,null}
2011-05-26 17:56:38.100:INFO::Started SocketConnector@0.0.0.0:8000 STARTING
2011-05-26 17:56:38.110:INFO::Started BlockingChannelConnector@0.0.0.0:8001 STARTING
2011-05-26 17:56:38.123:INFO::Started SelectChannelConnector@0.0.0.0:8002 STARTING
2011-05-26 17:57:05
Full thread dump Java HotSpot(TM) 64-Bit Server VM (19.1-b02-334 mixed mode):

"qtp466046775-27" prio=5 tid=1100ee800 nid=0x112510000 waiting on condition [11250f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <7f44e5d50> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
	at java.lang.Thread.run(Thread.java:680)

"qtp466046775-26" prio=5 tid=10c15f800 nid=0x11240d000 runnable [11240c000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.FileDispatcher.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:202)
	at sun.nio.ch.IOUtil.read(IOUtil.java:175)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243)
	- locked <7f46a83c8> (a java.lang.Object)
	at org.eclipse.jetty.io.nio.ChannelEndPoint.fill(ChannelEndPoint.java:162)
	- locked <7f462e8e0> (a java.nio.HeapByteBuffer)
	at org.eclipse.jetty.server.nio.BlockingChannelConnector$BlockingChannelEndPoint.fill(BlockingChannelConnector.java:233)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:289)
	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:214)
	at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:411)
	at org.eclipse.jetty.server.nio.BlockingChannelConnector$BlockingChannelEndPoint.run(BlockingChannelConnector.java:289)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
	at java.lang.Thread.run(Thread.java:680)

"qtp466046775-24" prio=5 tid=102a5d000 nid=0x112207000 waiting on condition [112206000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <7f44e5d50> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
	at java.lang.Thread.run(Thread.java:680)

"qtp466046775-23" prio=5 tid=110008000 nid=0x112001000 waiting on condition [112000000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <7f44e5d50> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
	at java.lang.Thread.run(Thread.java:680)

"qtp466046775-22" prio=5 tid=10d000000 nid=0x112104000 runnable [112103000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:129)
	at org.eclipse.jetty.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:388)
	at org.eclipse.jetty.io.bio.StreamEndPoint.fill(StreamEndPoint.java:132)
	at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.fill(SocketConnector.java:209)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:289)
	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:214)
	at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:411)
	at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:241)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
	at java.lang.Thread.run(Thread.java:680)

"qtp466046775-20" prio=5 tid=10182c800 nid=0x111d01000 runnable [111d00000]
   java.lang.Thread.State: RUNNABLE
	at java.lang.Throwable.fillInStackTrace(Native Method)
	- locked <7f3a1b090> (a java.net.SocketException)
	at java.lang.Throwable.<init>(Throwable.java:196)
	at java.lang.Exception.<init>(Exception.java:41)
	at java.io.IOException.<init>(IOException.java:41)
	at java.net.SocketException.<init>(SocketException.java:29)
	at java.net.SocketInputStream.read(SocketInputStream.java:113)
	at org.eclipse.jetty.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:388)
	at org.eclipse.jetty.io.bio.StreamEndPoint.fill(StreamEndPoint.java:132)
	at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.fill(SocketConnector.java:209)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:289)
	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:214)
	at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:411)
	at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:241)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
	at java.lang.Thread.run(Thread.java:680)

"qtp466046775-18 Acceptor0 SocketConnector@0.0.0.0:8000 STARTED" prio=5 tid=10c21a000 nid=0x111a04000 runnable [111a03000]
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainSocketImpl.socketAccept(Native Method)
	at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
	- locked <7f44e9f60> (a java.net.SocksSocketImpl)
	at java.net.ServerSocket.implAccept(ServerSocket.java:462)
	at java.net.ServerSocket.accept(ServerSocket.java:430)
	at org.eclipse.jetty.server.bio.SocketConnector.accept(SocketConnector.java:107)
	at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:830)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
	at java.lang.Thread.run(Thread.java:680)

"qtp466046775-17 Acceptor0 BlockingChannelConnector@0.0.0.0:8001 STARTING" prio=5 tid=10c219800 nid=0x111901000 runnable [111900000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
	- locked <7f44e8728> (a java.lang.Object)
	at org.eclipse.jetty.server.nio.BlockingChannelConnector.accept(BlockingChannelConnector.java:134)
	at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:830)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
	at java.lang.Thread.run(Thread.java:680)

"qtp466046775-16 Acceptor0 SelectChannelConnector@0.0.0.0:8002 STARTING" prio=5 tid=10c238800 nid=0x110f10000 waiting for monitor entry [110f0f000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:136)
	- waiting to lock <7f44e5d98> (a java.lang.Object)
	at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:92)
	at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:830)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
	at java.lang.Thread.run(Thread.java:680)

"qtp466046775-15 Acceptor1 SelectChannelConnector@0.0.0.0:8002 STARTING" prio=5 tid=10c237800 nid=0x110e0d000 runnable [110e0c000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
	- locked <7f44e5d98> (a java.lang.Object)
	at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:92)
	at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:830)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
	at java.lang.Thread.run(Thread.java:680)

"qtp466046775-14 Selector0 SelectChannelConnector@0.0.0.0:8002 STARTED" prio=5 tid=10c237000 nid=0x110d0a000 runnable [110d09000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
	at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:136)
	at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:69)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <7f44e2458> (a sun.nio.ch.Util$2)
	- locked <7f44e2470> (a java.util.Collections$UnmodifiableSet)
	- locked <7f44f4320> (a sun.nio.ch.KQueueSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:504)
	at org.eclipse.jetty.io.nio.SelectorManager.doSelect(SelectorManager.java:228)
	at org.eclipse.jetty.server.nio.SelectChannelConnector$1.run(SelectChannelConnector.java:268)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
	at java.lang.Thread.run(Thread.java:680)

"qtp466046775-13" prio=5 tid=10c236000 nid=0x110c07000 runnable [110c06000]
   java.lang.Thread.State: RUNNABLE
	at java.lang.Throwable.fillInStackTrace(Native Method)
	- locked <7f39f0198> (a java.net.SocketException)
	at java.lang.Throwable.<init>(Throwable.java:196)
	at java.lang.Exception.<init>(Exception.java:41)
	at java.io.IOException.<init>(IOException.java:41)
	at java.net.SocketException.<init>(SocketException.java:29)
	at java.net.SocketInputStream.read(SocketInputStream.java:113)
	at org.eclipse.jetty.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:388)
	at org.eclipse.jetty.io.bio.StreamEndPoint.fill(StreamEndPoint.java:132)
	at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.fill(SocketConnector.java:209)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:289)
	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:214)
	at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:411)
	at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:241)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
	at java.lang.Thread.run(Thread.java:680)

"qtp466046775-12 Selector1 SelectChannelConnector@0.0.0.0:8002 STARTED" prio=5 tid=10c235800 nid=0x110b04000 runnable [110b03000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
	at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:136)
	at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:69)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <7f44e2e50> (a sun.nio.ch.Util$2)
	- locked <7f44e2e68> (a java.util.Collections$UnmodifiableSet)
	- locked <7f44fa420> (a sun.nio.ch.KQueueSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:504)
	at org.eclipse.jetty.io.nio.SelectorManager.doSelect(SelectorManager.java:228)
	at org.eclipse.jetty.server.nio.SelectChannelConnector$1.run(SelectChannelConnector.java:268)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
	at java.lang.Thread.run(Thread.java:680)

"qtp466046775-11" prio=5 tid=10c14d000 nid=0x110a01000 waiting on condition [110a00000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at org.eclipse.jetty.server.nio.BlockingChannelConnector$1.run(BlockingChannelConnector.java:86)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
	at java.lang.Thread.run(Thread.java:680)

"Monitor Ctrl-Break" daemon prio=5 tid=1018d3800 nid=0x10db01000 runnable [10db00000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:129)
	at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
	- locked <7f46add00> (a java.io.InputStreamReader)
	at java.io.InputStreamReader.read(InputStreamReader.java:167)
	at java.io.BufferedReader.fill(BufferedReader.java:136)
	at java.io.BufferedReader.readLine(BufferedReader.java:299)
	- locked <7f46add00> (a java.io.InputStreamReader)
	at java.io.BufferedReader.readLine(BufferedReader.java:362)
	at com.intellij.rt.execution.application.AppMain$1.run(AppMain.java:85)
	at java.lang.Thread.run(Thread.java:680)

"Low Memory Detector" daemon prio=5 tid=101810800 nid=0x10cc01000 runnable [00000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread1" daemon prio=9 tid=10180f800 nid=0x10c901000 waiting on condition [00000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=9 tid=10c001000 nid=0x10bf04000 waiting on condition [00000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=9 tid=10c000000 nid=0x10be01000 waiting on condition [00000000]
   java.lang.Thread.State: RUNNABLE

"Surrogate Locker Thread (CMS)" daemon prio=5 tid=102948800 nid=0x10bb01000 waiting on condition [00000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=8 tid=102940000 nid=0x10b5f9000 in Object.wait() [10b5f8000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <7f44e1b68> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
	- locked <7f44e1b68> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=10293f000 nid=0x10b4f6000 in Object.wait() [10b4f5000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <7f44e5dc8> (a java.lang.ref.Reference$Lock)
	at java.lang.Object.wait(Object.java:485)
	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
	- locked <7f44e5dc8> (a java.lang.ref.Reference$Lock)

"main" prio=5 tid=102802000 nid=0x100501000 in Object.wait() [1004ff000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <7f44e5de0> (a java.lang.Object)
	at java.lang.Object.wait(Object.java:485)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.join(QueuedThreadPool.java:382)
	- locked <7f44e5de0> (a java.lang.Object)
	at org.eclipse.jetty.server.Server.join(Server.java:399)
	at com.codahale.jettyparserbug.Runner.main(Runner.java:32)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)

"VM Thread" prio=9 tid=10293a800 nid=0x10b3f3000 runnable 

"Gang worker#0 (Parallel GC Threads)" prio=9 tid=102803000 nid=0x102301000 runnable 

"Gang worker#1 (Parallel GC Threads)" prio=9 tid=102803800 nid=0x102404000 runnable 

"Gang worker#2 (Parallel GC Threads)" prio=9 tid=102804000 nid=0x102507000 runnable 

"Gang worker#3 (Parallel GC Threads)" prio=9 tid=102805000 nid=0x10260a000 runnable 

"Gang worker#4 (Parallel GC Threads)" prio=9 tid=102805800 nid=0x10270d000 runnable 

"Gang worker#5 (Parallel GC Threads)" prio=9 tid=102806000 nid=0x107102000 runnable 

"Gang worker#6 (Parallel GC Threads)" prio=9 tid=102806800 nid=0x107205000 runnable 

"Gang worker#7 (Parallel GC Threads)" prio=9 tid=102807800 nid=0x107308000 runnable 

"Concurrent Mark-Sweep GC Thread" prio=9 tid=1028e4800 nid=0x10b0ff000 runnable 
"Gang worker#0 (Parallel CMS Threads)" prio=9 tid=1028e3800 nid=0x10a6f9000 runnable 

"Gang worker#1 (Parallel CMS Threads)" prio=9 tid=1028e4000 nid=0x10a7fc000 runnable 

"VM Periodic Task Thread" prio=10 tid=101822000 nid=0x10cd04000 waiting on condition 

"Exception Catcher Thread" prio=10 tid=102800800 nid=0x10176a000 runnable 
JNI global references: 906

Heap
 par new generation   total 19136K, used 10516K [7f3000000, 7f44c0000, 7f44c0000)
  eden space 17024K,  61% used [7f3000000, 7f3a43a70, 7f40a0000)
  from space 2112K,   0% used [7f42b0000, 7f42b1800, 7f44c0000)
  to   space 2112K,   0% used [7f40a0000, 7f40a0000, 7f42b0000)
 concurrent mark-sweep generation total 63872K, used 1199K [7f44c0000, 7f8320000, 7fae00000)
 concurrent-mark-sweep perm gen total 21248K, used 7726K [7fae00000, 7fc2c0000, 800000000)

Show

Coda Hale added a comment - 26/May/11 8:00 PM And I can duplicate it with 7.4.2: 2011-05-26 17:56:38.049:INFO::jetty-7.4.2.v20110526 2011-05-26 17:56:38.089:INFO::started o.e.j.s.ServletContextHandler{/,null} 2011-05-26 17:56:38.100:INFO::Started SocketConnector@0.0.0.0:8000 STARTING 2011-05-26 17:56:38.110:INFO::Started BlockingChannelConnector@0.0.0.0:8001 STARTING 2011-05-26 17:56:38.123:INFO::Started SelectChannelConnector@0.0.0.0:8002 STARTING 2011-05-26 17:57:05 Full thread dump Java HotSpot(TM) 64-Bit Server VM (19.1-b02-334 mixed mode): "qtp466046775-27" prio=5 tid=1100ee800 nid=0x112510000 waiting on condition [11250f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <7f44e5d50> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558) at java.lang.Thread.run(Thread.java:680) "qtp466046775-26" prio=5 tid=10c15f800 nid=0x11240d000 runnable [11240c000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.FileDispatcher.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:202) at sun.nio.ch.IOUtil.read(IOUtil.java:175) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243) - locked <7f46a83c8> (a java.lang.Object) at org.eclipse.jetty.io.nio.ChannelEndPoint.fill(ChannelEndPoint.java:162) - locked <7f462e8e0> (a java.nio.HeapByteBuffer) at org.eclipse.jetty.server.nio.BlockingChannelConnector$BlockingChannelEndPoint.fill(BlockingChannelConnector.java:233) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:289) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:214) at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:411) at org.eclipse.jetty.server.nio.BlockingChannelConnector$BlockingChannelEndPoint.run(BlockingChannelConnector.java:289) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529) at java.lang.Thread.run(Thread.java:680) "qtp466046775-24" prio=5 tid=102a5d000 nid=0x112207000 waiting on condition [112206000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <7f44e5d50> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558) at java.lang.Thread.run(Thread.java:680) "qtp466046775-23" prio=5 tid=110008000 nid=0x112001000 waiting on condition [112000000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <7f44e5d50> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558) at java.lang.Thread.run(Thread.java:680) "qtp466046775-22" prio=5 tid=10d000000 nid=0x112104000 runnable [112103000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at org.eclipse.jetty.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:388) at org.eclipse.jetty.io.bio.StreamEndPoint.fill(StreamEndPoint.java:132) at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.fill(SocketConnector.java:209) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:289) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:214) at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:411) at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:241) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529) at java.lang.Thread.run(Thread.java:680) "qtp466046775-20" prio=5 tid=10182c800 nid=0x111d01000 runnable [111d00000] java.lang.Thread.State: RUNNABLE at java.lang.Throwable.fillInStackTrace(Native Method) - locked <7f3a1b090> (a java.net.SocketException) at java.lang.Throwable.<init>(Throwable.java:196) at java.lang.Exception.<init>(Exception.java:41) at java.io.IOException.<init>(IOException.java:41) at java.net.SocketException.<init>(SocketException.java:29) at java.net.SocketInputStream.read(SocketInputStream.java:113) at org.eclipse.jetty.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:388) at org.eclipse.jetty.io.bio.StreamEndPoint.fill(StreamEndPoint.java:132) at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.fill(SocketConnector.java:209) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:289) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:214) at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:411) at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:241) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529) at java.lang.Thread.run(Thread.java:680) "qtp466046775-18 Acceptor0 SocketConnector@0.0.0.0:8000 STARTED" prio=5 tid=10c21a000 nid=0x111a04000 runnable [111a03000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408) - locked <7f44e9f60> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:462) at java.net.ServerSocket.accept(ServerSocket.java:430) at org.eclipse.jetty.server.bio.SocketConnector.accept(SocketConnector.java:107) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:830) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529) at java.lang.Thread.run(Thread.java:680) "qtp466046775-17 Acceptor0 BlockingChannelConnector@0.0.0.0:8001 STARTING" prio=5 tid=10c219800 nid=0x111901000 runnable [111900000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152) - locked <7f44e8728> (a java.lang.Object) at org.eclipse.jetty.server.nio.BlockingChannelConnector.accept(BlockingChannelConnector.java:134) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:830) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529) at java.lang.Thread.run(Thread.java:680) "qtp466046775-16 Acceptor0 SelectChannelConnector@0.0.0.0:8002 STARTING" prio=5 tid=10c238800 nid=0x110f10000 waiting for monitor entry [110f0f000] java.lang.Thread.State: BLOCKED (on object monitor) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:136) - waiting to lock <7f44e5d98> (a java.lang.Object) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:92) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:830) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529) at java.lang.Thread.run(Thread.java:680) "qtp466046775-15 Acceptor1 SelectChannelConnector@0.0.0.0:8002 STARTING" prio=5 tid=10c237800 nid=0x110e0d000 runnable [110e0c000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152) - locked <7f44e5d98> (a java.lang.Object) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:92) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:830) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529) at java.lang.Thread.run(Thread.java:680) "qtp466046775-14 Selector0 SelectChannelConnector@0.0.0.0:8002 STARTED" prio=5 tid=10c237000 nid=0x110d0a000 runnable [110d09000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:136) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:69) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <7f44e2458> (a sun.nio.ch.Util$2) - locked <7f44e2470> (a java.util.Collections$UnmodifiableSet) - locked <7f44f4320> (a sun.nio.ch.KQueueSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:504) at org.eclipse.jetty.io.nio.SelectorManager.doSelect(SelectorManager.java:228) at org.eclipse.jetty.server.nio.SelectChannelConnector$1.run(SelectChannelConnector.java:268) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529) at java.lang.Thread.run(Thread.java:680) "qtp466046775-13" prio=5 tid=10c236000 nid=0x110c07000 runnable [110c06000] java.lang.Thread.State: RUNNABLE at java.lang.Throwable.fillInStackTrace(Native Method) - locked <7f39f0198> (a java.net.SocketException) at java.lang.Throwable.<init>(Throwable.java:196) at java.lang.Exception.<init>(Exception.java:41) at java.io.IOException.<init>(IOException.java:41) at java.net.SocketException.<init>(SocketException.java:29) at java.net.SocketInputStream.read(SocketInputStream.java:113) at org.eclipse.jetty.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:388) at org.eclipse.jetty.io.bio.StreamEndPoint.fill(StreamEndPoint.java:132) at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.fill(SocketConnector.java:209) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:289) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:214) at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:411) at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:241) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529) at java.lang.Thread.run(Thread.java:680) "qtp466046775-12 Selector1 SelectChannelConnector@0.0.0.0:8002 STARTED" prio=5 tid=10c235800 nid=0x110b04000 runnable [110b03000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:136) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:69) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <7f44e2e50> (a sun.nio.ch.Util$2) - locked <7f44e2e68> (a java.util.Collections$UnmodifiableSet) - locked <7f44fa420> (a sun.nio.ch.KQueueSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:504) at org.eclipse.jetty.io.nio.SelectorManager.doSelect(SelectorManager.java:228) at org.eclipse.jetty.server.nio.SelectChannelConnector$1.run(SelectChannelConnector.java:268) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529) at java.lang.Thread.run(Thread.java:680) "qtp466046775-11" prio=5 tid=10c14d000 nid=0x110a01000 waiting on condition [110a00000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.eclipse.jetty.server.nio.BlockingChannelConnector$1.run(BlockingChannelConnector.java:86) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529) at java.lang.Thread.run(Thread.java:680) "Monitor Ctrl-Break" daemon prio=5 tid=1018d3800 nid=0x10db01000 runnable [10db00000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264) at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158) - locked <7f46add00> (a java.io.InputStreamReader) at java.io.InputStreamReader.read(InputStreamReader.java:167) at java.io.BufferedReader.fill(BufferedReader.java:136) at java.io.BufferedReader.readLine(BufferedReader.java:299) - locked <7f46add00> (a java.io.InputStreamReader) at java.io.BufferedReader.readLine(BufferedReader.java:362) at com.intellij.rt.execution.application.AppMain$1.run(AppMain.java:85) at java.lang.Thread.run(Thread.java:680) "Low Memory Detector" daemon prio=5 tid=101810800 nid=0x10cc01000 runnable [00000000] java.lang.Thread.State: RUNNABLE "CompilerThread1" daemon prio=9 tid=10180f800 nid=0x10c901000 waiting on condition [00000000] java.lang.Thread.State: RUNNABLE "CompilerThread0" daemon prio=9 tid=10c001000 nid=0x10bf04000 waiting on condition [00000000] java.lang.Thread.State: RUNNABLE "Signal Dispatcher" daemon prio=9 tid=10c000000 nid=0x10be01000 waiting on condition [00000000] java.lang.Thread.State: RUNNABLE "Surrogate Locker Thread (CMS)" daemon prio=5 tid=102948800 nid=0x10bb01000 waiting on condition [00000000] java.lang.Thread.State: RUNNABLE "Finalizer" daemon prio=8 tid=102940000 nid=0x10b5f9000 in Object.wait() [10b5f8000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <7f44e1b68> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) - locked <7f44e1b68> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) "Reference Handler" daemon prio=10 tid=10293f000 nid=0x10b4f6000 in Object.wait() [10b4f5000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <7f44e5dc8> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:485) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) - locked <7f44e5dc8> (a java.lang.ref.Reference$Lock) "main" prio=5 tid=102802000 nid=0x100501000 in Object.wait() [1004ff000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <7f44e5de0> (a java.lang.Object) at java.lang.Object.wait(Object.java:485) at org.eclipse.jetty.util.thread.QueuedThreadPool.join(QueuedThreadPool.java:382) - locked <7f44e5de0> (a java.lang.Object) at org.eclipse.jetty.server.Server.join(Server.java:399) at com.codahale.jettyparserbug.Runner.main(Runner.java:32) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120) "VM Thread" prio=9 tid=10293a800 nid=0x10b3f3000 runnable "Gang worker#0 (Parallel GC Threads)" prio=9 tid=102803000 nid=0x102301000 runnable "Gang worker#1 (Parallel GC Threads)" prio=9 tid=102803800 nid=0x102404000 runnable "Gang worker#2 (Parallel GC Threads)" prio=9 tid=102804000 nid=0x102507000 runnable "Gang worker#3 (Parallel GC Threads)" prio=9 tid=102805000 nid=0x10260a000 runnable "Gang worker#4 (Parallel GC Threads)" prio=9 tid=102805800 nid=0x10270d000 runnable "Gang worker#5 (Parallel GC Threads)" prio=9 tid=102806000 nid=0x107102000 runnable "Gang worker#6 (Parallel GC Threads)" prio=9 tid=102806800 nid=0x107205000 runnable "Gang worker#7 (Parallel GC Threads)" prio=9 tid=102807800 nid=0x107308000 runnable "Concurrent Mark-Sweep GC Thread" prio=9 tid=1028e4800 nid=0x10b0ff000 runnable "Gang worker#0 (Parallel CMS Threads)" prio=9 tid=1028e3800 nid=0x10a6f9000 runnable "Gang worker#1 (Parallel CMS Threads)" prio=9 tid=1028e4000 nid=0x10a7fc000 runnable "VM Periodic Task Thread" prio=10 tid=101822000 nid=0x10cd04000 waiting on condition "Exception Catcher Thread" prio=10 tid=102800800 nid=0x10176a000 runnable JNI global references: 906 Heap par new generation total 19136K, used 10516K [7f3000000, 7f44c0000, 7f44c0000) eden space 17024K, 61% used [7f3000000, 7f3a43a70, 7f40a0000) from space 2112K, 0% used [7f42b0000, 7f42b1800, 7f44c0000) to space 2112K, 0% used [7f40a0000, 7f40a0000, 7f42b0000) concurrent mark-sweep generation total 63872K, used 1199K [7f44c0000, 7f8320000, 7fae00000) concurrent-mark-sweep perm gen total 21248K, used 7726K [7fae00000, 7fc2c0000, 800000000)

Hide

Permalink

Coda Hale added a comment - 26/May/11 8:22 PM

The issue seems to be in HttpParser.java, lines 287-298:

try
{
    filled=_endp.fill(_buffer);
    if (filled>0)
        progress++;
}
catch(IOException e)
{
    Log.debug(e);
    ioex=e;
    filled=-1;
}

_endp.fill(_buffer) throws a SocketException because the connection's been reset, but the parser state is never modified, so it just keeps looping back – each time it tries to read from the connection's socket, it throws an exception and then catches it and then soldiers on.

Show

Coda Hale added a comment - 26/May/11 8:22 PM The issue seems to be in HttpParser.java , lines 287-298: try { filled=_endp.fill(_buffer); if (filled>0) progress++; } catch (IOException e) { Log.debug(e); ioex=e; filled=-1; } _endp.fill(_buffer) throws a SocketException because the connection's been reset, but the parser state is never modified, so it just keeps looping back – each time it tries to read from the connection's socket, it throws an exception and then catches it and then soldiers on.

Hide

Permalink

Coda Hale added a comment - 26/May/11 8:27 PM

Re-throwing the IOException fixes this, so I've got a hot-patch for my problem.

Show

Coda Hale added a comment - 26/May/11 8:27 PM Re-throwing the IOException fixes this, so I've got a hot-patch for my problem.

Hide

Permalink

Coda Hale added a comment - 02/Jul/11 6:43 PM

This is still an issue for us in production – it doesn't seem fixed in 7.4.2 or 7.4.3.

Show

Coda Hale added a comment - 02/Jul/11 6:43 PM This is still an issue for us in production – it doesn't seem fixed in 7.4.2 or 7.4.3.

Hide

Permalink

Coda Hale added a comment - 25/Jul/11 6:54 PM

Doesn't seem fixed in 7.4.5, either.

Show

Coda Hale added a comment - 25/Jul/11 6:54 PM Doesn't seem fixed in 7.4.5, either.

Hide

Permalink

Michael Gorovoy added a comment - 04/Aug/11 3:57 PM - edited

Greetings,

Could you test your issue with the latest 7.5.0-SNAPSHOT available from http://goo.gl/eOQHm? Greg committed a series of changes to HttpParser code, including the area that you are referring to, that should have fixed this issue.

Cheers,
Michael

Show

Michael Gorovoy added a comment - 04/Aug/11 3:57 PM - edited Greetings, Could you test your issue with the latest 7.5.0-SNAPSHOT available from http://goo.gl/eOQHm? Greg committed a series of changes to HttpParser code, including the area that you are referring to, that should have fixed this issue. Cheers, Michael

Hide

Permalink

Michael Gorovoy added a comment - 04/Aug/11 3:59 PM

Going to mark this ticket as resolved pending confirmation that it has been fixed.

Show

Michael Gorovoy added a comment - 04/Aug/11 3:59 PM Going to mark this ticket as resolved pending confirmation that it has been fixed.

Hide

Permalink

Coda Hale added a comment - 04/Aug/11 4:29 PM

I can no longer duplicate this issue with BlockingChannelConnector or SelectChannelConnector, but this is still broken for SocketConnector.

FWIW, this is a very easy issue to test. I have a fully packaged test case, complete with repro steps here: https://github.com/codahale/jetty-parser-bug-example

Show

Coda Hale added a comment - 04/Aug/11 4:29 PM I can no longer duplicate this issue with BlockingChannelConnector or SelectChannelConnector , but this is still broken for SocketConnector . FWIW, this is a very easy issue to test. I have a fully packaged test case, complete with repro steps here: https://github.com/codahale/jetty-parser-bug-example

Hide

Permalink

Greg Wilkins added a comment - 21/Sep/11 12:33 AM

thanks for creating a test case, but I've tried to reproduce with 7.2.2, 7.4.2 and 7.5.2-SNAPSHOT, using firefox and chrome.... no joy.

So I'm going to reclose for now, as we certainly have the fix you suggest into the HttpParser.
Also, when 7.5.2 comes out, it will have a bit more debugging in it to detect busy loops and dump info.

If you can reproduce on 7.5.2, then please re-open.

Show

Greg Wilkins added a comment - 21/Sep/11 12:33 AM thanks for creating a test case, but I've tried to reproduce with 7.2.2, 7.4.2 and 7.5.2-SNAPSHOT, using firefox and chrome.... no joy. So I'm going to reclose for now, as we certainly have the fix you suggest into the HttpParser. Also, when 7.5.2 comes out, it will have a bit more debugging in it to detect busy loops and dump info. If you can reproduce on 7.5.2, then please re-open.

People

Assignee:

Greg Wilkins

Reporter:

Coda Hale

Vote (0)

Watch (1)

Dates

Created:

19/May/11 8:21 PM

Updated:

21/Sep/11 1:18 AM

Resolved:

21/Sep/11 1:18 AM