Jetty

CLONE -NullPointerException when closing connection

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: 6.1.2rc1
  • Fix Version/s: 6.1.6
  • Component/s: NIO
  • Labels:
    None
  • Environment:
    JDK 1.6.0, Solaris 9
  • Number of attachments :
    0

Description

I am getting an intermittent NPE when I try to close the output stream for a response. The channel is non-blocking and my code is running outside of the handler after a continuation was created. I previously wrote the header for an HTTP 1.0 style streaming response (i.e. no content length, non-chunked) on the connection and now I am trying to close the connection at the end of the document.

The exception I'm getting is:

java.lang.NullPointerException
        at org.mortbay.jetty.HttpFields$2.hasNext(HttpFields.java:267)
        at org.mortbay.jetty.HttpGenerator.completeHeader(HttpGenerator.java:379)
        at org.mortbay.jetty.HttpConnection.commitResponse(HttpConnection.java:529)
        at org.mortbay.jetty.HttpConnection$Output.close(HttpConnection.java:837)
        at com.electriccloud.accelerator.service.BuildNotifier.cleanup(BuildNotifier.java:77)

I have tried to trace the problem in the debugger, and it appears the NPE happens because destroy() has already been called on the connection so the attempt to commit a response is working on a partially destroyed object. As far as I can tell, destroy() is being called from the selector thread. My speculation is that the failure is caused by a race between the selector thread detecting a disconnect from the remote side and my thread deciding to close the channel.

I'm having a hard time narrowing down the problem to a small test case, but let me know if you want me to inject any logging because the problem is reproducible within 10-20 minutes.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Pankaj Arora added a comment -

2007-12-12 20:35:12.984::WARN: handle failed
java.lang.IllegalStateException
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:361)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:396)
at com.approuter.module.http.protocol.CIBoundedThreadPool$PoolThread.run(CIBoundedThreadPool.java:404)
java.lang.NullPointerException
at org.mortbay.jetty.HttpFields$1.hasMoreElements(HttpFields.java:230)

I am facing soemhting similar intermittently under heavy load. Looks like httpFields got destroyed before sending the resposne and while reading from them it cause Null pointer eception. I ma not sure why this is happening. Is there any work aord for this.
i am using 6.1.5 release.

Show
Pankaj Arora added a comment - 2007-12-12 20:35:12.984::WARN: handle failed java.lang.IllegalStateException at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:361) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:396) at com.approuter.module.http.protocol.CIBoundedThreadPool$PoolThread.run(CIBoundedThreadPool.java:404) java.lang.NullPointerException at org.mortbay.jetty.HttpFields$1.hasMoreElements(HttpFields.java:230) I am facing soemhting similar intermittently under heavy load. Looks like httpFields got destroyed before sending the resposne and while reading from them it cause Null pointer eception. I ma not sure why this is happening. Is there any work aord for this. i am using 6.1.5 release.
Hide
Permalink
Greg Wilkins added a comment -

I think you speculation is correct. We have had previous problems with the race between remote close and flushing the response.

The good news is that if we correctly detect the situation, all that will change is that the NPE will be
replaced by an IOException.

So in once sense, other than nasty log noise, your webapp is probably working as expected. However
we will see what we can do to improve this.

Previously we had avoided the race by delaying the call to destroy until the handlers had exited. Perhaps
this does not take into account continuations - so we will look at this again.

cheers

Show
Greg Wilkins added a comment - I think you speculation is correct. We have had previous problems with the race between remote close and flushing the response. The good news is that if we correctly detect the situation, all that will change is that the NPE will be replaced by an IOException. So in once sense, other than nasty log noise, your webapp is probably working as expected. However we will see what we can do to improve this. Previously we had avoided the race by delaying the call to destroy until the handlers had exited. Perhaps this does not take into account continuations - so we will look at this again. cheers
Hide
Permalink
Greg Wilkins added a comment -

Inspecting 6.1.6, I think we have protection in for this NPE now.

Please reopen if you see similar error with 6.1.6 or later.

Show
Greg Wilkins added a comment - Inspecting 6.1.6, I think we have protection in for this NPE now. Please reopen if you see similar error with 6.1.6 or later.
Hide
Permalink
Ben Tatham added a comment -

This is occurring for me in Jetty 6.1.27.

My case may be a bit strange though as my servlet does pause the request thread for a very long time, and wakes up after another thread may have caused the destroy. (I would use park, but my servlet also needs to run in Jetty5)

Thread [LiveAndHistoricalNpRetrievalStreamer-4] (Suspended (exception NullPointerException))
HttpFields$2.hasNext() line: 269
HttpGenerator.completeHeader(HttpFields, boolean) line: 397
HttpConnection.commitResponse(boolean) line: 632
HttpConnection$Output.close() line: 995
BufferedOutputStream(FilterOutputStream).close() line: 143
DataOutputStream(FilterOutputStream).close() line: 143
LiveNpRetrievalServlet$1(LiveAndHistoricalNpRetrievalStreamer).cleanupAndStop() line: 391
LiveNpRetrievalServlet$1(LiveAndHistoricalNpRetrievalStreamer).completed() line: 110
LiveNpRetrievalServlet$1.completed() line: 98
LiveNpRetrievalServlet$1(AbstractBufferedConsumer<T>).run() line: 289
Thread.run() line: 619

Show
Ben Tatham added a comment - This is occurring for me in Jetty 6.1.27. My case may be a bit strange though as my servlet does pause the request thread for a very long time, and wakes up after another thread may have caused the destroy. (I would use park, but my servlet also needs to run in Jetty5) Thread [LiveAndHistoricalNpRetrievalStreamer-4] (Suspended (exception NullPointerException)) HttpFields$2.hasNext() line: 269 HttpGenerator.completeHeader(HttpFields, boolean) line: 397 HttpConnection.commitResponse(boolean) line: 632 HttpConnection$Output.close() line: 995 BufferedOutputStream(FilterOutputStream).close() line: 143 DataOutputStream(FilterOutputStream).close() line: 143 LiveNpRetrievalServlet$1(LiveAndHistoricalNpRetrievalStreamer).cleanupAndStop() line: 391 LiveNpRetrievalServlet$1(LiveAndHistoricalNpRetrievalStreamer).completed() line: 110 LiveNpRetrievalServlet$1.completed() line: 98 LiveNpRetrievalServlet$1(AbstractBufferedConsumer<T>).run() line: 289 Thread.run() line: 619

People

  • Assignee:
    Greg Wilkins
    Reporter:
    Pankaj Arora
Vote (0)
Watch (1)

Dates

  • Created:
    Updated:
    Resolved: