Issue Details (XML | Word | Printable)

Key: JETTY-748
Type: Task Task
Status: Reopened Reopened
Priority: Minor Minor
Assignee: Greg Wilkins
Reporter: Greg Wilkins
Votes: 0
Watchers: 4
Operations

If you were logged in you would be able to see more operations.
Jetty

Assist Hadoop to port to jetty-6 and optimize

Created: 16/Oct/08 11:34 PM   Updated: 31/Aug/09 05:29 AM
Return to search
Component/s: HTTP
Affects Version/s: None
Fix Version/s: None

Time Tracking:
Not Specified

File Attachments: 1. Java Source File OpenCloseTest.java (2 kB)


Sort Order: Ascending order - Click to sort in descending order
Greg Wilkins made changes - 21/Oct/08 03:51 AM
Field Original Value New Value
Assignee Greg Wilkins [ gregw ]
[ Permalink | « Hide ]
Greg Wilkins added a comment - 14/Nov/08 05:13 AM

Added RandomAccessFileBuffer and test harness.
Updated AbstractGenerator so that such buffers passed with sendContent are efficiently sent.

now testing.... then will advise hadoop how to use.


[ Permalink | « Hide ]
Chris Douglas added a comment - 10/Dec/08 02:38 AM

We're hitting a wall with transferTo in Java SE 6 (HADOOP-4700). It looks like a particularly buggy Linux implementation, and most of the issues aren't fixed until JDK7. There are a few more things to try, but we may have to abandon this particular route for now. Has anyone had more success with this on Linux? Is it worth experimenting with blocking I/O, or is that a non-starter for Jetty6?

On the bright side, dropping the transferTo experiment means we can tune the Server and MapOutputServlet in more conventional ways. Where should we start?

There's currently an open issue (HADOOP-4744) that may be related to the Jetty6 migration. In allocating a medium/large cluster (~400-500 nodes), it looks like some of the jetty servers are failing to start. If there are strategies for troubleshooting this, any guidance would be appreciated.


[ Permalink | « Hide ]
Greg Wilkins added a comment - 10/Dec/08 05:02 PM

David,

can you have a quick look at these issues and summarize them to me.
Also can you make sure that the RandomAccessFileBuffer and it's support are
backported from jetty-7 to jetty-6.


Greg Wilkins made changes - 10/Dec/08 05:02 PM
Assignee Greg Wilkins [ gregw ] David Yu [ dyu ]
[ Permalink | « Hide ]
Devaraj Das added a comment - 11/Dec/08 01:10 PM

I recently commented on HADOOP-4744. Please have a look. Thanks!


[ Permalink | « Hide ]
Chris Douglas added a comment - 16/Dec/08 03:10 AM

We're seeing this pair of exceptions frequently:

org.mortbay.jetty.EofException
        at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:787)
        at org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:548)
        at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:569)
        at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:946)
        at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:646)
        at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:577)
        at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2872)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
        at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:324)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
        at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
        at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
Caused by: java.io.IOException: Broken pipe
        at sun.nio.ch.FileDispatcher.write0(Native Method)
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
        at sun.nio.ch.IOUtil.write(IOUtil.java:75)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
        at org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:169)
        at org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
        at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:721)
        ... 24 more

java.lang.IllegalStateException: Committed
        at org.mortbay.jetty.Response.resetBuffer(Response.java:994)
        at org.mortbay.jetty.Response.sendError(Response.java:240)
        at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2895)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
        at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:324)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
        at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
        at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)

This is running trunk without attempting to use RandomAccessFileBuffer. Have we misconfigured something? Is it worth worrying about?


[ Permalink | « Hide ]
Greg Wilkins added a comment - 16/Dec/08 07:07 AM

The Eof exception implies that your client is closing the connection while content is still being sent. Can this happen?
If so, then you can silently ignore(but log in debug) this exception (but best not to close while writing content).

The second exception looks like an attempt to generate an error page for the first exception - should not be a problem if you silently ignore the first exception.


[ Permalink | « Hide ]
Jan Bartel added a comment - 23/Dec/08 10:14 AM

Greg,

I checked in the change to AbstractGenerator.isBufferFull() see svn rev 4213. Please review.

Jan


[ Permalink | « Hide ]
David Yu added a comment - 01/Jul/09 03:42 AM

The invalid port returned from Connector.getLocalPort() was extremely random.
The accepted solution was simply to restart the connector and get the new valid port when its restarted.


David Yu made changes - 01/Jul/09 03:42 AM
Resolution Fixed [ 1 ]
Status Open [ 1 ] Resolved [ 5 ]
[ Permalink | « Hide ]
Owen O'Malley added a comment - 01/Jul/09 12:04 PM

That is an unfortunate response to an observed race condition.


[ Permalink | « Hide ]
Greg Wilkins added a comment - 02/Jul/09 03:40 AM

Owen,

We are working to remove this close race.
It's not yet 100% fixed, but we will address it soon.

I'll reopen this issue so that we can close it when we have the 100% fix, but I'll reduce the priority because the exception thrown is essentially just noise (in the case that we understand it).


Greg Wilkins made changes - 02/Jul/09 03:40 AM
Assignee David Yu [ dyu ] Greg Wilkins [ gregw ]
Status Resolved [ 5 ] Reopened [ 4 ]
Resolution Fixed [ 1 ]
Greg Wilkins made changes - 02/Jul/09 03:41 AM
Priority Major [ 3 ] Minor [ 4 ]
[ Permalink | « Hide ]
Greg Wilkins added a comment - 26/Aug/09 03:37 AM

Oh my previous comment was about the wrong race condition... I was talking about the issue if somebody closes a connection as it is being written to.

David was discussion a getLocalPort issue that I believe was fixed.
We have certainly not observed getLocalPort returning incorrect values and we heavily use it in our test harnesses.

But I'll reinvestigate anyway.


[ Permalink | « Hide ]
Greg Wilkins added a comment - 26/Aug/09 03:50 AM

https://issues.apache.org/jira/browse/HADOOP-4744 is closed, so I think the getLocalPort issue is indeed fixed.
The open issue associated with this is https://issues.apache.org/jira/browse/MAPREDUCE-258
Apparently int is not sufficient size for hadoop, so perhaps another approach is required.


[ Permalink | « Hide ]
Chris Douglas added a comment - 26/Aug/09 05:49 AM

As David said earlier, the "resolution" for HADOOP-4744 was to check if the race was lost, and retry. It looks like the race condition still exists.


[ Permalink | « Hide ]
Greg Wilkins added a comment - 26/Aug/09 08:50 AM

OK, so I've looked at this again, and I just cannot see how the localPort can be -1 unless the open fails somehow.

I've written a test harness to try to reproduce the problem, but it does not for me.
Could somebody who see the problem on hadoop run the OpenCloseTest and see if they can reproduce.

thanks


[ Permalink | « Hide ]
Greg Wilkins added a comment - 26/Aug/09 08:51 AM

pass this test a list of interface addresses on the command line


Greg Wilkins made changes - 26/Aug/09 08:51 AM
Attachment OpenCloseTest.java [ 44042 ]
[ Permalink | « Hide ]
Chris Douglas added a comment - 31/Aug/09 05:29 AM

I couldn't reproduce this with OpenCloseTest and our default interface. I'll try to reproduce in our specific use of Jetty in the next few weeks.
Atlassian JIRA (v4.0#466) | Report a problem | Request a feature | Contact administrators
Powered by a free Atlassian JIRA open source license for Codehaus. Try JIRA - bug tracking software for your team.