ArrayIndexOutOfBoundsException from JSP file which is printing/writing long lines

I haven't been able to produce testcase yet - a simple "dump lots of stuff w/o flushing" test passes.

However, the jsp works unmodified on Jetty 5.1.10, if this helps in any way.

Still trying to make testcase by destilling the problematic jsp..

Well, this isn't JSP specific, it seems.

I'll attach a war containing a small Servlet that at least in my environment gives this Exception reliably under specific circumstances:

Unpack fresh Jetty 6.0.1, put crash.war in webapps dir, start jetty, and hit
http://localhost:8080/crash/crashtest
with a browser, or wget it.

This outputs some 50 char text 5000 times with 20 ms delay. Disconnect the client (click stop, or Ctrl+C if you're wgetting).

At some point later, the ArrayIndexOutOfBoundsException comes. (at least in my world!).

The problem seems to be that the out.write will never throw an IOException as most ServletContainers do if the client side has lost interest (closed connection). Instead it will apparently fill and fill on some buffer(s), untill something probably fills up, but this isn't handled properly, and one gets an ArrayIndexOutOfBoundsException instead of a good'ol IOException.

One other point is that sometimes it takes a real long time after the disconnect before this exception comes, in particular with wget (does the browser send some "I'm off" message?). Check it out a bunch of times, sometimes you'll get up to 1400 more rounds before it crashes, the text is 50 chars, which at least is 70000 bytes.
IMHO, a verified closed connection (throwing IOExceptions at you when trying to write) shall propagate all the way instantly, letting the Servlet terminate its processing instantly.

Another point is why flushing doesn't fix this? It might seem like out.flush doesn't flush "all the way through"?

I think the problem goes deeper than just dropping an IOException or something like this: It might be that only having a slow client, or one that'll not accept in a while, will trig this behaviour, but I haven't tested this properly. I believe this since the problematic JSP in question did work on Resin and Jetty5, but in nearly every instance got AIOOB on jetty6. (wgetting the same stuff didn't give the Exception as often - might be that the outputted XML is hard to digest for firefox?). I thought about races, but it seems like only one thread is handling one request?

It seems to happen to both with the SelectChannel and Socket Connectors.

Both the java and class file is in the war, but if you want to compile it yourself: in webapps dir: mkdir crash, cd crash, unzip war, run:
javac -d WEB-INF/classes/ -cp ../../lib/servlet-api-2.5-6.0.1.jar endre/test/CrashTest.java

Aha, there's a comment field when attaching.. okay..!

This war-file will demonstrate (some aspect of) the problem I'm experiencing..

I also am hitting the same issue with jetty 6.0.1, and the issue is reproducible each and every time using local clients. I'm hitting jetty with firefox on a mac, and the issue pops up every time.

The problem in the test code is that it is using a PrintWriter (which is returned from response.getWriter()) to periodically send more data to the client, but the total time exceeds the time that the browser is prepared to keep the connection open. So, the browser closes the connection, but the client code continues to write data to it after it is closed. As the data is being written with a PrintWriter, no EOF exceptions can be thrown.

Rather, the client code should use writer.checkError() after each write() to flush the output and check for errors with the write.

Modifying the code to use checkError() solves the issue:
protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
PrintWriter out = (PrintWriter)resp.getWriter();
int sleeptime = 0;
boolean failure = false;
int i=0;
try {
for (; i < 5000 && !failure; i++) {
out.write(i+":Just writing something here. Kind regards, Endre");
failure = out.checkError();
if (!failure)

}
}
catch (Exception e)

if (failure)
System.err.println("Failure on iteration: "+i);
}

Check the web for more info on long-held http responses and the patterns for how to check for connection timeouts etc.

(Damn, I'd written a largish comment here, and it just went due to the "wiki markup help" button on the side there just crashes when you click it.)

• The initial problem with both this and JETTY-136 is about JSPs. I just tried to provoke the problem with a simpler test. A JSPWriter do throw IOExceptions on every output method, and the closing of the client should thus throw a SocketException all the way out.

• ArrayIndexOutOfBoundsException is at any rate not the correct thing to raise, in any codepath of a Servlet's accessing the Servlet API. This must be acknowledged. (Jetty 5 just correctly consumes all output, up to 4999, since there is no way to notify the client unless it does a checkError() - but Jetty 6 throws an AIOOBE).

• There must be some underlying problem at hand, which isn't properly demonstrated correctly by the actual killing of the client as such.

• If you rewrite my doGet method to take the OutputStream, wrap this in a OutputStreamWriter, then you immediately get the Exception. This is however an "org.mortbay.jetty.EofException" in both jetty5 and jetty6, wrapping the correct java.net.SocketException in jetty 5, and a java.io.IOException in jetty 6 (coming from the SocketChannel). The test-case is obviously too limited to point to the problem in this case.

• But, the fact remains that the JSP in question worked, and works, in both Tomcat and Resin AND JETTY 5, but didn't, and doesn't, work in jetty 6.

• • (The problem was somewhat mitigated by reducing the sizes of the lines (as it started out being one huge line). When I had it cut down to really really small lines, it actually went through most of the time. This is of course not an option.)

I believe there is some underlying problem with "resource handling", resources referring to array buffers. Apparently jetty 6 introduced some fancy "split buffers", and my hunch is that this is at fault.

Think "slow clients" instead of "clients disconnecting", or clients just choking a bit on some hard-to-decode xml. Maybe the test-case doesn't point to it correctly, but WHY does it throw a ArrayIndexOutOfBounds at any rate?!

I certainly can't use Jetty 6 as it stands (I'll use Jetty 5 for development due to its small-and-cleanness, and I guess Tomcat for prod..)

Erik,

I agree that ArrayIndexOutOfBounds is not good to expose to the servlet. I'll change that. However, I'm curious that you say that the same app worked in jetty5. If the server side is slow to respond, the browser client will timeout the connection, so presumably this was happening also in jetty5. That being the case, if the servlet tried to write more content to the connection after it was closed, the browser client would never receive it. So when you say it was working in jetty5, do you mean that the browser always received all the output you were expecting to see, or do you mean that there was no exception on the server side (but that output was missing)?
regards,
Jan

I forgot to say that if you had a test jsp that demonstrated the problem, that would be a big help.
regards

Ah, ok, re-reading your comment it seems that the test case you sent does not reproduce exactly the the problem you see with your jsp. I see. In that case, a test jsp would definitely be a big help.

John, I can't submit the JSP, as it is a part of a extremely tangled system, which queries a database and a JNI-based library of Autonomy. I hate it.

(I must admit that I've already used way too much time on trying to give a proper hunt-down of this problem... Thus I haven't got the time to clear out every call to these services and replace them with some dummies - with the risk that it actually is the random lags that occur during these queries that are part of the "race-ish" problems..!)

BUT, what I meant, is that yes: all output came, and everything was fine, on Jetty 5, Resin (that's where this thing has been running for several years, obviously without these problems occuring on every invocation), and Tomcat.

The point is that Jetty 6 just midway in the processing comes up with this overflow. It's not always the same place in the output.

The output is a somewhat big XML file around 127.48 KB to be exact ! (yes - it "renders" XML. ). Since firefox renders this visually as a "proper XML tree", and uses 100% CPU while doing it, I can only assume that it "hiccups" a little while receiving this (It is also based upon the fact that if wgetting the same URL, it on some occations "came through", while in firefox it always crashed) - this is however pure speculation.

I think that you should "hunt down" the pathway_s_ that may lead to ArrayIndexOutOfBoundsException, and see if there isn't some "race-ish" situations where one piece of the code assumes that the buffer is output and cleared (or whatever), but the other piece of the code hasn't come to this point yet, since the client didn't consume the output afterall (slowish-ish) - thus when writing that one more byte, the buffer is full, instead of good too take on more data. See, what I'm getting at, is that I just can't understand a code-path that can results in a AIOOBE due to a consumer not consuming anymore: such a code-path just have to get to this point also when it has a slow client?!

(On other possibility that I can't positively rule out (since I haven't been running journal on the testing), is that there might be errors in the resulting xml that is introduced by jetty6, which isn't introduced in the other servlet containers. This would make firefox break the connection, as it displays "an error in the xml" page (this is what invariably happens), and thus one get the AIOOBE (in this scenario actually because of the client has broken the connection, not the other way around). What I mean is that what if there is "interleaving" in the buffers? Repeats, or overwrites? What i REALLY mean is: is this thing synched properly, are the buffer-management 100%, and is the byte-accounting absolutely correct, even if both the production is in fits-and-starts (due to way to much requests out to different systems), and the consumption (due to "hard to parse" results?)

Jack, I might think that you skim the comments to fast.

I say that the test-case MIGHT not reproduce the same conditions as the JSP. It does however give the exact same Exception, obviously with a slightly different trace, and I, in my idiocy, thought that this COULD lead you folks to find the underlying fault.

Apparently it doesn't. Too bad. I still have this problem exclusively with Jetty6, not any other servlet container I've tried.

I've committed a fix to svn trunk. The ArrayIndexOutOfBounds exception will no longer occur if the connection is closed while it is still being written to. Please can you confirm this fixes your issue.

Joe, one more comment: if a browser gets data, it won't close the connection. I've never had a browser that is fed data regularly close on me. If you don't give it a byte in a long while, it'll close. But not if yoy give it data regularly. No browser I know of will close on the test case unless you stop the browser yourself.

But, as I mentioned, at least firefox will close midway in the xml-case if suddenly the xml is wrong. This could be the case (I haven't thought of it this way before now, thus I can't say for sure).

Okay, I think I got it. And I'm a sorry face now..

Forget the theory of the browser closing connection due to xml-faults.

I have re-written the JSP into a Servlet. This still fails in jetty6.

After way too much time on this shit, finally exchanging the way-to-many SQL-queries with a random sleep of 0-500 ms, it still happened (see, it didn't happen if I just removed them).

Then it suddenly struck me - let's time this:

endre@endrebox:~/temp$ time wget 'http://localhost:8080/<blah blah>'
--14:54:19--  http://localhost:8080/<blah blah>
           => `sttq?cid=1905&querytext=fradrag+i+skatt+for+forskning+og+utvikling&sessionid=ENDRE_TEST'
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:8080... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/xml]

    [  <=>                                                  ] 282,369       10.99K/s

14:54:49 (10.12 KB/s) - `<blah blah>' saved [282369]

real    0m30.013s
user    0m0.004s
sys     0m0.004s

Exactly 30 seconds. Oh no.

(Note that the final output shall be 366,090 bytes, not ~= 300,000

I did read the comment by Jeffrey in JETTY-136 about "maxIdleTime" being 30 seconds, but I assumed that this didn't apply to me: I output data all the time, just at a slow pace - it was never idle.

So, you folks just close the channel if it "idles out"??? But why would a connection that have had 10 kB/s running through it for 30 seconds be deemed idle?

Oh my god.

So even though a Servlet actually outputs data all the time, the channel/socket is deemed "idle" if it has been in operation for more than 30 seconds, and is forcibly closed down? (It is apparently cleanly closed, as the client is happy with the file he gets..)
Then, due to your absolutely fantastic handling of closed sockets, a buffer fills up (How can that possibly become the outcome?!). And then we get a ArrayIndexOutOfBoundsException?! (My point is, shouldn't just the "print" or "write" call block instaed of over-filling a buffer? I really can't seem to grasp this - why wouldn't this happen also if the client just happens to accept data very slowly?)

I would have appreciated several things different here. But most of all, the RuntimeException in question could have been some "SocketClosedDueToInactivityException" or some other intelligent stuff - given that there's actually a Servlet in progress here, happily outputting data, assuming that the Container it lives in doesn't stab it in its back by shutting down his socket w/o him knowing - and then giving an utterly bogus explaination about what happened.
EOFException, ArrayIndexOutOfBoundsException, ChannelClosedException, SocketException doesn't really cut it, don't you agree??

And why, if the point about "maxIdleTIme" (I don't believe this - the name is so god-damn wrong) is to halt runaway processes or whatever, don't you shove a rock-hard RuntimeException up its face upon trying to output more to the output sink, whether it is a PrintWriter or a OutputStream????

This was very bad.

Assigning to Greg to verify operation of the idle timeout on connections.

Endre - whilst I understand you are frustrated, please try and remember that we are spending our time to help you for free. And by the way, it's JAN - not John, Jack or Joe!

Jan, yes, and it's Endre (not Erik)!

Yes, I know you're using your time - and for that I am very happy - Jetty 5 gave me a very good platform for development.

It is just that when I hit this smack-in-my-face bug, used a whole bunch of hours to try track it down, realize that it's probably the servlet container's fault, tried to comment as good as I can, trying to make a test-case and such, it is "slightly annoying" to get "this ain't a bug, go read some stuff on the web about http connection, n00b - resolving /WONTFIX/" back.

But sorry for yelling..

Re: the maxIdleTime thing: how can Jetty 6 be able to serve a 1 MB file over a modem, if a slow but live connection is deemed "idle"?

Sorry for slow frustrating service on this one.....

Was it me that said this aint a bug?

Cause it is a bug.... if you are writing content, then the connection is not idle.

I will be fixed probably over the next 36 hours while have a planes trains and automobiles experience
getting home!!!!!

Ah, relief!

(But of course, the real pain has now subsided somewhat, since I just increased the timeout to 120 seconds.)

I still however worry hard about the AIOOBE, as I can't understand why that would be the end-result at any rate: If this can happen when the connection is closed, why couldn't it also happen in the right (wrong!) circumstances, for example if the producer (servlet) is really fast, outputting a dual-layer HD-DVD directly from memory, and the consumer is a TCP/IP enabled Commodore 64 over a acoustically coupled 9600 baud modem in a very noisy room? My absolutely main point is: why doesn't out.write/print/whatever block instead of going over bounds?

PS: have a nice trip home, then! 36 hrs "on the road"? Sounds really .. refreshing?!

Reopening for Greg.

I am checking in a version to trunk that resets the idle timeout when content is written.

However... I don't like the inefficiency of this.... so I am thinking that I will change jetty 6 to not apply idle timeout
unless something is actually trying to read content - this is the way blocking connectors have always worked and
I don't think it is in violation of the spec.

There is little point in jetty idling out and just closing the IO connection, as the process will continue until it does
a write. The browser will do exactly the same waiting and closing!

So I will work on another version that avoids write idle processing all together.

I have reconsidered my reconsideration.....

I have found a more efficient way to handle idle timeouts (with much less calling of System.currentTimeMillis() )
Also there is a usecase for programmable timers....

so I will not change the basic action for now. But I will leave this issue open as I want to check exactly why the aioobe
was happening !

The last comment was nice to see - I really wonder about that (the AIOOBE).

However, the ilde timeout stuff, I can't really grok it. At all.

See, I could understand if this was to try to kill runaway servlets or something, but if this was the case, any access to any API-method (provided by the container) should preferrably throw some runtime exception. The default should be set to never, as it probably is in heavy violation of the API-spec - but it is definately a interesting feature (maybe it should be possible to disable it for download servlets, e.g. by setting some special jetty-attribute on the ServletRequest?).

Basically, I don't quite get the entire idea behind "idle timeouts", at least as it stands. If you've lent a connection to a servlet, then that's it (And I've realized that that's also what JETTY-80 tried to point out - this whole long winding thread here is actually about an already identified since June unfixed bug - you opened JETTY-83 yourself as a follow-up). Taking away ("releasing") the connection from the servlet doesn't really help anything - if the servlet's fubarred, then I can't seem to understand the big point of reclaiming the small resource that a connection is, compared to a whole thread potentially spinning in some loop or whatever. The only thing that could fix this, would be to kill the thread by some way or the other, and until Isolates are here, there's no way to do that reliably. But at least throwing IOException on all write methods on OutputStream, and write/print methods on JSPWriter, is allowed by spec (But let this be some sane, informative extension of IOException, with a message that clearly states what have happened), but this "idle timeout" idea is not, in my opinion: Where does the servlet spec open for random killing of a servlet's connection to the client, regardless of whether it's taking 1 ms, or 1 week, between its byte-outputs?!

What's the thing about "programmable timers"?

The idle timeout is not to catch a bad servlet but to prevent and evil client doing a simple DOS attack.

In the blocking world... all is nice because you have timeouts on read and write provided by the JVM.
if you block for too long... then bam! IOException.

The problem is that once you go into non-blocking world, there are no JVM provided timeouts that
one can use and we have to implement them ourselves.

I had implemented a timeout that applied to any read - I have now extended writes to wakeup the
idle timeout... but I think that is the wrong direction. Instead I should apply the idle timeout only
to reads between requests and blocking reads for input and blocking writes for output. This will mimic
the behaviour of the current blocking IO in servlets.

The programmable timers is that there is a use-case to send 1xx responses to keep a connection open while
a slow servlet thinks about generating the response.

Okay, thanks for the lecture - I guess I didn't realize this difference.

Yes, blocking - that's what I wonder about, in particular in regard to the AIOOBE.

So basically, what you're saying is that yes it was a bug, since writing wasn't being considered as activity at all. However, one really have to have some way to "shut down a connection" in case some client opens a whole bunch of connections, but then never either write to or reads from them (both non-actions would be DoS-ing) - you'd stay there forever w/o knowing that the guy on the other end doesn't really give nor take your bytes?

Okay, in that case, I think I follow the logic - but then I definately agree that this should only apply if the client doesn't accept any bytes (thus having the servlet block), or the other way, that the servlet tries to read, but the client doesn't feed it bytes. These are basically both an issue of blocking, not about "idle", as such? Then why not call the feature something like "maxBlockingTime" or something that points slightly more directly at the underlying problem?

In any case, it would be a good to properly notify the servlet, as it was in the olden days with blocking: you'd get an IOException if this struck. Both the OutputStream and the JSPWriter gives this option - but not the PrintWriter, except via the checkError(). I suggest making this one or more extensions of IOException that is called something like "ConnectionClosedDueToClient[Read|Write]BlockTimeExceededException" or something, including a nice Exception message, that would give immediate understanding of what happened, to the developer.

One more thing on a tangent: the magic buffers stuff you've implemented now: do I understand that correctly if the point is to let a servlet finish earlier than the client actually accepting the bytes?

Last thing: this bug should then really not happen if using the SocketConnector instead of SelectChannel Connector, if I understand correctly?

I've checked in a new idle time fix. Idle timeouts now only apply to blocking operations and it is a much cleaner solutions and should be a little faster as well.

The "magic buffers" are there for many reasons, but one reason is to allow async flushing after a sync servlet has completed.

Will look at the AIOOBE now - but I think that actually has already been fixed from some other changes.

The AIOOBE has been fixed by resetting the buffer as write is called rather than as write exits. This prevents exceptions bypassing the write.

Sounds good. Will output actually block when all the buffers are full?

However, I really think that "maxIdleTime" is a wrong name for the parameter. The servlet isn't idling - on the contrary, it is desperate to output (or input) more stash, not just hang around. The connection isn't idiling either - it is blocked.

Deprecate the parameter (make it still work), and call it "maxBlockingTime" or something.

• .. or else I'll bet a lot of developers and administrators won't get what the configuration parameter "maxIdleTime" does, thus wasting time wondering about it, or potentially misconfiguring it.

It blocks when the buffer is full/empty for output/input

The name is not optimal, but it has been named that for long time in Jetty.
The primary purpose is for the idle time between one request and the next.
It could probably be called keepAliveTime - but I think I will stick with maxIdle
for histories sake! I will work on the doco to avoid confusion.

cheers

Sorry for not letting this go! (but I can't help it..! I tend to get somewhat hung up in names of interfaces, classes, variables and .. parameters!)

The name as you describe it in historic setting makes sense, and I remember it now - I even produced this development config file for Jetty5 here that sets it to 0 since it annoyinglys, repeatedly output some stupid exception after the specified amount of time. It was called "setIdleTimeMs" at that time, right?

Have you might re-used, or over-used, this specific parameter to one more, somewhat unrelated, setting, when it comes to Channels? The time a socket should idle between requests, vs. the time a channel should block when within a request, doesn't seem overly related to me.. (And people don't read docs unless they have to. Thus the names of parameters should be RIGHT! (and the docs should preferrably be good, too!))

I'm not sure if this is the same issue:
With revision 1239 I get an AIOOBE when writing 6554 lines of 10 characters to the writer without flushing.
In tomcat 5.5 it works fine. Reproduction is on http://www.greenhills.co.uk/mak/jetty/aioobe.war.

Martin,

that would be because I am an idiot as I don't reset the size of space within my inner loop....
so of course I wont flush.... except in a few lucky cases!

fix (and actually an optimization!) is about to be checked in....
but I will make a test harness first for this!

OK really fixed this time!!!!!
with a test harness and a check of code coverage and everything!!!!!!

Verified fixed in 1263, with both aioobe.war, and my actual application.
Thanks Greg!