CGI servlet hangs if CGI script fills STDERR buffer

We've been experiencing occasional hanging behaviour for a while now, but have only now been able to track down the cause.

It seems that the Jetty CGI servlet only reads from the CGI process' STDOUT handle, and STDERR is simply buffered. If that buffer becomes full, the CGI process will block when it tries to write to STDERR, and Jetty will block (waiting for data to be written to STDOUT). Since the CGI process is blocked, it will never write to (or close) STDOUT and then whole system hangs until something is killed.

The following perl script can be used to reproduce the problem...

#!/usr/bin/perl -w

use strict;

1. Flood the STDERR buffer
   foreach (1..5000) {
   print STDERR "mds \n";
   }

1. Write out some data
   use CGI;
   my $cgi = new CGI;
   print $cgi->header();

print "Hi!\n";

The specific line where I've seen this blocking behaviour is the following one (in CGI.java)

(line 369)
while( (b = is.read()) != -1 && b != (int) '\n' ) {

with the is.read() call being the culprit...

Not sure what exactly to suggest as a solution, but clearly the STDERR handle needs to be read (even if the output is ignored).

See also http://www.faqs.org/faqs/computer-lang/java/programmers/faq/

6. (Sect. 18) How do I execute a command from Java?

[*] Use

Runtime.getRuntime().exec( myCommandString )

where myCommandString is something like "/full/pathname/command". An
applet will need to be signed in order to allow this.

Note, there is a gotcha associated with reading output from commands.
When the runtime exec's the process, it passes to it 3 streams, for
stdin, stdout, and stderr; the out and err are buffered but the buffer
size isn't very big. When your process runs, it reads (if needed) from
in, and writes to out and err. If it doesn't write more than the
buffer-size, it can run to completion.

But if it tries to write more data to one or the other stream than the
buffer can hold, the write blocks, and your process hangs, waiting for
you to empty the buffer so it can write some more.

So after the exec call, get the streams, and read from them in a loop
until they both hit end-of-stream (don't block on either one, just read
whatever is available from each, each loop iteration). Then when the
streams have ended, call the process.waitFor() method to let it finish
dying.