Process .getText() GDK method blocks on some operating systems.

Details

  • Type: Improvement Improvement
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: 1.5.5
  • Fix Version/s: 1.6-rc-1
  • Component/s: groovy-jdk
  • Labels:
    None
  • Environment:
    Windows, perhaps others. See JavaDoc for java.lang.Process.

Description

Pulled from the USER email group on 11/14/2007:

> i am executing maven reports inside a certain folder with
> 

new File('.').eachDir{
  //ignore config-dirs
  if(it.name.startsWith("."))
    return
  println "Executing site:deploy task für $it/pom.xml"
  def processOutput="mvn.bat -f $it/pom.xml site-deploy".execute().text
  print processOutput
}

>
> in some cases somehow the process hangs, i don't know why because the processOutput only gets displayed after
> the process exited. how is it possible to print out continously the stdout/stderr of the process to the console
> with "anyCommand".execute()?

And this is from the JavaDoc for java.lang.Process:

"Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block, and even deadlock."

Two simple solutions come to mind:
(1) .getText() actively reads and buffers stderr and stdout in two separate threads.
(2) Subclass java.lang.Process so that stderr and stdout are streams are overridden and read into buffered streams using separate threads.

And, for extra credit, there are issues with synchronizing output between stderr and stdout. If you are reading both in separate threads and printing to console or a log file, the outputs can come in the wrong order depending on when the threads were executed, and in what order. Maybe there is a better way to solve this problem.

Perhaps there is an easy way to pipe both into a StringBuffer???

I am calling this a "critical" bug because being able to run external processes simply and elegantly is pretty central to any scripting-language-type uses, and the current implementation is clearly broken (but in a subtle way).

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Jürgen Hermann added a comment -

I'd favour this syntax:

"cmd".execute() << { input } >> { output } >> { error }

with input being optional and if error is missing, both streams being synchronized to the output closure (if you want to discard stderr only, just add >> {} at the end).

Also, there is another issue to be taken care of, namely whether the processing should be line-wise (text, the default) or block-wise (binary).

Show
Jürgen Hermann added a comment - I'd favour this syntax: "cmd".execute() << { input } >> { output } >> { error } with input being optional and if error is missing, both streams being synchronized to the output closure (if you want to discard stderr only, just add >> {} at the end). Also, there is another issue to be taken care of, namely whether the processing should be line-wise (text, the default) or block-wise (binary).
Hide
Permalink
Paul King added a comment -

add code tags

Show
Paul King added a comment - add code tags
Hide
Permalink
Paul King added a comment -

There are now some additional methods in DGM to support things like below:

proc1 = 'ls'.execute()
proc2 = 'tr -d o'.execute()
proc3 = 'tr -d e'.execute()
proc4 = 'tr -d i'.execute()
proc1 | proc2 | proc3 | proc4
proc4.waitFor()
if (proc4.exitValue())
    print proc4.err.text
else
    print proc4.text

Which will work if the streams are small and if run in a directory containing 'testfile.groovy' will output 'tstfl.grvy'.

And to consume streams you could do something like this:

def sout = new StringBuffer()
def serr = new StringBuffer()
proc2 = 'tr -d o'.execute()
proc3 = 'tr -d e'.execute()
proc4 = 'tr -d i'.execute()
proc4.consumeProcessOutput(sout, serr)
proc2 | proc3 | proc4
[proc2, proc3].each{ it.consumeProcessErrorStream(serr) }
proc2.withWriter { writer ->
    writer << 'testfile.groovy'
}
proc4.waitForOrKill(1000)
println 'sout: ' + sout
println 'serr: ' + serr

Which will output:

sout: tstfl.grvy
serr:
Show
Paul King added a comment - There are now some additional methods in DGM to support things like below: 
proc1 = 'ls'.execute()
proc2 = 'tr -d o'.execute()
proc3 = 'tr -d e'.execute()
proc4 = 'tr -d i'.execute()
proc1 | proc2 | proc3 | proc4
proc4.waitFor()
if (proc4.exitValue())
    print proc4.err.text
else
    print proc4.text
Which will work if the streams are small and if run in a directory containing 'testfile.groovy' will output 'tstfl.grvy'. And to consume streams you could do something like this: 
def sout = new StringBuffer()
def serr = new StringBuffer()
proc2 = 'tr -d o'.execute()
proc3 = 'tr -d e'.execute()
proc4 = 'tr -d i'.execute()
proc4.consumeProcessOutput(sout, serr)
proc2 | proc3 | proc4
[proc2, proc3].each{ it.consumeProcessErrorStream(serr) }
proc2.withWriter { writer ->
    writer << 'testfile.groovy'
}
proc4.waitForOrKill(1000)
println 'sout: ' + sout
println 'serr: ' + serr
Which will output: 
sout: tstfl.grvy
serr:
Hide
Permalink
Paul King added a comment -

pipeTo changed to support binary streams not just text streams, so now this works too:

def sout = new StringBuffer()
def serr = new StringBuffer()
proc1 = 'gzip -c'.execute()
proc2 = 'gunzip -c'.execute()
proc2.consumeProcessOutput(sout, serr)
proc1 | proc2
proc1.consumeProcessErrorStream(serr)
proc1.withWriter { writer ->
    writer << 'test text'
}
proc2.waitForOrKill(1000)
println 'sout: ' + sout // => test text
println 'serr: ' + serr
Show
Paul King added a comment - pipeTo changed to support binary streams not just text streams, so now this works too: 
def sout = new StringBuffer()
def serr = new StringBuffer()
proc1 = 'gzip -c'.execute()
proc2 = 'gunzip -c'.execute()
proc2.consumeProcessOutput(sout, serr)
proc1 | proc2
proc1.consumeProcessErrorStream(serr)
proc1.withWriter { writer ->
    writer << 'test text'
}
proc2.waitForOrKill(1000)
println 'sout: ' + sout // => test text
println 'serr: ' + serr
Hide
Permalink
Hans Dockter added a comment -

Two things:

It would be cool to have a fix for either of these in 1.5.5.

Show
Hans Dockter added a comment - Two things: It would be cool to have a fix for either of these in 1.5.5.
Hide
Permalink
Paul King added a comment -

Split off enhancements to get around this issue into GROOVY-2758.
This issue now remains as purely the enhancement of Groovy so that simply by using the straight getText() etc methods there would be no deadlock. This would involve changing the plumbing underneath a bit and perhaps using ProcessBuilder.

Show
Paul King added a comment - Split off enhancements to get around this issue into GROOVY-2758. This issue now remains as purely the enhancement of Groovy so that simply by using the straight getText() etc methods there would be no deadlock. This would involve changing the plumbing underneath a bit and perhaps using ProcessBuilder.
Hide
Permalink
blackdrag blackdrag added a comment -

having a non-blocking getText() means that err must be read too, because if err is full, getText) will still block. On the other hand we won't know if the user wants to use the err part. Therefor I think the consumeProcessOutput methods are ok for that. Meaning this issue should be closed now.. Paul, what do you think?

Show
blackdrag blackdrag added a comment - having a non-blocking getText() means that err must be read too, because if err is full, getText) will still block. On the other hand we won't know if the user wants to use the err part. Therefor I think the consumeProcessOutput methods are ok for that. Meaning this issue should be closed now.. Paul, what do you think?
Hide
Permalink
Paul King added a comment -

I guess so. I was planning to look more at ProcessBuilder to see if it gave us anything but I don't think it helps this case.

Show
Paul King added a comment - I guess so. I was planning to look more at ProcessBuilder to see if it gave us anything but I don't think it helps this case.

People

  • Assignee:
    Paul King
    Reporter:
    Jason Smith
Vote (3)
Watch (3)

Dates

  • Created:
    Updated:
    Resolved: