JRuby

Gaps in STDIN pipe stream if backtick is used

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: JRuby 1.4
  • Fix Version/s: JRuby 1.7.0.pre1
  • Component/s: Core Classes/Modules
  • Labels:
    None
  • Environment:
    Mac OSX, Ubuntu Karmic
  • Testcase included:
    yes
  • Number of attachments :
    1

Description

If you use a backtick operator, any data going through STDIN ends up going to that process during the time it is executing. Attached find an example that if you run it via:

cat test_file.txt | jruby test.rb

You will see "nil" whereas

cat test_file.txt | ruby test.rb

You will see "hello"

The reason for this is because of the presence of a `ls` on the first line of the script. Apparently STDIN gets corrupted due to this backtick.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Greg Fodor added a comment -

Actually it appears to be even trickier than I first gave it credit for. It appears that in MRI ruby, if you use a backtick and the child process never reads STDIN, the script keeps STDIN. If the child process reads STDIN, it gets forwarded.

If you change the above example to `cat` instead of `ls`, then ruby and jruby have the same behavior. (You can verify this further by changing it to `cat > stdout.txt` which will then cause stdout.txt to contain the contents of the piped file into the script.)

Show
Greg Fodor added a comment - Actually it appears to be even trickier than I first gave it credit for. It appears that in MRI ruby, if you use a backtick and the child process never reads STDIN, the script keeps STDIN. If the child process reads STDIN, it gets forwarded. If you change the above example to `cat` instead of `ls`, then ruby and jruby have the same behavior. (You can verify this further by changing it to `cat > stdout.txt` which will then cause stdout.txt to contain the contents of the piped file into the script.)
Hide
Permalink
Charles Oliver Nutter added a comment -

This appears to be working on JRuby 1.5. Can you give it a shot and reopen if you still see problems?

Show
Charles Oliver Nutter added a comment - This appears to be working on JRuby 1.5. Can you give it a shot and reopen if you still see problems?
Hide
Permalink
Greg Fodor added a comment -

No luck – just tried 1.5.0 dev snapshot from Mar 08 on my 10.6 Mac – same behavior.

Show
Greg Fodor added a comment - No luck – just tried 1.5.0 dev snapshot from Mar 08 on my 10.6 Mac – same behavior.
Hide
Permalink
Vladimir Sizikov added a comment -

I can reproduce with the latest jruby 1.5.0.dev on Windows just fine. So, I'm reopening.

Show
Vladimir Sizikov added a comment - I can reproduce with the latest jruby 1.5.0.dev on Windows just fine. So, I'm reopening.
Hide
Permalink
Vladimir Sizikov added a comment -

The same problem is not only with backtick, but with system() as well.

Show
Vladimir Sizikov added a comment - The same problem is not only with backtick, but with system() as well.
Hide
Permalink
Charles Oliver Nutter added a comment -

This may be a JDK bug:

~/projects/jruby/tmp ➔ pickjdk 2
New JDK: 1.5.0-leopard

~/projects/jruby/tmp ➔ cat test_file.txt | jruby test.rb 
nil

~/projects/jruby/tmp ➔ pickjdk 3
New JDK: 1.6.0

~/projects/jruby/tmp ➔ cat test_file.txt | jruby test.rb 
hello

~/projects/jruby/tmp ➔ pickjdk 4
New JDK: java-1.7.0-internal-2010_03_07

~/projects/jruby/tmp ➔ cat test_file.txt | jruby test.rb 
hello
Show
Charles Oliver Nutter added a comment - This may be a JDK bug: ~/projects/jruby/tmp ➔ pickjdk 2 New JDK: 1.5.0-leopard ~/projects/jruby/tmp ➔ cat test_file.txt | jruby test.rb nil ~/projects/jruby/tmp ➔ pickjdk 3 New JDK: 1.6.0 ~/projects/jruby/tmp ➔ cat test_file.txt | jruby test.rb hello ~/projects/jruby/tmp ➔ pickjdk 4 New JDK: java-1.7.0-internal-2010_03_07 ~/projects/jruby/tmp ➔ cat test_file.txt | jruby test.rb hello
Hide
Permalink
Charles Oliver Nutter added a comment -

I don't think this is a JDK bug anymore. I think what's actually happening is that the backquoted subprocess is just quicker to eat up the stdin stream in the Java 5 case, for whatever reason.

The problem here is not an easy one to solve. In order to allow backquoted subprocesses to properly receive input, we must spin up a thread that actively reads from the parent stdin and writes to the child stdin. If the child doesn't use that data, it just gets lost forever. If the JVM allowed us to properly inherit the stdio streams in the child process, there would also be no issue; the subprocess would consume data on the parent's stdin only if it wanted it.

I'll poke around for other ways to solve this.

Show
Charles Oliver Nutter added a comment - I don't think this is a JDK bug anymore. I think what's actually happening is that the backquoted subprocess is just quicker to eat up the stdin stream in the Java 5 case, for whatever reason. The problem here is not an easy one to solve. In order to allow backquoted subprocesses to properly receive input, we must spin up a thread that actively reads from the parent stdin and writes to the child stdin. If the child doesn't use that data, it just gets lost forever. If the JVM allowed us to properly inherit the stdio streams in the child process, there would also be no issue; the subprocess would consume data on the parent's stdin only if it wanted it. I'll poke around for other ways to solve this.
Hide
Permalink
Charles Oliver Nutter added a comment -

This is something we won't be able to fix in 1.5. The issues I describe in the previous comment can't be resolved without rewriting how we launch processes to use something really native, like posix_spawn, so that the child process actually gets the parent process's IO streams and no pumping is necessary. Java 7 also provides ways to launch subprocesses with various IO descriptor sharing models. In either case, it's not a fix we can make right now and not a fix that will work everywhere (posix_spawn isn't available on Windows and Java 7 isn't even released yet).

In the interim, you can work around this by using popen instead of system or ``, like this:

io = IO.popen('ls')
data = io.read

Since popen does not give the child process the parent's streams, this does not cause it to consume stdin inadvertently.

Show
Charles Oliver Nutter added a comment - This is something we won't be able to fix in 1.5. The issues I describe in the previous comment can't be resolved without rewriting how we launch processes to use something really native, like posix_spawn, so that the child process actually gets the parent process's IO streams and no pumping is necessary. Java 7 also provides ways to launch subprocesses with various IO descriptor sharing models. In either case, it's not a fix we can make right now and not a fix that will work everywhere (posix_spawn isn't available on Windows and Java 7 isn't even released yet). In the interim, you can work around this by using popen instead of system or ``, like this: io = IO.popen('ls') data = io.read Since popen does not give the child process the parent's streams, this does not cause it to consume stdin inadvertently.
Hide
Permalink
Charles Oliver Nutter added a comment -

For JRuby 1.7pre1 on Java 7, this should be fixed; TTY should be handled correctly. For other Java versions, we can't fix this.

Show
Charles Oliver Nutter added a comment - For JRuby 1.7pre1 on Java 7, this should be fixed; TTY should be handled correctly. For other Java versions, we can't fix this.

People

  • Assignee:
    Charles Oliver Nutter
    Reporter:
    Greg Fodor
Vote (0)
Watch (3)

Dates

  • Created:
    Updated:
    Resolved: