JRuby

Using backquotes to run commands with shell redirects sometimes passes the redirects to the called program

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Blocker Blocker
  • Resolution: Fixed
  • Affects Version/s: JRuby 1.0.0, JRuby 1.0.1, JRuby 1.0.2, JRuby 1.1b1
  • Fix Version/s: JRuby 1.1RC3
  • Component/s: Core Classes/Modules
  • Labels:
    None
  • Environment:
    Mac OS X 10.4.10 with Java 1.5, Linux with IBM Java 1.4.2
  • Testcase included:
    yes
  • Number of attachments :
    1

Description

Attached is a test case that uses backquotes to invoke a Ruby program named 'arguments.' The arguments program prints the contents of ARGV. This test case assumes the ruby and jruby commands are on your path.

Paste of the test case:

require "test/unit"

class BackquotesWithShellRedirectsTest < Test::Unit::TestCase
  def test_backquotes_with_redirects_pass_through_shell
    assert_equal "arguments: one two", `./arguments one two 2> /dev/null`, "wrong arguments when run straight up"
    assert_equal "arguments: three four", `ruby ./arguments three four 2> /dev/null`, "wrong arguments when run with ruby prefix"
    assert_equal "arguments: five six", `jruby ./arguments five six 2> /dev/null`, "wrong arguments when run with jruby prefix"
  end
end

Paste of the arguments program:

#!/usr/bin/env ruby

print "arguments: #{ARGV.join(' ')}"

This test case passes in MRI. It fails in JRuby, but at different points depending on which version of JRuby you're running. Here are some results.

JRuby 1.0, 1.0.1

fletcher@nugent ~/svn/jruby_bugs $ jruby test_backquotes_with_shell_redirects.rb 
Loaded suite test_backquotes_with_shell_redirects
Started
F
Finished in 0.2 seconds.

  1) Failure:
test_backquotes_with_redirects_pass_through_shell(BackquotesWithShellRedirectsTest) [test_backquotes_with_shell_redirects.rb:6]:
wrong arguments when run with ruby prefix.
<"arguments: three four"> expected but was
<"arguments: three four 2> /dev/null">.

1 tests, 2 assertions, 1 failures, 0 errors

JRuby 1.0.2, 1.1b1

fletcher@nugent ~/svn/jruby_bugs $ jruby test_backquotes_with_shell_redirects.rb 
Loaded suite test_backquotes_with_shell_redirects
Started
F
Finished in 0.132 seconds.

  1) Failure:
test_backquotes_with_redirects_pass_through_shell(BackquotesWithShellRedirectsTest) [test_backquotes_with_shell_redirects.rb:5]:
wrong arguments when run straight up.
<"arguments: one two"> expected but was
<"arguments: one two 2> /dev/null">.

1 tests, 1 assertions, 1 failures, 0 errors

This particular regression is preventing me from upgrading from JRuby 1.0 to 1.0.2. My acceptance test suite is using the first form of invocation (line 5 of the test case) to run my application, which is sensitive to the passed arguments. I'd rather not change my code to work around incorrect behavior if I don't have to, so I'm going to stick with 1.0 for now.

Issue Links

This issue relates to:
JRUBY-2235 Minor regression: Kernel.system('non-existing-file') produces some output but shouldn't Minor Closed

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Matt Fletcher added a comment -

I spent a little time looking through JRuby 1.0.2 code; in ShellLauncher.java, it looks like my case is falling in 'don't run in process, don't run in shell' category (line 212). This uses Java's Runtime#exec method to launch the program, which certainly isn't going to interpret shell redirects.

It looks like the reason my case is failing the shouldRunInShell check (line 308) is because the first argument (the program name) does exist. But I'm not clear on what this check is for so I'm not certain.

Show
Matt Fletcher added a comment - I spent a little time looking through JRuby 1.0.2 code; in ShellLauncher.java, it looks like my case is falling in 'don't run in process, don't run in shell' category (line 212). This uses Java's Runtime#exec method to launch the program, which certainly isn't going to interpret shell redirects. It looks like the reason my case is failing the shouldRunInShell check (line 308) is because the first argument (the program name) does exist. But I'm not clear on what this check is for so I'm not certain.
Hide
Permalink
Charles Oliver Nutter added a comment -

Ick. In other similar cases we have had to do the parsing of the command string ourselves, which is not going to scale. I half wonder if perhaps we should just fail gracefully in these cases and fall back on a slow out-of-process dispatch to the command shell.

I'm assigning this to Nick, since it's largely his code. Sorry Nick. We could also consider whether a JNA process launch might be a better compatibility option in some cases.

Show
Charles Oliver Nutter added a comment - Ick. In other similar cases we have had to do the parsing of the command string ourselves, which is not going to scale. I half wonder if perhaps we should just fail gracefully in these cases and fall back on a slow out-of-process dispatch to the command shell. I'm assigning this to Nick, since it's largely his code. Sorry Nick. We could also consider whether a JNA process launch might be a better compatibility option in some cases.
Hide
Permalink
Matt Fletcher added a comment -

From my perspective, I'd rather just always get a new process by default. I just filed another JIRA issue related to this problem (JRUBY-1579). It seems like it is quite difficult to poke into a command and decide if it is safe to run in the same VM.

I've been running into these problems because I have a system test suite that fires up my application in a variety of ways. I'm not sure what use cases benefit a lot from spawning new JRubys in the same instance and I'd hate to ignore those cases if they exist. In my case, I'd rather play it safe and always just get a new process, even if it is slower.

Show
Matt Fletcher added a comment - From my perspective, I'd rather just always get a new process by default. I just filed another JIRA issue related to this problem (JRUBY-1579). It seems like it is quite difficult to poke into a command and decide if it is safe to run in the same VM. I've been running into these problems because I have a system test suite that fires up my application in a variety of ways. I'm not sure what use cases benefit a lot from spawning new JRubys in the same instance and I'd hate to ignore those cases if they exist. In my case, I'd rather play it safe and always just get a new process, even if it is slower.
Hide
Permalink
Nick Sieger added a comment -

The first argument check was mostly for Windows, which had a pathname-with-spaces issue that I couldn't resolve unless I launched the program directly.

I could try to check for redirect operators on the command-line and launch the shell in that case.

Your best bet for a workaround is to prefix your command with "sh -c" or "bash -c" in which case it will get launched by the shell.

Show
Nick Sieger added a comment - The first argument check was mostly for Windows, which had a pathname-with-spaces issue that I couldn't resolve unless I launched the program directly. I could try to check for redirect operators on the command-line and launch the shell in that case. Your best bet for a workaround is to prefix your command with "sh -c" or "bash -c" in which case it will get launched by the shell.
Hide
Permalink
Stephen Bannasch added a comment -

I reported a similar issue in JRUBY-1676 which I marked as a duplicate of this issue.

I'm trying to use ara howards macaddr gem. This provides a way to get the mac address on windows, linux, and macosx. You can see the code here: macaddr.rb.

In it he uses code of this form (for testing on my mac systems – there are different commands for windows):

stdout = IO.popen("/sbin/ifconfig >2 /dev/null")

This works in MRI but not in JRuby.

Here's one simplification which shows the differences from:

MRI

mkdir emptydir
cd emptydir
irb
irb(main):002:0> p IO.popen("ls 2> /dev/null").readlines
[]
=> nil
irb(main):003:0> p IO.popen("ls ./ 2> /dev/null").readlines
[]
=> nil
irb(main):004:0> p IO.popen("/bin/ls").readlines
[]
=> nil
irb(main):005:0> p IO.popen("/bin/ls 2> /dev/null").readlines
[]
=> nil
irb(main):006:0> p IO.popen("/bin/ls ./ 2> /dev/null").readlines
[]
=> nil
irb(main):008:0* p IO.popen("ls abc").readlines
ls: abc: No such file or directory
[]
=> nil
irb(main):009:0> p IO.popen("ls abc 2> /dev/null").readlines
[]
=> nil
irb(main):011:0> p IO.popen("/bin/ls abc").readlines
ls: abc: No such file or directory
[]
=> nil
irb(main):012:0> p IO.popen("/bin/ls abc 2> /dev/null").readlines
[]
=> nil

and here's what happens in JRuby r5180:

mkdir emptydir
cd emptydir
jirb
irb(main):001:0> p IO.popen("ls").readlines
[]
=> nil
irb(main):002:0> p IO.popen("ls 2> /dev/null").readlines
[]
=> nil
irb(main):003:0> p IO.popen("ls ./ 2> /dev/null").readlines
[]
=> nil
irb(main):004:0> p IO.popen("/bin/ls").readlines
[]
=> nil
irb(main):005:0> p IO.popen("/bin/ls 2> /dev/null").readlines
["/dev/null\n"]
=> nil
irb(main):006:0> p IO.popen("/bin/ls ./ 2> /dev/null").readlines
["/dev/null\n", "\n", "./:\n"]
=> nil
irb(main):008:0* p IO.popen("ls abc").readlines
[]
=> nil
irb(main):009:0> p IO.popen("ls abc 2> /dev/null").readlines
[]
=> nil
irb(main):010:0> p IO.popen("ls abc 2> /dev/null").readlines
[]
=> nil
irb(main):011:0> p IO.popen("/bin/ls abc").readlines
[]
=> nil
irb(main):012:0> p IO.popen("/bin/ls abc 2> /dev/null").readlines
["/dev/null\n"]
Show
Stephen Bannasch added a comment - I reported a similar issue in JRUBY-1676 which I marked as a duplicate of this issue. I'm trying to use ara howards macaddr gem. This provides a way to get the mac address on windows, linux, and macosx. You can see the code here: macaddr.rb. In it he uses code of this form (for testing on my mac systems – there are different commands for windows): 
stdout = IO.popen("/sbin/ifconfig >2 /dev/null")
This works in MRI but not in JRuby. Here's one simplification which shows the differences from: MRI 
mkdir emptydir
cd emptydir
irb
irb(main):002:0> p IO.popen("ls 2> /dev/null").readlines
[]
=> nil
irb(main):003:0> p IO.popen("ls ./ 2> /dev/null").readlines
[]
=> nil
irb(main):004:0> p IO.popen("/bin/ls").readlines
[]
=> nil
irb(main):005:0> p IO.popen("/bin/ls 2> /dev/null").readlines
[]
=> nil
irb(main):006:0> p IO.popen("/bin/ls ./ 2> /dev/null").readlines
[]
=> nil
irb(main):008:0* p IO.popen("ls abc").readlines
ls: abc: No such file or directory
[]
=> nil
irb(main):009:0> p IO.popen("ls abc 2> /dev/null").readlines
[]
=> nil
irb(main):011:0> p IO.popen("/bin/ls abc").readlines
ls: abc: No such file or directory
[]
=> nil
irb(main):012:0> p IO.popen("/bin/ls abc 2> /dev/null").readlines
[]
=> nil
and here's what happens in JRuby r5180: 
mkdir emptydir
cd emptydir
jirb
irb(main):001:0> p IO.popen("ls").readlines
[]
=> nil
irb(main):002:0> p IO.popen("ls 2> /dev/null").readlines
[]
=> nil
irb(main):003:0> p IO.popen("ls ./ 2> /dev/null").readlines
[]
=> nil
irb(main):004:0> p IO.popen("/bin/ls").readlines
[]
=> nil
irb(main):005:0> p IO.popen("/bin/ls 2> /dev/null").readlines
["/dev/null\n"]
=> nil
irb(main):006:0> p IO.popen("/bin/ls ./ 2> /dev/null").readlines
["/dev/null\n", "\n", "./:\n"]
=> nil
irb(main):008:0* p IO.popen("ls abc").readlines
[]
=> nil
irb(main):009:0> p IO.popen("ls abc 2> /dev/null").readlines
[]
=> nil
irb(main):010:0> p IO.popen("ls abc 2> /dev/null").readlines
[]
=> nil
irb(main):011:0> p IO.popen("/bin/ls abc").readlines
[]
=> nil
irb(main):012:0> p IO.popen("/bin/ls abc 2> /dev/null").readlines
["/dev/null\n"]
Hide
Permalink
Stephen Bannasch added a comment -

Another simple example on a Mac:

from jirb:

`ls /Applications/Utilities/Java/Java\ Web\ Start.app`
ls: /Applications/Utilities/Java/Java: No such file or directory
ls: Start.app: No such file or directory
ls: Web: No such file or directory
=> ""

no combination of escaping the space characters seems to work.

I'd love any ideas of how to fix this.

Show
Stephen Bannasch added a comment - Another simple example on a Mac: from jirb: 
`ls /Applications/Utilities/Java/Java\ Web\ Start.app`
ls: /Applications/Utilities/Java/Java: No such file or directory
ls: Start.app: No such file or directory
ls: Web: No such file or directory
=> ""
no combination of escaping the space characters seems to work. I'd love any ideas of how to fix this.
Hide
Permalink
Stephen Bannasch added a comment -

Here's a work-around for the simpler problem with spaces in the path:

`p="/Applications/Utilities/Java/Java Web Start.app"; ls "$p"`
=> "Contents\n"

But it doesn't work for redirection:

irb(main):008:0>  `c="one two 2> /dev/null"; ./arguments.rb "$c"`
=> "arguments: one two 2> /dev/null"
Show
Stephen Bannasch added a comment - Here's a work-around for the simpler problem with spaces in the path: 
`p="/Applications/Utilities/Java/Java Web Start.app"; ls "$p"`
=> "Contents\n"
But it doesn't work for redirection: 
irb(main):008:0>  `c="one two 2> /dev/null"; ./arguments.rb "$c"`
=> "arguments: one two 2> /dev/null"
Hide
Permalink
Raphael Valyi added a comment -

Notice that an other issue with msmtmp Gmail sending via IO.popen has been open and seems to be related to an underlyning IO.popen bug. We might close it as duplicated if ever it really seems to be exactly the same problem.
see: http://jira.codehaus.org/browse/JRUBY-1878

Show
Raphael Valyi added a comment - Notice that an other issue with msmtmp Gmail sending via IO.popen has been open and seems to be related to an underlyning IO.popen bug. We might close it as duplicated if ever it really seems to be exactly the same problem. see: http://jira.codehaus.org/browse/JRUBY-1878
Hide
Permalink
Charles Oliver Nutter added a comment -

Unlikely to be fixed for 1.0.4, but we should fix for 1.1.

Show
Charles Oliver Nutter added a comment - Unlikely to be fixed for 1.0.4, but we should fix for 1.1.
Hide
Permalink
Charles Oliver Nutter added a comment -

Punting issues from 1.1 RC2 to 1.1 final.

Show
Charles Oliver Nutter added a comment - Punting issues from 1.1 RC2 to 1.1 final.
Hide
Permalink
Nick Sieger added a comment -

Marking as must-fix for 1.1

Show
Nick Sieger added a comment - Marking as must-fix for 1.1
Hide
Permalink
Vladimir Sizikov added a comment -

Fixed by Nick in rev. r6111 on trunk. I just like to report the revision number where fix was done.

Show
Vladimir Sizikov added a comment - Fixed by Nick in rev. r6111 on trunk. I just like to report the revision number where fix was done.
Hide
Permalink
Vladimir Sizikov added a comment -

The fix caused minor regression: JRUBY-2235.

Show
Vladimir Sizikov added a comment - The fix caused minor regression: JRUBY-2235.

People

Vote (1)
Watch (4)

Dates

  • Created:
    Updated:
    Resolved:
Atlassian JIRA (v4.4.3#663-r165197) | Report a problem
Powered by a free Atlassian JIRA open source license for Codehaus. Try JIRA - bug tracking software for your team.