Ruby environment variables do not always appear as Java environment variables to new processes

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Minor Minor
  • Resolution: Fixed
  • Affects Version/s: JRuby 1.0.0, JRuby 1.0.2, JRuby 1.1b1, JRuby 1.1.1
  • Fix Version/s: JRuby 1.6
  • Component/s: Core Classes/Modules
  • Labels:
    None
  • Environment:
    OS X 10.4.11 with Java 1.5
  • Testcase included:
    yes

Description

Here's another bug related to spawning new processes with backquotes. This time, if you set some variables in your Ruby program's ENV, new JRuby processes will not always see them as Java environment variables.

Attached is a test case demonstrating this behavior. The test case uses backquotes to start up a new JRuby program in a variety of ways; the new environment variables will show up in the new program in some cases, but not others.

Paste of test case:

require "test/unit"

class BackquotesEnvironmentTest < Test::Unit::TestCase
  def test_backquotes_pass_environment_variables_from_ruby_environment
    ENV["ONE"] = nil
    ENV["TWO"] = nil
    assert_equal "ONE:  TWO: ", `./other_program`, "variables should have been unset"

    ENV["ONE"] = "aaa"
    ENV["TWO"] = "bbb"
    results_with_variables_set = "ONE: aaa TWO: bbb"
    assert_equal results_with_variables_set, `./other_program`, "wrong variables when run straight up"
    assert_equal results_with_variables_set, `THREE=four jruby ./other_program`, "wrong variables when tricking backquotes"
    assert_equal results_with_variables_set, `jruby ./other_program`, "wrong variables when run with jruby"
    assert_equal results_with_variables_set, `./other_program.rb`, "wrong variables when run with rb suffix"
  end
end

Paste of result:

Loaded suite test_backquotes_pass_environment
Started
F
Finished in 3.363 seconds.

  1) Failure:
test_backquotes_pass_environment_variables_from_ruby_environment(BackquotesEnvironmentTest) [test_backquotes_pass_environment.rb:14]:
wrong variables when run with jruby.
<"ONE: aaa TWO: bbb"> expected but was
<"ONE:  TWO: ">.

1 tests, 4 assertions, 1 failures, 0 errors

Note that in the test failed on the line spawning the new application when it has a 'jruby' prefix. This is because that prefix causes JRuby to not actually spawn a new process. The previous two assertions demonstrate that when a new process is created, the variables do show up correctly. (See how I tricked JRuby into creating a new process on line 13?)

I ran into this problem because I, unfortunately, need to use a Java library that digs into environment variables to get some of its configuration data. My system tests were setting up the environment for my application and then firing it up with backqoutes, but since that Java library is using Java's accessors to the environment, it wasn't seeing my configuration. Bummer. (To get by for now I'm using the trick shown on line 13 of the above test case.)

  1. File
    other_program
    18/Nov/07 12:29 PM
    0.0 kB
    Matt Fletcher
  2. File
    other_program.rb
    18/Nov/07 12:29 PM
    0.1 kB
    Matt Fletcher
  3. File
    test_backquotes_pass_environment.rb
    18/Nov/07 12:29 PM
    0.8 kB
    Matt Fletcher

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Matt Fletcher added a comment -

What seems to be happening here is that variables set into Ruby's environment don't show up in the Java environment; I think this is because there is no way to set variables in the current Java environment (or at least I know of no way to do it). Somehow Ruby variables do end up in the environment of a new process. So in the cases where I'm actually getting a new process (`./other_program` and `THREE=four jruby ./other_program`), I get the new variables. But when I don't get a new process, my Java library can't see the variables.

Show
Matt Fletcher added a comment - What seems to be happening here is that variables set into Ruby's environment don't show up in the Java environment; I think this is because there is no way to set variables in the current Java environment (or at least I know of no way to do it). Somehow Ruby variables do end up in the environment of a new process. So in the cases where I'm actually getting a new process (`./other_program` and `THREE=four jruby ./other_program`), I get the new variables. But when I don't get a new process, my Java library can't see the variables.
Hide
Permalink
Charles Oliver Nutter added a comment -

Does still seem to be a problem in JRuby 1.1.1

Show
Charles Oliver Nutter added a comment - Does still seem to be a problem in JRuby 1.1.1
Hide
Permalink
Jay McGaffigan added a comment -

Ok so I took a spin investigating this bug and here is what I found out (With tom's help):

backtick has 2 modes of operation:

in-proc and out-of-proc.

In-proc backtick creates you a new runtime in the same JVM thus saving you from the startup times of a new JVM. This means the following: Your sub process has access to the ENV variable and things work ducky. However your ENV variables do NOT get promoted into the Environement of the main JVM thus java.lang.System.getenv will not see them. Further more java does not have an API for setting environment variables (that I could find and I googled a bunch). So I couldn't even modify the ENV hash in Jruby to promote these variables when ENV changes.

Out-of-proc backtick spawns a new JVM process. In spawning the JVM process all the ENV variables are put into the sub process's environment. And you can access these variables with java.lang.System.getenv.

Out of proc backtick occurs in 1 of 2 ways:
when you start your main jruby process you can use the -J-Djruby.launch.inproc=false flag. This is the most consistent way of getting this too happen. The other way is implicit. Depending on how you call your sub process from within backtick, jruby can decide on whether or not to launch the sub process as out of proc. The attached test case has an example of this happening. (The `THREE=four ./other_program.rb` call causes the sub process to be invoked out of process because jruby recognizes that the spawner wants to set a new environment variable for the sub process to have access to.

If you have the case where you need a sub process that uses java to access some environment variables promoted by the calling process then you should explicitly set the inproc flag to false OR do something like:
`#{ENV.keys.inject([]) {|array, key| array << "#{key}=#{ENV[key]}" }.join " "} ./other_program` this will always force the out of proc case.

Overall. I think there is nothing to fix here. maybe we should take the above statement and put it in the wiki. People need to understand the tradeoffs here.

Show
Jay McGaffigan added a comment - Ok so I took a spin investigating this bug and here is what I found out (With tom's help): backtick has 2 modes of operation: in-proc and out-of-proc. In-proc backtick creates you a new runtime in the same JVM thus saving you from the startup times of a new JVM. This means the following: Your sub process has access to the ENV variable and things work ducky. However your ENV variables do NOT get promoted into the Environement of the main JVM thus java.lang.System.getenv will not see them. Further more java does not have an API for setting environment variables (that I could find and I googled a bunch). So I couldn't even modify the ENV hash in Jruby to promote these variables when ENV changes. Out-of-proc backtick spawns a new JVM process. In spawning the JVM process all the ENV variables are put into the sub process's environment. And you can access these variables with java.lang.System.getenv. Out of proc backtick occurs in 1 of 2 ways: when you start your main jruby process you can use the -J-Djruby.launch.inproc=false flag. This is the most consistent way of getting this too happen. The other way is implicit. Depending on how you call your sub process from within backtick, jruby can decide on whether or not to launch the sub process as out of proc. The attached test case has an example of this happening. (The `THREE=four ./other_program.rb` call causes the sub process to be invoked out of process because jruby recognizes that the spawner wants to set a new environment variable for the sub process to have access to. If you have the case where you need a sub process that uses java to access some environment variables promoted by the calling process then you should explicitly set the inproc flag to false OR do something like: `#{ENV.keys.inject([]) {|array, key| array << "#{key}=#{ENV[key]}" }.join " "} ./other_program` this will always force the out of proc case. Overall. I think there is nothing to fix here. maybe we should take the above statement and put it in the wiki. People need to understand the tradeoffs here.
Hide
Permalink
Matt Fletcher added a comment -

Jay,

I agree with your technical assessment of the situation. I remember coming to the same conclusions myself.

I personally believe that the current behavior is inaccurate and, therefore, inappropriate. At one point I was creating a JRuby application that launched a lot of different types of programs. I experienced a lot of frustration until I learned about the jruby.launch.inproc flag. Although launching new instances in-process can be handy, it leads to subtle, unexpected, and inaccurate behaviors like what is described in this issue. On top of all this, MRI does not behave this way. I favor accuracy.

The question is: is running lots of little JRuby programs or running lots of other programs the more common use case?

Show
Matt Fletcher added a comment - Jay, I agree with your technical assessment of the situation. I remember coming to the same conclusions myself. I personally believe that the current behavior is inaccurate and, therefore, inappropriate. At one point I was creating a JRuby application that launched a lot of different types of programs. I experienced a lot of frustration until I learned about the jruby.launch.inproc flag. Although launching new instances in-process can be handy, it leads to subtle, unexpected, and inaccurate behaviors like what is described in this issue. On top of all this, MRI does not behave this way. I favor accuracy. The question is: is running lots of little JRuby programs or running lots of other programs the more common use case?
Hide
Permalink
Jay McGaffigan added a comment -

Matt,

You are making sense... After reading your post maybe we should change the "default" for the in proc flag. If the flag were set to "false" by default Jruby would be more in line with how MRI works. The flip side could be if people "REALLY" just wanted another runtime in the same process they could optimize that way. From the tests I ran with inproc set to false things behaved exactly like MRI (of course my test runs may not have been complete but I started with what you put up).

What do you think?

Show
Jay McGaffigan added a comment - Matt, You are making sense... After reading your post maybe we should change the "default" for the in proc flag. If the flag were set to "false" by default Jruby would be more in line with how MRI works. The flip side could be if people "REALLY" just wanted another runtime in the same process they could optimize that way. From the tests I ran with inproc set to false things behaved exactly like MRI (of course my test runs may not have been complete but I started with what you put up). What do you think?
Hide
Permalink
Charles Oliver Nutter added a comment -

Ok, I've finally caught up on this bug. I have a possible solution.

Some years ago, I proposed a "Kernel.ruby" method that would know how to launch a new Ruby instance in the cheapest way possible. So rather than spawning a new process, it could spin something up in-memory. If we added this to JRuby, in a way that was reasonably clear for new users (like require 'kernel-ruby' or something) we could consider turning off inproc launching for system/backtick/exec completely and recommend people use the new "ruby" method if they want to run a ruby script as cheaply as possible, with the normal launchers always spawning a new process.

This is one of those areas that always comes back as a painful discovery for users. We (I) added the in-process launching to reduce the cost of e.g. Rake's test/unit tasks, but at the same time we added confusion about whether scripts are running in-process or out. Ultimately, I'm not sure the savings from launching in-process have actually been worth it.

Would that fit with what you guys expect?

Show
Charles Oliver Nutter added a comment - Ok, I've finally caught up on this bug. I have a possible solution. Some years ago, I proposed a "Kernel.ruby" method that would know how to launch a new Ruby instance in the cheapest way possible. So rather than spawning a new process, it could spin something up in-memory. If we added this to JRuby, in a way that was reasonably clear for new users (like require 'kernel-ruby' or something) we could consider turning off inproc launching for system/backtick/exec completely and recommend people use the new "ruby" method if they want to run a ruby script as cheaply as possible, with the normal launchers always spawning a new process. This is one of those areas that always comes back as a painful discovery for users. We (I) added the in-process launching to reduce the cost of e.g. Rake's test/unit tasks, but at the same time we added confusion about whether scripts are running in-process or out. Ultimately, I'm not sure the savings from launching in-process have actually been worth it. Would that fit with what you guys expect?
Hide
Permalink
Matt Fletcher added a comment -

Jay: Yup, I definitely agree with you on defaulting the inproc property to false.

Charles: From my perspective, you have a great idea, since inproc would default to false, but in-process launching would still be accessible. But if I were someone who had been (probably unknowingly) banking on in-process launching, I'd be annoyed if things suddenly started running slower on me. It really is unfortunate that we don't know how many people depend on in-process launching vs. how many struggle with it.

Regardless, I think Charles has a solid suggestion that errs on the side of accuracy vs. performance.

Show
Matt Fletcher added a comment - Jay: Yup, I definitely agree with you on defaulting the inproc property to false. Charles: From my perspective, you have a great idea, since inproc would default to false, but in-process launching would still be accessible. But if I were someone who had been (probably unknowingly) banking on in-process launching, I'd be annoyed if things suddenly started running slower on me. It really is unfortunate that we don't know how many people depend on in-process launching vs. how many struggle with it. Regardless, I think Charles has a solid suggestion that errs on the side of accuracy vs. performance.
Hide
Permalink
Charles Oliver Nutter added a comment -

We've improved this greatly over the years, and aside from a few reported bugs we have not yet fixed, I consider this fixed. We'll deal with specific cases in new bugs.

Show
Charles Oliver Nutter added a comment - We've improved this greatly over the years, and aside from a few reported bugs we have not yet fixed, I consider this fixed. We'll deal with specific cases in new bugs.
Hide
Permalink
Matt Fletcher added a comment -

Charlie, since the posted test case still fails in JRuby 1.6.3, would you prefer to have a new JIRA issue opened or re-open this one? I'm guessing re-opening this one, since you wouldn't want to lose the history on the conversation.

Show
Matt Fletcher added a comment - Charlie, since the posted test case still fails in JRuby 1.6.3, would you prefer to have a new JIRA issue opened or re-open this one? I'm guessing re-opening this one, since you wouldn't want to lose the history on the conversation.

People

  • Assignee:
    Charles Oliver Nutter
    Reporter:
    Matt Fletcher
Vote (0)
Watch (2)

Dates

  • Created:
    Updated:
    Resolved: