Garbled Gem.user_home with jruby --1.9

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: JRuby 1.6.5
Fix Version/s: JRuby 1.6.7
Component/s: None
Labels:
None
Environment:
jruby 1.6.5 (ruby-1.8.7-p330) (2011-10-25 9dcd388) (Java HotSpot(TM) 64-Bit Server VM 1.7.0_01) [Windows 7-amd64-java]

Number of attachments :
0

Description

the result of jruby --1.9 -e 'puts Gem.user_home' is garbled, see below

C:\>chcp
Active code page: 1252

C:\>echo %USERPROFILE%
C:\Users\xxx

C:\>ruby --version && ruby -e 'puts Gem.user_home'
ruby 1.9.2p290 (2011-07-09) [i386-mingw32]
C:/Users/xxx

C:\>jruby --version && jruby --1.9 -e 'puts Gem.user_home'
jruby 1.6.5 (ruby-1.8.7-p330) (2011-10-25 9dcd388) (Java HotSpot(TM) 64-Bit Server VM 1.7.0_01) [Win
dows 7-amd64-java]
C:/C:₩Users₩xxx

C:\>

Activity

Ascending order - Click to sort in descending order

Hide

Permalink

ryenus added a comment - 12/Dec/11 9:50 AM

It turns out to be a bug in Rubygems, see https://github.com/rubygems/rubygems/pull/229

Show

ryenus added a comment - 12/Dec/11 9:50 AM It turns out to be a bug in Rubygems, see https://github.com/rubygems/rubygems/pull/229

Hide

Permalink

ryenus added a comment - 12/Dec/11 11:39 PM

I tested again on another box with a fresh downloaded jruby 1.6.5 and cannot reproduce the issue.

Show

ryenus added a comment - 12/Dec/11 11:39 PM I tested again on another box with a fresh downloaded jruby 1.6.5 and cannot reproduce the issue.

Hide

Permalink

ryenus added a comment - 10/Jan/12 12:52 AM

Now the reproducing condition is clear, and it's an encoding related issue.

the issue can be reproduced with jruby -J-Dfile.encoding=cp936 --1.9 -e "puts File.expand_path '~'".

and chcp doesn't help.

Show

ryenus added a comment - 10/Jan/12 12:52 AM Now the reproducing condition is clear, and it's an encoding related issue. the issue can be reproduced with jruby -J-Dfile.encoding=cp936 --1.9 -e "puts File.expand_path '~'". and chcp doesn't help.

Hide

Permalink

Charles Oliver Nutter added a comment - 18/Jan/12 1:56 AM

If you can reproduce this with some regularity, can you test it against JRuby master (1.6) or JRuby 1.6.6 (snapshots from http://ci.jruby.org/snapshot). There have been many encoding-related fixes recently that aren't in 1.6.5.

Show

Charles Oliver Nutter added a comment - 18/Jan/12 1:56 AM If you can reproduce this with some regularity, can you test it against JRuby master (1.6) or JRuby 1.6.6 (snapshots from http://ci.jruby.org/snapshot ). There have been many encoding-related fixes recently that aren't in 1.6.5.

Hide

Permalink

ryenus added a comment - 25/Jan/12 9:21 AM - edited

I just tested it against master and jruby-1_6, same result:

$ java -Dfile.encoding=cp936 -jar lib/jruby.jar --1.9 -e "puts RUBY_VERSION, File.expand_path('~')" && git br -v

1.9.3

E:/sandbox/jruby/C:&#8361;Users&#8361;roger

* master      0cd7377 Fix JRUBY-5967

Show

ryenus added a comment - 25/Jan/12 9:21 AM - edited I just tested it against master and jruby-1_6, same result: $ java -Dfile.encoding=cp936 -jar lib/jruby.jar --1.9 -e "puts RUBY_VERSION, File.expand_path('~')" && git br -v 1.9.3 E:/sandbox/jruby/C:₩Users₩roger * master 0cd7377 Fix JRUBY-5967

Hide

Permalink

ryenus added a comment - 06/Feb/12 10:11 AM

I think I've located the problem in JRuby (though very likely an external bug).

In org.jruby.util.OSEnvironment#getAsMapOfRubyStrings, the encoding is retrieved like this: Encoding encoding = runtime.getEncodingService().getLocaleEncoding();

that line of the code is buggy, when java.nio.charset.Charset.defaultCharset().toString() == "GBK", the above code returns CP949Encoding, I suppose it should return GBKEncoding

Specifically, the code 5c for '\' in GBK is different for that in CP949, instead in CP949 it's for the 'WON SIGN' (see here: http://www.fileformat.info/info/charset/x-IBM949/decode.htm, seems CP949 is asymmetric).

I'm just curious, since JVM can correctly report the name of the default charset, why don't we use that to find the correct encoding but use jcoding's code to detect it?

Show

ryenus added a comment - 06/Feb/12 10:11 AM I think I've located the problem in JRuby (though very likely an external bug). In org.jruby.util.OSEnvironment#getAsMapOfRubyStrings , the encoding is retrieved like this: Encoding encoding = runtime.getEncodingService().getLocaleEncoding(); that line of the code is buggy, when java.nio.charset.Charset.defaultCharset().toString() == "GBK" , the above code returns CP949Encoding , I suppose it should return GBKEncoding Specifically, the code 5c for '\' in GBK is different for that in CP949, instead in CP949 it's for the 'WON SIGN' (see here: http://www.fileformat.info/info/charset/x-IBM949/decode.htm , seems CP949 is asymmetric). I'm just curious, since JVM can correctly report the name of the default charset, why don't we use that to find the correct encoding but use jcoding's code to detect it?

Hide

Permalink

Hiro Asari added a comment - 06/Feb/12 10:50 AM

Hmm. This sounds very much like ~~JRUBY-6398~~. For that bug's fix, I decided to default to ASCII-8BIT if JVM doesn't have a known default Charset. Does that work for you? Can you test one from http://ci.jruby.org/snapshots/release/ ?

Thank you.

Show

Hiro Asari added a comment - 06/Feb/12 10:50 AM Hmm. This sounds very much like JRUBY-6398 . For that bug's fix, I decided to default to ASCII-8BIT if JVM doesn't have a known default Charset. Does that work for you? Can you test one from http://ci.jruby.org/snapshots/release/ ? Thank you.

Hide

Permalink

Hiro Asari added a comment - 06/Feb/12 11:04 AM

ryenus,

Sorry, I skipped over some details of your update above. I will look into this a bit further.

Show

Hiro Asari added a comment - 06/Feb/12 11:04 AM ryenus, Sorry, I skipped over some details of your update above. I will look into this a bit further.

Hide

Permalink

ryenus added a comment - 06/Feb/12 2:24 PM

OK, the bug is in jcodings, see https://github.com/jruby/jcodings/pull/2

Show

ryenus added a comment - 06/Feb/12 2:24 PM OK, the bug is in jcodings, see https://github.com/jruby/jcodings/pull/2

Hide

Permalink

ryenus added a comment - 06/Feb/12 2:27 PM

well, that one is probably different, I'm happy to take a look, but definitely later

Show

ryenus added a comment - 06/Feb/12 2:27 PM well, that one is probably different, I'm happy to take a look, but definitely later

Hide

Permalink

ryenus added a comment - 06/Feb/12 2:39 PM

[04:28:36 roger@R /forks/jcodings (fix-gbk)]
$ cp target/jcodings.jar /sandbox/jruby/build_lib/jcodings.jar

[04:29:08 roger@R /sandbox/jruby (master)]
$ ant clean jar
Buildfile: E:\sandbox\jruby\build.xml
...
BUILD SUCCESSFUL
Total time: 1 minute 58 seconds

[04:31:12 roger@R /sandbox/jruby (master)]
$ java -jar lib/jruby.jar --1.9 -e "puts File.expand_path '~'"
C:/Users/roger

[04:31:42 roger@R /sandbox/jruby (master)]
$ java -Dfile.encoding=gbk -jar lib/jruby.jar --1.9 -e "puts File.expand_path '~'"
C:/Users/roger

Oh, yeah!

Show

ryenus added a comment - 06/Feb/12 2:39 PM [04:28:36 roger@R /forks/jcodings (fix-gbk)] $ cp target/jcodings.jar /sandbox/jruby/build_lib/jcodings.jar [04:29:08 roger@R /sandbox/jruby (master)] $ ant clean jar Buildfile: E:\sandbox\jruby\build.xml ... BUILD SUCCESSFUL Total time: 1 minute 58 seconds [04:31:12 roger@R /sandbox/jruby (master)] $ java -jar lib/jruby.jar --1.9 -e "puts File.expand_path '~'" C:/Users/roger [04:31:42 roger@R /sandbox/jruby (master)] $ java -Dfile.encoding=gbk -jar lib/jruby.jar --1.9 -e "puts File.expand_path '~'" C:/Users/roger Oh, yeah!

Hide

Permalink

Hiro Asari added a comment - 06/Feb/12 7:24 PM

OK, I merged the pull request. I am not familiar with the release procedure for jcodings, so I'll let Tom and Marcin take care of that. Until the fix is released (I suppose it will be jcodings 1.0.7), the only way to get the fix is to build jruby from source after copying it to build_lib.

Show

Hiro Asari added a comment - 06/Feb/12 7:24 PM OK, I merged the pull request. I am not familiar with the release procedure for jcodings, so I'll let Tom and Marcin take care of that. Until the fix is released (I suppose it will be jcodings 1.0.7), the only way to get the fix is to build jruby from source after copying it to build_lib.

Hide

Permalink

ryenus added a comment - 12/Feb/12 8:42 AM

What about a test, e.g.: https://gist.github.com/1808755

Show

ryenus added a comment - 12/Feb/12 8:42 AM What about a test, e.g.: https://gist.github.com/1808755

Hide

Permalink

ryenus added a comment - 22/Feb/12 7:48 PM - edited

tested with fresh 1.6.7 release, problem fixed. this should have been in 1.6.7 changelog

Show

ryenus added a comment - 22/Feb/12 7:48 PM - edited tested with fresh 1.6.7 release, problem fixed. this should have been in 1.6.7 changelog

Hide

Permalink

Charles Oliver Nutter added a comment - 23/Feb/12 10:11 AM

ryenus: Thank you for following up!

Show

Charles Oliver Nutter added a comment - 23/Feb/12 10:11 AM ryenus: Thank you for following up!

People

Assignee:

Thomas E Enebo

Reporter:

ryenus

Vote (0)

Watch (0)

Dates

Created:

12/Dec/11 7:55 AM

Updated:

15/Mar/12 4:14 PM

Resolved:

22/Feb/12 7:48 PM