Problems in getting access to files who has Chinese charactors in name.

In order to get encoding, we can use system property. Please try this patch.

Here is for trunk.

I'm a bit confused.
The first .diff file seems to update standard jruby-1.1.2 source code, but the second isn't.
Then I update the second .diff by hand.

But nothing is improved.

F:\NetBeansProjects\jruby-1.1.2\lib>java -jar jruby.jar test.rb txt_中文
txt_??
中文字符串
test.rb:3:in `initialize': No such file or directory - File not found - txt_?? (Errno::ENOENT)
from test.rb:3:in `foreach'
from test.rb:3

In my opinion, your patch won't work because when executing ".getBytes("ISO-8859-1")", all the Chinese charaters will be replaced with "?" in the return byte streams.

Hi Tsing,

First patch is for JRuby 1.1.2, another one is for trunk. If you have 1.1.2 source, apply patch JRUBY-2812.diff only and try again.

Hi TAKAI Naoto, my English is poor, and until now I still can't understand what's the meaning of "trunk".
I have apply the first patch to JRuby1.1.2 only, but that bug is still there.
Test in WinXP:
F:\NetBeansProjects\jruby-1.1.2-old\lib>java -jar jruby.jar test.rb txt_en
txt_en
中文字符串
English string from text file.
中文字符串来自text文件。
F:\NetBeansProjects\jruby-1.1.2-old\lib>java -jar jruby.jar test.rb txt_中文
txt_??
中文字符串
test.rb:3:in `initialize': No such file or directory - File not found - txt_?? (Errno::ENOENT)
from test.rb:3:in `foreach'
from test.rb:3:in `foreach'
from test.rb:3
F:\NetBeansProjects\jruby-1.1.2-old\lib>java -jar jruby.jar 测试.rb txt_en
Error opening script file: F:/NetBeansProjects/jruby-1.1.2-old/lib/??.rb (文件名、目录名或卷标语法不正确。)
PS: test_en.rb==test_测试.rb==test.rb txt_en==txt_中文 All are encoded in GB2312.

As for the second patch, I have applied it to "jruby 1.1.3-dev (ruby 1.8.6 patchlevel 114) (2008-07-16 rev 7193) [x86-java]", but I can't build the jruby.jar, so I can't test it.
The version of NetBeans I'm using:
Product Version: NetBeans IDE 6.0 (Build 200711261600)
Java: 1.6.0_03; Java HotSpot(TM) Client VM 1.6.0_03-b05
System: Windows XP version 5.1 running on x86; GBK; zh_CN (nb)
Userdir: C:\Documents and Settings\Administrator\.netbeans\6.0

May you can tell me how to build it.

I still belive the problem occurs just after decoding the argvs with ".getByte("ISO-8859-1")".
These Chinese charactors have been replaced with "?" before encoding these bytes back to a string.

Recently I found that, if using Iconv to convert a string encoded in GB2312 in WinXP to UTF-8 first, then everything turns to be all right. (Of course so because after the conversion everything goes in the same way as my Linux which I mentioned in my first post.)
So, this bug is still a bug, but it no long matters much.

Thanks for that update, Tsing. We'll leave it open and perhaps someone will have a chance apply and test your patch.

There is also http://jira.codehaus.org/browse/JRUBY-3053 (seems like a related issue or even dup)

We have a patch...let's evaluate this for 1.1.6 since it will probably solve at least two issues.

Ok, here's a patch that might solve a whole bunch of string encoding problems at once. Basically, what this does is modify the RubyString.getUnicodeValue method to just use the default encoding for the platform. That should be the correct thing to do in 90% of cases, since most people will be putting non-ASCII characters into files on their own machines. It will also allow strings to come in from the system and back out to Java APIs without being damaged.

But the problem is that all the systems I have available for testing default to UTF-8. I need people on non-UTF-8 systems to test this extensively, and I really need help figuring out a way for us to test this regularly without having easy access to systems with other encodings (like GB-1252).

Please review patch! Try it out, try lots of stuff out.

I posted a patch with a possible fix, but it's rather dangerous to do this late in 1.1.6 cycle. I think it might be ok, but we need more testing than two days can give us. Punting to 1.1.7, hoping for community help.

Hi Charles, good news, that patch works.
The test sample works well on both my Linux(utf8) and XP(gb2312).
I will test more cases with that.
Well, maybe I should Google some Chinese characters problems which are complained on the web.

Hi Charles:
I Googled a tough one.

File Test.java:
public class Test{
public static String message = "你好"
public void setMessage(String s)

}

File hey.rb:
include Java
include_class("java.lang.System")
include_class("Test") #Please place Test.class and hey.rb under a same directory.
System.out.println(Test.message)#Works well.
a = Test.new
a.setMessage("测试中文输出结果")
System.out.println(Test.message)#Question marks displayed.

The Result:(jruby 1.1.5 (ruby 1.8.6 patchlevel 114) (2008-12-02 rev 6586) [x86-java])
你好
?????????????

That patch doesn't work.

Ok, this is a slightly different case. Currently when we get strings cross the Java/Ruby boundary, we're making the same mistake of assuming UTF-8. So the same change would need to happen there. I don't think this damns the getUnicodeValue patch, but we may need a new "getDecodedValue" method and go case-by-case to see which one should be called.

This bug is related to JRUBY-3053. We need a comprehensive solution for handling file encodings, and it's not going to be easy.

This is related to JRUBY-2812, which I've punted to 1.3. We'll include this bug and its test cases in that work.

Charles:
I have seen your post on JRUBY-2812 and I think I know what's going on about this bug. And "Fix Version:1.3", it seems there will be a long time before this problem fixed.
If you work out some fix I'll try some test cases for you.

Well hopefully 1.3 won't be that far out; really 1.2 was only three months, and this bug is already flagged to get worked on. I also think the Ruby 1.9 work we're doing will help this along, since it will force us to address default encoding, transcoding to Java String, and other details necessary to resolve this.

Test cases will be helpful...and I'm eager to work on this more and get it working well in 1.3. Plus if we get it fixed sooner, there's no reason you couldn't use a nightly build.

Tsing: It's still getting pushed back...we simply don't have the cycles to work on everything. But if you are able to get some simple test cases we'll be able to fix each case in turn.

With recent improvements in encoding handling, I'm fairly certain that this one is fixed.

(Original comment included output from Terminal.app, but apparently the encoding fails on JIRA.)