Unicode characters are lost when embedding JRuby even if the calling code performs no conversion to byte[]

Ugh, I swear I put it under Embedding, but it says None now. :-/

Verified today that this issue still occurs on JRuby 1.5.5.

I looked at this issue and concluded this is not a bug. This happened because encoding(s) used to compile or run Java code didn't meet with the terminal encoding to show Japanese characters.

According to http://download.oracle.com/javase/7/docs/technotes/guides/intl/encoding.doc.html, windows-1252 is "Windows Latin-1." I'm not a windows user but I don't think this encoding can handle Japanese characters correctly. Are you sure windows-1252 can show Japanese characters?

Well, I'm going to explain why you couldn't get a correct result with the test code. I tested this on JDK 1.6 on OSX. My terminal shows UTF-8, and default encoding of Java is Shift_JIS. This is a good combination since the correct encoding must be set explicitly to get correct result.

First of all, you just mentioned about unicode escape characters. But, you should have gotten the same result even though you used actual Japanese characters. Look at the code below.

package elm;

import javax.script.ScriptEngine;
import javax.script.ScriptEngineManager;
import javax.script.ScriptException;

public class Jruby4890 {
    String orig = "\u6625\u304C\u6765\u305F\u3002";
    String ref = "春が来た。";
    String scriptlet = "str = \"" + orig + "\" ; puts \"str: #{str}\" ; str";
    String scriptlet_ref = "str = \"" + ref + "\" ; puts \"str: #{str}\" ; str";

    private Jruby4890() throws ScriptException {
        ScriptEngine engine = new ScriptEngineManager().getEngineByExtension("rb");
        String result = (String) engine.eval(scriptlet);
        System.out.println("result: " + result);
        String result_ref = (String) engine.eval(scriptlet_ref);
        System.out.println("result_ref: " + result_ref);
    }

    public static void main(String[] args) throws ScriptException {
        new Jruby4890();
    }
}

Both result and result_ref showed exactly the same and incorrect characters. I tested this on Eclipse which has SJIS for its text file encoding. However, output window displayed UTF-8 characters. This is why the result was incorrect. (I don't paste the results since they will be likely to be converted incorrectly on this web page.)

Workaround is to use NKF library to convert to desirable encoding. For example,

String scriptlet_nkf = "require 'nkf'; str = \"" + orig + "\" ; puts \"str: #{str}\" ; NKF::nkf('-Sw', str)";

String result_nkf = (String) engine.eval(scriptlet_nkf);
System.out.println("result_nkf: " + result_nkf);

worked and showed the correct result.

I strongly recommend that you should try using javac and java command on the terminal.

For example, I compile and run above code as follows:

javac -J-Duser.language=en -encoding SJIS -d build -cp ~/DevSpace/jruby~main/lib/jruby.jar src/elm Jruby4890.java
java -Dfile.encoding=UTF-8 -cp ~/DevSpace/jruby~main/lib/jruby.jar:build elm.Jruby4890

This time, I got correct results from result and resutl_red and an exception from result_nkf. Why? I explicitely specified -Dfile.encoding=UTF-8 when I ran Java code. This encoding was used while Ruby code was evaluated, so "str" had UTF-8 as its encoding. NKF detected that input was not Shift_JIS and raised the exception.

Using NKF library helps you to know what encoding was applied to Ruby strings. And double check what encoding your terminal or output windows uses. Exactly the same encoding should be chosen to get the correct result.

As I explained, this is not a bug. I confirmed on JRuby 1.5.1, 1.5.5 and 1.6.0.dev.

Sigh, Japanese character in example code was converted incorrectly. Sorry. I wrote Japanese character there, which was the exact replacement of unicode escape characters.

"I'm not a windows user but I don't think this encoding can handle Japanese characters correctly. Are you sure windows-1252 can show Japanese characters?"

The windows-1252 encoding cannot show Japanese characters, which is why it is possible to reproduce this bug using that encoding.

I'm not sure why this question is even being asked though, because:

1. My value is passed as a Java string.
2. I am outputting it to a StringWriter.
3. I am returning it as a Java string.

None of these require encoding, and despite that, JRuby is mangling the value that goes in before it comes out. Even if you stretch the definition of encoding and say that Java strings really are encoded, but as UTF-16, #1 and #3 are both UTF-16 and JRuby is mangling those too.

I am going to reopen this because I believe that it's still a problem.

The core of the issue is that the terminal is not involved here, so the strings should come through "unharmed". If I were outputting to the terminal, the explanation given above would be acceptable, however I am outputting to a StringWriter, so no encoding should occur.

If you don't want to acknowledge this as a bug, at least acknowledge it as a problem, but call it an Improvement instead.

I just noticed this as reopened and running the various programs I really only see ?????. I moved this as something to look at for 1.7.1 because this seems related to other issues we have where the Ruby parser assumes file.encoding as a default_external encoding unless otherwise specified (1.9-wise). I know this issue was opened long before 1.9 support was there, but this had me scratching my head enough to add it to 1.7.1.