Encoding problem when using JRuby 1.7.0.preview1 + Nokogiri under Windows

Can you show versions of Nokogiri for each of the tests above?

Sure. I have updated my test snippet with the following line:

puts "Nokogiri version: #{Nokogiri::VERSION}"

Which gives:

$ java -jar jruby-complete-1.6.7.2.jar --1.9 test.rb
Nokogiri version: 1.5.2
XML string encoding: UTF-8
Parsed root name encoding: UTF-8

$ java -jar jruby-complete-1.7.0.preview1.jar test.rb
Nokogiri version: 1.5.2
XML string encoding: UTF-8
Parsed root name encoding: Window-1252

Also, even though I was pretty sure it would work in OS X, it actually doesn't either! (using iTerm2 with a terminal set to use UTF-8)

$ java -version
java version "1.6.0_31"
Java(TM) SE Runtime Environment (build 1.6.0_31-b04-415-11M3635)
Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01-415, mixed mode)

$ java -jar jruby-complete-1.7.0.preview1.jar test.rb
Nokogiri version: 1.5.2
XML string encoding: UTF-8
Parsed root name encoding: macRoman

Hello,

Would you try it on Nokogiri 1.5.3?

I got UTF-8s from both puts lines using the latest Nokogiri on OSX.

Tried with Nokogiri 1.5.3. No luck:

$ java -jar jruby-complete-1.7.0.preview1.jar test.rb
Nokogiri version: 1.5.3
XML string encoding: UTF-8
Parsed root name encoding: macRoman

Thanks for testing on the latest Nokogiri.

I got the same wrong result as yours on OSX using jruby-complete-1.7.0.preview1.jar .

It looks the odd result appears only when jruby-complete jar archive is used. I created jruby-compelete.jar in my local repo. This also gave me wrong result.

So, some setting to create jruby-complete jar archive is suspicious.

There's a new nokogiri release that has reportedly fixed a bunch of encoding issues. Perhaps you can give this a try or Yoko can confirm it's working ok now?

Nope. Still not working:

$ java -jar lib/jruby-complete-1.7.0.preview1.jar test.rb
Nokogiri version: 1.5.5
XML string encoding: UTF-8
Parsed root name encoding: macRoman

This seems JRuby's packaging related bug.

Only when jruby-complete.jar is used, the bug appears. If it is jruby.jar, there's no problem.

Ahh the MacRoman problem is probably related a bug (in my opinion) in Apple's JDK. It sets the default encoding to macRoman but it's actually UTF-8 for almost all IO pipes.

I can reproduce here, but if I specify the proper encoding as we do in the 'jruby' command it works ok:

system ~/projects/jruby $ java -Djruby.home=`pwd` -jar lib/jruby.jar test.rb
XML string encoding: UTF-8
Parsed root name encoding: macRoman

system ~/projects/jruby $ java -Dfile.encoding=UTF-8 -Djruby.home=`pwd` -jar lib/jruby.jar test.rb
XML string encoding: UTF-8
Parsed root name encoding: UTF-8

Java 7 also appears to fix the "macRoman" issue:

system ~/projects/jruby $ pickjdk 2
New JDK: 1.7.0.jdk

system ~/projects/jruby $ java -Djruby.home=`pwd` -jar lib/jruby.jar test.rb
XML string encoding: UTF-8
Parsed root name encoding: UTF-8

Perhaps we should actually modify JRuby to use UTF-8 on Mac unconditionally, since MacRoman is never right.

Sorry Charles but the issue is not Mac specific and also happens on Windows, I just tried again with the latest Nokogiri:

C:\Documents and Settings\Administrator\Desktop> java -jar jruby-complete-1.7.0.preview1.jar test.rb
Nokogiri version: 1.5.5
XML string encoding: UTF-8
Parsed root name encoding: Windows-1252

Explicitely specifying encoding using the file.encoding parameter does fix the problem but I see that as a workaround rather than a proper fix, wouldn't you agree?

It seems then that we should be using a different encoding for the results of these files. I'm not sure if this is a JRuby issue or a Nokogiri issue. JRuby appears to be using the default encoding (from 'locale encoding' probably) for the resulting Strings. Perhaps Nokogiri should be forcing UTF-8 rather than allowing JRuby to construct using that default encoding?

I'm not sure if this is a JRuby issue or a Nokogiri issue either Charly I just posted the issue here because it is working using JRuby 1.6.7.2 and not 1.7.0.preview1.

There's a good chance this will be fixed by Tom Enebo's rework of our transcoding logic, due to land soon. We'll see.

Marking for 1.7pre2.

Thanks for the update Charles, I'll try with 1.7pre2 when it's available.

Incidentally, if you can try JRuby master right now on Windows we might be able to confirm that it's working.

Recent nightly snapshots are here: http://ci.jruby.org/snapshots/master/

Tried with jruby-complete-1.7.0.preview2.dev.jar (and nokogiri 1.5.5): the problem still occurs.

I think this is the same problem as JRUBY-6694. When doing a 'new String(...)' it uses file.coding (or locale charset). So it is taking UTF-8 String and trying to make a Windows-1252 string. Windows-1252 cannot represent all characters in UTF-8 so you see crud (at times) and you see the wrong encoding. Yoko, I believe this is something you need to fix in nokogiri itself. This is what I did in a stringFor() method:

    ByteList bytes = new ByteList(value.getBytes(RubyEncoding.UTF8), UTF8Encoding.INSTANCE);
    RubyString string = RubyString.newString(runtime, bytes);

Ah, I see. 1.9 mode needs two steps to convert Java String to RubyString.

Currently, Nokogiri does

RubyString string = RubyString.newString(runtime, str);

only. It seems to work only for 1.8 mode.

Actually, above code fixed the problem. Thanks, Tom.

I pushed the change in rev. c3953c2 on Nokogiri master.

Luc, if you have a chance, try Nokogiri master.

Yoko, you can also try on any OS by doing:

jruby -J-Dfile.coding=Windows-1252 my_file.rb

I am going to resolve this since it is an issue with 1.9 string construction in Nokogiri and JRuby is closing out blockers (this will be fixed in next Nokogiri release.

I'm pretty sure it should be "file.encoding"

jruby -J-Dfile.encoding=Windows-1252 -Ilib ../Canna/src/snippet/JRUBY6685.rb 
XML string encoding: UTF-8
Parsed root name encoding: UTF-8

and

java -jar ~/Projects/nokogiri/tmp/jruby-complete.jar -J-Dfile.encoding=Windows-1252 -Ilib ../Canna/src/snippet/JRUBY6685.rb 
warning: -J-Dfile.encoding=Windows-1252 argument ignored (launched in same VM?)
XML string encoding: UTF-8
Parsed root name encoding: UTF-8

JRUBY6685.rb is a given example.
I used below to run the example:

jruby -v
jruby 1.7.0.preview2.dev (ruby-1.9.3-p203) (2012-07-25 aaceaa2) (Java HotSpot(TM) 64-Bit Server VM 1.6.0_33) [darwin-x86_64-java]

java -version
java version "1.6.0_33"
Java(TM) SE Runtime Environment (build 1.6.0_33-b03-424-10M3720)
Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03-424, mixed mode)

Hi, sorry for the late follow up, kids and vacations and stuff

I tried Yoko's changes in Nokogiri master and yes, everything seems to be fine now:

% jruby -v
jruby 1.7.0.preview2 (1.9.3p203) 2012-08-07 4a6bb0a on Java HotSpot(TM) 64-Bit Server VM 1.6.0_33-b03-424-11M3720 [darwin-x86_64]

% jruby test.rb
jruby test.rb
Nokogiri version: 1.5.6.rc1
XML string encoding: UTF-8
Parsed root name encoding: UTF-8

I need to test on Windows too but so far so good. Thanks!