Invalid byte sequence in utf-8

Just FYI, it works fine on my Mac OS X machine.

$ jruby -v; jruby -S gem install nokogiri
jruby 1.7.0.preview1 (ruby-1.9.3-p203) (2012-05-29 5e6c689) (Java HotSpot(TM) 64-Bit Server VM 1.7.0_04) [darwin-x86_64-java]
Fetching: nokogiri-1.5.2-java.gem (100%)
Successfully installed nokogiri-1.5.2-java
1 gem installed

On another Windows 7 machine I still get the same problem:

jruby 1.7.0.preview1 (ruby-1.9.3-p203) (2012-05-19 00c8c98) (Java HotSpot(TM) Client VM 1.7.0_04) [Windows 7-x86-java]

Fetching: nokogiri-1.5.2-java.gem (100%)
Invalid gemspec in [C:/Dev/SDK/jruby-1.7.0.preview1/lib/ruby/gems/shared/specifications/nokogiri-1.5.2-java.gemspec]: invalid byte sequence in UTF-8
Successfully installed nokogiri-1.5.2-java
1 gem installed

Same machine, but in a Ubuntu virtual box gues I don't have any problems:

 jruby -v; jruby --1.9 -S gem install nokogiri
jruby 1.7.0.preview1 (ruby-1.9.3-p203) (2012-05-19 00c8c98) (OpenJDK Client VM 1.6.0_20) [linux-i386-java]
Fetching: nokogiri-1.5.2-java.gem (100%)
Successfully installed nokogiri-1.5.2-java
1 gem installed

Removed the s.date from the gem spec file and tried "gem list".. The error is gone in XP. Tried the same in Win 7 but it din work

Just another data point - I'm able to reproduce this trying to install nokogiri-1.5.3 on the TorqueBox Windows CI server w/ JRuby 1.7.0.preview1

Window encoding issue...

My guess would be that one of the default encodings we're using on Windows is conflicting with UTF-8 or Ruby encoding behavior (or just improperly handled).

I do not currently have a Windows VM set up. I need:

• An error with -Xbacktrace.style=raw.
• A listing of what the default encodings are on Windows (Encoding.default_external, default_internal).
• The default java.nio.charset.Charset on an affected Windows machine.

Hope to get a Windows VM installed soon.

On a Windows 7 machine that has the invalid byte sequence problem:

Encoding.default_external => #<Encoding:Windows-1252>
Encoding.default_internal => nil
java.nio.charset.Charset.defaultCharset => #<Java:SunNioCs::MS1252:0xc77a748>

The setting -Xbacktrace.style=raw doesn't change the output since there's not a backtrace being printed.

Reproduced on my XP VM. Investigating.

It looks like the gemspec is getting mangled during unpacking. Here's part of the gemspec file from Windows:

"Nokogiri (\xE9?\xB8)

And here is the same on OS X:

"Nokogiri (\u{e9}\u{8b}\u{b8})

It seems like when the gemspec is getting written to disk, either via zlib calls or by serializing the in-memory spec, it's dumping incorrectly on Windows. Perhaps we're trying to encode it as Windows-1252 when it should remain UTF-8?

Should be fixed by JRUBY-6739; linking.

I am looking into this one today and I will state what the problem is without the actual fix. Zlib is not using default external encoding (which was an original though I had); it is using file.encoding property of Java itself. I can reproduce this problem on MacOS:

jruby -J-Dfile.encoding=Windows-1252 -S gem install nokogiri
Fetching: nokogiri-1.5.5-java.gem (100%)
Invalid gemspec in [/Users/enebo/work/jruby/lib/ruby/gems/shared/specifications/nokogiri-1.5.5-java.gemspec]: invalid byte sequence in UTF-8
Successfully installed nokogiri-1.5.5-java
1 gem installed

I suspect we are calling something in Java and doing a getBytes() and assuming it is the right encoding on the Ruby side. We shall see....

This seems to be a bug in our YAML support:

jruby -J-Dfile.encoding=Windows-1252 snippets/ym.rb

ym.rb:

# encoding: utf-8

require "yaml"

y = YAML.load <<EOS
--- 
description: "Nokogiri (\xE9\x8B\xB8) is an HTML, ..."
EOS

p y

I believe YAML should always be UTF-8 and not use locale ever. Looking into a fix.

Fixed in commit 407c88c. The underlying issue is that locale encoding (or file.coding) was being used to construct new strings. I did minor String construction gymnastics to generate UTF-8 strings instead. In theory, YAML could also produce utf-16 or utf-32 encodings, but I could see no way of knowing which one is present. Worst-case you get utf-8 but it still works. Possibly I am missing something about SnakeYAML.

Works for me on Windows 7 64-bit, compiled from master. Nokogiri installs, no byte-sequence errors.