JRuby

'incompatible character encodings: ASCII-8BIT and UTF-8' error thrown by jruby but not by ruby

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Incomplete
  • Affects Version/s: JRuby 1.6.5
  • Fix Version/s: None
  • Component/s: Ruby 1.9.2
  • Labels:
  • Environment:
    OS X 10.6.8; jruby 1.6.5 (ruby-1.9.2-p136) (2011-10-25 9dcd388) (Java HotSpot(TM) 64-Bit Server VM 1.6.0_29) [darwin-x86_64-java]; Rails 3.1.3; activerecord-jdbcsqlite3-adapter (1.2.1); activerecord-jdbc-adapter (1.2.1); jdbc-sqlite3 (3.7.2)
  • Number of attachments :
    0

Description

When trying to save an activerecord using the jdbcsqlite3-adapter, I get an 'incompatible character encodings: ASCII-8BIT and UTF-8' error.
raw stack trace: https://gist.github.com/1491694
string that caused error: "spec/testfiles/chars/arabic/Новая"

Looking into the issue myself, I believe the issue is caused by a line similar to: sql.gsub('?')

{strings.shift}

the encodings of the strings are similar to this pattern:
US-ASCII x 4
ASCII-8BIT
US-ASCII x 2
ASCII-8BIT x 2
US-ASCII x 12
UTF-8
US-ASCII x2
ASCII-8BIT
US-ASCII x 6

I tried to get the strings themselves, but when I did a join(), I got the same error.

When I tried to replicate the issue with a similar gsub and set of strings, I could not reproduce the error. My attempt follows:
—
usasciistr="Lorem".encode 'US-ASCII'
usasciistr.force_encoding('US-ASCII') #just in case that does something more than just encode. Force encoding doesn't do any conversion, unlike encode, but does force the specified encoding type to be set. Does it do anything else? i don't know.
ascii8bitstr="Ipsum".encode 'ASCII-8BIT'
ascii8bitstr.force_encoding('ASCII-8BIT')
utf8str="Новая".encode 'UTF-8'
utf8str.force_encoding('UTF-8')
sql='(?,?,?,?,?,?)'.encode 'US-ASCII' #because the sql strings seem to be encoded that way
sql.force_encoding('US-ASCII')
ary=[usasciistr,ascii8bitstr,usasciistr,usasciistr,utf8str,usasciistr]
sql.gsub('?'.encode('US-ASCII'))

{ary.shift}

#this is bassically the line of code that throws the error
—

I am glad to help solve this issue, and will lend a hand if you need more information.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Charles Oliver Nutter added a comment -

There's a good chance this is fixed on master and jruby-1_6 branches. Can you re-test with a build from http://ci.jruby.org/snapshots?

Show
Charles Oliver Nutter added a comment - There's a good chance this is fixed on master and jruby-1_6 branches. Can you re-test with a build from http://ci.jruby.org/snapshots?
Hide
Permalink
Charles Oliver Nutter added a comment -

No response after request for follow-up, and there have been many related fixes for jruby-1.6.6. Reopen if it's still broken with 1.6.6 or master, and try to produce a short reproduction for us.

Show
Charles Oliver Nutter added a comment - No response after request for follow-up, and there have been many related fixes for jruby-1.6.6. Reopen if it's still broken with 1.6.6 or master, and try to produce a short reproduction for us.
Hide
Permalink
Tom Beauvais added a comment -

I would like to reopen this issue. I have a fairly simple test case for a Rails app. The following fails with JRuby 1.7.0, but works with MRI.

You should be able to run this in the Rails console with any Rails Model
Foo.send(:sanitize_sql_for_conditions, ["foo=%s bar=%s", "ÅÆ", "'ÅÆ"])

The interesting thing is if you remove the single quote from the third array string then it works.

Show
Tom Beauvais added a comment - I would like to reopen this issue. I have a fairly simple test case for a Rails app. The following fails with JRuby 1.7.0, but works with MRI. You should be able to run this in the Rails console with any Rails Model Foo.send(:sanitize_sql_for_conditions, ["foo=%s bar=%s", "ÅÆ", "'ÅÆ"] ) The interesting thing is if you remove the single quote from the third array string then it works.
Hide
Permalink
Tom Beauvais added a comment -

Sorry, here's the stack trace

Encoding::CompatibilityError: incompatible character encodings: UTF-8 and ASCII-8BIT
from org/jruby/RubyString.java:1161:in `%'
from /Users/me/.rvm/gems/jruby-1.7.0/gems/activerecord-3.2.8/lib/active_record/sanitization.rb:121:in `sanitize_sql_array'
from /Users/me/.rvm/gems/jruby-1.7.0/gems/activerecord-3.2.8/lib/active_record/sanitization.rb:28:in `sanitize_sql_for_conditions'
from org/jruby/RubyBasicObject.java:1673:in `_send_'

Show
Tom Beauvais added a comment - Sorry, here's the stack trace Encoding::CompatibilityError: incompatible character encodings: UTF-8 and ASCII-8BIT from org/jruby/RubyString.java:1161:in `%' from /Users/me/.rvm/gems/jruby-1.7.0/gems/activerecord-3.2.8/lib/active_record/sanitization.rb:121:in `sanitize_sql_array' from /Users/me/.rvm/gems/jruby-1.7.0/gems/activerecord-3.2.8/lib/active_record/sanitization.rb:28:in `sanitize_sql_for_conditions' from org/jruby/RubyBasicObject.java:1673:in `_ send _'
Hide
Permalink
Tom Beauvais added a comment -

Seems to be easier to reproduce with the following:

jruby-1.7.0 :001 > Foo.send(:sanitize_sql_for_conditions, ["(foo = ? bar = ?)", "ÅÆ", "\"ÅÆ"])
Encoding::CompatibilityError: incompatible character encodings: UTF-8 and ASCII-8BIT
from org/jruby/RubyString.java:3014:in `gsub'
from /Users/me/.rvm/gems/jruby-1.7.0/gems/activerecord-3.2.8/lib/active_record/sanitization.rb:131:in `replace_bind_variables'
from /Users/me/.rvm/gems/jruby-1.7.0/gems/activerecord-3.2.8/lib/active_record/sanitization.rb:117:in `sanitize_sql_array'
from /Users/me/.rvm/gems/jruby-1.7.0/gems/activerecord-3.2.8/lib/active_record/sanitization.rb:28:in `sanitize_sql_for_conditions'

Show
Tom Beauvais added a comment - Seems to be easier to reproduce with the following: jruby-1.7.0 :001 > Foo.send(:sanitize_sql_for_conditions, ["(foo = ? bar = ?)", "ÅÆ", "\"ÅÆ"] ) Encoding::CompatibilityError: incompatible character encodings: UTF-8 and ASCII-8BIT from org/jruby/RubyString.java:3014:in `gsub' from /Users/me/.rvm/gems/jruby-1.7.0/gems/activerecord-3.2.8/lib/active_record/sanitization.rb:131:in `replace_bind_variables' from /Users/me/.rvm/gems/jruby-1.7.0/gems/activerecord-3.2.8/lib/active_record/sanitization.rb:117:in `sanitize_sql_array' from /Users/me/.rvm/gems/jruby-1.7.0/gems/activerecord-3.2.8/lib/active_record/sanitization.rb:28:in `sanitize_sql_for_conditions'
Hide
Permalink
Hiro Asari added a comment - 
$ jruby -S rails c
Loading development environment (Rails 3.2.8)
irb(main):001:0> RUBY_DESCRIPTION
=> "jruby 1.7.1.dev (1.9.3p286) 2012-11-07 6b13c46 on Java HotSpot(TM) 64-Bit Server VM 1.8.0-ea-b61 +indy [darwin-x86_64]"
irb(main):002:0> Bar.send(:sanitize_sql_for_conditions, ["(foo = ? bar = ?)", "ÅÆ", "\"ÅÆ"])
=> "(foo = 'ÅÆ' bar = '\"ÅÆ')"
Show
Hiro Asari added a comment - $ jruby -S rails c Loading development environment (Rails 3.2.8) irb(main):001:0> RUBY_DESCRIPTION => "jruby 1.7.1.dev (1.9.3p286) 2012-11-07 6b13c46 on Java HotSpot(TM) 64-Bit Server VM 1.8.0-ea-b61 +indy [darwin-x86_64]" irb(main):002:0> Bar.send(:sanitize_sql_for_conditions, ["(foo = ? bar = ?)", "ÅÆ", "\"ÅÆ"]) => "(foo = 'ÅÆ' bar = '\"ÅÆ')"
Hide
Permalink
Tom Beauvais added a comment -

Here I reduced it down a bit more, and included more version information

Loading development environment (Rails 3.2.8)
jruby-1.7.0 :001 > RUBY_DESCRIPTION
=> "jruby 1.7.0 (1.9.3p203) 2012-10-22 ff1ebbe on Java HotSpot(TM) 64-Bit Server VM 1.6.0_37-b06-434-11M3909 [darwin-x86_64]"
jruby-1.7.0 :002 > a = ["ÅÆ", "\"ÅÆ"]
=> ["ÅÆ", "\"ÅÆ"]
jruby-1.7.0 :003 >
jruby-1.7.0 :004 > "(foo = ? bar = ?)".gsub('?')

{Bar.connection.quote(a.shift)}

Creating scope :uuids. Overwriting existing method Address.uuids.
Encoding::CompatibilityError: incompatible character encodings: UTF-8 and ASCII-8BIT
from org/jruby/RubyString.java:3014:in `gsub'
from (irb):4:in `evaluate'
from org/jruby/RubyKernel.java:1065:in `eval'
from org/jruby/RubyKernel.java:1390:in `loop'
from org/jruby/RubyKernel.java:1173:in `catch'
from org/jruby/RubyKernel.java:1173:in `catch'

Show
Tom Beauvais added a comment - Here I reduced it down a bit more, and included more version information Loading development environment (Rails 3.2.8) jruby-1.7.0 :001 > RUBY_DESCRIPTION => "jruby 1.7.0 (1.9.3p203) 2012-10-22 ff1ebbe on Java HotSpot(TM) 64-Bit Server VM 1.6.0_37-b06-434-11M3909 [darwin-x86_64] " jruby-1.7.0 :002 > a = ["ÅÆ", "\"ÅÆ"] => ["ÅÆ", "\"ÅÆ"] jruby-1.7.0 :003 > jruby-1.7.0 :004 > "(foo = ? bar = ?)".gsub('?') {Bar.connection.quote(a.shift)} Creating scope :uuids. Overwriting existing method Address.uuids. Encoding::CompatibilityError: incompatible character encodings: UTF-8 and ASCII-8BIT from org/jruby/RubyString.java:3014:in `gsub' from (irb):4:in `evaluate' from org/jruby/RubyKernel.java:1065:in `eval' from org/jruby/RubyKernel.java:1390:in `loop' from org/jruby/RubyKernel.java:1173:in `catch' from org/jruby/RubyKernel.java:1173:in `catch'
Hide
Permalink
Tom Beauvais added a comment -

Same Error with Java 1.7

Loading development environment (Rails 3.2.8)
jruby-1.7.0 :001 > RUBY_DESCRIPTION
=> "jruby 1.7.0 (1.9.3p203) 2012-10-22 ff1ebbe on Java HotSpot(TM) 64-Bit Server VM 1.7.0_07-b10 [darwin-x86_64]"
jruby-1.7.0 :002 >
jruby-1.7.0 :003 > a = ["ÅÆ", "\"ÅÆ"]
=> ["ÅÆ", "\"ÅÆ"]
jruby-1.7.0 :004 >
jruby-1.7.0 :005 > "(foo = ? bar = ?)".gsub('?')

{Bar.connection.quote(a.shift)}

Encoding::CompatibilityError: incompatible character encodings: UTF-8 and ASCII-8BIT
from org/jruby/RubyString.java:3014:in `gsub'
from (irb):5:in `evaluate'
from org/jruby/RubyKernel.java:1065:in `eval'
from org/jruby/RubyKernel.java:1390:in `loop'
from org/jruby/RubyKernel.java:1173:in `catch'

Show
Tom Beauvais added a comment - Same Error with Java 1.7 Loading development environment (Rails 3.2.8) jruby-1.7.0 :001 > RUBY_DESCRIPTION => "jruby 1.7.0 (1.9.3p203) 2012-10-22 ff1ebbe on Java HotSpot(TM) 64-Bit Server VM 1.7.0_07-b10 [darwin-x86_64] " jruby-1.7.0 :002 > jruby-1.7.0 :003 > a = ["ÅÆ", "\"ÅÆ"] => ["ÅÆ", "\"ÅÆ"] jruby-1.7.0 :004 > jruby-1.7.0 :005 > "(foo = ? bar = ?)".gsub('?') {Bar.connection.quote(a.shift)} Encoding::CompatibilityError: incompatible character encodings: UTF-8 and ASCII-8BIT from org/jruby/RubyString.java:3014:in `gsub' from (irb):5:in `evaluate' from org/jruby/RubyKernel.java:1065:in `eval' from org/jruby/RubyKernel.java:1390:in `loop' from org/jruby/RubyKernel.java:1173:in `catch'
Hide
Permalink
Hiro Asari added a comment -

What's your locale?

$ locale
LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL=
$ jruby -S rails c
Loading development environment (Rails 3.2.8)
irb(main):001:0> RUBY_DESCRIPTION
=> "jruby 1.7.1.dev (1.9.3p286) 2012-11-07 6b13c46 on Java HotSpot(TM) 64-Bit Server VM 1.8.0-ea-b61 +indy [darwin-x86_64]"
irb(main):002:0> a = ["ÅÆ", "\"ÅÆ"]
=> ["ÅÆ", "\"ÅÆ"]
irb(main):003:0> "(foo = ? bar = ?)".gsub('?') {Bar.connection.quote(a.shift)}
=> "(foo = 'ÅÆ' bar = '\"ÅÆ')"
Show
Hiro Asari added a comment - What's your locale? $ locale LANG="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_CTYPE="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_ALL= $ jruby -S rails c Loading development environment (Rails 3.2.8) irb(main):001:0> RUBY_DESCRIPTION => "jruby 1.7.1.dev (1.9.3p286) 2012-11-07 6b13c46 on Java HotSpot(TM) 64-Bit Server VM 1.8.0-ea-b61 +indy [darwin-x86_64]" irb(main):002:0> a = ["ÅÆ", "\"ÅÆ"] => ["ÅÆ", "\"ÅÆ"] irb(main):003:0> "(foo = ? bar = ?)".gsub('?') {Bar.connection.quote(a.shift)} => "(foo = 'ÅÆ' bar = '\"ÅÆ')"
Hide
Permalink
Tom Beauvais added a comment -

Here's what I have for local. I see you're running with JRuby 1.7.1.dev and Java 1.8, any chance you can try with JRuby 1.7.0, and Java 1.7?

$ locale
LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL=

Show
Tom Beauvais added a comment - Here's what I have for local. I see you're running with JRuby 1.7.1.dev and Java 1.8, any chance you can try with JRuby 1.7.0, and Java 1.7? $ locale LANG="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_CTYPE="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_ALL=
Hide
Permalink
Hiro Asari added a comment -

I don't think that's it.

$ jruby -S rails c       
Loading development environment (Rails 3.2.8)
jruby-1.7.0 :001 > RUBY_DESCRIPTION
 => "jruby 1.7.0 (1.9.3p203) 2012-10-22 ff1ebbe on Java HotSpot(TM) 64-Bit Server VM 1.7.0_09-b05 [darwin-x86_64]" 
jruby-1.7.0 :002 > a = ["ÅÆ", "\"ÅÆ"]
 => ["ÅÆ", "\"ÅÆ"] 
jruby-1.7.0 :003 > "(foo = ? bar = ?)".gsub('?') {Bar.connection.quote(a.shift)}
 => "(foo = 'ÅÆ' bar = '\"ÅÆ')"
Show
Hiro Asari added a comment - I don't think that's it. $ jruby -S rails c Loading development environment (Rails 3.2.8) jruby-1.7.0 :001 > RUBY_DESCRIPTION => "jruby 1.7.0 (1.9.3p203) 2012-10-22 ff1ebbe on Java HotSpot(TM) 64-Bit Server VM 1.7.0_09-b05 [darwin-x86_64]" jruby-1.7.0 :002 > a = ["ÅÆ", "\"ÅÆ"] => ["ÅÆ", "\"ÅÆ"] jruby-1.7.0 :003 > "(foo = ? bar = ?)".gsub('?') {Bar.connection.quote(a.shift)} => "(foo = 'ÅÆ' bar = '\"ÅÆ')"
Hide
Permalink
Tom Beauvais added a comment -

Thanks for looking at this. Getting closer...
I created a brand new rails project with the default sqlite DB, ran my test, and I did not see the error. I then created a second project using mysql, and the error came back.

$ rails new Blog3 -d mysql
$ cd blog3
$ rake db:create
$ rails c
Loading development environment (Rails 3.2.8)
jruby-1.7.0 :001 > class Foo < ActiveRecord::Base
jruby-1.7.0 :002?> end
=> nil
jruby-1.7.0 :003 >
jruby-1.7.0 :004 > a = ["ÅÆ", "\"ÅÆ"]
=> ["ÅÆ", "\"ÅÆ"]
jruby-1.7.0 :006 > "(foo = ? bar = ?)".gsub('?')

{Foo.connection.quote(a.shift)}

(3.0ms) SET SQL_AUTO_IS_NULL=0
Encoding::CompatibilityError: incompatible character encodings: UTF-8 and ASCII-8BIT
from org/jruby/RubyString.java:3014:in `gsub'
from (irb):6:in `evaluate'
from org/jruby/RubyKernel.java:1065:in `eval'
from org/jruby/RubyKernel.java:1390:in `loop'
from org/jruby/RubyKernel.java:1173:in `catch'
from org/jruby/RubyKernel.java:1173:in `catch'

Show
Tom Beauvais added a comment - Thanks for looking at this. Getting closer... I created a brand new rails project with the default sqlite DB, ran my test, and I did not see the error. I then created a second project using mysql, and the error came back. $ rails new Blog3 -d mysql $ cd blog3 $ rake db:create $ rails c Loading development environment (Rails 3.2.8) jruby-1.7.0 :001 > class Foo < ActiveRecord::Base jruby-1.7.0 :002?> end => nil jruby-1.7.0 :003 > jruby-1.7.0 :004 > a = ["ÅÆ", "\"ÅÆ"] => ["ÅÆ", "\"ÅÆ"] jruby-1.7.0 :006 > "(foo = ? bar = ?)".gsub('?') {Foo.connection.quote(a.shift)} (3.0ms) SET SQL_AUTO_IS_NULL=0 Encoding::CompatibilityError: incompatible character encodings: UTF-8 and ASCII-8BIT from org/jruby/RubyString.java:3014:in `gsub' from (irb):6:in `evaluate' from org/jruby/RubyKernel.java:1065:in `eval' from org/jruby/RubyKernel.java:1390:in `loop' from org/jruby/RubyKernel.java:1173:in `catch' from org/jruby/RubyKernel.java:1173:in `catch'
Hide
Permalink
Tom Beauvais added a comment -

I found this on the jdbc-adapter

https://github.com/jruby/activerecord-jdbc-adapter/issues/229

Show
Tom Beauvais added a comment - I found this on the jdbc-adapter https://github.com/jruby/activerecord-jdbc-adapter/issues/229
Hide
Permalink
Hiro Asari added a comment -

That sounds promising.

Show
Hiro Asari added a comment - That sounds promising.
Hide
Permalink
Tom Beauvais added a comment -

Any chance you can confirm this using mysql, and get this reopened?

I also added comments to https://github.com/jruby/activerecord-jdbc-adapter/issues/229

Show
Tom Beauvais added a comment - Any chance you can confirm this using mysql, and get this reopened? I also added comments to https://github.com/jruby/activerecord-jdbc-adapter/issues/229

People

  • Assignee:
    Charles Oliver Nutter
    Reporter:
    Garrett Motzner
Vote (0)
Watch (2)

Dates

  • Created:
    Updated:
    Resolved: