Details

    • Type: Test Test
    • Status: Closed Closed
    • Priority: Major Major
    • Resolution: Not A Bug
    • Affects Version/s: JRuby 1.6.4
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Number of attachments :
      0

      Activity

      Hide
      Diego Plentz added a comment - - edited

      my bad, follow the link the test project https://github.com/plentz/jruby_report

      Show
      Diego Plentz added a comment - - edited my bad, follow the link the test project https://github.com/plentz/jruby_report
      Hide
      Thomas E Enebo added a comment -

      I have not looked at the actual test failures yet, but are the failures simplified at all away from having to wade through various dependent libraries? If it is possible to reduce the test cases it will be much easier for us to fix the issues. If not, we still appreciate the report...

      Show
      Thomas E Enebo added a comment - I have not looked at the actual test failures yet, but are the failures simplified at all away from having to wade through various dependent libraries? If it is possible to reduce the test cases it will be much easier for us to fix the issues. If not, we still appreciate the report...
      Hide
      Diego Plentz added a comment -

      Yup. It depends only on activemodel and rspec-rails(used to create the test). The README shows that it works well using MRI 1.9.2p290. I'm updating the project with one more failing case(and rails 3.1), but this time relate to ActiveRecord 3.1.0 and reload method. For the encoding problems, you could checkout this commit https://github.com/plentz/jruby_report/commit/7f8256d14b838cb1733f9841411b86360ccc4d30

      (I renamed the project to jruby_report since it don't have only encoding tests)

      Show
      Diego Plentz added a comment - Yup. It depends only on activemodel and rspec-rails(used to create the test). The README shows that it works well using MRI 1.9.2p290. I'm updating the project with one more failing case(and rails 3.1), but this time relate to ActiveRecord 3.1.0 and reload method. For the encoding problems, you could checkout this commit https://github.com/plentz/jruby_report/commit/7f8256d14b838cb1733f9841411b86360ccc4d30 (I renamed the project to jruby_report since it don't have only encoding tests)
      Hide
      Charles Oliver Nutter added a comment -

      Still fails on master. Investigating a bit.

      Show
      Charles Oliver Nutter added a comment - Still fails on master. Investigating a bit.
      Hide
      Charles Oliver Nutter added a comment -

      Here's the failures at present on master:

        1) Bar should know how to reload a record
           Failure/Error: bar = Bar.create(:name => "a")
           ActiveRecord::JDBCError:
             query does not return ResultSet
           # arjdbc/jdbc/RubyJdbcConnection.java:281:in `execute_query'
           # org/jruby/RubyProc.java:270:in `call'
           # org/jruby/RubyProc.java:258:in `call'
           # org/jruby/RubyHash.java:647:in `default'
           # org/jruby/RubyHash.java:996:in `[]'
           # org/jruby/RubyProc.java:270:in `call'
           # org/jruby/RubyProc.java:258:in `call'
           # org/jruby/RubyHash.java:647:in `default'
           # org/jruby/RubyHash.java:996:in `[]'
           # ./spec/models/bar_spec.rb:5:in `(root)'
           # org/jruby/RubyBasicObject.java:1743:in `instance_eval'
           # org/jruby/RubyArray.java:2307:in `collect'
           # org/jruby/RubyArray.java:2315:in `collect'
           # org/jruby/RubyArray.java:2307:in `collect'
           # org/jruby/RubyArray.java:2315:in `collect'
      
        2) Foo should do the right interpolation for error messages
           Failure/Error: lambda{I18n.t("errors.messages.record_invalid", :errors => foo.errors.full_messages)}.should_not raise_error
             expected no Exception, got #<Encoding::CompatibilityError: incompatible character encodings: UTF-8 and ASCII-8BIT>
           # ./spec/models/foo_spec.rb:9:in `(root)'
           # org/jruby/RubyBasicObject.java:1743:in `instance_eval'
           # org/jruby/RubyArray.java:2307:in `collect'
           # org/jruby/RubyArray.java:2315:in `collect'
           # org/jruby/RubyArray.java:2307:in `collect'
           # org/jruby/RubyArray.java:2315:in `collect'
      
        3) Foo should decode json even with special chars
           Failure/Error: ActiveSupport::JSON.decode({:message => ""}.to_json)['message'].should eq ""
             
             expected ""
                  got "\xC3\xA1"
             
             (compared using ==)
           # ./spec/models/foo_spec.rb:13:in `(root)'
           # org/jruby/RubyBasicObject.java:1743:in `instance_eval'
           # org/jruby/RubyArray.java:2307:in `collect'
           # org/jruby/RubyArray.java:2315:in `collect'
           # org/jruby/RubyArray.java:2307:in `collect'
           # org/jruby/RubyArray.java:2315:in `collect'
      
      Show
      Charles Oliver Nutter added a comment - Here's the failures at present on master: 1) Bar should know how to reload a record Failure/Error: bar = Bar.create(:name => "a") ActiveRecord::JDBCError: query does not return ResultSet # arjdbc/jdbc/RubyJdbcConnection.java:281:in `execute_query' # org/jruby/RubyProc.java:270:in `call' # org/jruby/RubyProc.java:258:in `call' # org/jruby/RubyHash.java:647:in `default' # org/jruby/RubyHash.java:996:in `[]' # org/jruby/RubyProc.java:270:in `call' # org/jruby/RubyProc.java:258:in `call' # org/jruby/RubyHash.java:647:in `default' # org/jruby/RubyHash.java:996:in `[]' # ./spec/models/bar_spec.rb:5:in `(root)' # org/jruby/RubyBasicObject.java:1743:in `instance_eval' # org/jruby/RubyArray.java:2307:in `collect' # org/jruby/RubyArray.java:2315:in `collect' # org/jruby/RubyArray.java:2307:in `collect' # org/jruby/RubyArray.java:2315:in `collect' 2) Foo should do the right interpolation for error messages Failure/Error: lambda{I18n.t("errors.messages.record_invalid", :errors => foo.errors.full_messages)}.should_not raise_error expected no Exception, got #<Encoding::CompatibilityError: incompatible character encodings: UTF-8 and ASCII-8BIT> # ./spec/models/foo_spec.rb:9:in `(root)' # org/jruby/RubyBasicObject.java:1743:in `instance_eval' # org/jruby/RubyArray.java:2307:in `collect' # org/jruby/RubyArray.java:2315:in `collect' # org/jruby/RubyArray.java:2307:in `collect' # org/jruby/RubyArray.java:2315:in `collect' 3) Foo should decode json even with special chars Failure/Error: ActiveSupport::JSON.decode({:message => ""}.to_json)['message'].should eq "" expected "" got "\xC3\xA1" (compared using ==) # ./spec/models/foo_spec.rb:13:in `(root)' # org/jruby/RubyBasicObject.java:1743:in `instance_eval' # org/jruby/RubyArray.java:2307:in `collect' # org/jruby/RubyArray.java:2315:in `collect' # org/jruby/RubyArray.java:2307:in `collect' # org/jruby/RubyArray.java:2315:in `collect'
      Hide
      Charles Oliver Nutter added a comment -

      I'm guessing the problem is that we're not observing encoding of the YAML file when loading it. Not sure where this code would go.

      ruby-1.9.2-p290 ~/projects/jruby_report $ ruby -v -ryaml -e "p YAML.load(File.read('config/locales/en.yml')).to_s.encoding"
      ruby 1.9.2p290 (2011-07-09 revision 32553) [x86_64-darwin10.8.0]
      #<Encoding:UTF-8>
      
      ruby-1.9.2-p290 ~/projects/jruby_report $ jruby -v --1.9 -ryaml -e "p YAML.load(File.read('config/locales/en.yml')).to_s.encoding"jruby 1.7.0.dev (ruby-1.9.2-p136) (2011-09-19 b9d59b0) (Java HotSpot(TM) 64-Bit Server VM 1.6.0_26) [darwin-x86_64-java]
      #<Encoding:ASCII-8BIT>
      
      Show
      Charles Oliver Nutter added a comment - I'm guessing the problem is that we're not observing encoding of the YAML file when loading it. Not sure where this code would go. ruby-1.9.2-p290 ~/projects/jruby_report $ ruby -v -ryaml -e "p YAML.load(File.read('config/locales/en.yml')).to_s.encoding" ruby 1.9.2p290 (2011-07-09 revision 32553) [x86_64-darwin10.8.0] #<Encoding:UTF-8> ruby-1.9.2-p290 ~/projects/jruby_report $ jruby -v --1.9 -ryaml -e "p YAML.load(File.read('config/locales/en.yml')).to_s.encoding"jruby 1.7.0.dev (ruby-1.9.2-p136) (2011-09-19 b9d59b0) (Java HotSpot(TM) 64-Bit Server VM 1.6.0_26) [darwin-x86_64-java] #<Encoding:ASCII-8BIT>
      Hide
      Charles Oliver Nutter added a comment -

      Ok, I think there's a number of issues at play in the first non-JDBC failure.

      My Psych impl was missing various bits of encoding logic. I'm adding those now, but they alone don't seem to fix this. Also, it appears that what comes out of the YAML file is encoded as UTF-8 regardless of my changes, so I don't think that's the source of problems. I'm going to put the missing logic in anyway.

      The actual issue may be that the string produced from the array of error messages is not encoded properly. I'm not sure how that list is subbed into the localized string yet, but if it's using something like Array#to_s that could be the problem...it does not appear that Array#to_s (or Hash#to_s) consider encoding at all right now. I'm going to work on that next.

      Show
      Charles Oliver Nutter added a comment - Ok, I think there's a number of issues at play in the first non-JDBC failure. My Psych impl was missing various bits of encoding logic. I'm adding those now, but they alone don't seem to fix this. Also, it appears that what comes out of the YAML file is encoded as UTF-8 regardless of my changes, so I don't think that's the source of problems. I'm going to put the missing logic in anyway. The actual issue may be that the string produced from the array of error messages is not encoded properly. I'm not sure how that list is subbed into the localized string yet, but if it's using something like Array#to_s that could be the problem...it does not appear that Array#to_s (or Hash#to_s) consider encoding at all right now. I'm going to work on that next.
      Hide
      Charles Oliver Nutter added a comment -

      Ok, the first encoding problem seems to have boiled down to Array#to_s not preserving/negotiating encodings properly. I modified the logic to match how MRI does it, via rb_str methods rather than through direct updates (as we did before).

      I've pushed that along with related tweaks in 7eaef8b5ef61d562c46d15f00610d1d9a876cca0 to my JRUBY-6033 topic branch available on http://github.com/headius/jruby.

      Also pushed to that branch is the parser-side encoding logic for Psych. I'm not sure if it actually plays a role in this bug or not.

      I'm going to call it a night on this one for now and hope that Diego is able to come up with a better reproduction of the second issue. It will probably be a similarly stupid lack-of-encoding-support somewhere.

      Show
      Charles Oliver Nutter added a comment - Ok, the first encoding problem seems to have boiled down to Array#to_s not preserving/negotiating encodings properly. I modified the logic to match how MRI does it, via rb_str methods rather than through direct updates (as we did before). I've pushed that along with related tweaks in 7eaef8b5ef61d562c46d15f00610d1d9a876cca0 to my JRUBY-6033 topic branch available on http://github.com/headius/jruby . Also pushed to that branch is the parser-side encoding logic for Psych. I'm not sure if it actually plays a role in this bug or not. I'm going to call it a night on this one for now and hope that Diego is able to come up with a better reproduction of the second issue. It will probably be a similarly stupid lack-of-encoding-support somewhere.
      Hide
      Diego Plentz added a comment -

      Charles,

      I've isolated the tests a bit more(removed AS:Json call) and looks like the problem isnt related with json parsing, but with json generation(changing the json engines with multi_json don't change the behavior fot the test tried OkJson and yajl).

      You could take a look at https://github.com/plentz/jruby_report/blob/8c22492579ba33e995c1c7d7c185b0d8251b724f/spec/models/json_spec.rb#L17

      Show
      Diego Plentz added a comment - Charles, I've isolated the tests a bit more(removed AS:Json call) and looks like the problem isnt related with json parsing, but with json generation(changing the json engines with multi_json don't change the behavior fot the test tried OkJson and yajl). You could take a look at https://github.com/plentz/jruby_report/blob/8c22492579ba33e995c1c7d7c185b0d8251b724f/spec/models/json_spec.rb#L17
      Hide
      Diego Plentz added a comment -

      Charles, the test with different charpoints that I commented wasnt pushed that time. That one is here now https://github.com/plentz/jruby_report/blob/72bd8b86b2/spec/models/json_spec.rb#L17

      Working now to isolate more the tests.

      Show
      Diego Plentz added a comment - Charles, the test with different charpoints that I commented wasnt pushed that time. That one is here now https://github.com/plentz/jruby_report/blob/72bd8b86b2/spec/models/json_spec.rb#L17 Working now to isolate more the tests.
      Hide
      Diego Plentz added a comment -

      update: here's one more case that looks like the same problem https://github.com/RubyMoney/money/issues/131

      Show
      Diego Plentz added a comment - update: here's one more case that looks like the same problem https://github.com/RubyMoney/money/issues/131
      Hide
      Diego Plentz added a comment -

      Charles/Thomas,

      I isolated the tests more and I think it's easier to find out the problems. Now we're down to 2 problems:

      • OkJson.engine should generate the same char codepoint in both implementations(actually it generates \u00C3\u00A1 for JRuby and \xC3\xA1 for MRI)
      • Yajl::Encoder should encode the YAML in both implementations, even with special chars (actually it only works in MRI)

      The updated code is here: https://github.com/plentz/jruby_report

      Show
      Diego Plentz added a comment - Charles/Thomas, I isolated the tests more and I think it's easier to find out the problems. Now we're down to 2 problems: OkJson.engine should generate the same char codepoint in both implementations(actually it generates \u00C3\u00A1 for JRuby and \xC3\xA1 for MRI) Yajl::Encoder should encode the YAML in both implementations, even with special chars (actually it only works in MRI) The updated code is here: https://github.com/plentz/jruby_report
      Hide
      Thomas E Enebo added a comment -

      Started examining this for a quick fix. When I explicitly use json/pure in money rspecs all the encoding issues go away. json-java fails and I suspect the native extension might need some additional love for string creation in 1.9 mode. Currently, it seems to make all strings it creates as ASCII-8BIT when the source string starts as UTF-8.

      This particular issue should probably be replaced by multiple individual issues (I suspect for each native extension), but I will leave it open until I get some traction on json-java.

      Show
      Thomas E Enebo added a comment - Started examining this for a quick fix. When I explicitly use json/pure in money rspecs all the encoding issues go away. json-java fails and I suspect the native extension might need some additional love for string creation in 1.9 mode. Currently, it seems to make all strings it creates as ASCII-8BIT when the source string starts as UTF-8. This particular issue should probably be replaced by multiple individual issues (I suspect for each native extension), but I will leave it open until I get some traction on json-java.
      Hide
      Thomas E Enebo added a comment -

      As I mentioned in last comment json_pure works right now. json-java will be fixed once this is merged:

      https://github.com/flori/json/pull/119

      For the ambitious OkJson you should be able to examine my commit and figure out what needs to change in this gem.

      Diego, can you update your example to use json-java (you will need to build it yourself [jrake jruby_gem]) and make sure this gem is at least functioning as you expect?

      Show
      Thomas E Enebo added a comment - As I mentioned in last comment json_pure works right now. json-java will be fixed once this is merged: https://github.com/flori/json/pull/119 For the ambitious OkJson you should be able to examine my commit and figure out what needs to change in this gem. Diego, can you update your example to use json-java (you will need to build it yourself [jrake jruby_gem] ) and make sure this gem is at least functioning as you expect?
      Hide
      Diego Plentz added a comment -

      Thomas,

      Thanks for that fix in the flori/json lib. I think that everything is fine now. OkJson also has been updated and I'm just waiting for a reply here https://github.com/brianmario/yajl-ruby/issues/97, but either way, looks like that none of the problems(https://github.com/intridea/multi_json/issues/25) are with Jruby itself, so I think its better to close this issue and open another if a new problem appears. Thanks!

      Show
      Diego Plentz added a comment - Thomas, Thanks for that fix in the flori/json lib. I think that everything is fine now. OkJson also has been updated and I'm just waiting for a reply here https://github.com/brianmario/yajl-ruby/issues/97 , but either way, looks like that none of the problems( https://github.com/intridea/multi_json/issues/25 ) are with Jruby itself, so I think its better to close this issue and open another if a new problem appears. Thanks!
      Hide
      Diego Plentz added a comment -

      Turns out that all the problems where in the related gems.

      Show
      Diego Plentz added a comment - Turns out that all the problems where in the related gems.
      Diego Plentz made changes -
      Field Original Value New Value
      Status Open [ 1 ] Resolved [ 5 ]
      Resolution Not A Bug [ 6 ]
      Charles Oliver Nutter made changes -
      Status Resolved [ 5 ] Closed [ 6 ]

        People

        • Assignee:
          Thomas E Enebo
          Reporter:
          Diego Plentz
        • Votes:
          0 Vote for this issue
          Watchers:
          2 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved: