JRuby

[1.9] Make String#gsub encoding aware for ruby files

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: None
  • Fix Version/s: JRuby 1.6RC3
  • Component/s: Ruby 1.9.2
  • Labels:
    None
  • Number of attachments :
    0

Description

Executing this script from a ruby file we raise an error but it should work:

p 'foo'.gsub(/[\x00-\x20%\x7F-\xFF]/, '')

org/jruby/RubyString.java:2843:in `gsub19': invalid multibyte escape: /[\x00-\x20%\x7F-\xFF]/ (RegexpError)
	from org/jruby/RubyString.java:2811:in `gsub19'
	from t.rb:3:in `(root)'

Be aware that the same error from the command line or irb is fine because of the encoding. I guess ruby takes the file encoding from the system and propagates it to the regular expression.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
David Calavera added a comment -

Btw, this bug causes 74 ActiveSupport tests fail.

Show
David Calavera added a comment - Btw, this bug causes 74 ActiveSupport tests fail.
Hide
Permalink
Thomas E Enebo added a comment -

The good news is the parser is recognizing the proper encodings for the string and the regexp:

s = 'foo'
r = /[\x00-\x20%\x7F-\xFF]/
p s.encoding
p r.encoding
p s.gsub(r, '')

You can see both encodings match between MRI and JRuby. The gsub itself does not like the combo between US-ASCII and ASCII-8BIT. Hmmm, looking into this....

Show
Thomas E Enebo added a comment - The good news is the parser is recognizing the proper encodings for the string and the regexp: s = 'foo' r = /[\x00-\x20%\x7F-\xFF]/ p s.encoding p r.encoding p s.gsub(r, '') You can see both encodings match between MRI and JRuby. The gsub itself does not like the combo between US-ASCII and ASCII-8BIT. Hmmm, looking into this....
Hide
Permalink
Thomas E Enebo added a comment -

I would not say this is a perfect patch (and I know this will allow a couple of more rubyspec's to be untagged in addition to the active_support errors), but it does make more things run than not. The propagation of encoding + options + kcode option values from the parser to Regexp is confusing (expecially because of 1.8 + 1.9 mode behaving differently). This all needs to change a bit. In MRI they use multiple 2 ints where we only use 1. They mask over bits we are still using to store things like EUC flag.

Show
Thomas E Enebo added a comment - I would not say this is a perfect patch (and I know this will allow a couple of more rubyspec's to be untagged in addition to the active_support errors), but it does make more things run than not. The propagation of encoding + options + kcode option values from the parser to Regexp is confusing (expecially because of 1.8 + 1.9 mode behaving differently). This all needs to change a bit. In MRI they use multiple 2 ints where we only use 1. They mask over bits we are still using to store things like EUC flag.
Hide
Permalink
David Calavera added a comment -

I'm affraid this patch makes things don't work at all :S

ActiveSupport tests don't even initialize because of it:

$ jdev --1.9 -S rake
(in /home/dcalavera/dev/rails3/activesupport)
/home/dcalavera/dev/jruby/bin/jruby -w -I"lib:test" "/home/dcalavera/.rvm/gems/jruby-1.6.0.RC1/gems/rake-0.8.7/lib/rake/rake_test_loader.rb" "test/clean_logger_test.rb" "test/transliterate_test.rb" "test/option_merger_test.rb" "test/callback_inheritance_test.rb" "test/isolation_test.rb" "test/message_encryptor_test.rb" "test/configurable_test.rb" "test/multibyte_chars_test.rb" "test/load_paths_test.rb" "test/multibyte_unicode_database_test.rb" "test/ordered_options_test.rb" "test/test_case_test.rb" "test/clean_backtrace_test.rb" "test/multibyte_utils_test.rb" "test/dependencies_test.rb" "test/deprecation_test.rb" "test/message_verifier_test.rb" "test/callbacks_test.rb" "test/file_update_checker_test.rb" "test/buffered_logger_test.rb" "test/secure_random_test.rb" "test/notifications_test.rb" "test/time_zone_test.rb" "test/caching_test.rb" "test/descendants_tracker_test.rb" "test/ordered_hash_test.rb" "test/log_subscriber_test.rb" "test/test_test.rb" "test/gzip_test.rb" "test/memoizable_test.rb" "test/rescuable_test.rb" "test/string_inquirer_test.rb" "test/inflector_test.rb" "test/i18n_test.rb" "test/whiny_nil_test.rb" "test/benchmarkable_test.rb" "test/flush_cache_on_private_memoization_test.rb" "test/lazy_load_hooks_test.rb" "test/concern_test.rb" "test/safe_buffer_test.rb" "test/json/encoding_test.rb" "test/json/decoding_test.rb" "test/deprecation/proxy_wrappers_test.rb" "test/core_ext/module_test.rb" "test/core_ext/object_and_class_ext_test.rb" "test/core_ext/time_ext_test.rb" "test/core_ext/duplicable_test.rb" "test/core_ext/name_error_test.rb" "test/core_ext/integer_ext_test.rb" "test/core_ext/base64_ext_test.rb" "test/core_ext/float_ext_test.rb" "test/core_ext/file_test.rb" "test/core_ext/range_ext_test.rb" "test/core_ext/numeric_ext_test.rb" "test/core_ext/string_ext_test.rb" "test/core_ext/hash_ext_test.rb" "test/core_ext/duration_test.rb" "test/core_ext/proc_test.rb" "test/core_ext/regexp_ext_test.rb" "test/core_ext/uri_ext_test.rb" "test/core_ext/array_ext_test.rb" "test/core_ext/load_error_test.rb" "test/core_ext/blank_test.rb" "test/core_ext/time_with_zone_test.rb" "test/core_ext/class_test.rb" "test/core_ext/kernel_test.rb" "test/core_ext/enumerable_test.rb" "test/core_ext/bigdecimal_test.rb" "test/core_ext/date_ext_test.rb" "test/core_ext/date_time_ext_test.rb" "test/core_ext/object/to_param_test.rb" "test/core_ext/object/to_query_test.rb" "test/core_ext/module/synchronization_test.rb" "test/core_ext/module/attr_internal_test.rb" "test/core_ext/module/attribute_accessor_test.rb" "test/core_ext/module/attribute_aliasing_test.rb" "test/core_ext/module/anonymous_test.rb" "test/core_ext/module/attr_accessor_with_default_test.rb" "test/core_ext/module/reachable_test.rb" "test/core_ext/class/delegating_attributes_test.rb" "test/core_ext/class/class_inheritable_attributes_test.rb" "test/core_ext/class/attribute_accessor_test.rb" "test/core_ext/class/attribute_test.rb" "test/xml_mini/rexml_engine_test.rb" "test/xml_mini/nokogirisax_engine_test.rb" "test/xml_mini/libxml_engine_test.rb" "test/xml_mini/jdom_engine_test.rb" "test/xml_mini/nokogiri_engine_test.rb" "test/xml_mini/libxmlsax_engine_test.rb" 
Isolation tests can test test-unit 1 only
Skipping memcached backed store tests. Start memcached and try again.
org/jruby/RubyString.java:2843:in `gsub19': incompatible encoding regexp match (ASCII-8BIT regexp with UTF-8 string) (Encoding::CompatibilityError)
	from org/jruby/RubyString.java:2811:in `gsub19'
	from /home/dcalavera/dev/rails3/activesupport/lib/active_support/testing/declarative.rb:26:in `test'
	from /home/dcalavera/dev/rails3/activesupport/test/json/decoding_test.rb:56:in `(class TestJSONDecoding)'
	from org/jruby/RubyHash.java:1172:in `each19'
	from /home/dcalavera/dev/rails3/activesupport/test/json/decoding_test.rb:55:in `(class TestJSONDecoding)'
	from org/jruby/RubyArray.java:1676:in `each'
	from /home/dcalavera/dev/rails3/activesupport/test/json/decoding_test.rb:54:in `(class TestJSONDecoding)'
	from /home/dcalavera/dev/rails3/activesupport/test/json/decoding_test.rb:7:in `(root)'
	from org/jruby/RubyKernel.java:1075:in `load19'
	from /home/dcalavera/dev/rails3/activesupport/test/json/decoding_test.rb:235:in `load'
	from /home/dcalavera/dev/rails3/activesupport/lib/active_support/dependencies.rb:227:in `load_dependency'
	from /home/dcalavera/dev/rails3/activesupport/lib/active_support/dependencies.rb:235:in `load'
	from /home/dcalavera/.rvm/gems/jruby-1.6.0.RC1/gems/rake-0.8.7/lib/rake/rake_test_loader.rb:5:in `__file__'
	from org/jruby/RubyArray.java:1676:in `each'
	from /home/dcalavera/.rvm/gems/jruby-1.6.0.RC1/gems/rake-0.8.7/lib/rake/rake_test_loader.rb:5:in `(root)'
Show
David Calavera added a comment - I'm affraid this patch makes things don't work at all :S ActiveSupport tests don't even initialize because of it: $ jdev --1.9 -S rake (in /home/dcalavera/dev/rails3/activesupport) /home/dcalavera/dev/jruby/bin/jruby -w -I"lib:test" "/home/dcalavera/.rvm/gems/jruby-1.6.0.RC1/gems/rake-0.8.7/lib/rake/rake_test_loader.rb" "test/clean_logger_test.rb" "test/transliterate_test.rb" "test/option_merger_test.rb" "test/callback_inheritance_test.rb" "test/isolation_test.rb" "test/message_encryptor_test.rb" "test/configurable_test.rb" "test/multibyte_chars_test.rb" "test/load_paths_test.rb" "test/multibyte_unicode_database_test.rb" "test/ordered_options_test.rb" "test/test_case_test.rb" "test/clean_backtrace_test.rb" "test/multibyte_utils_test.rb" "test/dependencies_test.rb" "test/deprecation_test.rb" "test/message_verifier_test.rb" "test/callbacks_test.rb" "test/file_update_checker_test.rb" "test/buffered_logger_test.rb" "test/secure_random_test.rb" "test/notifications_test.rb" "test/time_zone_test.rb" "test/caching_test.rb" "test/descendants_tracker_test.rb" "test/ordered_hash_test.rb" "test/log_subscriber_test.rb" "test/test_test.rb" "test/gzip_test.rb" "test/memoizable_test.rb" "test/rescuable_test.rb" "test/string_inquirer_test.rb" "test/inflector_test.rb" "test/i18n_test.rb" "test/whiny_nil_test.rb" "test/benchmarkable_test.rb" "test/flush_cache_on_private_memoization_test.rb" "test/lazy_load_hooks_test.rb" "test/concern_test.rb" "test/safe_buffer_test.rb" "test/json/encoding_test.rb" "test/json/decoding_test.rb" "test/deprecation/proxy_wrappers_test.rb" "test/core_ext/module_test.rb" "test/core_ext/object_and_class_ext_test.rb" "test/core_ext/time_ext_test.rb" "test/core_ext/duplicable_test.rb" "test/core_ext/name_error_test.rb" "test/core_ext/integer_ext_test.rb" "test/core_ext/base64_ext_test.rb" "test/core_ext/float_ext_test.rb" "test/core_ext/file_test.rb" "test/core_ext/range_ext_test.rb" "test/core_ext/numeric_ext_test.rb" "test/core_ext/string_ext_test.rb" "test/core_ext/hash_ext_test.rb" "test/core_ext/duration_test.rb" "test/core_ext/proc_test.rb" "test/core_ext/regexp_ext_test.rb" "test/core_ext/uri_ext_test.rb" "test/core_ext/array_ext_test.rb" "test/core_ext/load_error_test.rb" "test/core_ext/blank_test.rb" "test/core_ext/time_with_zone_test.rb" "test/core_ext/class_test.rb" "test/core_ext/kernel_test.rb" "test/core_ext/enumerable_test.rb" "test/core_ext/bigdecimal_test.rb" "test/core_ext/date_ext_test.rb" "test/core_ext/date_time_ext_test.rb" "test/core_ext/object/to_param_test.rb" "test/core_ext/object/to_query_test.rb" "test/core_ext/module/synchronization_test.rb" "test/core_ext/module/attr_internal_test.rb" "test/core_ext/module/attribute_accessor_test.rb" "test/core_ext/module/attribute_aliasing_test.rb" "test/core_ext/module/anonymous_test.rb" "test/core_ext/module/attr_accessor_with_default_test.rb" "test/core_ext/module/reachable_test.rb" "test/core_ext/class/delegating_attributes_test.rb" "test/core_ext/class/class_inheritable_attributes_test.rb" "test/core_ext/class/attribute_accessor_test.rb" "test/core_ext/class/attribute_test.rb" "test/xml_mini/rexml_engine_test.rb" "test/xml_mini/nokogirisax_engine_test.rb" "test/xml_mini/libxml_engine_test.rb" "test/xml_mini/jdom_engine_test.rb" "test/xml_mini/nokogiri_engine_test.rb" "test/xml_mini/libxmlsax_engine_test.rb" Isolation tests can test test-unit 1 only Skipping memcached backed store tests. Start memcached and try again. org/jruby/RubyString.java:2843:in `gsub19': incompatible encoding regexp match (ASCII-8BIT regexp with UTF-8 string) (Encoding::CompatibilityError) from org/jruby/RubyString.java:2811:in `gsub19' from /home/dcalavera/dev/rails3/activesupport/lib/active_support/testing/declarative.rb:26:in `test' from /home/dcalavera/dev/rails3/activesupport/test/json/decoding_test.rb:56:in `(class TestJSONDecoding)' from org/jruby/RubyHash.java:1172:in `each19' from /home/dcalavera/dev/rails3/activesupport/test/json/decoding_test.rb:55:in `(class TestJSONDecoding)' from org/jruby/RubyArray.java:1676:in `each' from /home/dcalavera/dev/rails3/activesupport/test/json/decoding_test.rb:54:in `(class TestJSONDecoding)' from /home/dcalavera/dev/rails3/activesupport/test/json/decoding_test.rb:7:in `(root)' from org/jruby/RubyKernel.java:1075:in `load19' from /home/dcalavera/dev/rails3/activesupport/test/json/decoding_test.rb:235:in `load' from /home/dcalavera/dev/rails3/activesupport/lib/active_support/dependencies.rb:227:in `load_dependency' from /home/dcalavera/dev/rails3/activesupport/lib/active_support/dependencies.rb:235:in `load' from /home/dcalavera/.rvm/gems/jruby-1.6.0.RC1/gems/rake-0.8.7/lib/rake/rake_test_loader.rb:5:in `__file__' from org/jruby/RubyArray.java:1676:in `each' from /home/dcalavera/.rvm/gems/jruby-1.6.0.RC1/gems/rake-0.8.7/lib/rake/rake_test_loader.rb:5:in `(root)'
Hide
Permalink
Thomas E Enebo added a comment -

Ok. Back to work then...

Show
Thomas E Enebo added a comment - Ok. Back to work then...
Hide
Permalink
Thomas E Enebo added a comment -

I am marking this fixed. I believe this was also fixed for RC2. My mistake in evaluating it was that JRuby on the Mac will default to UTF-8 on the command-line because of our launcher settings. The error thrown is expected with using UTF-8. When we are ASCII or US-ASCII this works the same as MRI.

Show
Thomas E Enebo added a comment - I am marking this fixed. I believe this was also fixed for RC2. My mistake in evaluating it was that JRuby on the Mac will default to UTF-8 on the command-line because of our launcher settings. The error thrown is expected with using UTF-8. When we are ASCII or US-ASCII this works the same as MRI.
Hide
Permalink
Pedro Teixeira added a comment -

currently seeing this error with JRuby --1.9, rails 3 running on ubuntu:

incompatible encoding regexp match (UTF-8 regexp with ASCII-8BIT string) (Encoding::CompatibilityError)
org/jruby/RubyString.java:2889:in `gsub'
org/jruby/RubyString.java:2869:in `gsub!'
/home/pedro/.rvm/gems/jruby-1.6.2@itsm/gems/activesupport-3.0.7/lib/active_support/inflector/inflections.rb:135:in `pluralize'
org/jruby/RubyArray.java:1602:in `each'
/home/pedro/.rvm/gems/jruby-1.6.2@itsm/gems/activesupport-3.0.7/lib/active_support/inflector/inflections.rb:135:in `pluralize'

would there be any tips on this?

Show
Pedro Teixeira added a comment - currently seeing this error with JRuby --1.9, rails 3 running on ubuntu: incompatible encoding regexp match (UTF-8 regexp with ASCII-8BIT string) (Encoding::CompatibilityError) org/jruby/RubyString.java:2889:in `gsub' org/jruby/RubyString.java:2869:in `gsub!' /home/pedro/.rvm/gems/jruby-1.6.2@itsm/gems/activesupport-3.0.7/lib/active_support/inflector/inflections.rb:135:in `pluralize' org/jruby/RubyArray.java:1602:in `each' /home/pedro/.rvm/gems/jruby-1.6.2@itsm/gems/activesupport-3.0.7/lib/active_support/inflector/inflections.rb:135:in `pluralize' would there be any tips on this?

People

  • Assignee:
    Thomas E Enebo
    Reporter:
    David Calavera
Vote (0)
Watch (1)

Dates

  • Created:
    Updated:
    Resolved: