JRuby (please use github issues at http://bugs.jruby.org)
  1. JRuby (please use github issues at http://bugs.jruby.org)
  2. JRUBY-6200

[1.9] Loading some Unicode characters results in non-printable characters on Windows

    Details

    • Type: Bug Bug
    • Status: Closed Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: JRuby 1.6.5
    • Fix Version/s: JRuby 1.6.6
    • Component/s: Ruby 1.9.3, Windows
    • Labels:
      None
    • Environment:
      Windows 7 64-bit, JRuby 1.6.5 in 1.9.2 mode
    • Number of attachments :
      0

      Description

      So I'm trying to run Cucumber on JRuby in my Rails 3.1.1 application.

      Here's the trace:

      D:\jruby-1.6.5\bin\jruby.exe --1.9 -e $stdout.sync=true;$stderr.sync=true;load($0=ARGV.shift) C:/Users/Patrick/RubymineProjects/sparkbank/script/cucumber C:/Users/Patrick/RubymineProjects/sparkbank/features --format Teamcity::Cucumber::Formatter --expand --color -r features
      Testing started at 1:59 AM ...
      LoadError: load error: gherkin/i18n -- org.yaml.snakeyaml.reader.ReaderException: special characters are not allowed
        require at org/jruby/RubyKernel.java:1047
         (root) at D:/jruby-1.6.5/lib/ruby/gems/1.9/gems/gherkin-2.6.2-java/lib/gherkin/lexer/i18n_lexer.rb:1
        require at org/jruby/RubyKernel.java:1047
         (root) at D:/jruby-1.6.5/lib/ruby/gems/1.9/gems/gherkin-2.6.2-java/lib/gherkin/lexer/i18n_lexer.rb:2
        require at org/jruby/RubyKernel.java:1047
         (root) at D:/jruby-1.6.5/lib/ruby/gems/1.9/gems/cucumber-1.1.2/lib/cucumber/ast/table.rb:3
        require at org/jruby/RubyKernel.java:1047
         (root) at D:/jruby-1.6.5/lib/ruby/gems/1.9/gems/cucumber-1.1.2/lib/cucumber/ast/step_invocation.rb:7
        require at org/jruby/RubyKernel.java:1047
         (root) at D:/jruby-1.6.5/lib/ruby/gems/1.9/gems/cucumber-1.1.2/lib/cucumber/ast.rb:2
        require at org/jruby/RubyKernel.java:1047
         (root) at D:/jruby-1.6.5/lib/ruby/gems/1.9/gems/cucumber-1.1.2/lib/cucumber/parser.rb:6
        require at org/jruby/RubyKernel.java:1047
         (root) at D:/jruby-1.6.5/lib/ruby/gems/1.9/gems/cucumber-1.1.2/lib/cucumber.rb:8
           load at org/jruby/RubyKernel.java:1073
         (root) at -e:1
      
      Process finished with exit code 1
      

      Here's my Gemfile.lock

      GEM
        remote: http://rubygems.org/
        specs:
          actionmailer (3.1.1)
            actionpack (= 3.1.1)
            mail (~> 2.3.0)
          actionpack (3.1.1)
            activemodel (= 3.1.1)
            activesupport (= 3.1.1)
            builder (~> 3.0.0)
            erubis (~> 2.7.0)
            i18n (~> 0.6)
            rack (~> 1.3.2)
            rack-cache (~> 1.1)
            rack-mount (~> 0.8.2)
            rack-test (~> 0.6.1)
            sprockets (~> 2.0.2)
          activemodel (3.1.1)
            activesupport (= 3.1.1)
            builder (~> 3.0.0)
            i18n (~> 0.6)
          activerecord (3.1.1)
            activemodel (= 3.1.1)
            activesupport (= 3.1.1)
            arel (~> 2.2.1)
            tzinfo (~> 0.3.29)
          activerecord-jdbc-adapter (1.2.0)
          activerecord-jdbcsqlite3-adapter (1.2.0)
            activerecord-jdbc-adapter (~> 1.2.0)
            jdbc-sqlite3 (~> 3.7.2)
          activeresource (3.1.1)
            activemodel (= 3.1.1)
            activesupport (= 3.1.1)
          activesupport (3.1.1)
            multi_json (~> 1.0)
          addressable (2.2.6)
          arel (2.2.1)
          bcrypt-ruby (3.0.1)
          bcrypt-ruby (3.0.1-java)
          bcrypt-ruby (3.0.1-x86-mingw32)
          bouncy-castle-java (1.5.0146.1)
          bourbon (1.1.0)
            sass (>= 3.1)
          builder (3.0.0)
          capistrano (2.9.0)
            highline
            net-scp (>= 1.0.0)
            net-sftp (>= 2.0.0)
            net-ssh (>= 2.0.14)
            net-ssh-gateway (>= 1.1.0)
          capybara (1.1.1)
            mime-types (>= 1.16)
            nokogiri (>= 1.3.3)
            rack (>= 1.0.0)
            rack-test (>= 0.5.4)
            selenium-webdriver (~> 2.0)
            xpath (~> 0.1.4)
          childprocess (0.2.2)
            ffi (~> 1.0.6)
          coffee-rails (3.1.1)
            coffee-script (>= 2.2.0)
            railties (~> 3.1.0)
          coffee-script (2.2.0)
            coffee-script-source
            execjs
          coffee-script-source (1.1.3)
          cucumber (1.1.2)
            builder (>= 2.1.2)
            diff-lcs (>= 1.1.2)
            gherkin (~> 2.6.2)
            json (>= 1.4.6)
            term-ansicolor (>= 1.0.6)
          cucumber-rails (1.2.0)
            capybara (>= 1.1.1)
            cucumber (>= 1.1.1)
            nokogiri (>= 1.5.0)
          database_cleaner (0.6.7)
          diff-lcs (1.1.3)
          erubis (2.7.0)
          execjs (1.2.9)
            multi_json (~> 1.0)
          factory_girl (2.2.0)
            activesupport
          factory_girl_rails (1.3.0)
            factory_girl (~> 2.2.0)
            railties (>= 3.0.0)
          faraday (0.6.1)
            addressable (~> 2.2.4)
            multipart-post (~> 1.1.0)
            rack (>= 1.1.0, < 2)
          ffi (1.0.9)
          ffi (1.0.9-java)
          ffi (1.0.9-x86-mingw32)
          gherkin (2.6.2)
            json (>= 1.4.6)
          gherkin (2.6.2-java)
            json (>= 1.4.6)
          gherkin (2.6.2-x86-mingw32)
            json (>= 1.4.6)
          haml (3.1.3)
          haml-rails (0.3.4)
            actionpack (~> 3.0)
            activesupport (~> 3.0)
            haml (~> 3.0)
            railties (~> 3.0)
          highline (1.6.5)
          hike (1.2.1)
          i18n (0.6.0)
          jdbc-sqlite3 (3.7.2)
          jquery-rails (1.0.17)
            railties (~> 3.0)
            thor (~> 0.14)
          jruby-openssl (0.7.4)
            bouncy-castle-java
          json (1.6.1)
          json (1.6.1-java)
          json_pure (1.6.1)
          launchy (2.0.5)
            addressable (~> 2.2.6)
          libv8 (3.3.10.2)
          mail (2.3.0)
            i18n (>= 0.4.0)
            mime-types (~> 1.16)
            treetop (~> 1.4.8)
          maruku (0.6.0)
            syntax (>= 1.0.0)
          mime-types (1.17.2)
          multi_json (1.0.3)
          multipart-post (1.1.3)
          mysql (2.8.1)
          mysql (2.8.1-x86-mingw32)
          net-scp (1.0.4)
            net-ssh (>= 1.99.1)
          net-sftp (2.0.5)
            net-ssh (>= 2.0.9)
          net-ssh (2.2.1)
          net-ssh-gateway (1.1.0)
            net-ssh (>= 1.99.1)
          nokogiri (1.5.0)
          nokogiri (1.5.0-java)
          nokogiri (1.5.0-x86-mingw32)
          oauth (0.4.5)
          oauth2 (0.4.1)
            faraday (~> 0.6.1)
            multi_json (>= 0.0.5)
          polyamorous (0.5.0)
            activerecord (~> 3.0)
          polyglot (0.3.3)
          rack (1.3.5)
          rack-cache (1.1)
            rack (>= 0.4)
          rack-mount (0.8.3)
            rack (>= 1.0.0)
          rack-ssl (1.3.2)
            rack
          rack-test (0.6.1)
            rack (>= 1.0)
          rails (3.1.1)
            actionmailer (= 3.1.1)
            actionpack (= 3.1.1)
            activerecord (= 3.1.1)
            activeresource (= 3.1.1)
            activesupport (= 3.1.1)
            bundler (~> 1.0)
            railties (= 3.1.1)
          rails-footnotes (3.7.5)
            rails (>= 3.0.0)
          railties (3.1.1)
            actionpack (= 3.1.1)
            activesupport (= 3.1.1)
            rack-ssl (~> 1.3.2)
            rake (>= 0.8.7)
            rdoc (~> 3.4)
            thor (~> 0.14.6)
          rake (0.9.2.2)
          rdoc (3.11)
            json (~> 1.4)
          rspec (2.7.0)
            rspec-core (~> 2.7.0)
            rspec-expectations (~> 2.7.0)
            rspec-mocks (~> 2.7.0)
          rspec-core (2.7.1)
          rspec-expectations (2.7.0)
            diff-lcs (~> 1.1.2)
          rspec-mocks (2.7.0)
          rspec-rails (2.7.0)
            actionpack (~> 3.0)
            activesupport (~> 3.0)
            railties (~> 3.0)
            rspec (~> 2.7.0)
          rubyzip (0.9.4)
          sass (3.1.10)
          sass-rails (3.1.4)
            actionpack (~> 3.1.0)
            railties (~> 3.1.0)
            sass (>= 3.1.4)
            sprockets (~> 2.0.0)
            tilt (~> 1.3.2)
          selenium-webdriver (2.10.0)
            childprocess (>= 0.2.1)
            ffi (= 1.0.9)
            json_pure
            rubyzip
          shoulda (3.0.0.beta2)
            shoulda-context (~> 1.0.0.beta1)
            shoulda-matchers (~> 1.0.0.beta1)
          shoulda-context (1.0.0)
          shoulda-matchers (1.0.0)
          sorcery (0.7.4)
            bcrypt-ruby (~> 3.0.0)
            oauth (~> 0.4.4)
            oauth (~> 0.4.4)
            oauth2 (~> 0.4.1)
            oauth2 (~> 0.4.1)
          sprockets (2.0.3)
            hike (~> 1.2)
            rack (~> 1.0)
            tilt (~> 1.1, != 1.3.0)
          sqlite3 (1.3.4)
          squeel (0.9.3)
            activerecord (~> 3.0)
            activesupport (~> 3.0)
            polyamorous (~> 0.5.0)
          syntax (1.0.0)
          term-ansicolor (1.0.7)
          therubyracer (0.9.9)
            libv8 (~> 3.3.10)
          thor (0.14.6)
          tilt (1.3.3)
          treetop (1.4.10)
            polyglot
            polyglot (>= 0.3.1)
          tzinfo (0.3.31)
          uglifier (1.0.4)
            execjs (>= 0.3.0)
            multi_json (>= 1.0.2)
          xpath (0.1.4)
            nokogiri (~> 1.3)
      
      PLATFORMS
        java
        ruby
        x86-mingw32
      
      DEPENDENCIES
        activerecord-jdbcsqlite3-adapter
        bourbon
        capistrano
        capybara (>= 1.1.1)
        coffee-rails (~> 3.1.0)
        cucumber-rails
        database_cleaner
        factory_girl_rails (>= 1.2.0)
        haml (>= 3.1.2)
        haml-rails (>= 0.3.4)
        jquery-rails
        jruby-openssl
        launchy (>= 2.0.5)
        maruku
        mysql
        rails (= 3.1.1)
        rails-footnotes (>= 3.7)
        rspec-rails (>= 2.6.1)
        sass-rails (~> 3.1.0)
        shoulda (>= 3.0.0.beta2)
        sorcery
        sqlite3
        squeel
        therubyracer (>= 0.8.2)
        uglifier
      

        Issue Links

          Activity

          Hide
          Aung Maw added a comment -

          Hi Hiro, This is what I have in environment.rb. No special characters. Thanks.

          1. Load the rails application
            require File.expand_path('../application', _FILE_)
            $CLASSPATH << "file:///# {Rails.root}

            /lib/sqljdbc4.jar"

          2. Initialize the rails application
            Globalidm3::Application.initialize!
          Show
          Aung Maw added a comment - Hi Hiro, This is what I have in environment.rb. No special characters. Thanks. Load the rails application require File.expand_path('../application', _ FILE _) $CLASSPATH << "file:///# {Rails.root} /lib/sqljdbc4.jar" Initialize the rails application Globalidm3::Application.initialize!
          Hide
          Hiro Asari added a comment -

          Thank you, Aung. I believe we have identified the root cause and working towards a resolution now.

          Show
          Hiro Asari added a comment - Thank you, Aung. I believe we have identified the root cause and working towards a resolution now.
          Hide
          Charles Oliver Nutter added a comment -

          I believe the String case may also be fixed right now. See the following, where I first use an IO and then use an already-read String:

          system ~/projects/jruby $ jruby --1.9 -e "require 'yaml'; puts YAML.load(open('lib/ruby/gems/shared/gems/gherkin-2.7.2-java/lib/gherkin/i18n.yml'))['zh-CN']['feature']"
          功能
          
          system ~/projects/jruby $ jruby --1.9 -e "require 'yaml'; puts YAML.load(File.read('lib/ruby/gems/shared/gems/gherkin-2.7.2-java/lib/gherkin/i18n.yml'))['zh-CN']['feature']"
          功能
          

          The additional case Hiro found with \ufffd may be a separate issue...in fact, it may be that YAML disallows this character in the YAML stream. I will open a separate bug for investigating that.

          Show
          Charles Oliver Nutter added a comment - I believe the String case may also be fixed right now. See the following, where I first use an IO and then use an already-read String: system ~/projects/jruby $ jruby --1.9 -e "require 'yaml'; puts YAML.load(open('lib/ruby/gems/shared/gems/gherkin-2.7.2-java/lib/gherkin/i18n.yml'))['zh-CN']['feature']" 功能 system ~/projects/jruby $ jruby --1.9 -e "require 'yaml'; puts YAML.load(File.read('lib/ruby/gems/shared/gems/gherkin-2.7.2-java/lib/gherkin/i18n.yml'))['zh-CN']['feature']" 功能 The additional case Hiro found with \ufffd may be a separate issue...in fact, it may be that YAML disallows this character in the YAML stream. I will open a separate bug for investigating that.
          Hide
          Charles Oliver Nutter added a comment -

          Oh, FWIW, both of those did output the proper characters to my console...JIRA just mucked them up.

          Show
          Charles Oliver Nutter added a comment - Oh, FWIW, both of those did output the proper characters to my console...JIRA just mucked them up.
          Hide
          Charles Oliver Nutter added a comment -

          Another note: In another bug I discovered that the Gherkin spec was actually failing because it wasn't getting set to the proper encoding by Zlib::GzipReader. As a result, we attempted to transcode it to a Java String (UTF-16BE) from ASCII-8BIT, which mangled the characters in Aslak's last name. Fixing that (773a155) cleaned up remaining String decoding issues. So that wasn't directly an issue with PsychParser, but it has been fixed as well.

          Show
          Charles Oliver Nutter added a comment - Another note: In another bug I discovered that the Gherkin spec was actually failing because it wasn't getting set to the proper encoding by Zlib::GzipReader. As a result, we attempted to transcode it to a Java String (UTF-16BE) from ASCII-8BIT, which mangled the characters in Aslak's last name. Fixing that (773a155) cleaned up remaining String decoding issues. So that wasn't directly an issue with PsychParser, but it has been fixed as well.

            People

            • Assignee:
              Charles Oliver Nutter
              Reporter:
              Patrick Ma
            • Votes:
              2 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: