JRuby

AST Node positions and content incorrect when passing down Multibyte content to lexer

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Blocker Blocker
  • Resolution: Fixed
  • Affects Version/s: JRuby 1.1RC2
  • Fix Version/s: JRuby 1.4
  • Component/s: Parser
  • Labels:
    None
  • Number of attachments :
    2

Description

In previous versions of JRuby I'd be able to pass in a Reader to the LexerSource. Then it'd read from that. Now it takes in an InputStream and reads one byte at a time casting the read byte to char. This breaks multibyte string encodings. I'm getting lots of problems with Japanese users. I'm basically passing down a ByteArrayInputStream with the string content as bytes. The problem seems to be that the lexer is no longer multibyte aware.

Attachments

  1. Text File
    LexerSource.java.patch
    26/Mar/08 10:33 AM
    1 kB
    Chris Williams
  2. Java Source File
    ReaderLexerSource.java
    26/Mar/08 10:33 AM
    8 kB
    Chris Williams

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Chris Williams added a comment -

Patch for LexerSource to allow users to send in a Reader. New ReadrLexerSource class coming in another attachment.

Show
Chris Williams added a comment - Patch for LexerSource to allow users to send in a Reader. New ReadrLexerSource class coming in another attachment.
Hide
Permalink
Thomas E Enebo added a comment -

We changed the lexer to be non-character aware because Ruby itself is not character aware (in 1.8 at least). We ended up doing this to support Rubys byte-bucket String behavior. I believe netbeans solves the problem by converting strings back to UTF-8 on the way out (back to IDE). I believe they can make that assumption because they know the encoding of the characters before it gets sent to be parsed. Is it possible for you to do the same thing?

Show
Thomas E Enebo added a comment - We changed the lexer to be non-character aware because Ruby itself is not character aware (in 1.8 at least). We ended up doing this to support Rubys byte-bucket String behavior. I believe netbeans solves the problem by converting strings back to UTF-8 on the way out (back to IDE). I believe they can make that assumption because they know the encoding of the characters before it gets sent to be parsed. Is it possible for you to do the same thing?
Hide
Permalink
James Le Cuirot added a comment -

I'm confused about what Thomas meant by "on the way out" but if a conversion to UTF-8 is needed, would something like the following work in the RDT source? Given a char array called fContents...

lexerSource = LexerSource.getSource("filename", new ByteArrayInputStream(String.valueOf(fContents).getBytes("UTF-8")), null, config);

I'd like to see this resolved because it's complicating packaging on Gentoo Linux.

Show
James Le Cuirot added a comment - I'm confused about what Thomas meant by "on the way out" but if a conversion to UTF-8 is needed, would something like the following work in the RDT source? Given a char array called fContents... lexerSource = LexerSource.getSource("filename", new ByteArrayInputStream(String.valueOf(fContents).getBytes("UTF-8")), null, config); I'd like to see this resolved because it's complicating packaging on Gentoo Linux.
Hide
Permalink
Thomas E Enebo added a comment -

Internally the lexer assumes all content is just a big stream of bytes. Ruby 1.8 strings require this. So if you send a UTF8 string into the lexer it will process that stream like it is raw (or ISO-8859-1). If you read anything back from the generated AST, you know it originally came from a UTF8 source. So you can just re-encode the strings you get out of the AST as UTF8 from ISO-8859-1. That is what I meant by on the way out.

Show
Thomas E Enebo added a comment - Internally the lexer assumes all content is just a big stream of bytes. Ruby 1.8 strings require this. So if you send a UTF8 string into the lexer it will process that stream like it is raw (or ISO-8859-1). If you read anything back from the generated AST, you know it originally came from a UTF8 source. So you can just re-encode the strings you get out of the AST as UTF8 from ISO-8859-1. That is what I meant by on the way out.
Hide
Permalink
Christopher Williams added a comment -

Hmm, it's been quite a while since I reported this, so I'm a little rusty as to the exact issue. But I think essentially it's not just a matter of the AST content being mangled as a result, but also the positions reported for the nodes being off by the extra byte lengths.

Show
Christopher Williams added a comment - Hmm, it's been quite a while since I reported this, so I'm a little rusty as to the exact issue. But I think essentially it's not just a matter of the AST content being mangled as a result, but also the positions reported for the nodes being off by the extra byte lengths.
Hide
Permalink
Christopher Williams added a comment -

A related issue that I brought up in IRC is from this original bug report on our system: http://support.aptana.com/asap/browse/ROR-833

The ruby source is

module ApplicationHelper
  def £
    :foo
  end
end

I tried a simple test to make sure none of the RDT code was interfering in mangling the source or it's encoding. But it still gets a SyntaxException complaining about the pound character. Here's the test code:

String src = "$KCODE = \"utf8\"\nmodule ApplicationHelper\n  def £\n    :foo\n  end\nend";
ParserConfiguration config = new ParserConfiguration(0, true, false);
LexerSource source = LexerSource.getSource("filename", new ByteArrayInputStream(String.valueOf(src).getBytes("UTF-8")), null, config);
RubyParserResult result = new RubyParser().getDefaultRubyParser().parse(config, source);
assertNotNull(result);
Show
Christopher Williams added a comment - A related issue that I brought up in IRC is from this original bug report on our system: http://support.aptana.com/asap/browse/ROR-833 The ruby source is 
module ApplicationHelper
  def £
    :foo
  end
end
I tried a simple test to make sure none of the RDT code was interfering in mangling the source or it's encoding. But it still gets a SyntaxException complaining about the pound character. Here's the test code: 
String src = "$KCODE = \"utf8\"\nmodule ApplicationHelper\n  def £\n    :foo\n  end\nend";
ParserConfiguration config = new ParserConfiguration(0, true, false);
LexerSource source = LexerSource.getSource("filename", new ByteArrayInputStream(String.valueOf(src).getBytes("UTF-8")), null, config);
RubyParserResult result = new RubyParser().getDefaultRubyParser().parse(config, source);
assertNotNull(result);
Hide
Permalink
Christopher Williams added a comment -

As for the last issue I commented on here, this is against JRuby 1.1.4.

Show
Christopher Williams added a comment - As for the last issue I commented on here, this is against JRuby 1.1.4.
Hide
Permalink
Thomas E Enebo added a comment -

Ok, the last example is not valid Ruby so that example is not a good one for this bug (I am fairly certain we have multi-byte bugs in lexer). This also did not work in Ruby 1.9:

% ruby -Ku ~/jruby/scripts/name.rb
/Users/enebo/jruby/scripts/name.rb:2: Invalid char `\243' in expression
/Users/enebo/jruby/scripts/name.rb:3: syntax error, unexpected tSYMBEG
   :foo
    ^
/Users/enebo/jruby/scripts/name.rb:5: syntax error, unexpected kEND, expecting $end

Any other examples?

Show
Thomas E Enebo added a comment - Ok, the last example is not valid Ruby so that example is not a good one for this bug (I am fairly certain we have multi-byte bugs in lexer). This also did not work in Ruby 1.9: 
% ruby -Ku ~/jruby/scripts/name.rb
/Users/enebo/jruby/scripts/name.rb:2: Invalid char `\243' in expression
/Users/enebo/jruby/scripts/name.rb:3: syntax error, unexpected tSYMBEG
   :foo
    ^
/Users/enebo/jruby/scripts/name.rb:5: syntax error, unexpected kEND, expecting $end
Any other examples?
Hide
Permalink
James Le Cuirot added a comment -

Are you saying it's not valid Ruby despite the fact that it works (for me, at least) in 1.8.6? That's okay, I'm just checking.

Show
James Le Cuirot added a comment - Are you saying it's not valid Ruby despite the fact that it works (for me, at least) in 1.8.6? That's okay, I'm just checking.
Hide
Permalink
Thomas E Enebo added a comment -

You are right...For some reason it got saved as iso8859_1 encoded file. When I make it utf-8 things work in both Ruby 1.8.6 and 1.9. The example was fine...My editing skills were not

Show
Thomas E Enebo added a comment - You are right...For some reason it got saved as iso8859_1 encoded file. When I make it utf-8 things work in both Ruby 1.8.6 and 1.9. The example was fine...My editing skills were not
Hide
Permalink
Thomas E Enebo added a comment -

Fix for syntax error aspect of this in commit 8237. This should make strings and identifiers work. Please report back on this about position info...something tells me this is still broken since we return bytes from a stream and we increment accordingly.

Show
Thomas E Enebo added a comment - Fix for syntax error aspect of this in commit 8237. This should make strings and identifiers work. Please report back on this about position info...something tells me this is still broken since we return bytes from a stream and we increment accordingly.
Hide
Permalink
Thomas E Enebo added a comment -

The multibyte parsing part of this was fixed in the past. The column-offset bug remains. A few months ago we forked all IDE parsing aspects from the main JRuby project into a seperate project JRubyParser: http://kenai.com/projects/jruby-parser

I am closing this out since first half of this (part which affects runtime execution) has been fixed already.

Show
Thomas E Enebo added a comment - The multibyte parsing part of this was fixed in the past. The column-offset bug remains. A few months ago we forked all IDE parsing aspects from the main JRuby project into a seperate project JRubyParser: http://kenai.com/projects/jruby-parser I am closing this out since first half of this (part which affects runtime execution) has been fixed already.

People

Vote (1)
Watch (3)

Dates

  • Created:
    Updated:
    Resolved:
Atlassian JIRA (v4.4.3#663-r165197) | Report a problem
Powered by a free Atlassian JIRA open source license for Codehaus. Try JIRA - bug tracking software for your team.