JRuby

Parsing issues with long lines

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Minor Minor
  • Resolution: Fixed
  • Affects Version/s: JRuby 1.6.7, JRuby 1.7.0.pre2
  • Fix Version/s: JRuby 1.7.0.RC1
  • Component/s: Parser
  • Labels:
    None
  • Environment:
    Mac 10.7.4 with rvm 1.15.5
  • Number of attachments :
    0

Description

There seems to be a parsing bug in JRuby when the first line of a Ruby file is very long (~8000 characters or more). I created a gist with an example file here: https://gist.github.com/3497860 The first line creates a long array of hashes. Depending on how long the first line is or what the subsequent line is I'll see different errors such as SyntaxErrors or NameErrors. I saw on my system when the second line was very short the program would complete without any errors or output. This all seems to be related to a parse problem. If you remove one hash from the array then it parses correctly. I tried a few variants of the file on MRI and I couldn't reproduce the problem there.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Charles Oliver Nutter added a comment -

Fun! Reproduced here.

Show
Charles Oliver Nutter added a comment - Fun! Reproduced here.
Hide
Permalink
Charles Oliver Nutter added a comment -

I took a quick look through lexer/parser stuff and did not see where there might be an ~8000 byte limit on line length. Obviously there must be something...I suspect Tom Enebo can probably figure it out in no time at all.

Show
Charles Oliver Nutter added a comment - I took a quick look through lexer/parser stuff and did not see where there might be an ~8000 byte limit on line length. Obviously there must be something...I suspect Tom Enebo can probably figure it out in no time at all.
Hide
Permalink
Thomas E Enebo added a comment -

Fixed in commit [master 4aa8812]. The basic issue is we examine the incoming main file to look for shebang lines (and we skip top of files for -x). Since we are receiving a stream we need to potentially back up the stream after doing this processing so the parser gets a nice full stream. Our old logic was not safe on how it used mark.

This is pretty ugly stuff. I would like the parser to handle all of this, but it would take a lot of refactoring so I compromised

Show
Thomas E Enebo added a comment - Fixed in commit [master 4aa8812] . The basic issue is we examine the incoming main file to look for shebang lines (and we skip top of files for -x). Since we are receiving a stream we need to potentially back up the stream after doing this processing so the parser gets a nice full stream. Our old logic was not safe on how it used mark. This is pretty ugly stuff. I would like the parser to handle all of this, but it would take a lot of refactoring so I compromised

People

  • Assignee:
    Thomas E Enebo
    Reporter:
    Jason Gilman
Vote (0)
Watch (4)

Dates

  • Created:
    Updated:
    Resolved: