ajax_scaffold generator fails due to regular expression 'dot' matching not picking up \r characters

I looked at this a bit and it appears to be a problem related to regular expressions and line terminations. There are a pair of regular expressions in erb.rb that look for start and end tags in a template file, but these seem to make some unfortunate assumptions about line terminators:

stag_reg = /(.*?)(<%%|<%=|<%#|<%-|<%|\n|\z)/
etag_reg = /(.*?)(%%>|-%>|%>|\n|\z)/

The template file 'unit_test.rb' from the ajax_scaffold distribution is opened in binary mode and has Windows '\r\n' line terminators. The '.' in the regular expressions doesn't appear to match the '\r', which eventually leads to the attempted subscripting of nil. Altering these regular expressions to look for '\r\n' in their second groups gets past the problem:

stag_reg = /(.*?)(<%%|<%=|<%#|<%-|<%|\n|\r\n|\z)/
etag_reg = /(.*?)(%%>|-%>|%>|\n|\r\n|\z)/

Probably not an ideal solution...

Also, there's still a problem even after making these changes:

uninitialized constant SO_KEEPALIVE

Yup, here on the Mac at least the carriage return handling with regular expression dot metachars seems to be the trouble:

In MRI irb:

irb(main):001:0> %r{.}.match("\r")[0]
=> "\r"
irb(main):002:0>


in JRuby jirb:

irb(main):001:0> %r{.}.match("\r")[0]
NoMethodError: undefined method `[]' for nil:NilClass
from (irb):1:in `method_missing'
from (irb):1
from /Users/mcovarru/Documents/workspace/jruby/lib/ruby/1.8/irb.rb:150:in `eval_input'
from /Users/mcovarru/Documents/workspace/jruby/lib/ruby/1.8/irb.rb:70:in `signal_status'
from /Users/mcovarru/Documents/workspace/jruby/lib/ruby/1.8/irb.rb:189:in `eval_input'
from /Users/mcovarru/Documents/workspace/jruby/lib/ruby/1.8/irb.rb:70:in `each_top_level_statement'
from /Users/mcovarru/Documents/workspace/jruby/lib/ruby/1.8/irb.rb:190:in `loop'
from /Users/mcovarru/Documents/workspace/jruby/lib/ruby/1.8/irb.rb:190:in `catch'
from /Users/mcovarru/Documents/workspace/jruby/lib/ruby/1.8/irb.rb:190:in `eval_input'
from /Users/mcovarru/Documents/workspace/jruby/lib/ruby/1.8/irb.rb:70:in `start'
from /Users/mcovarru/Documents/workspace/jruby/bin/jirb:13:in `catch'
from /Users/mcovarru/Documents/workspace/jruby/lib/ruby/1.8/irb.rb:71:in `start'
from /Users/mcovarru/Documents/workspace/jruby/bin/jirb:13
irb(main):002:0>

This is likely a duplicate of another bug reported here...

Er, I see this is a regex issue now; renaming as appropriate.

Ola Bini posted a possible fix for this to the list, where we replace . with . plus \r before compiling incoming regex. Java's regex . will never match \r.

There's another potential fix for this: Pattern.DOTALL. This causes line-terminator characters to be matched by .

So we have a few possible solutions...I'll look into it a bit today.

I suspect (but have not confirmed) that Pattern.DOTALL will match '\n', which I don't think we want.

It doesn't seem to have been recorded here before, but there seems to be a duplicate at JRUBY-123 (with a patch).

More results:

DOTALL does not do what we need. It causes both \r and \n to be matched by .

However, UNIX_LINES appears to do the trick. It allows \r to be matched by . while \n remains unmatched. This fixes Miguel's very simple case in the 25/Jul/06 09:39 PM comment above, and potentially others.

I'll attach a patch that fixes the two locations where we instantiate regexp. One of those locations (in the AST) may go away soon because it's not doing appropriate checks for unicode flags.

Very cool, I never knew about UNIX_LINES. It at least sounds like it should do the trick.

Sets regex to use UNIX_LINES flag so \r will be matched by .

Raphael Gillet has reported that the ajax scaffolding generator is now working well. There appear to be some other issues, but the original regex problem seems to be solved:

Ajax Scaffold generator now works quite well on Ola Bini's test_arj.
Some Ajax functionality is missing. Not sure why yet.

I will mark this bug resolved, and I have recommended that if Raphael can narrow down any additional bugs that he go ahead and report them separately.

Resolving as fixed; sometimes the littlest changes make the biggest difference.

Closing 0.9.1 resolved issues.