[1.9] Unicode characters in regexp in unicode-encoded file do not parse

Charles Oliver Nutter added a comment - 04/Jan/11 4:22 AM Possible fix pushed to master. The problem appears to be that when pulling multibyte characters off the stream, StreamTerm.parseStringIntoBuffer was using the first byte as a codepoint for the whole character, and appending the resulting character bytes to the buffer. It then proceeded to the next byte and did the same thing again. The result was a totally garbled string. My fix was to get the encoded length associated with the first byte and append that many bytes to the buffer. There's probably a utility method for this somewhere I don't know about. commit e94175ed4335d251ef21fc4ec68d46aae801380a Author: Charles Oliver Nutter <headius@headius.com> Date: Tue Jan 4 04:19:09 2011 -0600 Possible fix for JRUBY-5301: [1.9] Unicode characters in regexp in unicode-encoded file do not parse * When reading multi-byte characters from stream, use encoded length associated with first byte to read remaining bytes. Old code was using first byte as codepoint, and garbling the string. diff --git a/src/org/jruby/lexer/yacc/StringTerm.java b/src/org/jruby/lexer/yacc/StringTerm.java index cc6ff85..77404e8 100644 --- a/src/org/jruby/lexer/yacc/StringTerm.java +++ b/src/org/jruby/lexer/yacc/StringTerm.java @@ -300,7 +300,13 @@ public class StringTerm extends StrTerm { if (buffer.getEncoding() != encoding) { mixedEscape(lexer, buffer.getEncoding(), encoding); } - lexer.tokenAddMBC(c, buffer, buffer.getEncoding()); // No EOF here? + // read bytes for length of character + int length = encoding.length((byte)c); + buffer.append((byte)c); + for (int off = 0; off < length - 1; off++) { + buffer.append(src.read()); + } + continue; } else if (qwords && Character.isWhitespace(c)) { src.unread(c); break;