Unicode characters cause source locations to be off in Groovy editor

The problem comes around from org.codehaus.groovy.antlr.UnicodeEscapingReader, which will automatically read extra characters if the appropriate escape sequence is used. So, multiple characters can be consumed by the reader even though the GroovyLexer receives a single character.

See also GRECLIPSE-824.

Bumping this up to major since many users are hitting this.

I have a solution locally that will work for all locations that are not on the same line as the unicode escape sequence. For example, the following will show an underline correctly:

"\u00E9"
jj

But, the following would not. It would show the underline 5 characters earlier than it should:

"\u00E9" + jj

This will solve most of the cases that we are interested in, since a single escape sequence would not mess up locations for the entire file.

I am trying to find a balance between minimal changes to the underlying Groovy parser and completeness of the solution. The solution I have requires a very small change. I have an idea on how to make a complete solution, but it would require too many changes to the parser to keep things simple.

I will sketch out my solution and the complete solution below.

Actually, I have reworked things a bit and I have a full solution in my workspace.

Here is a sketch of the solution:

1. Changes to UnicodeEscapingReader to keep track of the number of escape sequences found in the current line as well as the total amount found in the file (up to the current location).
2. Changes to SourceBuffer so that when items are added to the lineEndings array, it queries the UnicodeEscapingReader instance to find the offset to add for the total number of escape sequences found in the file.
3. This correctly computes the source locations for the lines after an escape sequence, but the escape sequence line itself is not computed correctly.
4. So, in addition to that, create a UnicodeLexerSharedInputState subclass of LexerSharedInputState that queries the UnicodeEscapingReader for an extra offset when the getColumn() and getTokenStartColumn() methods are called.
5. Changes to ErrorRecoveredCSTParserPlugin and AntlrParserPlugin so that UnicodeLexerSharedInputState is used, instead of LexerSharedInputState.

There is negligible overhead to this solution. I will write some tests and then commit.

I committed the fix with regression tests.