Compiler support for regression testing

This is not a release blocker. Defer to 2.1.

Sounds good to me, but I think it's a front-end sort of issue.

It will indeed become a frontend issue once we have a design for this.

Yoav has been working on validating the successes of _MustFailCompile tests, so can testify how easy it is to get spurious passes on those. However, this should not block the 2.1.0 release. Defer to 2.1.1.

Yoav has, I believe, greatly improved testing. He can decide if it's sufficient for our purposes; I suspect it probably is.

The design is simple:
I have 3 kind of markers:

"// ... ERR"  - marks an error
"// ... ShouldNotBeERR" - the compiler reports an error, but it shouldn't
"// ... ShouldBeErr"- the compiler doesn't report an error, but it should

I check that there is an error in that line (but I don't check a specific error message, because those change often).
So it reduces the risk of an accidental passing test.

See RunTestSuite.

This is the design:
I added 3 kinds of annotations:

@ERR
@ShouldNotBeERR
@ShouldBeErr

I also added another phase called:
ErrChecker
that is on when passing the flag CHECK_ERR_MARKERS.

I also added to ErrorInfo 3 new {{kind}}s:

  public static final int ERR_MARKER        = 8;  // A line was marked with @ERR
  public static final int SHOULD_NOT_BE_ERR_MARKER    = 9;  // A line was marked with @ShouldNotBeERR
  public static final int SHOULD_BE_ERR_MARKER    = 10; // A line was marked with @ShouldBeErr

The phase ErrChecker does the following:
it iterates over all the annotations, searching for one of the above 3 annotations.
then it issues the corresponding error in that line position.

RunTestSuite takes all errors and compares them with errors issued by the compiler.
Read the documentation in RunTestSuite.

bulk move of all unresolved issues from 2.1.1 to 2.1.2

defer remaining 2.1.2 issues to 2.2.0

Yoav, let's figure out if we're done with this issue and need to change the nightly test scripts, or if there's more work to be done on this in the frontend.

Ongoing work. Defer to 2.2.1.

Need to determine what else needs to be done to be able to close this issue. Igor, could you summarize?

I think you can close it.
If you run with RunTestSuite you can have specific error markers on the exact line that should fail compilation.
(Although it is not a specific error message, writing "// ERR" on the desired line is specific enough.)

We've had too many of these succeed with the wrong error (e.g., a syntax error instead of a type error, etc). I'd be more comfortable with having a more precise control over the expected messages.

There was some work on assigning error codes to messages. Has that progressed at all? If we had a way of specifying the expected code (e.g., // ERR:T2194 for type error with code 2194), the checking would be much more precise, and this issue could be closed.