The current approach of marking tests that are supposed to fail compilation with the _MustFailCompile suffix is too coarse-grained, because there is no way to specify a specific error to check for. So some _MustFailCompile tests passed for the wrong reasons (i.e., unrelated compilation errors).
I suggest we enlist compiler support in checking for specific errors. The compiler could succeed (and produce a pre-defined file, or simply remove the offending code from the original test case) if a particular error is present on its error queue at the end of the compilation (or, better yet, if the specified set of errors is exactly what's on the compiler's error queue).
The test infrastructure already supports adding per-test compile options. We could make use of this feature to specify the expected errors to the compiler. For example, we could add an -ENSURE_ERROR=id option to check for a specific error identifier, and, as an interim measure, an -ENSURE_ERROR_MESSAGE="message" to check that a particular message pattern appears on the error queue. The _MustFailCompile would be removed from the test names, and the compiler will ensure that compilation succeeds appropriately if the specified error is (or would have been) present, and the resulting code will be something that runs normally (perhaps simply a run() method with return true; as the body).
Until this is done, we will have "accidental" successes of _MustFailCompile tests.