There is no way to publicly set the compatVersion instance variable from within Java code. Instead, the only way to do so is through the "--compat" command line argument or the jruby.compat.version property setting.
Adding a RubyInstanceConfig#setCompatVersion method would be nice. Here's the code:
public void setCompatVersion(CompatVersion newCompatVersion) {
compatVersion = newCompatVersion;
}