Currently JRuby disables JIT compilation if tracing is enabled (by having a trace function installed). This works ok if tracing is enabled before calling any JITed code, but it fails to work correctly if tracing is enabled mid-call, such as if a JITed method calls code that causes tracing to be enabled. The return path out of the JITed code, since it's unaware of tracing, will not call trace functions appropriately.
And it's not a simple matter of adding tracing to the compiler. When the compiler optimizes methods to use normal Java local variables (stack-allocated) instead of DynamicScope heap-allocated variables, tracing is impossible since stack-allocated vars can't be placed into a usable binding, required for many (or perhaps all?) set_trace_func events.
The furthest we could go would be to enabled tracing in heap-allocated methods and disable stack-based optimizations when tracing is in use. But this comes back to the first problem again. We can't know ahead of time whether a stack-based method might be executing when tracing gets turned on.
Because of these problems, the test_trace_func unit test has been temporarily disabled until we can fix this problem.
The least-intrusive option I see right now is to require users to enable tracing at the command line, and add tracing to heap-allocated method compilation. This is the only way to know tracing is being used before any compilation happens, and by adding tracing into heap-based methods traced code can still get most of the benefit of compilation, without trace-breaking optimizations.
In addition, various fixes have come in to allow compiled code to be debugged/trace with more to come.