Ok, I have a conundrum here.
The new backtrace logic uses the actual Java trace of an exception to build the Ruby trace. However it defaults to excluding frames unless it knows about them, which is why arbitrary Java calls disappear.
I'm not sure how best to turn off that behavior for subsets of the Java trace. For cases where you simply call from Ruby into Java, it's easy; we just see that there's a marker in the stack indicating we've left Ruby and start aggregating those frames. But then I'm not sure how to handle cases where we call from Java back into Ruby, such as for an interface implementation. There's no clear marker to indicate we're back in "Ruby land" and should start filtering.
One thought I had would be to default to including frames unless explicitly excluded. So we'd omit any org.jruby stuff that's not a core method, rubyjit.* stuff, rubyobj.* stuff, and packages for any of the libraries we ship. But that might allow too much through, since if we call through a java.* class as part of our internal impl, it will show up in backtraces. The more internal stuff we show in backtraces the more likely we are to screw up anyone expecting caller() to produce similar results to MRI (and I won't get into the politics of why they shouldn't do that).
A third option would be to have Java exceptions in Ruby land include the Java trace as well, so when they log they log both our Ruby trace and the Java trace that goes with it.
A fourth option would be to have a clear marker when going from Ruby to Java that turns on full traces, and any call back through org.jruby reenables filtering. This is unlikely to be perfect, but it's closer.