I've committed a fix for this on trunk (r5360). Proxies are cached in a Weak/Concurrent Map-like structure, which should provide reasonably good performance (not verified yet). The old JavaObject instanceCache has been removed, as it serves no real purpose; since that was synchronized (whereas the new cache uses concurrent techniques), I would expect a net performance gain, though again, I haven't verified this.
One issue I've noted in the code is that the new cache holds weak (or soft) references to both the underlying Java object and its proxy. This is necessary because the proxy has a strong reference to the Java object, so nothing would ever be gc'ed if we kept a strong reference to the proxy. That means that at some point an incorrect proxy could be returned from Java, unless the user keeps a reference on the Ruby side to any proxies passed to Java. There's really not much that can be done about that, at least under the current architecture.
I'm leaving this issue (and the related/duplicate
JRUBY-1513 and JRUBY-1735 open pending the addition of some unit tests, and verification that this actually solves the problems reported. It would also be good to do some profiling, especially WRT memory consumption. I've currently got the cache using soft references, but this is a parameter to the cache and can be changed easily to use weak references if the memory hit is too great.
[As an interesting side note, it appears that at some point JavaObject was meant to be the base class for all "proxy" classes; the old instanceCache did for JavaObjects exactly what I've done for proxies, if somewhat less performantly. However, under the current architecture, it appears to serve no good purpose.]
[A second side note: I spent some time the past few days looking at eliminating out extra layer of wrapping, and in fact making JavaObject the base class for all proxy classes (but not interface modules), and also hard-wiring it as the Rubified version of java.lang.Object. I like the idea as a transitional step in our JI overhaul, simplifying and eliminating a lot of cruft. But it may be a bit much in the midst of a beta cycle. Also, after next week (when I get a few more extra days), I'm not sure when I'll next have much time.]