Here's a test case that reproduces on OS X: https://gist.github.com/975179
You have to wait for 32 or more iterations to see evidence of the leak, but it's there. The key to reproducing it seem to be the use of FFI::Function#attach to a constant. This leads me to suspect that the problem revolves around the WeakHashMap "refmap" in org.jruby.ext.ffi.AbstractInvoker. Here's my hypothesis:
The WeakHashMap holds a strong reference to the AbstractInvoker as long as the associated DynamicMethod key is reachable. The DynamicMethod is reachable so long as the object it's attached to is reachable. When attached to a Ruby constant, that constant is reachable as long as its runtime is reachable. And the runtime is always reachable because there's a reference to it in the AbstractInvoker held in the WeakHashMap.
This is similar to the memory leak I just tracked down in the JRuby JSON library: https://github.com/flori/json/pull/74. I'm having doubts about the patch I submitted there though, so I'd appreciate having someone take a second look at both this and that.