memory leak in rubygems

Can you describe the leak more? I ran your benchmark, and it stayed at a consistent speed and memory usage (1.6.0.dev).

I don't understand why it's leaking. What OS did you test against?
Here's an updated test script which prints out memory usage after each iteration
http://gist.github.com/750957

When run on windows 2003 32 bit (before applying previously proposed patch)

jruby 1.5.6 (ruby 1.8.7 patchlevel 249) (2010-12-03 9cf97c3) (Java HotSpot(TM) Client VM 1.6.0_22) [x86-java]
loop 1 time 0.4530 mem: 16252928
loop 2 time 0.1720 mem: 16318464
loop 3 time 0.1560 mem: 16318464
loop 4 time 0.1400 mem: 16318464
loop 5 time 0.1410 mem: 16879616
loop 6 time 0.1090 mem: 20127744
loop 7 time 0.1250 mem: 23183360
loop 8 time 0.1250 mem: 26370048
loop 9 time 0.1250 mem: 29200384
loop 10 time 0.1250 mem: 32256000
...
loop 161 time 0.0940 mem: 498311168
loop 162 time 0.0940 mem: 501497856
loop 163 time 0.0940 mem: 504553472
loop 164 time 0.0940 mem: 506855424
loop 165 time 0.1090 mem: 506855424
loop 166 time 0.0930 mem: 506855424

after applying proposed patch

jruby 1.5.6 (ruby 1.8.7 patchlevel 249) (2010-12-03 9cf97c3) (Java HotSpot(TM) Client VM 1.6.0_22) [x86-java]
loop: 1 time: 0.4530 mem: 16252928
loop: 2 time: 0.1720 mem: 16318464
loop: 3 time: 0.1570 mem: 16318464
loop: 4 time: 0.1410 mem: 16318464
loop: 5 time: 0.1410 mem: 16318464
loop: 6 time: 0.1410 mem: 16318464
loop: 7 time: 0.1250 mem: 16318464
...
loop: 185 time: 0.0940 mem: 16318464

see the same leak occurring on Windows XP 32bit (even on master). If I had to venture a guess I would say that since our DL+Win32API implementation is implemented in terms of FFI, we are not releasing something at an FFI level. However, since I see no explicit way of cleaning up the Win32API instance itself Win32API appears to rely on GC to clean up resources

Your patch seems reasonable to me but I wonder what explains that saving the object as a constant will cause it to never be freed/released....Even when terminating the runtime? I would like to understand that a little bit better. We may be able to solve this issue without accepting the patch (which I believe works around an issue).

I think I was able to simulate basically what happens in the Win32API case, but on OS X, using the following patch and your test script:

https://gist.github.com/798256

However, on JRuby master (to become 1.6.0.RC2) I don't see any leaking. I think we should re-test this on Windows to see if it is still a problem.

Moving off of 1.6.0 (and without more work not for 1.6.1). What we need is for someone to dig into this and isolate the leak. My personal opinion is this is not related to rubygems as much as it is to WIN32API not freeing resources...So if anyone wants to dig into this we would love it?

Here's a simplified test case: https://gist.github.com/974975

If you remove the assignment to the constant SHGetFolderPath, it doesn't leak.

Here's a test case that reproduces on OS X: https://gist.github.com/975179

You have to wait for 32 or more iterations to see evidence of the leak, but it's there. The key to reproducing it seem to be the use of FFI::Function#attach to a constant. This leads me to suspect that the problem revolves around the WeakHashMap "refmap" in org.jruby.ext.ffi.AbstractInvoker. Here's my hypothesis:

The WeakHashMap holds a strong reference to the AbstractInvoker as long as the associated DynamicMethod key is reachable. The DynamicMethod is reachable so long as the object it's attached to is reachable. When attached to a Ruby constant, that constant is reachable as long as its runtime is reachable. And the runtime is always reachable because there's a reference to it in the AbstractInvoker held in the WeakHashMap.

This is similar to the memory leak I just tracked down in the JRuby JSON library: https://github.com/flori/json/pull/74. I'm having doubts about the patch I submitted there though, so I'd appreciate having someone take a second look at both this and that.

This issue appears more generic than just constants - I'm seeing DynamicMethod instances themselves being held onto in org.jruby.RubyModule's methods map. So, just like with constants, this means that the Ruby is always keeping a strong reference to the DynamicMethod around and thus the DynamicMethod key in the refmap WeakHashMap will never get garbage collected as long as the Ruby is around. The Ruby will stay around as long as the refmap entry exists.

I opened https://github.com/jruby/jruby/issues/203 before seeing the issue here but they will likely be fixed by the same thing, even though this JIRA is about a more specific manifestation of this leak.

I've sent a pull request (https://github.com/jruby/jruby/pull/205) to fix this issue. Here's the output of the OS X test case provided by John before and after the fix:

before:
jruby 1.7.0.dev (ruby-1.9.3-p139) (2012-05-04 b0c4eaa) (Java HotSpot(TM) 64-Bit Server VM 1.6.0_31) [darwin-x86_64-java]
loop: 1 time: 0.7300 mem: 85000192
loop: 2 time: 0.8800 mem: 85000192
loop: 3 time: 0.2900 mem: 85000192
...
loop: 50 time: 0.1200 mem: 226222080
loop: 51 time: 0.3400 mem: 230547456
loop: 52 time: 0.1000 mem: 234803200

after:
jruby 1.7.0.preview1 (ruby-1.9.3-p203) (2012-06-12 1a7fe2f) (Java HotSpot(TM) 64-Bit Server VM 1.6.0_31) [darwin-x86_64-java]
loop: 1 time: 1.3800 mem: 85000192
loop: 2 time: 1.2400 mem: 85000192
loop: 3 time: 0.4000 mem: 85000192
...
loop: 50 time: 0.1500 mem: 85000192
loop: 51 time: 0.1300 mem: 85000192
loop: 52 time: 0.1000 mem: 85000192
...
loop: 125 time: 0.0800 mem: 85000192

fixed in commit cfbb64d (patch by Ben Browning)