DummyDynamicScope should never be used for backref storage

This is probably the inspector not properly inspecting some piece of code here that needs a scope. Definitely needs to be fixed. Any progress on that spec?

I could not reproduce this, either by running it directly, with -X+C, as a .class, or as a .class inside a jar file.

I've got it narrowed down, but determined that compiling and jarring platform.rb is needed, but there's a few other things needed as well I haven't found out yet.

This is my test app:

require 'lib/java/gemini.jar'   #gemini does not work
require 'lib/java/platform.jar' #platform works
require 'platform'
puts Platform.using_osx?

Here's the stack:

DummyDynamicScope.java:49:in `getBackRef': java.lang.RuntimeException: DummyDynamicScope should never be used for backref storage
        from RubyRegexp.java:1392:in `updateBackRef'
        from RubyRegexp.java:1384:in `updateBackRef'
        from RubyRegexp.java:1374:in `search'
        from RubyRegexp.java:1289:in `op_match'
        from RuntimeHelpers.java:1217:in `match3'
        from platform.rb:20:in `method__4$RUBY$using_osx_p_'
        from platform#using_osx_p_:-1:in `call'
        from platform#using_osx_p_:-1:in `call'
        from WrapperMethod.java:54:in `call'
        from CachingCallSite.java:258:in `cacheAndCall'
        from CachingCallSite.java:77:in `call'
        from main.rb:4:in `__file__'
        from main.rb:-1:in `load'
        from Ruby.java:592:in `runScript'
        from Ruby.java:514:in `runNormally'
        from Ruby.java:360:in `runFromMain'
        from Main.java:268:in `run'
        from Main.java:113:in `run'
        from Main.java:97:in `main'

I'm not sure if that helps or not. I'm going to start removing classes and such from the gemini.jar I've compiled. I'll provide it as an attachment if you're in a hurry to squash this.

Thanks for the patience (:

gemini.jar has platform.rb in it. Compiling and jarring platform.rb is not enough to produce the error. If gemini.jar is built without compiled files (just jarred .rb files), the error is also not produced.

Here's another data point that may help with troubleshooting. Right now on trunk, you get a similar error as follows;

surfboard:jruby[git:master]$ bin/jruby --1.9 test/externals/ruby1.9/ruby/test_time.rb 
Loaded suite test/externals/ruby1.9/ruby/test_time
Started
.EEFDummyDynamicScope.java:49:in `getBackRef': java.lang.RuntimeException: DummyDynamicScope should never be used for backref storage
	from RubyRational.java:369:in `convertCommon'
	from RubyRational.java:354:in `convert'
	from org/jruby/RubyRational$s_method_multi$RUBYINVOKER$convert.gen:-1:in `call'
	from RubyClass.java:584:in `finvoke'
	from RuntimeHelpers.java:412:in `invoke'
	from RubyKernel.java:346:in `new_rational'
	from org/jruby/RubyKernel$s_method_multi$RUBYINVOKER$new_rational.gen:-1:in `call'
	from CachingCallSite.java:330:in `cacheAndCall'
	from CachingCallSite.java:189:in `call'
	from test_time.rb:139:in `method__16$RUBY$test_at_rational'
	from test_time#test_at_rational:-1:in `call'
	from RubyClass.java:429:in `finvoke'
	from RubyObject.java:1436:in `send'
	from org/jruby/RubyObject$i_method_multi$RUBYINVOKER$send.gen:-1:in `call'
	from JavaMethod.java:259:in `call'
	from CachingCallSite.java:147:in `call'
	from CallOneArgNode.java:57:in `interpret'
	from NewlineNode.java:104:in `interpret'
	from BlockNode.java:71:in `interpret'
	from RescueNode.java:225:in `executeBody'
	from RescueNode.java:121:in `interpretWithoutJavaExceptions'
	from RescueNode.java:112:in `interpret'
	from EnsureNode.java:96:in `interpret'
	from BeginNode.java:83:in `interpret'
	from NewlineNode.java:104:in `interpret'
	from BlockNode.java:71:in `interpret'
	from InterpretedMethod.java:173:in `call'
	from DefaultMethod.java:169:in `call'
	from CachingCallSite.java:147:in `call'
	from CallOneArgNode.java:57:in `interpret'
	from DAsgnNode.java:110:in `interpret'
	from NewlineNode.java:104:in `interpret'
	from BlockNode.java:71:in `interpret'
	from Interpreted19Block.java:186:in `evalBlockBody'
	from Interpreted19Block.java:138:in `yield'
	from Block.java:194:in `yield'
	from RubyArray.java:1580:in `each'
	from RubyArray.java:1587:in `each19'
	from org/jruby/RubyArray$i_method_0_0$RUBYFRAMEDINVOKER$each19.gen:-1:in `call'
	from CachingCallSite.java:116:in `callBlock'
	from CachingCallSite.java:123:in `call'
	from CallNoArgBlockNode.java:64:in `interpret'
	from NewlineNode.java:104:in `interpret'
	from Interpreted19Block.java:186:in `evalBlockBody'
	from Interpreted19Block.java:138:in `yield'
	from Block.java:194:in `yield'
	from RubyArray.java:1580:in `each'
	from RubyArray.java:1587:in `each19'
	from org/jruby/RubyArray$i_method_0_0$RUBYFRAMEDINVOKER$each19.gen:-1:in `call'
	from CachingCallSite.java:300:in `cacheAndCall'
	from CachingCallSite.java:118:in `callBlock'
	from CachingCallSite.java:123:in `call'
	from CallNoArgBlockNode.java:64:in `interpret'
	from NewlineNode.java:104:in `interpret'
	from BlockNode.java:71:in `interpret'
	from InterpretedMethod.java:173:in `call'
	from DefaultMethod.java:169:in `call'
	from CachingCallSite.java:310:in `cacheAndCall'
	from CachingCallSite.java:149:in `call'
	from FCallOneArgNode.java:36:in `interpret'
	from NewlineNode.java:104:in `interpret'
	from BlockNode.java:71:in `interpret'
	from InterpretedMethod.java:173:in `call'
	from DefaultMethod.java:169:in `call'
	from CachingCallSite.java:310:in `cacheAndCall'
	from CachingCallSite.java:149:in `call'
	from CallOneArgNode.java:57:in `interpret'
	from DAsgnNode.java:110:in `interpret'
	from NewlineNode.java:104:in `interpret'
	from BlockNode.java:71:in `interpret'
	from Interpreted19Block.java:186:in `evalBlockBody'
	from Interpreted19Block.java:174:in `yield'
	from Interpreted19Block.java:106:in `call'
	from Block.java:89:in `call'
	from RubyProc.java:221:in `call'
	from RubyProc.java:204:in `call'
	from Ruby.java:2611:in `tearDown'
	from Main.java:274:in `run'
	from Main.java:117:in `run'
	from Main.java:97:in `main'

1.9 compiler is still in pretty weak shape, waiting for more runs against the interpreter to confirm we've got the logic right. But I will make a mental note of this problem and mark it as 1.9 also.

Still having trouble reproducing this. Logan: Can you try to reconfirm and hunt one of us down on IRC to show us how to produce this?

I'll make the time! When are you guys typically on nowadays? The 8 PM nights seem largely over now (:

1.9 mode does not current use the compiler right now and it appears this is happening for 1.8 code. So this is strictly a JIT issue.

Attached file should reproduce this issue (at least, as far as I am experiencing it – I am assuming this is the same issue).

I've had reports of people hitting this problem with DataMapper (and, specifically dm-validations, our validations plugin). I made a reduction from our dm-validations code, and got it down to a few lines: http://gist.github.com/234230

And to confirm, if I run with JIT off (-X-C) or in 1.9 mode, then I do not see this issue using the reproduction I attached.

I have committed in e6c7ee9 a change to make DummyDynamicScope only warn (in verbose mode) if backref is set/get'ed, rather than raising a Java-level RuntimeException. Since this generally will only affect bodies of code that are using send to call backref-aware methods, the silent swallowing should ideally be ok.

Those of you that have seen this issue, please give things another go against JRuby master.

Alex: I didn't realize your reduction actually included a very simple version...I should have read it more carefully.

Based on your reduction3, I have a trivial reduced case:

../jruby-1.4.0/bin/jruby -e "def foo(value, expected); Object; blah = value.send(:=~, expected); end; foo('foo', /foo/)"

The problem here is that the compiler optimization to eliminate local scopes does not do so for "send". Normally, it only detects methods known to set or get $_{, like the literal =} method itself. When you call those methods via send, it does not see them, and therefore does not prepare a scope for them. It does not always fire because often the calling method does have a scope, and the backref ends up getting set there.

The constant lookup here is what triggers the error. A method that does not appear to set or get $~ but does access a constant gets a DummyDynamicScope just for constant-scoping purposes. This results in the method having a scope, but a scope that refuses to accept $~, so it errors out.

This is currently hard to fix without also causing a deoptimization for any methods doing .send, which is very heavily used for metaprogrammed code. I do have, in theory, a better way to handle backref logic; but I have not had spare cycles to try to put it in place.

The temporary fix is not great, because it masks two bugs: the first is the fact that we're not properly preparing for $~ when you use send, and the second is that we're potentially overwriting a previous call's $~ when no constant is present to trigger the error.

Not sure how to deal with this at the moment. $~ sucks.

I am downgrading this to "major" and making the warning only appear in debug mode, since it isn't something the user can fix and won't affect 99% of users. There's a small chance that code which sends a backref or lastline-mutating method could overwrite a previous call's backref or lastline, and that still needs to be remedied. But I'm more and more comfortable saying backref and lastline should not be expected to work across things like evals and sends, since it requires a level of stack manipulation we can't really do.

The fix for this will likely require a rework of how we handle backref. It needs to be local to the containing method scope (mostly) but mutable by any calls that happen within the body. My current strategy for this would be to have specific named calls (like =~) use one of two ways to set/get backref/lastline:

• Via a thread-local slot that's used only for the duration of the call. It would be populated with the current backref local variable before the call and used to set that local var after the call completes, avoiding the requirement for a separate heap structure.
• Via an additional "out" parameter passed on the call stack. In this case, an IRubyObject-implementing carrier object would be prepared for backref-utilizing methods and passed to all calls likely to use backref. Calls that don't use it would simply ignore the extra value.

At any rate, this is not something easily fixable in 1.5, and the impact to users is extremely low and unlikely to happen.

I upgraded to 1.9 and now I got this new Warning message
":1 warning: DummyDynamicScope should never be used for backref storage"
a lot. I assume that its somewhere in my code, but how can I now find the reason(s)?

Alex: If you are getting this "a lot" then I want to look into it. It indicates that some code is over-optimizing somewhere.

I'll dig up this warning and see if I can come up with a way to investigate it.

I'm going to call this fixed.

There were recently some key fixes to the compiler (the inspector, really) that ensured scopes would be available for backref methods in more cases. This bug is also very old, and the compiler has been patched numerous times since then for similar issues.

If this exact issue still exists with a simple reproduction, feel free to reopen.