groovy

Closure.memoize fails on Google App Engine

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: None
  • Fix Version/s: 1.8-beta-3
  • Component/s: groovy-jdk
  • Labels:
    None
  • Number of attachments :
    1

Description

Running the following code on the Groovy Web Console:

cl = {a, b ->
    sleep(3000) // simulate some time consuming processing
    a + b
}
mem = cl.memoize()

def callClosure(a, b) {
    def start = System.currentTimeMillis()

    mem(a, b)

    println "Inputs(a = $a, b = $b) - took ${System.currentTimeMillis() - start} msecs."
}

callClosure(1, 2)
callClosure(1, 2)
callClosure(2, 3)
callClosure(2, 3)
callClosure(3, 4)
callClosure(3, 4)

callClosure(1, 2)
callClosure(2, 3)
callClosure(3, 4)

(taken from http://roshandawrani.wordpress.com/2010/10/18/groovy-new-feature-closures-can-now-memorize-their-results/)

Throws the following exception (when caught and printed to stdout)

java.security.AccessControlException: access denied (java.lang.RuntimePermission accessDeclaredMembers)
	at com.google.appengine.runtime.Request.process-<unknown>(Request.java)
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:355)
	at java.security.AccessController.checkPermission(AccessController.java:567)
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
	at java.lang.SecurityManager.checkMemberAccess(SecurityManager.java:1679)
	at java.lang.Class.checkMemberAccess(Class.java:2174)
	at java.lang.Class.getDeclaredMethods(Class.java:1807)
	at java.lang.Class.getEnclosingMethod(Class.java:919)
	at sun.reflect.generics.scope.ClassScope.computeEnclosingScope(ClassScope.java:50)
	at sun.reflect.generics.scope.AbstractScope.getEnclosingScope(AbstractScope.java:74)
	at sun.reflect.generics.scope.AbstractScope.lookup(AbstractScope.java:90)
	at sun.reflect.generics.factory.CoreReflectionFactory.findTypeVariable(CoreReflectionFactory.java:109)
	at sun.reflect.generics.visitor.Reifier.visitTypeVariableSignature(Reifier.java:165)
	at sun.reflect.generics.tree.TypeVariableSignature.accept(TypeVariableSignature.java:43)
	at sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
	at sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
	at sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
	at sun.reflect.generics.repository.ClassRepository.getSuperclass(ClassRepository.java:84)
	at java.lang.Class.getGenericSuperclass(Class.java:694)
	at com.sun.beans.TypeResolver.prepare(TypeResolver.java:274)
	at com.sun.beans.TypeResolver.<init>(TypeResolver.java:243)
	at com.sun.beans.TypeResolver.resolve(TypeResolver.java:172)
	at com.sun.beans.TypeResolver.resolveInClass(TypeResolver.java:93)
	at java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:385)
	at java.beans.MethodDescriptor.setMethod(MethodDescriptor.java:116)
	at java.beans.MethodDescriptor.<init>(MethodDescriptor.java:74)
	at java.beans.MethodDescriptor.<init>(MethodDescriptor.java:58)
	at java.beans.Introspector.getTargetMethodInfo(Introspector.java:1181)
	at java.beans.Introspector.getBeanInfo(Introspector.java:408)
	at java.beans.Introspector.getBeanInfo(Introspector.java:180)
	at groovy.lang.MetaClassImpl$15.run(MetaClassImpl.java:2937)
	at java.security.AccessController.doPrivileged(AccessController.java:63)
	at groovy.lang.MetaClassImpl.addProperties(MetaClassImpl.java:2935)
	at groovy.lang.MetaClassImpl.initialize(MetaClassImpl.java:2918)
	at org.codehaus.groovy.reflection.ClassInfo.getMetaClassUnderLock(ClassInfo.java:166)
	at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:182)
	at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:214)
	at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:751)
	at groovy.lang.GroovyObjectSupport.<init>(GroovyObjectSupport.java:32)
	at groovy.lang.Closure.<init>(Closure.java:106)
	at groovy.lang.Closure.<init>(Closure.java:117)
	at org.codehaus.groovy.runtime.memoize.Memoize$1.<init>(Memoize.java:55)
	at org.codehaus.groovy.runtime.memoize.Memoize.buildMemoizeFunction(Memoize.java:55)
	at groovy.lang.Closure.memoize(Closure.java:530)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:43)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:88)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:885)
	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:39)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:40)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120)
	at Script1.run(Script1.groovy:7)
	at groovy.lang.GroovyShell.evaluate(GroovyShell.java:576)
	at groovy.lang.GroovyShell.evaluate(GroovyShell.java:614)
	at groovy.lang.GroovyShell.evaluate(GroovyShell.java:585)
	at groovy.lang.GroovyShell$evaluate.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:40)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:124)
	at executor.run(executor.groovy:25)
	at groovy.util.GroovyScriptEngine.run(GroovyScriptEngine.java:515)
	at groovy.servlet.GroovyServlet$1.call(GroovyServlet.java:120)
	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:99)
	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.access$300(GroovyCategorySupport.java:61)
	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:239)
	at groovy.servlet.GroovyServlet.service(GroovyServlet.java:129)
	at groovyx.gaelyk.GaelykServlet.super$5$service(GaelykServlet.groovy)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:43)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:88)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1055)
	at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:127)
	at groovyx.gaelyk.GaelykServlet$_service_closure1.doCall(GaelykServlet.groovy:68)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:43)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:88)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:885)
	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:149)
	at groovyx.gaelyk.GaelykServlet$_service_closure1.doCall(GaelykServlet.groovy)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:43)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:88)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:885)
	at groovy.lang.Closure.call(Closure.java:288)
	at groovy.lang.Closure.call(Closure.java:282)
	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:111)
	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:250)
	at org.codehaus.groovy.runtime.DefaultGroovyMethods.use(DefaultGroovyMethods.java:415)
	at org.codehaus.groovy.runtime.dgm$740.invoke(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoMetaMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:307)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:51)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:153)
	at groovyx.gaelyk.GaelykServlet.service(GaelykServlet.groovy:67)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:806)
	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:43)
	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:229)
	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:52)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:128)
	at groovyx.gaelyk.routes.RoutesFilter.doFilter(RoutesFilter.groovy:156)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:326)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
	at com.google.net.rpc.impl.BlockingApplicationHandler.handleRequest(BlockingApplicationHandler.java:24)
	at com.google.net.rpc.impl.RpcUtil.runRpcInApplication(RpcUtil.java:418)
	at com.google.net.rpc.impl.Server$RpcTask.runInContext(Server.java:572)
	at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:448)
	at com.google.tracing.TraceContext.runInContext(TraceContext.java:688)
	at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:326)
	at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:318)
	at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:446)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:636)

Seems to go bang when GroovyObjectSupport calls InvokerHelper.getMetaClass in the constructor on line 32

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Roshan Dawrani added a comment -

Attaching a patch that has the workaround for making memoize() work on google app engine.

The java.lang.RuntimePermission accessDeclaredMembers permission check was failing at the following place in OpenJDK VM:

java.lang.SecurityManager
    if (which != Member.PUBLIC) {
        Class stack[] = getClassContext();
        /*
         * stack depth of 4 should be the caller of one of the
         * methods in java.lang.Class that invoke checkMember
         * access. The stack should look like:
         *
         * someCaller                        [3]
         * java.lang.Class.someReflectionAPI [2]
         * java.lang.Class.checkMemberAccess [1]
         * SecurityManager.checkMemberAccess [0]
         *
         */
        if ((stack.length<4) ||
            (stack[3].getClassLoader() != clazz.getClassLoader())) {
            checkPermission(SecurityConstants.CHECK_MEMBER_ACCESS_PERMISSION);
        }
    }

The clazz here was the anonymous inner class used in Memoize.buildMemoizeFunction() and Memoize.buildSoftReferenceMemoizeFunction() calls and the JVM was bombing at the above permission check when initializing the metaclass of the AIC because of the classloader that was loading the AIC.

So, a potential workaround was to make a change that will influence the class loader that loads the inner classes needed by Memoize() - because everywhere else, the initializing of metaclasses had no issues.

That's what the patch does - by converting the anonymous inner classes into non-anonymous inner classes, so they get loaded before buildMemoizeFunction() and buildSoftReferenceMemoizeFunction() try creating their instances.

I am no classloader expert and I have no way of proving the above on GAE, but that was the reasoning I went by and the change made makes memoize() functions work - I have deployed the patched version on my local copy of groovy webconsole - http://roshandawrani.appspot.com/, where it can be checked out

Show
Roshan Dawrani added a comment - Attaching a patch that has the workaround for making memoize() work on google app engine. The java.lang.RuntimePermission accessDeclaredMembers permission check was failing at the following place in OpenJDK VM: java.lang.SecurityManager if (which != Member.PUBLIC) { Class stack[] = getClassContext(); /* * stack depth of 4 should be the caller of one of the * methods in java.lang. Class that invoke checkMember * access. The stack should look like: * * someCaller [3] * java.lang. Class .someReflectionAPI [2] * java.lang. Class .checkMemberAccess [1] * SecurityManager .checkMemberAccess [0] * */ if ((stack.length<4) || (stack[3].getClassLoader() != clazz.getClassLoader())) { checkPermission(SecurityConstants.CHECK_MEMBER_ACCESS_PERMISSION); } } The clazz here was the anonymous inner class used in Memoize.buildMemoizeFunction() and Memoize.buildSoftReferenceMemoizeFunction() calls and the JVM was bombing at the above permission check when initializing the metaclass of the AIC because of the classloader that was loading the AIC. So, a potential workaround was to make a change that will influence the class loader that loads the inner classes needed by Memoize() - because everywhere else, the initializing of metaclasses had no issues. That's what the patch does - by converting the anonymous inner classes into non-anonymous inner classes, so they get loaded before buildMemoizeFunction() and buildSoftReferenceMemoizeFunction() try creating their instances. I am no classloader expert and I have no way of proving the above on GAE, but that was the reasoning I went by and the change made makes memoize() functions work - I have deployed the patched version on my local copy of groovy webconsole - http://roshandawrani.appspot.com/ , where it can be checked out
Hide
Permalink
Roshan Dawrani added a comment -

Sorry - attached the wrong patch earlier.

Show
Roshan Dawrani added a comment - Sorry - attached the wrong patch earlier.
Hide
Permalink
Roshan Dawrani added a comment -

Why exactly is the issue assigned to Guillaume with no comments on the fix?

Should I take it as that I am not to do any commits now?

If that is the case, someone should have had the courtesy to say so.

Else, it is the rudest thing I saw!

Show
Roshan Dawrani added a comment - Why exactly is the issue assigned to Guillaume with no comments on the fix? Should I take it as that I am not to do any commits now? If that is the case, someone should have had the courtesy to say so. Else, it is the rudest thing I saw!
Hide
Permalink
blackdrag blackdrag added a comment -

sorry Roshan, I did not intend to be rude. You commented your patch in a way that sounded as if you are not sure it will fix the problem, so I assigned it to Guillaume, since he can actually test it. He is on the road, otherwise I would not have done it in JIRA.

If on the other hand you think that will fix the issue for sure, then feel free to commit.

Show
blackdrag blackdrag added a comment - sorry Roshan, I did not intend to be rude. You commented your patch in a way that sounded as if you are not sure it will fix the problem, so I assigned it to Guillaume, since he can actually test it. He is on the road, otherwise I would not have done it in JIRA. If on the other hand you think that will fix the issue for sure, then feel free to commit.
Hide
Permalink
Roshan Dawrani added a comment - - edited

It is something that is absolutely not locally reproducible and absolutely not debuggable on GAE at that deep level in JVM. So, a bit of "workaround" flavor to it! It's not a straight-forward thing to fix anyway. More of troubleshooting.

I even deployed a copy of groovy webconsole under my own google app engine account to demonstrate that it works - not sure why should you think that the it has not fixed the issue. It can be tested at http://roshandawrani.appspot.com/ right away, if anyone wants to. I even tweeted(where it was reported) asking people to test it. Anything more I should have done?

The only help I later need is that when groovy has my fix, the patched version will need to be deployed to official webconsole because I don't have the rights to do that.

Show
Roshan Dawrani added a comment - - edited It is something that is absolutely not locally reproducible and absolutely not debuggable on GAE at that deep level in JVM. So, a bit of "workaround" flavor to it! It's not a straight-forward thing to fix anyway. More of troubleshooting. I even deployed a copy of groovy webconsole under my own google app engine account to demonstrate that it works - not sure why should you think that the it has not fixed the issue. It can be tested at http://roshandawrani.appspot.com/ right away, if anyone wants to. I even tweeted(where it was reported) asking people to test it. Anything more I should have done? The only help I later need is that when groovy has my fix, the patched version will need to be deployed to official webconsole because I don't have the rights to do that.
Hide
Permalink
blackdrag blackdrag added a comment -

ah, ok, then I did misread something in an earlier comment. Fell free to commit and close the issue then

Show
blackdrag blackdrag added a comment - ah, ok, then I did misread something in an earlier comment. Fell free to commit and close the issue then
Hide
Permalink
Roshan Dawrani added a comment -

I will commit the fix (since you don't have any comments on the patch, as such)

But we will close the issue after the groovy web console app is deployed with the patched groovy and it is verified there. I guess that will need to wait for Guillaume - or do you have the rights to do that?

Show
Roshan Dawrani added a comment - I will commit the fix (since you don't have any comments on the patch, as such) But we will close the issue after the groovy web console app is deployed with the patched groovy and it is verified there. I guess that will need to wait for Guillaume - or do you have the rights to do that?
Hide
Permalink
Roshan Dawrani added a comment -

Patch applied. To be verified on webconsole once it has the patched groovy.

Show
Roshan Dawrani added a comment - Patch applied. To be verified on webconsole once it has the patched groovy.
Hide
Permalink
Roshan Dawrani added a comment -

Its verification has been pending for quite some time now.

Can the latest groovy (trunk) be pushed to groovy web console?

Show
Roshan Dawrani added a comment - Its verification has been pending for quite some time now. Can the latest groovy (trunk) be pushed to groovy web console?
Hide
Permalink
Guillaume Laforge added a comment -

Sorry Roshan for that, this has slipped somehow, I'll try and have a look tomorrow.
And ping me on IM if ever I forget again

Show
Guillaume Laforge added a comment - Sorry Roshan for that, this has slipped somehow, I'll try and have a look tomorrow. And ping me on IM if ever I forget again
Hide
Permalink
Guillaume Laforge added a comment -

Groovy Web Console updated!
I had some other changes I had in parallel and had not finished them and didn't want to upgrade till I was fully done, but that's now okay.
And memoize does work fine now.
Well done and thank you!

Show
Guillaume Laforge added a comment - Groovy Web Console updated! I had some other changes I had in parallel and had not finished them and didn't want to upgrade till I was fully done, but that's now okay. And memoize does work fine now. Well done and thank you!

People

  • Assignee:
    Roshan Dawrani
    Reporter:
    Tim Yates
Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved: