Deadlock in groovy.lang.MetaClassImpl under heavy concurrent load

As far as I can tell Object@5f70bea5 is the BEANINFO_CACHE used by the Introspector internally to sync the cache access. And PropertyDescriptor#getReadMethod is synchronized. Looks like this usage of Introspector and PropertyDescriptor can lead to a deadlock. The question is what we can do about this.

One way would be to replace the Introspector completely with... well I don't have anything at hand. The other way would be to pay a big performance penalty and sync in the Introspector, so only one MetaClassImpl can access it. But even then this deadlock could happen in combination with custom other code working with the Introspector. I would use the BEANINFO_CACHE, but it is internal.

Well I really don't have enough knowledge of the source and platform to give any advice
It would be nice to see a fix though

It is happening to us too running under groovy 1.7.8 inside a grails (v1.3.7) app in tomcat 6.0.32, java version 1.6.0_21.

Stacktrace below if it helps.

"integration-lido-scheduler_Worker-2":
at java.beans.Introspector.getPublicDeclaredMethods(Introspector.java:1277)

• waiting to lock <0x00007f88c51224d0> (a java.lang.Object)
  at java.beans.Introspector.internalFindMethod(Introspector.java:1312)
  at java.beans.Introspector.findMethod(Introspector.java:1383)
  at java.beans.PropertyDescriptor.getWriteMethod(PropertyDescriptor.java:246)
• locked <0x00007f88d4804e80> (a java.beans.PropertyDescriptor)
  at groovy.lang.MetaClassImpl.applyPropertyDescriptors(MetaClassImpl.java:2227)
  at groovy.lang.MetaClassImpl.setupProperties(MetaClassImpl.java:1995)
  at groovy.lang.MetaClassImpl.addProperties(MetaClassImpl.java:2950)
  at groovy.lang.MetaClassImpl.initialize(MetaClassImpl.java:2921)
• locked <0x00007f88e64d0948> (a groovy.lang.ExpandoMetaClass)
  at groovy.lang.ExpandoMetaClass.initialize(ExpandoMetaClass.java:463)
• locked <0x00007f88e64d0948> (a groovy.lang.ExpandoMetaClass)
  at org.codehaus.groovy.reflection.ClassInfo.getMetaClassUnderLock(ClassInfo.java:166)
  at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:182)
  at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:210)
  at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:751)
  at groovy.lang.GroovyObjectSupport.<init>(GroovyObjectSupport.java:32)
  at groovy.lang.Closure.<init>(Closure.java:100)

"integration-lido-scheduler_Worker-3":
at java.beans.PropertyDescriptor.getReadMethod(PropertyDescriptor.java:158)

• waiting to lock <0x00007f88d4804e80> (a java.beans.PropertyDescriptor)
  at java.beans.Introspector.processPropertyDescriptors(Introspector.java:683)
  at java.beans.Introspector.getTargetPropertyInfo(Introspector.java:615)
  at java.beans.Introspector.getBeanInfo(Introspector.java:407)
  at java.beans.Introspector.getBeanInfo(Introspector.java:164)
• locked <0x00007f88c51224d0> (a java.lang.Object)
  at groovy.lang.MetaClassImpl$15.run(MetaClassImpl.java:2940)
  at java.security.AccessController.doPrivileged(Native Method)
  at groovy.lang.MetaClassImpl.addProperties(MetaClassImpl.java:2938)
  at groovy.lang.MetaClassImpl.initialize(MetaClassImpl.java:2921)
• locked <0x00007f88e64dad40> (a groovy.lang.ExpandoMetaClass)
  at groovy.lang.ExpandoMetaClass.initialize(ExpandoMetaClass.java:463)
• locked <0x00007f88e64dad40> (a groovy.lang.ExpandoMetaClass)
  at org.codehaus.groovy.reflection.ClassInfo.getMetaClassUnderLock(ClassInfo.java:166)
  at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:182)
  at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:210)
  at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:751)
  at groovy.lang.GroovyObjectSupport.<init>(GroovyObjectSupport.java:32)
  at groovy.lang.Closure.<init>(Closure.java:100)

We got the same problem using Groovy 1.8.3
I attached a simple testcase to reproduce the issue.

GroovyConcurrentDeadLockTestCase.java is a Test Case to rperoduce the problem

This looks like it could be due to this JDK boog:

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7064279

Bad news is that it is only shown as being fixed in JDK 7. I'd suggest testing to see if that does indeed solve this problem, and if so then follow up with the OpenJDK folks to make sure it gets into their releases.

I've backported the fixed java.beans.Introspector from OpenJDK 7 so that it should work with OpenJDK 6 (and maybe 5 but I haven't tried it). I've not done any meaningful testing so y'all will have to let me know whether it works or not. This should be a reasonable temporary workaround for folks who can't move to JDK 7 yet.

https://github.com/jimwhite/jdk7-introspector

This orignal bug is related to http://bugs.sun.com/view_bug.do?bug_id=6963811

Java Release Fixed 7(b102), 6u23(b02) (Bug ID:2194979) , 5.0u36(b31) (Bug ID:2224607)

remove my inefficient test case

Bug #6963811 is the right bug but they did not do the right fix, just scrambled up the code to make it worse. So even though they think they fixed it in 2010, they didn't. That's why bug #7064279 is seen in 2011. So you still need to use JDK 7 or my patch for JDK 6. Has someone made a bug report to Oracle or OpenJDK on this?

For JDK 7+, people affected by this JDK bug should hopefully be able to upgrade to the latest updates.
It'd be interesting to hear feedback from people suffering from this problem once this upgrade is done.

For JDK 6 and below, users should see if they can use Jim's patch.
I don't have a clear understanding (read "I need a lawyer to confirm") on whether we'd really be allowed to include Jim's patch in the Groovy sources, so for now, I prefer avoiding integrating it in Groovy.
Perhaps it'd be nice if the patch were available somehow in a Maven repository as a pre-built module people could put on their classpath?

For Groovy 3, we might investigate implementing a faster introspector for our needs.

For now, I'll close the issue as "won't fix", and we can revisit the topic if the problem is still reproducible for some of you with JDK 7+ or with Jim's patch.

Jochen's filed an issue for JDK 6 to Oracle, and we'll add a comment to the issue with a link once the issue is reviewed and has a reference we can point at.