groovy
  1. groovy
  2. GROOVY-5019

Deadlock in groovy.lang.MetaClassImpl under heavy concurrent load

    Details

    • Type: Bug Bug
    • Status: Closed Closed
    • Priority: Critical Critical
    • Resolution: Won't Fix
    • Affects Version/s: 1.7.10
    • Fix Version/s: None
    • Component/s: Groovysh
    • Labels:
      None
    • Number of attachments :
      1

      Description

      Sometimes we run into deadlocks in groovy shell. Looking at the threaddump below we see that threads pool-1-thread-1103 and pool-1-thread-1104 are deadlocked. I've looked in the issue list and similar issues looks be resolved but we're still experiencing it (for example GROOVY-4050).
      The calls are made to different instance of GroovyShell.

      "pool-1-thread-1104" - Thread t@219188
         java.lang.Thread.State: BLOCKED on java.beans.PropertyDescriptor@3db8a1e8 owned by: pool-1-thread-1103
              at java.beans.PropertyDescriptor.getReadMethod(Unknown Source)
              at java.beans.Introspector.processPropertyDescriptors(Unknown Source)
              at java.beans.Introspector.getTargetPropertyInfo(Unknown Source)
              at java.beans.Introspector.getBeanInfo(Unknown Source)
              at java.beans.Introspector.getBeanInfo(Unknown Source)
              - locked java.lang.Object@5f70bea5
              at groovy.lang.MetaClassImpl$15.run(MetaClassImpl.java:2940)
              at java.security.AccessController.doPrivileged(Native Method)
              at groovy.lang.MetaClassImpl.addProperties(MetaClassImpl.java:2938)
              at groovy.lang.MetaClassImpl.initialize(MetaClassImpl.java:2921)
              - locked groovy.lang.MetaClassImpl@4426073e
              at org.codehaus.groovy.reflection.ClassInfo.getMetaClassUnderLock(ClassInfo.java:166)
              at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:182)
              at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:210)
              at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:751)
              at groovy.lang.GroovyObjectSupport.<init>(GroovyObjectSupport.java:32)
              at groovy.lang.Script.<init>(Script.java:40)
              at groovy.lang.Script.<init>(Script.java:37)
              at Script1.<init>(Script1.groovy)
              at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
              at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
              at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
              at java.lang.reflect.Constructor.newInstance(Unknown Source)
              at java.lang.Class.newInstance0(Unknown Source)
              at java.lang.Class.newInstance(Unknown Source)
              at org.codehaus.groovy.runtime.InvokerHelper.createScript(InvokerHelper.java:408)
              at groovy.lang.GroovyShell.parse(GroovyShell.java:743)
              at groovy.lang.GroovyShell.evaluate(GroovyShell.java:578)
              at groovy.lang.GroovyShell.evaluate(GroovyShell.java:618)
              at groovy.lang.GroovyShell.evaluate(GroovyShell.java:589)
      
      
      -------
      
      "pool-1-thread-1103" - Thread t@219187
         java.lang.Thread.State: BLOCKED on java.lang.Object@5f70bea5 owned by: pool-1-thread-1104
              at java.beans.Introspector.getPublicDeclaredMethods(Unknown Source)
              at java.beans.Introspector.internalFindMethod(Unknown Source)
              at java.beans.Introspector.findMethod(Unknown Source)
              at java.beans.Introspector.findMethod(Unknown Source)
              at java.beans.PropertyDescriptor.getReadMethod(Unknown Source)
              - locked java.beans.PropertyDescriptor@3db8a1e8
              at groovy.lang.MetaClassImpl.applyPropertyDescriptors(MetaClassImpl.java:2215)
              at groovy.lang.MetaClassImpl.setupProperties(MetaClassImpl.java:1995)
              at groovy.lang.MetaClassImpl.addProperties(MetaClassImpl.java:2950)
              at groovy.lang.MetaClassImpl.initialize(MetaClassImpl.java:2921)
              - locked groovy.lang.MetaClassImpl@71ea1b61
              at org.codehaus.groovy.reflection.ClassInfo.getMetaClassUnderLock(ClassInfo.java:166)
              at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:182)
              at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:210)
              at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:751)
              at groovy.lang.GroovyObjectSupport.<init>(GroovyObjectSupport.java:32)
              at groovy.lang.Script.<init>(Script.java:40)
              at groovy.lang.Script.<init>(Script.java:37)
              at Script1.<init>(Script1.groovy)
              at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
              at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
              at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
              at java.lang.reflect.Constructor.newInstance(Unknown Source)
              at java.lang.Class.newInstance0(Unknown Source)
              at java.lang.Class.newInstance(Unknown Source)
              at org.codehaus.groovy.runtime.InvokerHelper.createScript(InvokerHelper.java:408)
              at groovy.lang.GroovyShell.parse(GroovyShell.java:743)
              at groovy.lang.GroovyShell.evaluate(GroovyShell.java:578)
              at groovy.lang.GroovyShell.evaluate(GroovyShell.java:618)
              at groovy.lang.GroovyShell.evaluate(GroovyShell.java:589)
      

        Activity

        Hide
        blackdrag blackdrag added a comment -

        As far as I can tell Object@5f70bea5 is the BEANINFO_CACHE used by the Introspector internally to sync the cache access. And PropertyDescriptor#getReadMethod is synchronized. Looks like this usage of Introspector and PropertyDescriptor can lead to a deadlock. The question is what we can do about this.

        One way would be to replace the Introspector completely with... well I don't have anything at hand. The other way would be to pay a big performance penalty and sync in the Introspector, so only one MetaClassImpl can access it. But even then this deadlock could happen in combination with custom other code working with the Introspector. I would use the BEANINFO_CACHE, but it is internal.

        Show
        blackdrag blackdrag added a comment - As far as I can tell Object@5f70bea5 is the BEANINFO_CACHE used by the Introspector internally to sync the cache access. And PropertyDescriptor#getReadMethod is synchronized. Looks like this usage of Introspector and PropertyDescriptor can lead to a deadlock. The question is what we can do about this. One way would be to replace the Introspector completely with... well I don't have anything at hand. The other way would be to pay a big performance penalty and sync in the Introspector, so only one MetaClassImpl can access it. But even then this deadlock could happen in combination with custom other code working with the Introspector. I would use the BEANINFO_CACHE, but it is internal.
        Hide
        Peter Liljenberg added a comment -

        Well I really don't have enough knowledge of the source and platform to give any advice
        It would be nice to see a fix though

        Show
        Peter Liljenberg added a comment - Well I really don't have enough knowledge of the source and platform to give any advice It would be nice to see a fix though
        Hide
        Andy Shen added a comment -

        It is happening to us too running under groovy 1.7.8 inside a grails (v1.3.7) app in tomcat 6.0.32, java version 1.6.0_21.

        Stacktrace below if it helps.

        "integration-lido-scheduler_Worker-2":
        at java.beans.Introspector.getPublicDeclaredMethods(Introspector.java:1277)

        • waiting to lock <0x00007f88c51224d0> (a java.lang.Object)
          at java.beans.Introspector.internalFindMethod(Introspector.java:1312)
          at java.beans.Introspector.findMethod(Introspector.java:1383)
          at java.beans.PropertyDescriptor.getWriteMethod(PropertyDescriptor.java:246)
        • locked <0x00007f88d4804e80> (a java.beans.PropertyDescriptor)
          at groovy.lang.MetaClassImpl.applyPropertyDescriptors(MetaClassImpl.java:2227)
          at groovy.lang.MetaClassImpl.setupProperties(MetaClassImpl.java:1995)
          at groovy.lang.MetaClassImpl.addProperties(MetaClassImpl.java:2950)
          at groovy.lang.MetaClassImpl.initialize(MetaClassImpl.java:2921)
        • locked <0x00007f88e64d0948> (a groovy.lang.ExpandoMetaClass)
          at groovy.lang.ExpandoMetaClass.initialize(ExpandoMetaClass.java:463)
        • locked <0x00007f88e64d0948> (a groovy.lang.ExpandoMetaClass)
          at org.codehaus.groovy.reflection.ClassInfo.getMetaClassUnderLock(ClassInfo.java:166)
          at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:182)
          at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:210)
          at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:751)
          at groovy.lang.GroovyObjectSupport.<init>(GroovyObjectSupport.java:32)
          at groovy.lang.Closure.<init>(Closure.java:100)

        "integration-lido-scheduler_Worker-3":
        at java.beans.PropertyDescriptor.getReadMethod(PropertyDescriptor.java:158)

        • waiting to lock <0x00007f88d4804e80> (a java.beans.PropertyDescriptor)
          at java.beans.Introspector.processPropertyDescriptors(Introspector.java:683)
          at java.beans.Introspector.getTargetPropertyInfo(Introspector.java:615)
          at java.beans.Introspector.getBeanInfo(Introspector.java:407)
          at java.beans.Introspector.getBeanInfo(Introspector.java:164)
        • locked <0x00007f88c51224d0> (a java.lang.Object)
          at groovy.lang.MetaClassImpl$15.run(MetaClassImpl.java:2940)
          at java.security.AccessController.doPrivileged(Native Method)
          at groovy.lang.MetaClassImpl.addProperties(MetaClassImpl.java:2938)
          at groovy.lang.MetaClassImpl.initialize(MetaClassImpl.java:2921)
        • locked <0x00007f88e64dad40> (a groovy.lang.ExpandoMetaClass)
          at groovy.lang.ExpandoMetaClass.initialize(ExpandoMetaClass.java:463)
        • locked <0x00007f88e64dad40> (a groovy.lang.ExpandoMetaClass)
          at org.codehaus.groovy.reflection.ClassInfo.getMetaClassUnderLock(ClassInfo.java:166)
          at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:182)
          at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:210)
          at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:751)
          at groovy.lang.GroovyObjectSupport.<init>(GroovyObjectSupport.java:32)
          at groovy.lang.Closure.<init>(Closure.java:100)
        Show
        Andy Shen added a comment - It is happening to us too running under groovy 1.7.8 inside a grails (v1.3.7) app in tomcat 6.0.32, java version 1.6.0_21. Stacktrace below if it helps. "integration-lido-scheduler_Worker-2": at java.beans.Introspector.getPublicDeclaredMethods(Introspector.java:1277) waiting to lock <0x00007f88c51224d0> (a java.lang.Object) at java.beans.Introspector.internalFindMethod(Introspector.java:1312) at java.beans.Introspector.findMethod(Introspector.java:1383) at java.beans.PropertyDescriptor.getWriteMethod(PropertyDescriptor.java:246) locked <0x00007f88d4804e80> (a java.beans.PropertyDescriptor) at groovy.lang.MetaClassImpl.applyPropertyDescriptors(MetaClassImpl.java:2227) at groovy.lang.MetaClassImpl.setupProperties(MetaClassImpl.java:1995) at groovy.lang.MetaClassImpl.addProperties(MetaClassImpl.java:2950) at groovy.lang.MetaClassImpl.initialize(MetaClassImpl.java:2921) locked <0x00007f88e64d0948> (a groovy.lang.ExpandoMetaClass) at groovy.lang.ExpandoMetaClass.initialize(ExpandoMetaClass.java:463) locked <0x00007f88e64d0948> (a groovy.lang.ExpandoMetaClass) at org.codehaus.groovy.reflection.ClassInfo.getMetaClassUnderLock(ClassInfo.java:166) at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:182) at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:210) at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:751) at groovy.lang.GroovyObjectSupport.<init>(GroovyObjectSupport.java:32) at groovy.lang.Closure.<init>(Closure.java:100) "integration-lido-scheduler_Worker-3": at java.beans.PropertyDescriptor.getReadMethod(PropertyDescriptor.java:158) waiting to lock <0x00007f88d4804e80> (a java.beans.PropertyDescriptor) at java.beans.Introspector.processPropertyDescriptors(Introspector.java:683) at java.beans.Introspector.getTargetPropertyInfo(Introspector.java:615) at java.beans.Introspector.getBeanInfo(Introspector.java:407) at java.beans.Introspector.getBeanInfo(Introspector.java:164) locked <0x00007f88c51224d0> (a java.lang.Object) at groovy.lang.MetaClassImpl$15.run(MetaClassImpl.java:2940) at java.security.AccessController.doPrivileged(Native Method) at groovy.lang.MetaClassImpl.addProperties(MetaClassImpl.java:2938) at groovy.lang.MetaClassImpl.initialize(MetaClassImpl.java:2921) locked <0x00007f88e64dad40> (a groovy.lang.ExpandoMetaClass) at groovy.lang.ExpandoMetaClass.initialize(ExpandoMetaClass.java:463) locked <0x00007f88e64dad40> (a groovy.lang.ExpandoMetaClass) at org.codehaus.groovy.reflection.ClassInfo.getMetaClassUnderLock(ClassInfo.java:166) at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:182) at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:210) at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:751) at groovy.lang.GroovyObjectSupport.<init>(GroovyObjectSupport.java:32) at groovy.lang.Closure.<init>(Closure.java:100)
        Hide
        Sebastien Arod added a comment -

        We got the same problem using Groovy 1.8.3
        I attached a simple testcase to reproduce the issue.

        Show
        Sebastien Arod added a comment - We got the same problem using Groovy 1.8.3 I attached a simple testcase to reproduce the issue.
        Hide
        Sebastien Arod added a comment - - edited

        GroovyConcurrentDeadLockTestCase.java is a Test Case to rperoduce the problem

        Show
        Sebastien Arod added a comment - - edited GroovyConcurrentDeadLockTestCase.java is a Test Case to rperoduce the problem
        Hide
        Jim White added a comment -

        This looks like it could be due to this JDK boog:

        http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7064279

        Bad news is that it is only shown as being fixed in JDK 7. I'd suggest testing to see if that does indeed solve this problem, and if so then follow up with the OpenJDK folks to make sure it gets into their releases.

        Show
        Jim White added a comment - This looks like it could be due to this JDK boog: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7064279 Bad news is that it is only shown as being fixed in JDK 7. I'd suggest testing to see if that does indeed solve this problem, and if so then follow up with the OpenJDK folks to make sure it gets into their releases.
        Hide
        Jim White added a comment -

        I've backported the fixed java.beans.Introspector from OpenJDK 7 so that it should work with OpenJDK 6 (and maybe 5 but I haven't tried it). I've not done any meaningful testing so y'all will have to let me know whether it works or not. This should be a reasonable temporary workaround for folks who can't move to JDK 7 yet.

        https://github.com/jimwhite/jdk7-introspector

        Show
        Jim White added a comment - I've backported the fixed java.beans.Introspector from OpenJDK 7 so that it should work with OpenJDK 6 (and maybe 5 but I haven't tried it). I've not done any meaningful testing so y'all will have to let me know whether it works or not. This should be a reasonable temporary workaround for folks who can't move to JDK 7 yet. https://github.com/jimwhite/jdk7-introspector
        Hide
        Vincent Lagorce added a comment -

        This orignal bug is related to http://bugs.sun.com/view_bug.do?bug_id=6963811

        Java Release Fixed 7(b102), 6u23(b02) (Bug ID:2194979) , 5.0u36(b31) (Bug ID:2224607)

        Show
        Vincent Lagorce added a comment - This orignal bug is related to http://bugs.sun.com/view_bug.do?bug_id=6963811 Java Release Fixed 7(b102), 6u23(b02) (Bug ID:2194979) , 5.0u36(b31) (Bug ID:2224607)
        Hide
        Vincent Lagorce added a comment - - edited

        remove my inefficient test case

        Show
        Vincent Lagorce added a comment - - edited remove my inefficient test case
        Hide
        Jim White added a comment -

        Bug #6963811 is the right bug but they did not do the right fix, just scrambled up the code to make it worse. So even though they think they fixed it in 2010, they didn't. That's why bug #7064279 is seen in 2011. So you still need to use JDK 7 or my patch for JDK 6. Has someone made a bug report to Oracle or OpenJDK on this?

        Show
        Jim White added a comment - Bug #6963811 is the right bug but they did not do the right fix, just scrambled up the code to make it worse. So even though they think they fixed it in 2010, they didn't. That's why bug #7064279 is seen in 2011. So you still need to use JDK 7 or my patch for JDK 6. Has someone made a bug report to Oracle or OpenJDK on this?
        Hide
        Guillaume Laforge added a comment -

        For JDK 7+, people affected by this JDK bug should hopefully be able to upgrade to the latest updates.
        It'd be interesting to hear feedback from people suffering from this problem once this upgrade is done.

        For JDK 6 and below, users should see if they can use Jim's patch.
        I don't have a clear understanding (read "I need a lawyer to confirm") on whether we'd really be allowed to include Jim's patch in the Groovy sources, so for now, I prefer avoiding integrating it in Groovy.
        Perhaps it'd be nice if the patch were available somehow in a Maven repository as a pre-built module people could put on their classpath?

        For Groovy 3, we might investigate implementing a faster introspector for our needs.

        For now, I'll close the issue as "won't fix", and we can revisit the topic if the problem is still reproducible for some of you with JDK 7+ or with Jim's patch.

        Jochen's filed an issue for JDK 6 to Oracle, and we'll add a comment to the issue with a link once the issue is reviewed and has a reference we can point at.

        Show
        Guillaume Laforge added a comment - For JDK 7+, people affected by this JDK bug should hopefully be able to upgrade to the latest updates. It'd be interesting to hear feedback from people suffering from this problem once this upgrade is done. For JDK 6 and below, users should see if they can use Jim's patch. I don't have a clear understanding (read "I need a lawyer to confirm") on whether we'd really be allowed to include Jim's patch in the Groovy sources, so for now, I prefer avoiding integrating it in Groovy. Perhaps it'd be nice if the patch were available somehow in a Maven repository as a pre-built module people could put on their classpath? For Groovy 3, we might investigate implementing a faster introspector for our needs. For now, I'll close the issue as "won't fix", and we can revisit the topic if the problem is still reproducible for some of you with JDK 7+ or with Jim's patch. Jochen's filed an issue for JDK 6 to Oracle, and we'll add a comment to the issue with a link once the issue is reviewed and has a reference we can point at.

          People

          • Assignee:
            Unassigned
            Reporter:
            Peter Liljenberg
          • Votes:
            1 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: