jira.codehaus.org

  • Log In Access more options
    • Online Help
    • Keyboard Shortcuts
    • About JIRA
    • JIRA Credits
    • What?s New
  • Dashboards Access more options (Alt+d)
  • Projects Access more options (Alt+p)
  • Issues Access more options (Alt+i)
  • groovy
  • GROOVY-5019

Deadlock in groovy.lang.MetaClassImpl under heavy concurrent load

  • Log In
  • Views
    • XML
    • Word
    • Printable

Details

  • Type: Bug Bug
  • Status: Open Open
  • Priority: Critical Critical
  • Resolution: Unresolved
  • Affects Version/s: 1.7.10
  • Fix Version/s: None
  • Component/s: Groovysh
  • Labels:
    None

Description

Sometimes we run into deadlocks in groovy shell. Looking at the threaddump below we see that threads pool-1-thread-1103 and pool-1-thread-1104 are deadlocked. I've looked in the issue list and similar issues looks be resolved but we're still experiencing it (for example GROOVY-4050).
The calls are made to different instance of GroovyShell.

"pool-1-thread-1104" - Thread t@219188
java.lang.Thread.State: BLOCKED on java.beans.PropertyDescriptor@3db8a1e8 owned by: pool-1-thread-1103
at java.beans.PropertyDescriptor.getReadMethod(Unknown Source)
at java.beans.Introspector.processPropertyDescriptors(Unknown Source)
at java.beans.Introspector.getTargetPropertyInfo(Unknown Source)
at java.beans.Introspector.getBeanInfo(Unknown Source)
at java.beans.Introspector.getBeanInfo(Unknown Source)

  • locked java.lang.Object@5f70bea5
    at groovy.lang.MetaClassImpl$15.run(MetaClassImpl.java:2940)
    at java.security.AccessController.doPrivileged(Native Method)
    at groovy.lang.MetaClassImpl.addProperties(MetaClassImpl.java:2938)
    at groovy.lang.MetaClassImpl.initialize(MetaClassImpl.java:2921)
  • locked groovy.lang.MetaClassImpl@4426073e
    at org.codehaus.groovy.reflection.ClassInfo.getMetaClassUnderLock(ClassInfo.java:166)
    at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:182)
    at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:210)
    at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:751)
    at groovy.lang.GroovyObjectSupport.<init>(GroovyObjectSupport.java:32)
    at groovy.lang.Script.<init>(Script.java:40)
    at groovy.lang.Script.<init>(Script.java:37)
    at Script1.<init>(Script1.groovy)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
    at java.lang.reflect.Constructor.newInstance(Unknown Source)
    at java.lang.Class.newInstance0(Unknown Source)
    at java.lang.Class.newInstance(Unknown Source)
    at org.codehaus.groovy.runtime.InvokerHelper.createScript(InvokerHelper.java:408)
    at groovy.lang.GroovyShell.parse(GroovyShell.java:743)
    at groovy.lang.GroovyShell.evaluate(GroovyShell.java:578)
    at groovy.lang.GroovyShell.evaluate(GroovyShell.java:618)
    at groovy.lang.GroovyShell.evaluate(GroovyShell.java:589)

-------

"pool-1-thread-1103" - Thread t@219187
java.lang.Thread.State: BLOCKED on java.lang.Object@5f70bea5 owned by: pool-1-thread-1104
at java.beans.Introspector.getPublicDeclaredMethods(Unknown Source)
at java.beans.Introspector.internalFindMethod(Unknown Source)
at java.beans.Introspector.findMethod(Unknown Source)
at java.beans.Introspector.findMethod(Unknown Source)
at java.beans.PropertyDescriptor.getReadMethod(Unknown Source)

  • locked java.beans.PropertyDescriptor@3db8a1e8
    at groovy.lang.MetaClassImpl.applyPropertyDescriptors(MetaClassImpl.java:2215)
    at groovy.lang.MetaClassImpl.setupProperties(MetaClassImpl.java:1995)
    at groovy.lang.MetaClassImpl.addProperties(MetaClassImpl.java:2950)
    at groovy.lang.MetaClassImpl.initialize(MetaClassImpl.java:2921)
  • locked groovy.lang.MetaClassImpl@71ea1b61
    at org.codehaus.groovy.reflection.ClassInfo.getMetaClassUnderLock(ClassInfo.java:166)
    at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:182)
    at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:210)
    at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:751)
    at groovy.lang.GroovyObjectSupport.<init>(GroovyObjectSupport.java:32)
    at groovy.lang.Script.<init>(Script.java:40)
    at groovy.lang.Script.<init>(Script.java:37)
    at Script1.<init>(Script1.groovy)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
    at java.lang.reflect.Constructor.newInstance(Unknown Source)
    at java.lang.Class.newInstance0(Unknown Source)
    at java.lang.Class.newInstance(Unknown Source)
    at org.codehaus.groovy.runtime.InvokerHelper.createScript(InvokerHelper.java:408)
    at groovy.lang.GroovyShell.parse(GroovyShell.java:743)
    at groovy.lang.GroovyShell.evaluate(GroovyShell.java:578)
    at groovy.lang.GroovyShell.evaluate(GroovyShell.java:618)
    at groovy.lang.GroovyShell.evaluate(GroovyShell.java:589)
  • Options
    • Sort By Name
    • Sort By Date
    • Ascending
    • Descending
    • Download All

Attachments

  1. Java Source File
    GroovyConcurrentDeadLockTestCase.java
    28/Oct/11 4:24 AM
    2 kB
    Sebastien Arod

Activity

Ascending order - Click to sort in descending order
  • All
  • Comments
  • Work Log
  • History
  • Activity
Hide
Permalink
blackdrag blackdrag added a comment - 14/Sep/11 9:23 AM

As far as I can tell Object@5f70bea5 is the BEANINFO_CACHE used by the Introspector internally to sync the cache access. And PropertyDescriptor#getReadMethod is synchronized. Looks like this usage of Introspector and PropertyDescriptor can lead to a deadlock. The question is what we can do about this.

One way would be to replace the Introspector completely with... well I don't have anything at hand. The other way would be to pay a big performance penalty and sync in the Introspector, so only one MetaClassImpl can access it. But even then this deadlock could happen in combination with custom other code working with the Introspector. I would use the BEANINFO_CACHE, but it is internal.

Show
blackdrag blackdrag added a comment - 14/Sep/11 9:23 AM As far as I can tell Object@5f70bea5 is the BEANINFO_CACHE used by the Introspector internally to sync the cache access. And PropertyDescriptor#getReadMethod is synchronized. Looks like this usage of Introspector and PropertyDescriptor can lead to a deadlock. The question is what we can do about this. One way would be to replace the Introspector completely with... well I don't have anything at hand. The other way would be to pay a big performance penalty and sync in the Introspector, so only one MetaClassImpl can access it. But even then this deadlock could happen in combination with custom other code working with the Introspector. I would use the BEANINFO_CACHE, but it is internal.
Hide
Permalink
Peter Liljenberg added a comment - 22/Sep/11 9:25 AM

Well I really don't have enough knowledge of the source and platform to give any advice
It would be nice to see a fix though

Show
Peter Liljenberg added a comment - 22/Sep/11 9:25 AM Well I really don't have enough knowledge of the source and platform to give any advice It would be nice to see a fix though
Hide
Permalink
Andy Shen added a comment - 24/Oct/11 5:52 PM

It is happening to us too running under groovy 1.7.8 inside a grails (v1.3.7) app in tomcat 6.0.32, java version 1.6.0_21.

Stacktrace below if it helps.

"integration-lido-scheduler_Worker-2":
at java.beans.Introspector.getPublicDeclaredMethods(Introspector.java:1277)

  • waiting to lock <0x00007f88c51224d0> (a java.lang.Object)
    at java.beans.Introspector.internalFindMethod(Introspector.java:1312)
    at java.beans.Introspector.findMethod(Introspector.java:1383)
    at java.beans.PropertyDescriptor.getWriteMethod(PropertyDescriptor.java:246)
  • locked <0x00007f88d4804e80> (a java.beans.PropertyDescriptor)
    at groovy.lang.MetaClassImpl.applyPropertyDescriptors(MetaClassImpl.java:2227)
    at groovy.lang.MetaClassImpl.setupProperties(MetaClassImpl.java:1995)
    at groovy.lang.MetaClassImpl.addProperties(MetaClassImpl.java:2950)
    at groovy.lang.MetaClassImpl.initialize(MetaClassImpl.java:2921)
  • locked <0x00007f88e64d0948> (a groovy.lang.ExpandoMetaClass)
    at groovy.lang.ExpandoMetaClass.initialize(ExpandoMetaClass.java:463)
  • locked <0x00007f88e64d0948> (a groovy.lang.ExpandoMetaClass)
    at org.codehaus.groovy.reflection.ClassInfo.getMetaClassUnderLock(ClassInfo.java:166)
    at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:182)
    at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:210)
    at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:751)
    at groovy.lang.GroovyObjectSupport.<init>(GroovyObjectSupport.java:32)
    at groovy.lang.Closure.<init>(Closure.java:100)

"integration-lido-scheduler_Worker-3":
at java.beans.PropertyDescriptor.getReadMethod(PropertyDescriptor.java:158)

  • waiting to lock <0x00007f88d4804e80> (a java.beans.PropertyDescriptor)
    at java.beans.Introspector.processPropertyDescriptors(Introspector.java:683)
    at java.beans.Introspector.getTargetPropertyInfo(Introspector.java:615)
    at java.beans.Introspector.getBeanInfo(Introspector.java:407)
    at java.beans.Introspector.getBeanInfo(Introspector.java:164)
  • locked <0x00007f88c51224d0> (a java.lang.Object)
    at groovy.lang.MetaClassImpl$15.run(MetaClassImpl.java:2940)
    at java.security.AccessController.doPrivileged(Native Method)
    at groovy.lang.MetaClassImpl.addProperties(MetaClassImpl.java:2938)
    at groovy.lang.MetaClassImpl.initialize(MetaClassImpl.java:2921)
  • locked <0x00007f88e64dad40> (a groovy.lang.ExpandoMetaClass)
    at groovy.lang.ExpandoMetaClass.initialize(ExpandoMetaClass.java:463)
  • locked <0x00007f88e64dad40> (a groovy.lang.ExpandoMetaClass)
    at org.codehaus.groovy.reflection.ClassInfo.getMetaClassUnderLock(ClassInfo.java:166)
    at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:182)
    at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:210)
    at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:751)
    at groovy.lang.GroovyObjectSupport.<init>(GroovyObjectSupport.java:32)
    at groovy.lang.Closure.<init>(Closure.java:100)
Show
Andy Shen added a comment - 24/Oct/11 5:52 PM It is happening to us too running under groovy 1.7.8 inside a grails (v1.3.7) app in tomcat 6.0.32, java version 1.6.0_21. Stacktrace below if it helps. "integration-lido-scheduler_Worker-2": at java.beans.Introspector.getPublicDeclaredMethods(Introspector.java:1277)
  • waiting to lock <0x00007f88c51224d0> (a java.lang.Object) at java.beans.Introspector.internalFindMethod(Introspector.java:1312) at java.beans.Introspector.findMethod(Introspector.java:1383) at java.beans.PropertyDescriptor.getWriteMethod(PropertyDescriptor.java:246)
  • locked <0x00007f88d4804e80> (a java.beans.PropertyDescriptor) at groovy.lang.MetaClassImpl.applyPropertyDescriptors(MetaClassImpl.java:2227) at groovy.lang.MetaClassImpl.setupProperties(MetaClassImpl.java:1995) at groovy.lang.MetaClassImpl.addProperties(MetaClassImpl.java:2950) at groovy.lang.MetaClassImpl.initialize(MetaClassImpl.java:2921)
  • locked <0x00007f88e64d0948> (a groovy.lang.ExpandoMetaClass) at groovy.lang.ExpandoMetaClass.initialize(ExpandoMetaClass.java:463)
  • locked <0x00007f88e64d0948> (a groovy.lang.ExpandoMetaClass) at org.codehaus.groovy.reflection.ClassInfo.getMetaClassUnderLock(ClassInfo.java:166) at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:182) at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:210) at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:751) at groovy.lang.GroovyObjectSupport.<init>(GroovyObjectSupport.java:32) at groovy.lang.Closure.<init>(Closure.java:100)
"integration-lido-scheduler_Worker-3": at java.beans.PropertyDescriptor.getReadMethod(PropertyDescriptor.java:158)
  • waiting to lock <0x00007f88d4804e80> (a java.beans.PropertyDescriptor) at java.beans.Introspector.processPropertyDescriptors(Introspector.java:683) at java.beans.Introspector.getTargetPropertyInfo(Introspector.java:615) at java.beans.Introspector.getBeanInfo(Introspector.java:407) at java.beans.Introspector.getBeanInfo(Introspector.java:164)
  • locked <0x00007f88c51224d0> (a java.lang.Object) at groovy.lang.MetaClassImpl$15.run(MetaClassImpl.java:2940) at java.security.AccessController.doPrivileged(Native Method) at groovy.lang.MetaClassImpl.addProperties(MetaClassImpl.java:2938) at groovy.lang.MetaClassImpl.initialize(MetaClassImpl.java:2921)
  • locked <0x00007f88e64dad40> (a groovy.lang.ExpandoMetaClass) at groovy.lang.ExpandoMetaClass.initialize(ExpandoMetaClass.java:463)
  • locked <0x00007f88e64dad40> (a groovy.lang.ExpandoMetaClass) at org.codehaus.groovy.reflection.ClassInfo.getMetaClassUnderLock(ClassInfo.java:166) at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:182) at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:210) at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:751) at groovy.lang.GroovyObjectSupport.<init>(GroovyObjectSupport.java:32) at groovy.lang.Closure.<init>(Closure.java:100)
Hide
Permalink
Sebastien Arod added a comment - 28/Oct/11 3:28 AM

We got the same problem using Groovy 1.8.3
I attached a simple testcase to reproduce the issue.

Show
Sebastien Arod added a comment - 28/Oct/11 3:28 AM We got the same problem using Groovy 1.8.3 I attached a simple testcase to reproduce the issue.
Hide
Permalink
Sebastien Arod added a comment - 28/Oct/11 3:29 AM - edited

GroovyConcurrentDeadLockTestCase.java is a Test Case to rperoduce the problem

Show
Sebastien Arod added a comment - 28/Oct/11 3:29 AM - edited GroovyConcurrentDeadLockTestCase.java is a Test Case to rperoduce the problem
Hide
Permalink
Jim White added a comment - 22/May/12 8:58 AM

This looks like it could be due to this JDK boog:

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7064279

Bad news is that it is only shown as being fixed in JDK 7. I'd suggest testing to see if that does indeed solve this problem, and if so then follow up with the OpenJDK folks to make sure it gets into their releases.

Show
Jim White added a comment - 22/May/12 8:58 AM This looks like it could be due to this JDK boog: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7064279 Bad news is that it is only shown as being fixed in JDK 7. I'd suggest testing to see if that does indeed solve this problem, and if so then follow up with the OpenJDK folks to make sure it gets into their releases.

People

  • Assignee:
    Unassigned
    Reporter:
    Peter Liljenberg
Vote (1)
Watch (5)

Dates

  • Created:
    14/Sep/11 8:45 AM
    Updated:
    Yesterday 8:58 AM
  • Atlassian JIRA (v5.0.4#731-sha1:3aa7374)
  • Report a problem
  • Powered by a free Atlassian JIRA open source license for Codehaus. Try JIRA - bug tracking software for your team.