groovy

deadlock in EMC.performOperationOnMetaClass

Details

  • Type: Sub-task Sub-task
  • Status: Open Open
  • Priority: Critical Critical
  • Resolution: Unresolved
  • Affects Version/s: 1.8.6
  • Fix Version/s: None
  • Component/s: None
  • Labels:
    None
  • Environment:
    1.8.6 the bug did not exist in 1.7.8
  • Number of attachments :
    1

Description

quoting from mail

in a customer project that just upgraded to grails 2.1 we hit a
deadlock at EMC.performOperationOnMetaClass line 809.

It is most likely to have been introduced with commit
cdc39843d361a1d6c519432913c5d30f20782e68
by Lari Hotari, trying to fix 3557.

Looking at
protected synchronized void performOperationOnMetaClass(Callable c) {
try {
writeLock.lock();
...

synchronized plus a lock looks indeed strange... the GROOVY-3557 removed all synchronized in that class, except the one on this method... MetaClass#initialize is the only other place I see this synchronized still being available... to keep compatibility, this is sadly required. Now... the question is why it does deadlock, and if it can be prevented

makes me think that a mixture of locks and synchronized
is not a good thing to have.

in general more than one lock is dangerous.

I would suggest that when it comes to concurrency issues in the core,
we should include Vaclav in the design.

see also:
http://grails.1312388.n4.nabble.com/Thread-Blocking-at-Groovy-ExpandoMetaclass-method-in-a-Grails-App-td4572990.html
https://jira.codehaus.org/browse/GROOVY-5249
https://jira.codehaus.org/browse/GROOVY-3557

any opinions or should I file an issue right away?

We need an issue showing the traces of the deadlock... right now I have a hard imagining how that synchronized could cause the deadlock. So please fill one

bye blackdrag

Providing a trace is difficult since no Exception is thrown.
Also, the deadlock appearance is not deterministic.
But when it comes, it invariantly stops at this method.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
blackdrag blackdrag added a comment -

You can for example use JConsole to get the traces. Another solution is to use the ThreadMXBean like in for example: 

ThreadMXBean mxBean = ManagementFactory.getThreadMXBean();
                        long[] ids = mxBean.findMonitorDeadlockedThreads();
                        if (ids==null || ids.length==0) continue;

Once ids is not null and has a length bigger zero, you found deadlocking threads and can print their trace

Show
blackdrag blackdrag added a comment - You can for example use JConsole to get the traces. Another solution is to use the ThreadMXBean like in for example: ThreadMXBean mxBean = ManagementFactory.getThreadMXBean(); long [] ids = mxBean.findMonitorDeadlockedThreads(); if (ids== null || ids.length==0) continue ; Once ids is not null and has a length bigger zero, you found deadlocking threads and can print their trace
Hide
Permalink
Lari Hotari added a comment -

In unixes (Linux,Macosx,etc) you can simply send the SIGQUIT (3) signal to the java process and it will output the threaddumps to System.out .

For example:
kill -3 [processid]

Show
Lari Hotari added a comment - In unixes (Linux,Macosx,etc) you can simply send the SIGQUIT (3) signal to the java process and it will output the threaddumps to System.out . For example: kill -3 [processid]
Hide
Permalink
Dierk Koenig added a comment -

deadlock in metaclass

Show
Dierk Koenig added a comment - deadlock in metaclass
Hide
Permalink
Dierk Koenig added a comment -

I attach the stack traces as requested.
On the application side it seems to be related to:
class A {
public myPublic()

{ myPrivate() }

private myPrivate() {}
}
class B extends A {
public myPublic()

{ super() }

}

Making myPrivate a public method makes this specific reproducible deadlock disappear.
(we now have other ones that appear only occasionally)

I'd like to see all locking being removed from the metaclass and would even strive for a design where the MC is immutable.

Show
Dierk Koenig added a comment - I attach the stack traces as requested. On the application side it seems to be related to: class A { public myPublic() { myPrivate() } private myPrivate() {} } class B extends A { public myPublic() { super() } } Making myPrivate a public method makes this specific reproducible deadlock disappear. (we now have other ones that appear only occasionally) I'd like to see all locking being removed from the metaclass and would even strive for a design where the MC is immutable.
Hide
Permalink
blackdrag blackdrag added a comment -

Dierk, my idea of MOP2 is that the meta class will not really be mutable, yes.

Show
blackdrag blackdrag added a comment - Dierk, my idea of MOP2 is that the meta class will not really be mutable, yes.
Hide
Permalink
blackdrag blackdrag added a comment -

that making the private method public makes the deadlock go away is an extremely strange fact

Show
blackdrag blackdrag added a comment - that making the private method public makes the deadlock go away is an extremely strange fact
Hide
Permalink
Dierk Koenig added a comment -

glad to hear!

Show
Dierk Koenig added a comment - glad to hear!
Hide
Permalink
blackdrag blackdrag added a comment -

looking at the traces I seem not to get the information I was hoping for... Thread[main,5,main] surely is the blocked one, but by whom?

Show
blackdrag blackdrag added a comment - looking at the traces I seem not to get the information I was hoping for... Thread [main,5,main] surely is the blocked one, but by whom?
Hide
Permalink
blackdrag blackdrag added a comment -

After looking at it for a longer time now, I can still not see any thread that should be blocking this main,5,main thread

Show
blackdrag blackdrag added a comment - After looking at it for a longer time now, I can still not see any thread that should be blocking this main,5,main thread

People

  • Assignee:
    Unassigned
    Reporter:
    Dierk Koenig
Vote (0)
Watch (3)

Dates

  • Created:
    Updated: