groovy

Deadlock when GroovyClassLoader is used in multi-threaded environment

Details

  • Type: Bug Bug
  • Status: Open Open
  • Priority: Blocker Blocker
  • Resolution: Unresolved
  • Affects Version/s: 1.7.9
  • Fix Version/s: None
  • Component/s: None
  • Environment:
    Multiple OS: development server: Ubuntu 32-bit, desktop machine: Windows 7 64-bit, Java 1.6 Update 21
  • Testcase included:
    yes
  • Number of attachments :
    5

Description

When multiple threads uses GroovyClassLoader to get Groovy classes or just work with them (instantiation, etc.) and an other(s) threads change the sources (followed by clearCache call), then deadlock can happen related to synchronizations on InnerLoader and HashMap (GroovyClassLoader.sourceCache)

I attached simple test scenation when 3 threads load Groovy classes and instantiate them and 3 other threads replace sources and call GroocyClassLoader.clearCache(). There is a synchronization on writeToFile to be ensure that the same file is not being written at the same time but this synchronization does not take part in a deadlock.

Please just run the test and wait a moment. Deadlock happens usually in 1 second at this test scenario. I hope this test will be also useful in Your future development as a standard test case.

  2. Java Source File
    GroovyEngineTest.java
    19/Mar/11 5:38 PM
    5 kB
    Szymon Kuklewicz
  3. Java Source File
    GroovyEngineTest.java
    19/Mar/11 7:51 AM
    6 kB
    Szymon Kuklewicz
  4. Text File
    ReentrantReadWriteLock_for_the_cache_cleaning.patch
    21/Mar/11 3:08 PM
    6 kB
    Guillaume Laforge
  5. Text File
    temporary_hack_for_deadlock_in_gcl.patch
    23/Mar/11 9:07 AM
    0.7 kB
    Guillaume Laforge

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
blackdrag blackdrag added a comment -

The system I tested was a 4 core System with Ubuntu 64bit and JDK 1.6 update 22.

I cannot reproduce the problem with your test program on my system using trunk. And using 1.7.10 it is the same

Show
blackdrag blackdrag added a comment - The system I tested was a 4 core System with Ubuntu 64bit and JDK 1.6 update 22. I cannot reproduce the problem with your test program on my system using trunk. And using 1.7.10 it is the same
Hide
Permalink
Szymon Kuklewicz added a comment -

Test program with extracted groovy-all-1.7.9.jar (embeddable).

Show
Szymon Kuklewicz added a comment - Test program with extracted groovy-all-1.7.9.jar (embeddable).
Hide
Permalink
Szymon Kuklewicz added a comment -

I uploaded test.jar that you can use to test. It contains compiled class GroovyEngineTest and extracted groovy-all-1.7.9.jar from groovy-1.7.9/embeddable directory. All you need to do just type "java -jar GroovyEngineTest.jar" when downloaded. Sorry for this big upload but I just wanted to create as similar environment as possible.

I asked developers from my company to do some more tests using this jar so I'll prepare detailed results in 2-3 days. It should be over a dozen machine tested.

Show
Szymon Kuklewicz added a comment - I uploaded test.jar that you can use to test. It contains compiled class GroovyEngineTest and extracted groovy-all-1.7.9.jar from groovy-1.7.9/embeddable directory. All you need to do just type "java -jar GroovyEngineTest.jar" when downloaded. Sorry for this big upload but I just wanted to create as similar environment as possible. I asked developers from my company to do some more tests using this jar so I'll prepare detailed results in 2-3 days. It should be over a dozen machine tested.
Hide
Permalink
Szymon Kuklewicz added a comment -

While waiting for test answers, I send you stack traces. Maybe you'll interested anyway. This is a dump from VisualVM.

Found one Java-level deadlock:
=============================
"Thread-8":
waiting to lock monitor 0x0818ae9c (object 0x9fb55368, a
java.util.HashMap),
which is held by "Thread-6"
"Thread-6":
waiting to lock monitor 0x7010334c (object 0x9ec04a58, a
groovy.lang.GroovyClassLoader$InnerLoader),
which is held by "Thread-7"
"Thread-7":
waiting to lock monitor 0x0818ae9c (object 0x9fb55368, a
java.util.HashMap),
which is held by "Thread-6"

Java stack information for the threads listed above:
===================================================
"Thread-8":
at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:776)

  • waiting to lock <0x9fb55368> (a java.util.HashMap)
    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:737)
    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:793)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
    at
    test.GroovyEngineTest.compileAndInstantiate(GroovyEngineTest.java:100)
    at test.GroovyEngineTest.access$200(GroovyEngineTest.java:19)
    at test.GroovyEngineTest$2.run(GroovyEngineTest.java:123)
    "Thread-6":
    at java.lang.Class.getDeclaredFields0(Native Method)
    at java.lang.Class.privateGetDeclaredFields(Class.java:2291)
    at java.lang.Class.getDeclaredFields(Class.java:1743)
    at
    org.codehaus.groovy.vmplugin.v5.Java5.configureClassNode(Java5.java:313)
    at org.codehaus.groovy.ast.ClassNode.lazyClassInit(ClassNode.java:263)
  • locked <0x9f097de8> (a java.lang.Object)
    at org.codehaus.groovy.ast.ClassNode.getInterfaces(ClassNode.java:341)
    at
    org.codehaus.groovy.ast.ClassNode.declaresInterface(ClassNode.java:929)
    at
    org.codehaus.groovy.ast.ClassNode.implementsInterface(ClassNode.java:909)
    at
    org.codehaus.groovy.classgen.AsmClassGenerator.doConvertAndCast(AsmClassGenerator.java:3842)
    at
    org.codehaus.groovy.classgen.AsmClassGenerator.doConvertAndCast(AsmClassGenerator.java:3837)
    at
    org.codehaus.groovy.classgen.AsmClassGenerator.storeThisInstanceField(AsmClassGenerator.java:2840)
    at
    org.codehaus.groovy.classgen.AsmClassGenerator.visitFieldExpression(AsmClassGenerator.java:2766)
    at
    org.codehaus.groovy.ast.expr.FieldExpression.visit(FieldExpression.java:38)
    at
    org.codehaus.groovy.classgen.AsmClassGenerator.evaluateEqual(AsmClassGenerator.java:4050)
    at
    org.codehaus.groovy.classgen.AsmClassGenerator.visitBinaryExpression(AsmClassGenerator.java:1485)
    at
    org.codehaus.groovy.ast.expr.BinaryExpression.visit(BinaryExpression.java:49)
    at
    org.codehaus.groovy.classgen.AsmClassGenerator.visitAndAutoboxBoolean(AsmClassGenerator.java:4122)
    at
    org.codehaus.groovy.classgen.AsmClassGenerator.visitExpressionStatement(AsmClassGenerator.java:1466)
    at
    org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:40)
    at
    org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:35)
    at
    org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:165)
    at
    org.codehaus.groovy.classgen.AsmClassGenerator.visitBlockStatement(AsmClassGenerator.java:738)
    at
    org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:69)
    at
    org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClassCodeContainer(ClassCodeVisitorSupport.java:101)
    at
    org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitConstructorOrMethod(ClassCodeVisitorSupport.java:112)
    at
    org.codehaus.groovy.classgen.AsmClassGenerator.visitStdMethod(AsmClassGenerator.java:626)
    at
    org.codehaus.groovy.classgen.AsmClassGenerator.visitConstructorOrMethod(AsmClassGenerator.java:601)
    at
    org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitConstructor(ClassCodeVisitorSupport.java:119)
    at
    org.codehaus.groovy.classgen.AsmClassGenerator.visitConstructor(AsmClassGenerator.java:688)
    at org.codehaus.groovy.ast.ClassNode.visitContents(ClassNode.java:1035)
    at
    org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClass(ClassCodeVisitorSupport.java:50)
    at
    org.codehaus.groovy.classgen.AsmClassGenerator.visitClass(AsmClassGenerator.java:276)
    at
    org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:748)
    at
    org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:942)
    at
    org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:519)
    at
    org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:497)
    at
    org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:474)
    at
    groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:306)
    at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:283)
  • locked <0x9fb55368> (a java.util.HashMap)
    at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:267)
    at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:263)
    at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:777)
  • locked <0x9fb55368> (a java.util.HashMap)
    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:737)
    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:793)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
    at
    test.GroovyEngineTest.compileAndInstantiate(GroovyEngineTest.java:100)
    at test.GroovyEngineTest.access$200(GroovyEngineTest.java:19)
    at test.GroovyEngineTest$2.run(GroovyEngineTest.java:123)
    "Thread-7":
    at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:776)
  • waiting to lock <0x9fb55368> (a java.util.HashMap)
    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:737)
    at
    groovy.lang.GroovyClassLoader$InnerLoader.loadClass(GroovyClassLoader.java:449)
    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:793)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:169)
    at test.A2.class$(test.A2)
    at test.A2.$get$$class$test$E2(test.A2)
    at test.A2.<init>(test.A2:4)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at
    sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at
    sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at
    test.GroovyEngineTest.compileAndInstantiate(GroovyEngineTest.java:101)
    at test.GroovyEngineTest.access$200(GroovyEngineTest.java:19)
    at test.GroovyEngineTest$2.run(GroovyEngineTest.java:123)

Found 1 deadlock.

Show
Szymon Kuklewicz added a comment - While waiting for test answers, I send you stack traces. Maybe you'll interested anyway. This is a dump from VisualVM. Found one Java-level deadlock: ============================= "Thread-8": waiting to lock monitor 0x0818ae9c (object 0x9fb55368, a java.util.HashMap), which is held by "Thread-6" "Thread-6": waiting to lock monitor 0x7010334c (object 0x9ec04a58, a groovy.lang.GroovyClassLoader$InnerLoader), which is held by "Thread-7" "Thread-7": waiting to lock monitor 0x0818ae9c (object 0x9fb55368, a java.util.HashMap), which is held by "Thread-6" Java stack information for the threads listed above: =================================================== "Thread-8": at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:776) waiting to lock <0x9fb55368> (a java.util.HashMap) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:737) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:793) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) at test.GroovyEngineTest.compileAndInstantiate(GroovyEngineTest.java:100) at test.GroovyEngineTest.access$200(GroovyEngineTest.java:19) at test.GroovyEngineTest$2.run(GroovyEngineTest.java:123) "Thread-6": at java.lang.Class.getDeclaredFields0(Native Method) at java.lang.Class.privateGetDeclaredFields(Class.java:2291) at java.lang.Class.getDeclaredFields(Class.java:1743) at org.codehaus.groovy.vmplugin.v5.Java5.configureClassNode(Java5.java:313) at org.codehaus.groovy.ast.ClassNode.lazyClassInit(ClassNode.java:263) locked <0x9f097de8> (a java.lang.Object) at org.codehaus.groovy.ast.ClassNode.getInterfaces(ClassNode.java:341) at org.codehaus.groovy.ast.ClassNode.declaresInterface(ClassNode.java:929) at org.codehaus.groovy.ast.ClassNode.implementsInterface(ClassNode.java:909) at org.codehaus.groovy.classgen.AsmClassGenerator.doConvertAndCast(AsmClassGenerator.java:3842) at org.codehaus.groovy.classgen.AsmClassGenerator.doConvertAndCast(AsmClassGenerator.java:3837) at org.codehaus.groovy.classgen.AsmClassGenerator.storeThisInstanceField(AsmClassGenerator.java:2840) at org.codehaus.groovy.classgen.AsmClassGenerator.visitFieldExpression(AsmClassGenerator.java:2766) at org.codehaus.groovy.ast.expr.FieldExpression.visit(FieldExpression.java:38) at org.codehaus.groovy.classgen.AsmClassGenerator.evaluateEqual(AsmClassGenerator.java:4050) at org.codehaus.groovy.classgen.AsmClassGenerator.visitBinaryExpression(AsmClassGenerator.java:1485) at org.codehaus.groovy.ast.expr.BinaryExpression.visit(BinaryExpression.java:49) at org.codehaus.groovy.classgen.AsmClassGenerator.visitAndAutoboxBoolean(AsmClassGenerator.java:4122) at org.codehaus.groovy.classgen.AsmClassGenerator.visitExpressionStatement(AsmClassGenerator.java:1466) at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:40) at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:35) at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:165) at org.codehaus.groovy.classgen.AsmClassGenerator.visitBlockStatement(AsmClassGenerator.java:738) at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:69) at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClassCodeContainer(ClassCodeVisitorSupport.java:101) at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitConstructorOrMethod(ClassCodeVisitorSupport.java:112) at org.codehaus.groovy.classgen.AsmClassGenerator.visitStdMethod(AsmClassGenerator.java:626) at org.codehaus.groovy.classgen.AsmClassGenerator.visitConstructorOrMethod(AsmClassGenerator.java:601) at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitConstructor(ClassCodeVisitorSupport.java:119) at org.codehaus.groovy.classgen.AsmClassGenerator.visitConstructor(AsmClassGenerator.java:688) at org.codehaus.groovy.ast.ClassNode.visitContents(ClassNode.java:1035) at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClass(ClassCodeVisitorSupport.java:50) at org.codehaus.groovy.classgen.AsmClassGenerator.visitClass(AsmClassGenerator.java:276) at org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:748) at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:942) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:519) at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:497) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:474) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:306) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:283) locked <0x9fb55368> (a java.util.HashMap) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:267) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:263) at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:777) locked <0x9fb55368> (a java.util.HashMap) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:737) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:793) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) at test.GroovyEngineTest.compileAndInstantiate(GroovyEngineTest.java:100) at test.GroovyEngineTest.access$200(GroovyEngineTest.java:19) at test.GroovyEngineTest$2.run(GroovyEngineTest.java:123) "Thread-7": at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:776) waiting to lock <0x9fb55368> (a java.util.HashMap) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:737) at groovy.lang.GroovyClassLoader$InnerLoader.loadClass(GroovyClassLoader.java:449) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:793) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:169) at test.A2.class$(test.A2) at test.A2.$get$$class$test$E2(test.A2) at test.A2.<init>(test.A2:4) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at test.GroovyEngineTest.compileAndInstantiate(GroovyEngineTest.java:101) at test.GroovyEngineTest.access$200(GroovyEngineTest.java:19) at test.GroovyEngineTest$2.run(GroovyEngineTest.java:123) Found 1 deadlock.
Hide
Permalink
blackdrag blackdrag added a comment -

ok, using the jar I was able to reproduce the deadlock... From a short look I am not so sure what to do about this. As far as Groovy is concerned no synchronization on the loaders is directly used. The reason that there is a lock on the class loader is because of Class#forName0, a native method - afaik. The reason that the Threads is blocked is that Class#getDeclaredFields0 is also using such a lock, again a native method.

It looks to me that adding a synchronized modifier to loadClass(String,boolean) is maybe the only way to prevent the deadlock to happen. Of course that is bad for speed then.

Show
blackdrag blackdrag added a comment - ok, using the jar I was able to reproduce the deadlock... From a short look I am not so sure what to do about this. As far as Groovy is concerned no synchronization on the loaders is directly used. The reason that there is a lock on the class loader is because of Class#forName0, a native method - afaik. The reason that the Threads is blocked is that Class#getDeclaredFields0 is also using such a lock, again a native method. It looks to me that adding a synchronized modifier to loadClass(String,boolean) is maybe the only way to prevent the deadlock to happen. Of course that is bad for speed then.
Hide
Permalink
blackdrag blackdrag added a comment -

Szymon would you be able to modify a Groovy 1.7 like that and test it?

Show
blackdrag blackdrag added a comment - Szymon would you be able to modify a Groovy 1.7 like that and test it?
Hide
Permalink
Szymon Kuklewicz added a comment -

Yes, I did as you said but it didn't help. In fact there is another problem but I think the same fundamental as the previous one.

I overridden GroovyClassLoader to add synchronized modifier

classLoader = new GroovyClassLoader() {
@Override
protected synchronized Class loadClass(String name, boolean resolve) throws ClassNotFoundException

{ return super.loadClass(name, resolve); }

};

but there is now deadlock between locks on GroovyClassLoader and GroovyClassLoader.InnerLoader (previous one was between InnerLoader and sourceCache):

Java-level deadlocks have been detected
Deadlock:

Thread-8 is waiting to lock groovy.lang.GroovyClassLoader$InnerLoader@671ff436 which is held by Thread-1
Thread-1 is waiting to lock test.GroovyEngineTest$1@2fe4cbc4 which is held by Thread-8

Thread stacks

Thread-1 [BLOCKED; waiting to lock test.GroovyEngineTest$1@2fe4cbc4]
java.lang.ClassLoader.loadClass(ClassLoader.java:292)
groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:696)
groovy.lang.GroovyClassLoader$InnerLoader.loadClass(GroovyClassLoader.java:449)
groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:793)
java.lang.ClassLoader.loadClass(ClassLoader.java:248)
java.lang.Class.getDeclaredConstructors0(native method)
java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
java.lang.Class.getConstructor0(Class.java:2699)
java.lang.Class.getConstructor(Class.java:1657)
test.GroovyEngineTest.compileAndInstantiate(GroovyEngineTest.java:106)
test.GroovyEngineTest.access$200(GroovyEngineTest.java:19)
test.GroovyEngineTest$3.run(GroovyEngineTest.java:128)

Thread-8 [BLOCKED; waiting to lock groovy.lang.GroovyClassLoader$InnerLoader@671ff436]
java.lang.Class.getDeclaredFields0(native method)
java.lang.Class.privateGetDeclaredFields(Class.java:2291)
java.lang.Class.getDeclaredFields(Class.java:1743)
org.codehaus.groovy.vmplugin.v5.Java5.configureClassNode(Java5.java:313)
org.codehaus.groovy.ast.ClassNode.lazyClassInit(ClassNode.java:263)
org.codehaus.groovy.ast.ClassNode.getInterfaces(ClassNode.java:341)
org.codehaus.groovy.ast.ClassNode.declaresInterface(ClassNode.java:929)
org.codehaus.groovy.ast.ClassNode.implementsInterface(ClassNode.java:909)
org.codehaus.groovy.classgen.AsmClassGenerator.doConvertAndCast(AsmClassGenerator.java:3842)
org.codehaus.groovy.classgen.AsmClassGenerator.doConvertAndCast(AsmClassGenerator.java:3837)
org.codehaus.groovy.classgen.AsmClassGenerator.storeThisInstanceField(AsmClassGenerator.java:2840)
org.codehaus.groovy.classgen.AsmClassGenerator.visitFieldExpression(AsmClassGenerator.java:2766)
org.codehaus.groovy.ast.expr.FieldExpression.visit(FieldExpression.java:38)
org.codehaus.groovy.classgen.AsmClassGenerator.evaluateEqual(AsmClassGenerator.java:4050)
org.codehaus.groovy.classgen.AsmClassGenerator.visitBinaryExpression(AsmClassGenerator.java:1485)
org.codehaus.groovy.ast.expr.BinaryExpression.visit(BinaryExpression.java:49)
org.codehaus.groovy.classgen.AsmClassGenerator.visitAndAutoboxBoolean(AsmClassGenerator.java:4122)
org.codehaus.groovy.classgen.AsmClassGenerator.visitExpressionStatement(AsmClassGenerator.java:1466)
org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:40)
org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:35)
org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:165)
org.codehaus.groovy.classgen.AsmClassGenerator.visitBlockStatement(AsmClassGenerator.java:738)
org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:69)
org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClassCodeContainer(ClassCodeVisitorSupport.java:101)
org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitConstructorOrMethod(ClassCodeVisitorSupport.java:112)
org.codehaus.groovy.classgen.AsmClassGenerator.visitStdMethod(AsmClassGenerator.java:626)
org.codehaus.groovy.classgen.AsmClassGenerator.visitConstructorOrMethod(AsmClassGenerator.java:601)
org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitConstructor(ClassCodeVisitorSupport.java:119)
org.codehaus.groovy.classgen.AsmClassGenerator.visitConstructor(AsmClassGenerator.java:688)
org.codehaus.groovy.ast.ClassNode.visitContents(ClassNode.java:1035)
org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClass(ClassCodeVisitorSupport.java:50)
org.codehaus.groovy.classgen.AsmClassGenerator.visitClass(AsmClassGenerator.java:276)
org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:748)
org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:942)
org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:519)
org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:497)
org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:474)
groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:306)
groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:283)
groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:267)
groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:263)
groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:777)
groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:737)
groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:793)
test.GroovyEngineTest$1.loadClass(GroovyEngineTest.java:29)
java.lang.ClassLoader.loadClass(ClassLoader.java:248)
test.GroovyEngineTest.compileAndInstantiate(GroovyEngineTest.java:105)
test.GroovyEngineTest.access$200(GroovyEngineTest.java:19)
test.GroovyEngineTest$3.run(GroovyEngineTest.java:128)

I tried also to override InnerLoader to add the modifier to method of this class but couldn't do it due to protected constuctor of ClassCollector.

I regret that you cannot reproduce this deadlock using your code. Maybe it help you if I sent you most recent version of my test class.

Show
Szymon Kuklewicz added a comment - Yes, I did as you said but it didn't help. In fact there is another problem but I think the same fundamental as the previous one. I overridden GroovyClassLoader to add synchronized modifier classLoader = new GroovyClassLoader() { @Override protected synchronized Class loadClass(String name, boolean resolve) throws ClassNotFoundException { return super.loadClass(name, resolve); } }; but there is now deadlock between locks on GroovyClassLoader and GroovyClassLoader.InnerLoader (previous one was between InnerLoader and sourceCache): Java-level deadlocks have been detected Deadlock: Thread-8 is waiting to lock groovy.lang.GroovyClassLoader$InnerLoader@671ff436 which is held by Thread-1 Thread-1 is waiting to lock test.GroovyEngineTest$1@2fe4cbc4 which is held by Thread-8 Thread stacks Thread-1 [BLOCKED; waiting to lock test.GroovyEngineTest$1@2fe4cbc4] java.lang.ClassLoader.loadClass(ClassLoader.java:292) groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:696) groovy.lang.GroovyClassLoader$InnerLoader.loadClass(GroovyClassLoader.java:449) groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:793) java.lang.ClassLoader.loadClass(ClassLoader.java:248) java.lang.Class.getDeclaredConstructors0(native method) java.lang.Class.privateGetDeclaredConstructors(Class.java:2389) java.lang.Class.getConstructor0(Class.java:2699) java.lang.Class.getConstructor(Class.java:1657) test.GroovyEngineTest.compileAndInstantiate(GroovyEngineTest.java:106) test.GroovyEngineTest.access$200(GroovyEngineTest.java:19) test.GroovyEngineTest$3.run(GroovyEngineTest.java:128) Thread-8 [BLOCKED; waiting to lock groovy.lang.GroovyClassLoader$InnerLoader@671ff436] java.lang.Class.getDeclaredFields0(native method) java.lang.Class.privateGetDeclaredFields(Class.java:2291) java.lang.Class.getDeclaredFields(Class.java:1743) org.codehaus.groovy.vmplugin.v5.Java5.configureClassNode(Java5.java:313) org.codehaus.groovy.ast.ClassNode.lazyClassInit(ClassNode.java:263) org.codehaus.groovy.ast.ClassNode.getInterfaces(ClassNode.java:341) org.codehaus.groovy.ast.ClassNode.declaresInterface(ClassNode.java:929) org.codehaus.groovy.ast.ClassNode.implementsInterface(ClassNode.java:909) org.codehaus.groovy.classgen.AsmClassGenerator.doConvertAndCast(AsmClassGenerator.java:3842) org.codehaus.groovy.classgen.AsmClassGenerator.doConvertAndCast(AsmClassGenerator.java:3837) org.codehaus.groovy.classgen.AsmClassGenerator.storeThisInstanceField(AsmClassGenerator.java:2840) org.codehaus.groovy.classgen.AsmClassGenerator.visitFieldExpression(AsmClassGenerator.java:2766) org.codehaus.groovy.ast.expr.FieldExpression.visit(FieldExpression.java:38) org.codehaus.groovy.classgen.AsmClassGenerator.evaluateEqual(AsmClassGenerator.java:4050) org.codehaus.groovy.classgen.AsmClassGenerator.visitBinaryExpression(AsmClassGenerator.java:1485) org.codehaus.groovy.ast.expr.BinaryExpression.visit(BinaryExpression.java:49) org.codehaus.groovy.classgen.AsmClassGenerator.visitAndAutoboxBoolean(AsmClassGenerator.java:4122) org.codehaus.groovy.classgen.AsmClassGenerator.visitExpressionStatement(AsmClassGenerator.java:1466) org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:40) org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:35) org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:165) org.codehaus.groovy.classgen.AsmClassGenerator.visitBlockStatement(AsmClassGenerator.java:738) org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:69) org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClassCodeContainer(ClassCodeVisitorSupport.java:101) org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitConstructorOrMethod(ClassCodeVisitorSupport.java:112) org.codehaus.groovy.classgen.AsmClassGenerator.visitStdMethod(AsmClassGenerator.java:626) org.codehaus.groovy.classgen.AsmClassGenerator.visitConstructorOrMethod(AsmClassGenerator.java:601) org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitConstructor(ClassCodeVisitorSupport.java:119) org.codehaus.groovy.classgen.AsmClassGenerator.visitConstructor(AsmClassGenerator.java:688) org.codehaus.groovy.ast.ClassNode.visitContents(ClassNode.java:1035) org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClass(ClassCodeVisitorSupport.java:50) org.codehaus.groovy.classgen.AsmClassGenerator.visitClass(AsmClassGenerator.java:276) org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:748) org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:942) org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:519) org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:497) org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:474) groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:306) groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:283) groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:267) groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:263) groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:777) groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:737) groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:793) test.GroovyEngineTest$1.loadClass(GroovyEngineTest.java:29) java.lang.ClassLoader.loadClass(ClassLoader.java:248) test.GroovyEngineTest.compileAndInstantiate(GroovyEngineTest.java:105) test.GroovyEngineTest.access$200(GroovyEngineTest.java:19) test.GroovyEngineTest$3.run(GroovyEngineTest.java:128) I tried also to override InnerLoader to add the modifier to method of this class but couldn't do it due to protected constuctor of ClassCollector. I regret that you cannot reproduce this deadlock using your code. Maybe it help you if I sent you most recent version of my test class.
Hide
Permalink
Szymon Kuklewicz added a comment -

New version of test class

Show
Szymon Kuklewicz added a comment - New version of test class
Hide
Permalink
blackdrag blackdrag added a comment -

I am sorry, I was not explaining enough I think... this loadClass method has to be synchronized on InnerLoader as well as normal GCL. And also the synchronization on the sourceCache should be replaced with a synchronization on "this"... maybe the whole method it occurs in

Show
blackdrag blackdrag added a comment - I am sorry, I was not explaining enough I think... this loadClass method has to be synchronized on InnerLoader as well as normal GCL. And also the synchronization on the sourceCache should be replaced with a synchronization on "this"... maybe the whole method it occurs in
Hide
Permalink
Szymon Kuklewicz added a comment -

I did some more research. Please tell if I am wrong ...

I extended test class to get some log info when class is put into GCL.classCache (by setClassCacheEntry) and also just before trying to instantiate previously compiled classes. Look at the logs i received ...

class registered test.C5@5e29c58e by Thread-1
at java.lang.Thread.dumpStack(Thread.java:1249)
at test.GroovyEngineTest$1.setClassCacheEntry(GroovyEngineTest.java:30)
at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:314)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:283)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:267)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:263)
at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:777)
at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:737)
at groovy.lang.GroovyClassLoader$InnerLoader.loadClass(GroovyClassLoader.java:449)
at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:793)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at test.C4.class$(test.C4)
at test.C4.$get$$class$test$C5(test.C4)
at test.C4.<init>(test.C4:4)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at test.GroovyEngineTest.compileAndInstantiate(GroovyEngineTest.java:111)
at test.GroovyEngineTest.access$200(GroovyEngineTest.java:19)
at test.GroovyEngineTest$3.run(GroovyEngineTest.java:133)
class registered test.C5@5e29c58e by Thread-1
at java.lang.Thread.dumpStack(Thread.java:1249)
at test.GroovyEngineTest$1.setClassCacheEntry(GroovyEngineTest.java:30)
at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:744)
at groovy.lang.GroovyClassLoader$InnerLoader.loadClass(GroovyClassLoader.java:449)
at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:793)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at test.C4.class$(test.C4)
at test.C4.$get$$class$test$C5(test.C4)
at test.C4.<init>(test.C4:4)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at test.GroovyEngineTest.compileAndInstantiate(GroovyEngineTest.java:111)
at test.GroovyEngineTest.access$200(GroovyEngineTest.java:19)
at test.GroovyEngineTest$3.run(GroovyEngineTest.java:133)

I combined info above with deadlock dump:

Thread-8 is waiting to lock groovy.lang.GroovyClassLoader$InnerLoader@5e29c58e which is held by Thread-1
Thread-1 is waiting to lock java.util.HashMap@24bb6086 which is held by Thread-8

Thread-1 [BLOCKED; waiting to lock java.util.HashMap@24bb6086]
groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:776)
groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:737)
groovy.lang.GroovyClassLoader$InnerLoader.loadClass(GroovyClassLoader.java:449)
groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:793)
java.lang.ClassLoader.loadClass(ClassLoader.java:248)
java.lang.Class.getDeclaredFields0(native method)
java.lang.Class.privateGetDeclaredFields(Class.java:2291)
java.lang.Class.getDeclaredFields(Class.java:1743)
org.codehaus.groovy.reflection.CachedClass$1$1.run(CachedClass.java:47)
java.security.AccessController.doPrivileged(native method)
org.codehaus.groovy.reflection.CachedClass$1.initValue(CachedClass.java:44)
org.codehaus.groovy.reflection.CachedClass$1.initValue(CachedClass.java:42)
org.codehaus.groovy.util.LazyReference.getLocked(LazyReference.java:46)
org.codehaus.groovy.util.LazyReference.get(LazyReference.java:33)
org.codehaus.groovy.reflection.CachedClass.getFields(CachedClass.java:253)
groovy.lang.MetaClassImpl.addFields(MetaClassImpl.java:2108)
groovy.lang.MetaClassImpl.inheritFields(MetaClassImpl.java:2103)
groovy.lang.MetaClassImpl.setupProperties(MetaClassImpl.java:1993)
groovy.lang.MetaClassImpl.addProperties(MetaClassImpl.java:2950)
groovy.lang.MetaClassImpl.initialize(MetaClassImpl.java:2921)
org.codehaus.groovy.reflection.ClassInfo.getMetaClassUnderLock(ClassInfo.java:166)
org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:182)
org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:210)
org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:751)
org.codehaus.groovy.runtime.callsite.CallSiteArray.createCallConstructorSite(CallSiteArray.java:69)
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:52)
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:190)
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:194)
test.C4.<init>(test.C4:4)
sun.reflect.NativeConstructorAccessorImpl.newInstance0(native method)
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
java.lang.reflect.Constructor.newInstance(Constructor.java:513)
test.GroovyEngineTest.compileAndInstantiate(GroovyEngineTest.java:111)
test.GroovyEngineTest.access$200(GroovyEngineTest.java:19)
test.GroovyEngineTest$3.run(GroovyEngineTest.java:133)

Thread-8 [BLOCKED; waiting to lock groovy.lang.GroovyClassLoader$InnerLoader@5e29c58e]
java.lang.Class.getDeclaredFields0(native method)
java.lang.Class.privateGetDeclaredFields(Class.java:2291)
java.lang.Class.getDeclaredFields(Class.java:1743)
org.codehaus.groovy.vmplugin.v5.Java5.configureClassNode(Java5.java:313)
org.codehaus.groovy.ast.ClassNode.lazyClassInit(ClassNode.java:263)
org.codehaus.groovy.ast.ClassNode.getInterfaces(ClassNode.java:341)
org.codehaus.groovy.ast.ClassNode.declaresInterface(ClassNode.java:929)
org.codehaus.groovy.ast.ClassNode.implementsInterface(ClassNode.java:909)
org.codehaus.groovy.classgen.AsmClassGenerator.doConvertAndCast(AsmClassGenerator.java:3842)
org.codehaus.groovy.classgen.AsmClassGenerator.doConvertAndCast(AsmClassGenerator.java:3837)
org.codehaus.groovy.classgen.AsmClassGenerator.storeThisInstanceField(AsmClassGenerator.java:2840)
org.codehaus.groovy.classgen.AsmClassGenerator.visitFieldExpression(AsmClassGenerator.java:2766)
org.codehaus.groovy.ast.expr.FieldExpression.visit(FieldExpression.java:38)
org.codehaus.groovy.classgen.AsmClassGenerator.evaluateEqual(AsmClassGenerator.java:4050)
org.codehaus.groovy.classgen.AsmClassGenerator.visitBinaryExpression(AsmClassGenerator.java:1485)
org.codehaus.groovy.ast.expr.BinaryExpression.visit(BinaryExpression.java:49)
org.codehaus.groovy.classgen.AsmClassGenerator.visitAndAutoboxBoolean(AsmClassGenerator.java:4122)
org.codehaus.groovy.classgen.AsmClassGenerator.visitExpressionStatement(AsmClassGenerator.java:1466)
org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:40)
org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:35)
org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:165)
org.codehaus.groovy.classgen.AsmClassGenerator.visitBlockStatement(AsmClassGenerator.java:738)
org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:69)
org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClassCodeContainer(ClassCodeVisitorSupport.java:101)
org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitConstructorOrMethod(ClassCodeVisitorSupport.java:112)
org.codehaus.groovy.classgen.AsmClassGenerator.visitStdMethod(AsmClassGenerator.java:626)
org.codehaus.groovy.classgen.AsmClassGenerator.visitConstructorOrMethod(AsmClassGenerator.java:601)
org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitConstructor(ClassCodeVisitorSupport.java:119)
org.codehaus.groovy.classgen.AsmClassGenerator.visitConstructor(AsmClassGenerator.java:688)
org.codehaus.groovy.ast.ClassNode.visitContents(ClassNode.java:1035)
org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClass(ClassCodeVisitorSupport.java:50)
org.codehaus.groovy.classgen.AsmClassGenerator.visitClass(AsmClassGenerator.java:276)
org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:748)
org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:942)
org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:519)
org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:497)
org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:474)
groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:306)
groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:283)
groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:267)
groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:263)
groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:777)
groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:737)
groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:793)
java.lang.ClassLoader.loadClass(ClassLoader.java:248)
test.GroovyEngineTest.compileAndInstantiate(GroovyEngineTest.java:109)
test.GroovyEngineTest.access$200(GroovyEngineTest.java:19)
test.GroovyEngineTest$3.run(GroovyEngineTest.java:133)

My interpretation is that it was Thread-1 which created InnerLoader@5e29c58e and then used it to compile some classes. However, when completed, during instantiation of just received class, it locks on this InnerLoader and waits for sourceCache (because more compilation was needed). Until now all is right. What is wrong I think is behave of Thread-8. Thread-8 also compiled some classes, it must had created it's own InnerLoader but during compilation, in particular during invocation of org.codehaus.groovy.vmplugin.v5.Java5#configureClassNode it calls native methods of class, which ClassLoader is InnerLoader@5e29c58e - I mean class compiled by another Thread/InnerLoader. Thread-8 tries to lock 3 resources: it's own InnerLoader, sourceCache and the other InnerLoader.

If I am right, no combination of extra synchronization on InnerLoader, GCL or replacement of sourceCache synchronization to "this" will help. With all this improvements, Thread-1 will still lock on it's InnerLoader and then GroovyClassLoader (instead of sourceCache) and Thread-8 will lock on it's InnerLoader, then GCL and futher on another InnerLoader. Switching synchronization of sourceCache to "this" won't help because GCL does not take part in the deadlock. Switching synchronization of sourceCache to InnerLoader could help but such change would not also be logical of course.

What I can suggest is to improve package "vmplugin.v5" not to call native methods on classes not from it's own InnerLoader. Maybe an information this vmplugin gathers should be cached directly by original Thread (compilation process). This is just a guess. I really don't known what this vmplugin does.

Show
Szymon Kuklewicz added a comment - I did some more research. Please tell if I am wrong ... I extended test class to get some log info when class is put into GCL.classCache (by setClassCacheEntry) and also just before trying to instantiate previously compiled classes. Look at the logs i received ... class registered test.C5@5e29c58e by Thread-1 at java.lang.Thread.dumpStack(Thread.java:1249) at test.GroovyEngineTest$1.setClassCacheEntry(GroovyEngineTest.java:30) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:314) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:283) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:267) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:263) at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:777) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:737) at groovy.lang.GroovyClassLoader$InnerLoader.loadClass(GroovyClassLoader.java:449) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:793) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:169) at test.C4.class$(test.C4) at test.C4.$get$$class$test$C5(test.C4) at test.C4.<init>(test.C4:4) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at test.GroovyEngineTest.compileAndInstantiate(GroovyEngineTest.java:111) at test.GroovyEngineTest.access$200(GroovyEngineTest.java:19) at test.GroovyEngineTest$3.run(GroovyEngineTest.java:133) class registered test.C5@5e29c58e by Thread-1 at java.lang.Thread.dumpStack(Thread.java:1249) at test.GroovyEngineTest$1.setClassCacheEntry(GroovyEngineTest.java:30) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:744) at groovy.lang.GroovyClassLoader$InnerLoader.loadClass(GroovyClassLoader.java:449) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:793) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:169) at test.C4.class$(test.C4) at test.C4.$get$$class$test$C5(test.C4) at test.C4.<init>(test.C4:4) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at test.GroovyEngineTest.compileAndInstantiate(GroovyEngineTest.java:111) at test.GroovyEngineTest.access$200(GroovyEngineTest.java:19) at test.GroovyEngineTest$3.run(GroovyEngineTest.java:133) I combined info above with deadlock dump: Thread-8 is waiting to lock groovy.lang.GroovyClassLoader$InnerLoader@5e29c58e which is held by Thread-1 Thread-1 is waiting to lock java.util.HashMap@24bb6086 which is held by Thread-8 Thread-1 [BLOCKED; waiting to lock java.util.HashMap@24bb6086] groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:776) groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:737) groovy.lang.GroovyClassLoader$InnerLoader.loadClass(GroovyClassLoader.java:449) groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:793) java.lang.ClassLoader.loadClass(ClassLoader.java:248) java.lang.Class.getDeclaredFields0(native method) java.lang.Class.privateGetDeclaredFields(Class.java:2291) java.lang.Class.getDeclaredFields(Class.java:1743) org.codehaus.groovy.reflection.CachedClass$1$1.run(CachedClass.java:47) java.security.AccessController.doPrivileged(native method) org.codehaus.groovy.reflection.CachedClass$1.initValue(CachedClass.java:44) org.codehaus.groovy.reflection.CachedClass$1.initValue(CachedClass.java:42) org.codehaus.groovy.util.LazyReference.getLocked(LazyReference.java:46) org.codehaus.groovy.util.LazyReference.get(LazyReference.java:33) org.codehaus.groovy.reflection.CachedClass.getFields(CachedClass.java:253) groovy.lang.MetaClassImpl.addFields(MetaClassImpl.java:2108) groovy.lang.MetaClassImpl.inheritFields(MetaClassImpl.java:2103) groovy.lang.MetaClassImpl.setupProperties(MetaClassImpl.java:1993) groovy.lang.MetaClassImpl.addProperties(MetaClassImpl.java:2950) groovy.lang.MetaClassImpl.initialize(MetaClassImpl.java:2921) org.codehaus.groovy.reflection.ClassInfo.getMetaClassUnderLock(ClassInfo.java:166) org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:182) org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:210) org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:751) org.codehaus.groovy.runtime.callsite.CallSiteArray.createCallConstructorSite(CallSiteArray.java:69) org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:52) org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:190) org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:194) test.C4.<init>(test.C4:4) sun.reflect.NativeConstructorAccessorImpl.newInstance0(native method) sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) java.lang.reflect.Constructor.newInstance(Constructor.java:513) test.GroovyEngineTest.compileAndInstantiate(GroovyEngineTest.java:111) test.GroovyEngineTest.access$200(GroovyEngineTest.java:19) test.GroovyEngineTest$3.run(GroovyEngineTest.java:133) Thread-8 [BLOCKED; waiting to lock groovy.lang.GroovyClassLoader$InnerLoader@5e29c58e] java.lang.Class.getDeclaredFields0(native method) java.lang.Class.privateGetDeclaredFields(Class.java:2291) java.lang.Class.getDeclaredFields(Class.java:1743) org.codehaus.groovy.vmplugin.v5.Java5.configureClassNode(Java5.java:313) org.codehaus.groovy.ast.ClassNode.lazyClassInit(ClassNode.java:263) org.codehaus.groovy.ast.ClassNode.getInterfaces(ClassNode.java:341) org.codehaus.groovy.ast.ClassNode.declaresInterface(ClassNode.java:929) org.codehaus.groovy.ast.ClassNode.implementsInterface(ClassNode.java:909) org.codehaus.groovy.classgen.AsmClassGenerator.doConvertAndCast(AsmClassGenerator.java:3842) org.codehaus.groovy.classgen.AsmClassGenerator.doConvertAndCast(AsmClassGenerator.java:3837) org.codehaus.groovy.classgen.AsmClassGenerator.storeThisInstanceField(AsmClassGenerator.java:2840) org.codehaus.groovy.classgen.AsmClassGenerator.visitFieldExpression(AsmClassGenerator.java:2766) org.codehaus.groovy.ast.expr.FieldExpression.visit(FieldExpression.java:38) org.codehaus.groovy.classgen.AsmClassGenerator.evaluateEqual(AsmClassGenerator.java:4050) org.codehaus.groovy.classgen.AsmClassGenerator.visitBinaryExpression(AsmClassGenerator.java:1485) org.codehaus.groovy.ast.expr.BinaryExpression.visit(BinaryExpression.java:49) org.codehaus.groovy.classgen.AsmClassGenerator.visitAndAutoboxBoolean(AsmClassGenerator.java:4122) org.codehaus.groovy.classgen.AsmClassGenerator.visitExpressionStatement(AsmClassGenerator.java:1466) org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:40) org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:35) org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:165) org.codehaus.groovy.classgen.AsmClassGenerator.visitBlockStatement(AsmClassGenerator.java:738) org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:69) org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClassCodeContainer(ClassCodeVisitorSupport.java:101) org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitConstructorOrMethod(ClassCodeVisitorSupport.java:112) org.codehaus.groovy.classgen.AsmClassGenerator.visitStdMethod(AsmClassGenerator.java:626) org.codehaus.groovy.classgen.AsmClassGenerator.visitConstructorOrMethod(AsmClassGenerator.java:601) org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitConstructor(ClassCodeVisitorSupport.java:119) org.codehaus.groovy.classgen.AsmClassGenerator.visitConstructor(AsmClassGenerator.java:688) org.codehaus.groovy.ast.ClassNode.visitContents(ClassNode.java:1035) org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClass(ClassCodeVisitorSupport.java:50) org.codehaus.groovy.classgen.AsmClassGenerator.visitClass(AsmClassGenerator.java:276) org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:748) org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:942) org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:519) org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:497) org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:474) groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:306) groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:283) groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:267) groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:263) groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:777) groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:737) groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:793) java.lang.ClassLoader.loadClass(ClassLoader.java:248) test.GroovyEngineTest.compileAndInstantiate(GroovyEngineTest.java:109) test.GroovyEngineTest.access$200(GroovyEngineTest.java:19) test.GroovyEngineTest$3.run(GroovyEngineTest.java:133) My interpretation is that it was Thread-1 which created InnerLoader@5e29c58e and then used it to compile some classes. However, when completed, during instantiation of just received class, it locks on this InnerLoader and waits for sourceCache (because more compilation was needed). Until now all is right. What is wrong I think is behave of Thread-8. Thread-8 also compiled some classes, it must had created it's own InnerLoader but during compilation, in particular during invocation of org.codehaus.groovy.vmplugin.v5.Java5#configureClassNode it calls native methods of class, which ClassLoader is InnerLoader@5e29c58e - I mean class compiled by another Thread/InnerLoader. Thread-8 tries to lock 3 resources: it's own InnerLoader, sourceCache and the other InnerLoader. If I am right, no combination of extra synchronization on InnerLoader, GCL or replacement of sourceCache synchronization to "this" will help. With all this improvements, Thread-1 will still lock on it's InnerLoader and then GroovyClassLoader (instead of sourceCache) and Thread-8 will lock on it's InnerLoader, then GCL and futher on another InnerLoader. Switching synchronization of sourceCache to "this" won't help because GCL does not take part in the deadlock. Switching synchronization of sourceCache to InnerLoader could help but such change would not also be logical of course. What I can suggest is to improve package "vmplugin.v5" not to call native methods on classes not from it's own InnerLoader. Maybe an information this vmplugin gathers should be cached directly by original Thread (compilation process). This is just a guess. I really don't known what this vmplugin does.
Hide
Permalink
blackdrag blackdrag added a comment -

I should maybe first explain why we have InnerLoader. Each compilation get's its own InnerLoader. If you for example compile A and it depends on B and A,B is not compiled and you do a loadClass for A, then an innerLoader will be created and A will be added to the compilation. During compilation it will find that B is also to be compiled and adds it to the same compilation process. A and B will share the same class loader in the end. Now if recompilaltion is enabled and you change the file for A, the we need to define new A. You cannot define a new class with the name of an existing class in the same loader. Each classname, classloader pair must be unique. Thus we need a new loader for this. That's why we have a new loader per compilation. Would we kill inner loader, then we would kill the recompilation feature as well.

If you look at the native methods being called there is one case:

at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:793)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at test.C4.class$(test.C4)

forName does no locking, but forName0 does. class$ is under our control... maybe we could let it call loadClass directly and so avoid the lock since GCL#loadClass ClassLoader#loadClass here won't impose it. That might be one part we have under our control.

For the other...

at java.lang.Class.getDeclaredFields0(Native Method)
at java.lang.Class.privateGetDeclaredFields(Class.java:2291)
at java.lang.Class.getDeclaredFields(Class.java:1743)
at
org.codehaus.groovy.vmplugin.v5.Java5.configureClassNode(Java5.java:313)
at org.codehaus.groovy.ast.ClassNode.lazyClassInit(ClassNode.java:263)

if you look at what this lazyClassInit and configureClassNode is, you will see that this is about populating a ClassNode by information from reflection. There is no real alternative for that. Simply using a different loader is not going to work, since we don't use any loader by ourselves and does so in the code of reflection, locking the loader of the class the class was defined in to get the information.

btw... there is one more... getConstructor0 is also locking.... also on the Innerloader.

But if that really resolves the deadlock problem... no idea. Normally, if there is a deadlock you have 3 strategies you can follow:
(1) reduce the number of places doing the locking
since this is in native code that we have not really any influence over we have no real chance here unless we can avoid the call at all
(2) reduce the number of different locks
This is normally the best chance, but since the locking happens on different InnerLoader we don't have a chance with this.
(3) guard more places with synchronization
More or less this applies only if you have the synchronization under your control. We would need this lock being set before InnerLoader is set... we could do that for vm5plugin and let it synchronize on the parent GCL. We could have GCL#loadClass synchronized... but if that really helps... no idea.

Show
blackdrag blackdrag added a comment - I should maybe first explain why we have InnerLoader. Each compilation get's its own InnerLoader. If you for example compile A and it depends on B and A,B is not compiled and you do a loadClass for A, then an innerLoader will be created and A will be added to the compilation. During compilation it will find that B is also to be compiled and adds it to the same compilation process. A and B will share the same class loader in the end. Now if recompilaltion is enabled and you change the file for A, the we need to define new A. You cannot define a new class with the name of an existing class in the same loader. Each classname, classloader pair must be unique. Thus we need a new loader for this. That's why we have a new loader per compilation. Would we kill inner loader, then we would kill the recompilation feature as well. If you look at the native methods being called there is one case: at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:793) at java.lang. ClassLoader .loadClass( ClassLoader .java:248) at java.lang. Class .forName0(Native Method) at java.lang. Class .forName( Class .java:169) at test.C4.class$(test.C4) forName does no locking, but forName0 does. class$ is under our control... maybe we could let it call loadClass directly and so avoid the lock since GCL#loadClass ClassLoader#loadClass here won't impose it. That might be one part we have under our control. For the other... at java.lang. Class .getDeclaredFields0(Native Method) at java.lang. Class .privateGetDeclaredFields( Class .java:2291) at java.lang. Class .getDeclaredFields( Class .java:1743) at org.codehaus.groovy.vmplugin.v5.Java5.configureClassNode(Java5.java:313) at org.codehaus.groovy.ast.ClassNode.lazyClassInit(ClassNode.java:263) if you look at what this lazyClassInit and configureClassNode is, you will see that this is about populating a ClassNode by information from reflection. There is no real alternative for that. Simply using a different loader is not going to work, since we don't use any loader by ourselves and does so in the code of reflection, locking the loader of the class the class was defined in to get the information. btw... there is one more... getConstructor0 is also locking.... also on the Innerloader. But if that really resolves the deadlock problem... no idea. Normally, if there is a deadlock you have 3 strategies you can follow: (1) reduce the number of places doing the locking since this is in native code that we have not really any influence over we have no real chance here unless we can avoid the call at all (2) reduce the number of different locks This is normally the best chance, but since the locking happens on different InnerLoader we don't have a chance with this. (3) guard more places with synchronization More or less this applies only if you have the synchronization under your control. We would need this lock being set before InnerLoader is set... we could do that for vm5plugin and let it synchronize on the parent GCL. We could have GCL#loadClass synchronized... but if that really helps... no idea.
Hide
Permalink
blackdrag blackdrag added a comment -

Only problem.... how to avoid Class#forName? I need at least initially to get the loader somehow and for that I already need a class.

Show
blackdrag blackdrag added a comment - Only problem.... how to avoid Class#forName? I need at least initially to get the loader somehow and for that I already need a class.
Hide
Permalink
Szymon Kuklewicz added a comment -

When I wrote previous comment, I knew the purpose that InnerLoaders are created for so I have no doubts that InnerLoader should stay.

If you really want to mess up with "java.lang.Class" you can try using so called "java agents". For example JRebel uses that to modify bytecode for java reflection package. But changing java.lang.Class sounds like we were to close too the edge. It may not be possible.

I think no general fix can be provided, we are not living in ideal world. The only thing I see to avoid the problem is to fix this particular case: stop vmplugin from calling Class#getDeclaredFields and others which would lock on InnerLoader. Can't such change really be done? As I suspect this plugin gathers information from reflections. Can't this information be gathered immediately by original Thread/InnerLoader and cached in some Map<Class, ClassReflectionData>? If we did it, we would also add some condition to the method Java5#configureClassNode to use this cache if class loader of specified class is InnerLoader and such loader is different than InnerLoader associated with current thread.

Show
Szymon Kuklewicz added a comment - When I wrote previous comment, I knew the purpose that InnerLoaders are created for so I have no doubts that InnerLoader should stay. If you really want to mess up with "java.lang.Class" you can try using so called "java agents". For example JRebel uses that to modify bytecode for java reflection package. But changing java.lang.Class sounds like we were to close too the edge. It may not be possible. I think no general fix can be provided, we are not living in ideal world. The only thing I see to avoid the problem is to fix this particular case: stop vmplugin from calling Class#getDeclaredFields and others which would lock on InnerLoader. Can't such change really be done? As I suspect this plugin gathers information from reflections. Can't this information be gathered immediately by original Thread/InnerLoader and cached in some Map<Class, ClassReflectionData>? If we did it, we would also add some condition to the method Java5#configureClassNode to use this cache if class loader of specified class is InnerLoader and such loader is different than InnerLoader associated with current thread.
Hide
Permalink
blackdrag blackdrag added a comment -

You misunderstood, I cannot mess with the java.lang.Class class itself. If the central point is to ensure we lock on the parent GCL, before any lock on the InnerLoader happens, then I think there is no real way to resolve this, since this provides a problem for class loading done from generated classes.

Then about the ClassNode init... Where would it be done, if not like now? We would have to cache the ClassNode we created in a different Thread. ClassNodes are not exactly light structures and they are not thread safe... also the compiler now does not keep them. Each compilation run makes a completely new compiler atm. This would mean a big change in the design of the compiler to realize this. And we would get a compiler that requires more memory, even if not running. The alternative would be the ClassReflectionData you spoke of... actually we have something like that already. I wonder if, if we would let the compiler use ClassInfo/CachedClass objects we would really save something. The problem is that it is not the part that does the compilation, that is causing the blocking call of getDeclaredFields0. It is another thread. So we would have to generate those directly after compilation I guess... But ClassInfo/CachedClass is a lazy structure, which means the info might have been collected before we reach the compiler part that needs the information. So this may still lead to deadlock and we still have to save a lot of information not lazy.

I am worried about the memory profile

Actually I wonder if we could instead use GroovyScriptEngine.

Show
blackdrag blackdrag added a comment - You misunderstood, I cannot mess with the java.lang.Class class itself. If the central point is to ensure we lock on the parent GCL, before any lock on the InnerLoader happens, then I think there is no real way to resolve this, since this provides a problem for class loading done from generated classes. Then about the ClassNode init... Where would it be done, if not like now? We would have to cache the ClassNode we created in a different Thread. ClassNodes are not exactly light structures and they are not thread safe... also the compiler now does not keep them. Each compilation run makes a completely new compiler atm. This would mean a big change in the design of the compiler to realize this. And we would get a compiler that requires more memory, even if not running. The alternative would be the ClassReflectionData you spoke of... actually we have something like that already. I wonder if, if we would let the compiler use ClassInfo/CachedClass objects we would really save something. The problem is that it is not the part that does the compilation, that is causing the blocking call of getDeclaredFields0. It is another thread. So we would have to generate those directly after compilation I guess... But ClassInfo/CachedClass is a lazy structure, which means the info might have been collected before we reach the compiler part that needs the information. So this may still lead to deadlock and we still have to save a lot of information not lazy. I am worried about the memory profile Actually I wonder if we could instead use GroovyScriptEngine.
Hide
Permalink
Szymon Kuklewicz added a comment -

ClassInfo/CachedClass info won't be collected if you use WeakHashMap or even better if you hook a reference directly into Class (just like __timestamp). WeakHashMap would cause more memory usage and is not thread safe. Memory usage will be probably doubled for any Groovy class definition.

How GroovyScriptEngine can help? I'm not familiar with class and what is a difference between GroovyScriptEngine and GroovyClassLoader from user's point of view.

Show
Szymon Kuklewicz added a comment - ClassInfo/CachedClass info won't be collected if you use WeakHashMap or even better if you hook a reference directly into Class (just like __timestamp). WeakHashMap would cause more memory usage and is not thread safe. Memory usage will be probably doubled for any Groovy class definition. How GroovyScriptEngine can help? I'm not familiar with class and what is a difference between GroovyScriptEngine and GroovyClassLoader from user's point of view.
Hide
Permalink
blackdrag blackdrag added a comment -

Szymon, before we actually advice something could you explain a little bit more your use case? Why are you forcing cache clearing?

Show
blackdrag blackdrag added a comment - Szymon, before we actually advice something could you explain a little bit more your use case? Why are you forcing cache clearing?
Hide
Permalink
Szymon Kuklewicz added a comment -

We are using Groovy as main language in are web framework jPALIO. Users use dedicated IDE to manage remote source code which is stored in database and compiled directly on remote server. When developer saves objects, new version is saved in database and then GroovyClassLoader.clearCache is called as we want future request to run fresh version of application code. Full cache is cleared to remove also potential dependencies. There are multiple developers and web users working simultaneously on the same server.

Show
Szymon Kuklewicz added a comment - We are using Groovy as main language in are web framework jPALIO. Users use dedicated IDE to manage remote source code which is stored in database and compiled directly on remote server. When developer saves objects, new version is saved in database and then GroovyClassLoader.clearCache is called as we want future request to run fresh version of application code. Full cache is cleared to remove also potential dependencies. There are multiple developers and web users working simultaneously on the same server.
Hide
Permalink
blackdrag blackdrag added a comment -

clearCache is not really intended to force recompilation. Normally the intension is o enable recompilation in the compiler configuration and set an recompilation interval that fits your needs. According to our tests here this prevents the deadlock too. But I wonder.. if you want everything to reload, wouldn't it be the most simple version to just use a completely new GroovyClassLoader?

Show
blackdrag blackdrag added a comment - clearCache is not really intended to force recompilation. Normally the intension is o enable recompilation in the compiler configuration and set an recompilation interval that fits your needs. According to our tests here this prevents the deadlock too. But I wonder.. if you want everything to reload, wouldn't it be the most simple version to just use a completely new GroovyClassLoader?
Hide
Permalink
Guillaume Laforge added a comment -

Like Jochen, I think the best and cleanest approach would actually be to use a brand new GroovyClassLoader each time you need a clean recompilation, for a clean state. That's what we'd recommend usually.
That said, the GroovyClassLoader indeed has some deadlocks when used the way you've done it. I'm going to attach a patch which adds a ReentrantReadWriteLock: read locks wherever we synchronize on the sourceCache and classCache, and a write lock on the code of the clearCache() method. With that setup, I don't get any more blocked threads.
If you could try to recompile Groovy with that patch applied, and could come back to us with feedback, that's something we could integrate.

Show
Guillaume Laforge added a comment - Like Jochen, I think the best and cleanest approach would actually be to use a brand new GroovyClassLoader each time you need a clean recompilation, for a clean state. That's what we'd recommend usually. That said, the GroovyClassLoader indeed has some deadlocks when used the way you've done it. I'm going to attach a patch which adds a ReentrantReadWriteLock: read locks wherever we synchronize on the sourceCache and classCache, and a write lock on the code of the clearCache() method. With that setup, I don't get any more blocked threads. If you could try to recompile Groovy with that patch applied, and could come back to us with feedback, that's something we could integrate.
Hide
Permalink
Guillaume Laforge added a comment -

Usage of a ReentrantReadWriteLock.

Show
Guillaume Laforge added a comment - Usage of a ReentrantReadWriteLock.
Hide
Permalink
Szymon Kuklewicz added a comment -

I'm applying the patch but before I do it, I want to ask why synchronized statements were left so now are both ReadWriteLock and standard synchronization are used?

Show
Szymon Kuklewicz added a comment - I'm applying the patch but before I do it, I want to ask why synchronized statements were left so now are both ReadWriteLock and standard synchronization are used?
Hide
Permalink
blackdrag blackdrag added a comment -

because during the read locks there will be still concurrent modification of the maps when removing entries. Let us first see if it works, we can then still improve from there.

Show
blackdrag blackdrag added a comment - because during the read locks there will be still concurrent modification of the maps when removing entries. Let us first see if it works, we can then still improve from there.
Hide
Permalink
Szymon Kuklewicz added a comment -

Deadlock:

Thread-3 is waiting to lock groovy.lang.GroovyClassLoader$InnerLoader@2321b59a which is held by Thread-7
Thread-7 is waiting to lock java.util.HashMap@4bb963c4 which is held by Thread-3

Thread stacks

Thread-3 [BLOCKED; waiting to lock groovy.lang.GroovyClassLoader$InnerLoader@2321b59a]
java.lang.Class.getDeclaredFields0(native method)
java.lang.Class.privateGetDeclaredFields(Class.java:2291)
java.lang.Class.getDeclaredFields(Class.java:1743)
org.codehaus.groovy.vmplugin.v5.Java5.configureClassNode(Java5.java:313)
org.codehaus.groovy.ast.ClassNode.lazyClassInit(ClassNode.java:263)
org.codehaus.groovy.ast.ClassNode.getInterfaces(ClassNode.java:341)
org.codehaus.groovy.ast.ClassNode.declaresInterface(ClassNode.java:929)
org.codehaus.groovy.ast.ClassNode.implementsInterface(ClassNode.java:909)
org.codehaus.groovy.classgen.AsmClassGenerator.doConvertAndCast(AsmClassGenerator.java:3842)
org.codehaus.groovy.classgen.AsmClassGenerator.doConvertAndCast(AsmClassGenerator.java:3837)
org.codehaus.groovy.classgen.AsmClassGenerator.storeThisInstanceField(AsmClassGenerator.java:2840)
org.codehaus.groovy.classgen.AsmClassGenerator.visitFieldExpression(AsmClassGenerator.java:2766)
org.codehaus.groovy.ast.expr.FieldExpression.visit(FieldExpression.java:38)
org.codehaus.groovy.classgen.AsmClassGenerator.evaluateEqual(AsmClassGenerator.java:4050)
org.codehaus.groovy.classgen.AsmClassGenerator.visitBinaryExpression(AsmClassGenerator.java:1485)
org.codehaus.groovy.ast.expr.BinaryExpression.visit(BinaryExpression.java:49)
org.codehaus.groovy.classgen.AsmClassGenerator.visitAndAutoboxBoolean(AsmClassGenerator.java:4122)
org.codehaus.groovy.classgen.AsmClassGenerator.visitExpressionStatement(AsmClassGenerator.java:1466)
org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:40)
org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:35)
org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:165)
org.codehaus.groovy.classgen.AsmClassGenerator.visitBlockStatement(AsmClassGenerator.java:738)
org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:69)
org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClassCodeContainer(ClassCodeVisitorSupport.java:101)
org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitConstructorOrMethod(ClassCodeVisitorSupport.java:112)
org.codehaus.groovy.classgen.AsmClassGenerator.visitStdMethod(AsmClassGenerator.java:626)
org.codehaus.groovy.classgen.AsmClassGenerator.visitConstructorOrMethod(AsmClassGenerator.java:601)
org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitConstructor(ClassCodeVisitorSupport.java:119)
org.codehaus.groovy.classgen.AsmClassGenerator.visitConstructor(AsmClassGenerator.java:688)
org.codehaus.groovy.ast.ClassNode.visitContents(ClassNode.java:1035)
org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClass(ClassCodeVisitorSupport.java:50)
org.codehaus.groovy.classgen.AsmClassGenerator.visitClass(AsmClassGenerator.java:276)
org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:748)
org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:942)
org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:519)
org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:497)
org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:474)
groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:315)
groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:289)
groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:270)
groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:266)
groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:808)
groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:765)
groovy.lang.GroovyClassLoader$InnerLoader.loadClass(GroovyClassLoader.java:458)
groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:827)
java.lang.ClassLoader.loadClass(ClassLoader.java:248)
java.lang.Class.forName0(native method)
java.lang.Class.forName(Class.java:169)
test.B2.class$(test.B2)
test.B2.$get$$class$test$B4(test.B2)
test.B2.<init>(test.B2:5)
sun.reflect.NativeConstructorAccessorImpl.newInstance0(native method)
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
java.lang.reflect.Constructor.newInstance(Constructor.java:513)
test.GroovyEngineTest.compileAndInstantiate(GroovyEngineTest.java:105)
test.GroovyEngineTest.access$200(GroovyEngineTest.java:21)
test.GroovyEngineTest$2.run(GroovyEngineTest.java:128)

Thread-7 [BLOCKED; waiting to lock java.util.HashMap@4bb963c4]
groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:807)
groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:765)
groovy.lang.GroovyClassLoader$InnerLoader.loadClass(GroovyClassLoader.java:458)
groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:827)
java.lang.ClassLoader.loadClass(ClassLoader.java:248)
java.lang.Class.forName0(native method)
java.lang.Class.forName(Class.java:169)
test.C5.class$(test.C5)
test.C5.$get$$class$test$E5(test.C5)
test.C5.<init>(test.C5:4)
sun.reflect.NativeConstructorAccessorImpl.newInstance0(native method)
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
java.lang.reflect.Constructor.newInstance(Constructor.java:513)
test.GroovyEngineTest.compileAndInstantiate(GroovyEngineTest.java:105)
test.GroovyEngineTest.access$200(GroovyEngineTest.java:21)
test.GroovyEngineTest$2.run(GroovyEngineTest.java:128)

I'm sorry but patch didn't help.

Answering your previous suggestion to create new GroovyClassLoader instead of clearCache ... I have doubts that need to be resolved first:

1) It does not look good because there's no point for InnerLoader to exist anymore (because without clearing cache same class won't be loaded twice)
2) If I did it, I would like to create a chain of GroovyClassLoader where old one always points to new one (as parent) so threads any time they use GroovyClassLoader first time, this class loader is mapped to this thread and any future usages of loader uses mapped GroovyClassLoader (to end of life of such thread) and moreover if there is a class missing in a loader, the class should be taken from future loaders (following parents path) - GroovyClassLoader does not have possibility to change parent loader

Show
Szymon Kuklewicz added a comment - Deadlock: Thread-3 is waiting to lock groovy.lang.GroovyClassLoader$InnerLoader@2321b59a which is held by Thread-7 Thread-7 is waiting to lock java.util.HashMap@4bb963c4 which is held by Thread-3 Thread stacks Thread-3 [BLOCKED; waiting to lock groovy.lang.GroovyClassLoader$InnerLoader@2321b59a] java.lang.Class.getDeclaredFields0(native method) java.lang.Class.privateGetDeclaredFields(Class.java:2291) java.lang.Class.getDeclaredFields(Class.java:1743) org.codehaus.groovy.vmplugin.v5.Java5.configureClassNode(Java5.java:313) org.codehaus.groovy.ast.ClassNode.lazyClassInit(ClassNode.java:263) org.codehaus.groovy.ast.ClassNode.getInterfaces(ClassNode.java:341) org.codehaus.groovy.ast.ClassNode.declaresInterface(ClassNode.java:929) org.codehaus.groovy.ast.ClassNode.implementsInterface(ClassNode.java:909) org.codehaus.groovy.classgen.AsmClassGenerator.doConvertAndCast(AsmClassGenerator.java:3842) org.codehaus.groovy.classgen.AsmClassGenerator.doConvertAndCast(AsmClassGenerator.java:3837) org.codehaus.groovy.classgen.AsmClassGenerator.storeThisInstanceField(AsmClassGenerator.java:2840) org.codehaus.groovy.classgen.AsmClassGenerator.visitFieldExpression(AsmClassGenerator.java:2766) org.codehaus.groovy.ast.expr.FieldExpression.visit(FieldExpression.java:38) org.codehaus.groovy.classgen.AsmClassGenerator.evaluateEqual(AsmClassGenerator.java:4050) org.codehaus.groovy.classgen.AsmClassGenerator.visitBinaryExpression(AsmClassGenerator.java:1485) org.codehaus.groovy.ast.expr.BinaryExpression.visit(BinaryExpression.java:49) org.codehaus.groovy.classgen.AsmClassGenerator.visitAndAutoboxBoolean(AsmClassGenerator.java:4122) org.codehaus.groovy.classgen.AsmClassGenerator.visitExpressionStatement(AsmClassGenerator.java:1466) org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:40) org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:35) org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:165) org.codehaus.groovy.classgen.AsmClassGenerator.visitBlockStatement(AsmClassGenerator.java:738) org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:69) org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClassCodeContainer(ClassCodeVisitorSupport.java:101) org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitConstructorOrMethod(ClassCodeVisitorSupport.java:112) org.codehaus.groovy.classgen.AsmClassGenerator.visitStdMethod(AsmClassGenerator.java:626) org.codehaus.groovy.classgen.AsmClassGenerator.visitConstructorOrMethod(AsmClassGenerator.java:601) org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitConstructor(ClassCodeVisitorSupport.java:119) org.codehaus.groovy.classgen.AsmClassGenerator.visitConstructor(AsmClassGenerator.java:688) org.codehaus.groovy.ast.ClassNode.visitContents(ClassNode.java:1035) org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClass(ClassCodeVisitorSupport.java:50) org.codehaus.groovy.classgen.AsmClassGenerator.visitClass(AsmClassGenerator.java:276) org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:748) org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:942) org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:519) org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:497) org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:474) groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:315) groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:289) groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:270) groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:266) groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:808) groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:765) groovy.lang.GroovyClassLoader$InnerLoader.loadClass(GroovyClassLoader.java:458) groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:827) java.lang.ClassLoader.loadClass(ClassLoader.java:248) java.lang.Class.forName0(native method) java.lang.Class.forName(Class.java:169) test.B2.class$(test.B2) test.B2.$get$$class$test$B4(test.B2) test.B2.<init>(test.B2:5) sun.reflect.NativeConstructorAccessorImpl.newInstance0(native method) sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) java.lang.reflect.Constructor.newInstance(Constructor.java:513) test.GroovyEngineTest.compileAndInstantiate(GroovyEngineTest.java:105) test.GroovyEngineTest.access$200(GroovyEngineTest.java:21) test.GroovyEngineTest$2.run(GroovyEngineTest.java:128) Thread-7 [BLOCKED; waiting to lock java.util.HashMap@4bb963c4] groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:807) groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:765) groovy.lang.GroovyClassLoader$InnerLoader.loadClass(GroovyClassLoader.java:458) groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:827) java.lang.ClassLoader.loadClass(ClassLoader.java:248) java.lang.Class.forName0(native method) java.lang.Class.forName(Class.java:169) test.C5.class$(test.C5) test.C5.$get$$class$test$E5(test.C5) test.C5.<init>(test.C5:4) sun.reflect.NativeConstructorAccessorImpl.newInstance0(native method) sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) java.lang.reflect.Constructor.newInstance(Constructor.java:513) test.GroovyEngineTest.compileAndInstantiate(GroovyEngineTest.java:105) test.GroovyEngineTest.access$200(GroovyEngineTest.java:21) test.GroovyEngineTest$2.run(GroovyEngineTest.java:128) I'm sorry but patch didn't help. Answering your previous suggestion to create new GroovyClassLoader instead of clearCache ... I have doubts that need to be resolved first: 1) It does not look good because there's no point for InnerLoader to exist anymore (because without clearing cache same class won't be loaded twice) 2) If I did it, I would like to create a chain of GroovyClassLoader where old one always points to new one (as parent) so threads any time they use GroovyClassLoader first time, this class loader is mapped to this thread and any future usages of loader uses mapped GroovyClassLoader (to end of life of such thread) and moreover if there is a class missing in a loader, the class should be taken from future loaders (following parents path) - GroovyClassLoader does not have possibility to change parent loader
Hide
Permalink
blackdrag blackdrag added a comment -

If I see it right, then basically the problem is that we have an class and a dependency is outdated, but the class itself is also actually outdated, it exists only because he test program got a reference to it. The right thing to do for that would actually be to return the old version of the class here. A crazy idea would be to let the InnerLoader have a copy of the classCache from the time it was created and then answer loadClass requests directly from there instead of causing a fatal recompilation.

We will produce a patch later today for that I think.

About using a new GCL... your point (1) sounds like if you are afraid a dependency to a class will not exist anymore. There is a classCache that keeps references to defined classes. As long as this exists the InnerLoader used to defined them cannot be collected. So if you have a reference to one of those classes or loaders, all of them are not collected, since every InnerLoader references the GCL. One important part for it not blowing up though is that if none of the classes is referenced anymore that this all can be collected. If not your test would otherwise cause memory problems over time. And that is why (2) makes imho no sense.

Show
blackdrag blackdrag added a comment - If I see it right, then basically the problem is that we have an class and a dependency is outdated, but the class itself is also actually outdated, it exists only because he test program got a reference to it. The right thing to do for that would actually be to return the old version of the class here. A crazy idea would be to let the InnerLoader have a copy of the classCache from the time it was created and then answer loadClass requests directly from there instead of causing a fatal recompilation. We will produce a patch later today for that I think. About using a new GCL... your point (1) sounds like if you are afraid a dependency to a class will not exist anymore. There is a classCache that keeps references to defined classes. As long as this exists the InnerLoader used to defined them cannot be collected. So if you have a reference to one of those classes or loaders, all of them are not collected, since every InnerLoader references the GCL. One important part for it not blowing up though is that if none of the classes is referenced anymore that this all can be collected. If not your test would otherwise cause memory problems over time. And that is why (2) makes imho no sense.
Hide
Permalink
Guillaume Laforge added a comment -

Could you please try this new patch?
It's more of a hack for now, but if it's working for your use case, that'd be interesting to know.
Thanks in advance.

Show
Guillaume Laforge added a comment - Could you please try this new patch? It's more of a hack for now, but if it's working for your use case, that'd be interesting to know. Thanks in advance.
Hide
Permalink
Szymon Kuklewicz added a comment -

Well

With recent patch it's much better now. Average time to deadlock raised from 2-3 seconds to 20-40 seconds. It was good shot

I didn't have time to analyze stack traces because i'm going home soon but I hope the new deadlock is about resources that this time we control.

Multiple threads with such stack
groovy.lang.GroovyClassLoader.recompile(URL, String, Class)
groovy.lang.GroovyClassLoader.loadClass(String, boolean, boolean, boolean)
groovy.lang.GroovyClassLoader.loadClass(String, boolean)
java.lang.ClassLoader.loadClass(String)
test.GroovyEngineTest.compileAndInstantiate()
test.GroovyEngineTest.access$200(GroovyEngineTest)
test.GroovyEngineTest$2.run()

and last 3 were different

Thread-2 [WAITING] CPU time: 0:03
sun.misc.Unsafe.park(boolean, long)
java.util.concurrent.locks.LockSupport.park(Object)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(int)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(int)
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock()
groovy.lang.GroovyClassLoader.getClassCacheEntry(String)
groovy.lang.GroovyClassLoader.loadClass(String, boolean, boolean, boolean)
groovy.lang.GroovyClassLoader$InnerLoader.loadClass(String, boolean, boolean, boolean)
groovy.lang.GroovyClassLoader.loadClass(String, boolean)
java.lang.ClassLoader.loadClass(String)
java.lang.Class.getDeclaredConstructors0(boolean)
java.lang.Class.privateGetDeclaredConstructors(boolean)
java.lang.Class.getConstructor0(Class[], int)
java.lang.Class.getConstructor(Class[])
test.GroovyEngineTest.compileAndInstantiate()
test.GroovyEngineTest.access$200(GroovyEngineTest)
test.GroovyEngineTest$2.run()

Thread-3 [BLOCKED] CPU time: 0:02
java.lang.Class.getDeclaredMethods0(boolean)
java.lang.Class.privateGetDeclaredMethods(boolean)
java.lang.Class.getDeclaredMethods()
groovy.lang.GroovyClassLoader.doParseClass(GroovyCodeSource)
groovy.lang.GroovyClassLoader.parseClass(GroovyCodeSource, boolean)
groovy.lang.GroovyClassLoader.parseClass(GroovyCodeSource)
groovy.lang.GroovyClassLoader.parseClass(InputStream, String)
groovy.lang.GroovyClassLoader.recompile(URL, String, Class)
groovy.lang.GroovyClassLoader.loadClass(String, boolean, boolean, boolean)
groovy.lang.GroovyClassLoader.loadClass(String, boolean)
java.lang.ClassLoader.loadClass(String)
test.GroovyEngineTest.compileAndInstantiate()
test.GroovyEngineTest.access$200(GroovyEngineTest)
test.GroovyEngineTest$2.run()

Thread-8 [WAITING] CPU time: 0:02
sun.misc.Unsafe.park(boolean, long)
java.util.concurrent.locks.LockSupport.park(Object)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer$Node, int)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(int)
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock()
groovy.lang.GroovyClassLoader.clearCache()
test.GroovyEngineTest.writeAndClearCache()
test.GroovyEngineTest.access$100(GroovyEngineTest)
test.GroovyEngineTest$2.run()

Anyway, I changed in our application server that anytime some source change, new instance of GroocyClassLoader is created. Of course few days must pass to know if that helped. In real environment deadlocks are less frequently than this test.

Show
Szymon Kuklewicz added a comment - Well With recent patch it's much better now. Average time to deadlock raised from 2-3 seconds to 20-40 seconds. It was good shot I didn't have time to analyze stack traces because i'm going home soon but I hope the new deadlock is about resources that this time we control. Multiple threads with such stack groovy.lang.GroovyClassLoader.recompile(URL, String, Class) groovy.lang.GroovyClassLoader.loadClass(String, boolean, boolean, boolean) groovy.lang.GroovyClassLoader.loadClass(String, boolean) java.lang.ClassLoader.loadClass(String) test.GroovyEngineTest.compileAndInstantiate() test.GroovyEngineTest.access$200(GroovyEngineTest) test.GroovyEngineTest$2.run() and last 3 were different Thread-2 [WAITING] CPU time: 0:03 sun.misc.Unsafe.park(boolean, long) java.util.concurrent.locks.LockSupport.park(Object) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(int) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(int) java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock() groovy.lang.GroovyClassLoader.getClassCacheEntry(String) groovy.lang.GroovyClassLoader.loadClass(String, boolean, boolean, boolean) groovy.lang.GroovyClassLoader$InnerLoader.loadClass(String, boolean, boolean, boolean) groovy.lang.GroovyClassLoader.loadClass(String, boolean) java.lang.ClassLoader.loadClass(String) java.lang.Class.getDeclaredConstructors0(boolean) java.lang.Class.privateGetDeclaredConstructors(boolean) java.lang.Class.getConstructor0(Class[], int) java.lang.Class.getConstructor(Class[]) test.GroovyEngineTest.compileAndInstantiate() test.GroovyEngineTest.access$200(GroovyEngineTest) test.GroovyEngineTest$2.run() Thread-3 [BLOCKED] CPU time: 0:02 java.lang.Class.getDeclaredMethods0(boolean) java.lang.Class.privateGetDeclaredMethods(boolean) java.lang.Class.getDeclaredMethods() groovy.lang.GroovyClassLoader.doParseClass(GroovyCodeSource) groovy.lang.GroovyClassLoader.parseClass(GroovyCodeSource, boolean) groovy.lang.GroovyClassLoader.parseClass(GroovyCodeSource) groovy.lang.GroovyClassLoader.parseClass(InputStream, String) groovy.lang.GroovyClassLoader.recompile(URL, String, Class) groovy.lang.GroovyClassLoader.loadClass(String, boolean, boolean, boolean) groovy.lang.GroovyClassLoader.loadClass(String, boolean) java.lang.ClassLoader.loadClass(String) test.GroovyEngineTest.compileAndInstantiate() test.GroovyEngineTest.access$200(GroovyEngineTest) test.GroovyEngineTest$2.run() Thread-8 [WAITING] CPU time: 0:02 sun.misc.Unsafe.park(boolean, long) java.util.concurrent.locks.LockSupport.park(Object) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer$Node, int) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(int) java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock() groovy.lang.GroovyClassLoader.clearCache() test.GroovyEngineTest.writeAndClearCache() test.GroovyEngineTest.access$100(GroovyEngineTest) test.GroovyEngineTest$2.run() Anyway, I changed in our application server that anytime some source change, new instance of GroocyClassLoader is created. Of course few days must pass to know if that helped. In real environment deadlocks are less frequently than this test.
Hide
Permalink
Szymon Kuklewicz added a comment -

Bad news. We encountered next deadlocks on our application server. Patches you gave me weren't there uploaded but new GroovyClassLoader was created after any modification, cache was not cleared and even recompilation was disabled!

Thread ID: 50
java.lang.ClassLoader.loadClass(ClassLoader.java:292)
groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:696)
groovy.lang.GroovyClassLoader$InnerLoader.loadClass(GroovyClassLoader.java:449)
groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:793)
java.lang.ClassLoader.loadClass(ClassLoader.java:248)
java.lang.Class.getDeclaredConstructors0(Native Method)
java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
java.lang.Class.getConstructor0(Class.java:2699)
java.lang.Class.newInstance0(Class.java:326)
java.lang.Class.newInstance(Class.java:308)
palio.pelements.PObject.executeAsRoot(PObject.java:273)
palio.pelements.PPage.executeBody(PPage.java:285)
palio.pelements.PPage.writeBody(PPage.java:236)
html.run.buildResponse(run.java:359)
html.run.service(run.java:443)
javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

Thread ID: 48
java.lang.Class.getDeclaredFields0(Native Method)
java.lang.Class.privateGetDeclaredFields(Class.java:2291)
java.lang.Class.getDeclaredFields(Class.java:1743)
org.codehaus.groovy.vmplugin.v5.Java5.configureClassNode(Java5.java:313)
org.codehaus.groovy.ast.ClassNode.lazyClassInit(ClassNode.java:263)
org.codehaus.groovy.ast.ClassNode.getUnresolvedSuperClass(ClassNode.java:957)
org.codehaus.groovy.ast.ClassNode.getUnresolvedSuperClass(ClassNode.java:952)
org.codehaus.groovy.control.ResolveVisitor.checkCyclicInheritence(ResolveVisitor.java:1309)
org.codehaus.groovy.control.ResolveVisitor.visitClass(ResolveVisitor.java:1286)
org.codehaus.groovy.control.ResolveVisitor.startResolving(ResolveVisitor.java:148)
org.codehaus.groovy.control.CompilationUnit$6.call(CompilationUnit.java:574)
org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:814)
org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:511)
org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:487)
org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:464)
groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:306)
groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:283)
palio.compiler.groovy.GroovyEngine$MyGroovyClassLoader.parseClass(GroovyEngine.java:75)
groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:267)
groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:263)
palio.compiler.groovy.GroovyEngine$MyGroovyClassLoader.parseClass(GroovyEngine.java:69)
groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:777)
groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:737)
groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:793)
palio.compiler.groovy.GroovyEngine$MyGroovyClassLoader.loadClass(GroovyEngine.java:61)
java.lang.ClassLoader.loadClass(ClassLoader.java:248)
palio.compiler.groovy.GroovyEngine.getCompiledClass(GroovyEngine.java:236)
palio.pelements.PObject.precompile(PObject.java:308)
palio.pelements.PObject.executeAsRoot(PObject.java:268)
palio.pelements.PPage.executeBody(PPage.java:285)
palio.pelements.PPage.writeBody(PPage.java:236)
html.run.buildResponse(run.java:359)
html.run.service(run.java:443)
javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

I know that last patch probably fix it and with combination of disabled recompilation/no cache clearing there should be OK but I would like to ask you if you could prepare LightweightGroovyClassLoader where recompilation is not possible, clearing cache is not possible and thus ... InnerLoader is not used? That would rid off deadlocks for sure.

Show
Szymon Kuklewicz added a comment - Bad news. We encountered next deadlocks on our application server. Patches you gave me weren't there uploaded but new GroovyClassLoader was created after any modification, cache was not cleared and even recompilation was disabled! Thread ID: 50 java.lang.ClassLoader.loadClass(ClassLoader.java:292) groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:696) groovy.lang.GroovyClassLoader$InnerLoader.loadClass(GroovyClassLoader.java:449) groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:793) java.lang.ClassLoader.loadClass(ClassLoader.java:248) java.lang.Class.getDeclaredConstructors0(Native Method) java.lang.Class.privateGetDeclaredConstructors(Class.java:2389) java.lang.Class.getConstructor0(Class.java:2699) java.lang.Class.newInstance0(Class.java:326) java.lang.Class.newInstance(Class.java:308) palio.pelements.PObject.executeAsRoot(PObject.java:273) palio.pelements.PPage.executeBody(PPage.java:285) palio.pelements.PPage.writeBody(PPage.java:236) html.run.buildResponse(run.java:359) html.run.service(run.java:443) javax.servlet.http.HttpServlet.service(HttpServlet.java:717) Thread ID: 48 java.lang.Class.getDeclaredFields0(Native Method) java.lang.Class.privateGetDeclaredFields(Class.java:2291) java.lang.Class.getDeclaredFields(Class.java:1743) org.codehaus.groovy.vmplugin.v5.Java5.configureClassNode(Java5.java:313) org.codehaus.groovy.ast.ClassNode.lazyClassInit(ClassNode.java:263) org.codehaus.groovy.ast.ClassNode.getUnresolvedSuperClass(ClassNode.java:957) org.codehaus.groovy.ast.ClassNode.getUnresolvedSuperClass(ClassNode.java:952) org.codehaus.groovy.control.ResolveVisitor.checkCyclicInheritence(ResolveVisitor.java:1309) org.codehaus.groovy.control.ResolveVisitor.visitClass(ResolveVisitor.java:1286) org.codehaus.groovy.control.ResolveVisitor.startResolving(ResolveVisitor.java:148) org.codehaus.groovy.control.CompilationUnit$6.call(CompilationUnit.java:574) org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:814) org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:511) org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:487) org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:464) groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:306) groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:283) palio.compiler.groovy.GroovyEngine$MyGroovyClassLoader.parseClass(GroovyEngine.java:75) groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:267) groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:263) palio.compiler.groovy.GroovyEngine$MyGroovyClassLoader.parseClass(GroovyEngine.java:69) groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:777) groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:737) groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:793) palio.compiler.groovy.GroovyEngine$MyGroovyClassLoader.loadClass(GroovyEngine.java:61) java.lang.ClassLoader.loadClass(ClassLoader.java:248) palio.compiler.groovy.GroovyEngine.getCompiledClass(GroovyEngine.java:236) palio.pelements.PObject.precompile(PObject.java:308) palio.pelements.PObject.executeAsRoot(PObject.java:268) palio.pelements.PPage.executeBody(PPage.java:285) palio.pelements.PPage.writeBody(PPage.java:236) html.run.buildResponse(run.java:359) html.run.service(run.java:443) javax.servlet.http.HttpServlet.service(HttpServlet.java:717) I know that last patch probably fix it and with combination of disabled recompilation/no cache clearing there should be OK but I would like to ask you if you could prepare LightweightGroovyClassLoader where recompilation is not possible, clearing cache is not possible and thus ... InnerLoader is not used? That would rid off deadlocks for sure.
Hide
Permalink
Guillaume Laforge added a comment -

Could you try on your app server with the patch as well?

Show
Guillaume Laforge added a comment - Could you try on your app server with the patch as well?
Hide
Permalink
Szymon Kuklewicz added a comment -

Yes. Of course. I started server using Groovy with patch included and there haven't been any deadlock since yesterday (but were not been heavily loaded that time)

Show
Szymon Kuklewicz added a comment - Yes. Of course. I started server using Groovy with patch included and there haven't been any deadlock since yesterday (but were not been heavily loaded that time)
Hide
Permalink
Guillaume Laforge added a comment -

Keep me updated, so that I can see whether I should include that patch in Groovy or not.

Show
Guillaume Laforge added a comment - Keep me updated, so that I can see whether I should include that patch in Groovy or not.
Hide
Permalink
Guillaume Laforge added a comment -

Still no deadlock so far with that latest patch?

Show
Guillaume Laforge added a comment - Still no deadlock so far with that latest patch?
Hide
Permalink
Szymon Kuklewicz added a comment -

No more deadlock so far. Normally should already happen few times so I think the second patch really helps. Thanks

Show
Szymon Kuklewicz added a comment - No more deadlock so far. Normally should already happen few times so I think the second patch really helps. Thanks
Hide
Permalink
Guillaume Laforge added a comment -

Okie dokie, good to know! Thanks for reporting that news back to me.
That patch is kinda... "interesting", in the sense that it's more of a hack to circumvent the weird locking issues happening inside JDK native code... but well, so far that's the sole workaround we've found, without too huge changes in our codebase.
Hopefully we'll find a better solution at some point.
So in the meantime, we'll certainly add that patch in the next versions.

Show
Guillaume Laforge added a comment - Okie dokie, good to know! Thanks for reporting that news back to me. That patch is kinda... "interesting", in the sense that it's more of a hack to circumvent the weird locking issues happening inside JDK native code... but well, so far that's the sole workaround we've found, without too huge changes in our codebase. Hopefully we'll find a better solution at some point. So in the meantime, we'll certainly add that patch in the next versions.
Hide
Permalink
Szymon Kuklewicz added a comment -

Unfortunately we encountered a deadlock. This time it's deadlock between outer and inner loader. Because it's another kind of deadlock it explains why it shows much rarlier now. Please ignore fact that we use overriden implementation of GCL - MyGroovyClassLoader is only for extra security (restricted access only to dictionary packages) and does interfere with any synchronizations.

Thread t@218
java.lang.Thread.State: BLOCKED
at java.lang.ClassLoader.loadClass(ClassLoader.java:292)

  • waiting to lock <158f69e2> (a palio.compiler.groovy.GroovyEngine$MyGroovyClassLoader) owned by Thread t@215
    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:732)
    at groovy.lang.GroovyClassLoader$InnerLoader.loadClass(GroovyClassLoader.java:466)
    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:835)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:169)
    at eway.ui.web.portal.widget.coupon.CouponWidgetController.class$(eway.ui.web.portal.widget.coupon.CouponWidgetController.groovy)
    at eway.ui.web.portal.widget.coupon.CouponWidgetController.$get$$class$palio$Groovy(eway.ui.web.portal.widget.coupon.CouponWidgetController.groovy)
    at eway.ui.web.portal.widget.coupon.CouponWidgetController.<clinit>(eway.ui.web.portal.widget.coupon.CouponWidgetController.groovy:16)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at java.lang.Class.newInstance0(Class.java:355)
    at java.lang.Class.newInstance(Class.java:308)
    at palio.pelements.PObject.executeAsRoot(PObject.java:273)
    at palio.pelements.PPage.executeBody(PPage.java:285)
    at palio.pelements.PPage.writeBody(PPage.java:236)
    at html.run.buildResponse(run.java:359)
    at html.run.service(run.java:443)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

Locked ownable synchronizers:

  • None

Thread t@215
java.lang.Thread.State: BLOCKED
at java.lang.Class.forName0(Native Method)

  • waiting to lock <1b234cfc> (a groovy.lang.GroovyClassLoader$InnerLoader) owned by Thread t@218
    at java.lang.Class.forName(Class.java:247)
    at sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:95)
    at sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:107)
    at sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:31)
    at sun.reflect.annotation.AnnotationParser.parseSig(AnnotationParser.java:370)
    at sun.reflect.annotation.AnnotationParser.parseAnnotation(AnnotationParser.java:181)
    at sun.reflect.annotation.AnnotationParser.parseAnnotations2(AnnotationParser.java:69)
    at sun.reflect.annotation.AnnotationParser.parseAnnotations(AnnotationParser.java:52)
    at java.lang.reflect.Method.declaredAnnotations(Method.java:693)
  • locked <1c207eea> (a java.lang.reflect.Method)
    at java.lang.reflect.Method.getDeclaredAnnotations(Method.java:686)
    at java.lang.reflect.AccessibleObject.getAnnotations(AccessibleObject.java:175)
    at org.codehaus.groovy.vmplugin.v5.Java5.configureClassNode(Java5.java:325)
    at org.codehaus.groovy.ast.ClassNode.lazyClassInit(ClassNode.java:263)
  • locked <12a7a16f> (a java.lang.Object)
    at org.codehaus.groovy.ast.ClassNode.getInterfaces(ClassNode.java:341)
    at org.codehaus.groovy.ast.ClassNode.declaresInterface(ClassNode.java:929)
    at org.codehaus.groovy.ast.ClassNode.implementsInterface(ClassNode.java:909)
    at org.codehaus.groovy.ast.ClassNode.isDerivedFromGroovyObject(ClassNode.java:899)
    at org.codehaus.groovy.classgen.AsmClassGenerator.loadWrapper(AsmClassGenerator.java:3225)
    at org.codehaus.groovy.classgen.AsmClassGenerator.makeCallSite(AsmClassGenerator.java:2183)
    at org.codehaus.groovy.classgen.AsmClassGenerator.makeCall(AsmClassGenerator.java:2019)
    at org.codehaus.groovy.classgen.AsmClassGenerator.makeCall(AsmClassGenerator.java:2005)
    at org.codehaus.groovy.classgen.AsmClassGenerator.makeInvokeMethodCall(AsmClassGenerator.java:1990)
    at org.codehaus.groovy.classgen.AsmClassGenerator.visitMethodCallExpression(AsmClassGenerator.java:2342)
    at org.codehaus.groovy.ast.expr.MethodCallExpression.visit(MethodCallExpression.java:75)
    at org.codehaus.groovy.classgen.AsmClassGenerator.visitAndAutoboxBoolean(AsmClassGenerator.java:4122)
    at org.codehaus.groovy.classgen.AsmClassGenerator.evaluateExpression(AsmClassGenerator.java:1447)
    at org.codehaus.groovy.classgen.AsmClassGenerator.visitReturnStatement(AsmClassGenerator.java:1408)
    at org.codehaus.groovy.ast.stmt.ReturnStatement.visit(ReturnStatement.java:47)
    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClassCodeContainer(ClassCodeVisitorSupport.java:101)
    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitConstructorOrMethod(ClassCodeVisitorSupport.java:112)
    at org.codehaus.groovy.classgen.AsmClassGenerator.visitStdMethod(AsmClassGenerator.java:626)
    at org.codehaus.groovy.classgen.AsmClassGenerator.visitConstructorOrMethod(AsmClassGenerator.java:601)
    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitMethod(ClassCodeVisitorSupport.java:123)
    at org.codehaus.groovy.classgen.AsmClassGenerator.visitMethod(AsmClassGenerator.java:696)
    at org.codehaus.groovy.ast.ClassNode.visitContents(ClassNode.java:1039)
    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClass(ClassCodeVisitorSupport.java:50)
    at org.codehaus.groovy.classgen.AsmClassGenerator.visitClass(AsmClassGenerator.java:276)
    at org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:748)
    at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:942)
    at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:519)
    at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:497)
    at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:474)
    at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:315)
    at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:289)
  • locked <3b5787fa> (a java.util.HashMap)
    at palio.compiler.groovy.GroovyEngine$MyGroovyClassLoader.parseClass(GroovyEngine.java:75)
    at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:270)
    at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:266)
    at palio.compiler.groovy.GroovyEngine$MyGroovyClassLoader.parseClass(GroovyEngine.java:69)
    at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:816)
  • locked <3b5787fa> (a java.util.HashMap)
    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:773)
    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:835)
    at palio.compiler.groovy.GroovyEngine$MyGroovyClassLoader.loadClass(GroovyEngine.java:61)
  • locked <158f69e2> (a palio.compiler.groovy.GroovyEngine$MyGroovyClassLoader)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
    at palio.compiler.groovy.GroovyEngine.getCompiledClass(GroovyEngine.java:236)
    at palio.pelements.PObject.precompile(PObject.java:308)
    at palio.pelements.PObject.executeAsRoot(PObject.java:268)
    at palio.pelements.PPage.executeBody(PPage.java:285)
    at palio.pelements.PPage.writeBody(PPage.java:236)
    at html.run.buildResponse(run.java:359)
    at html.run.service(run.java:443)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

Locked ownable synchronizers:

  • None
Show
Szymon Kuklewicz added a comment - Unfortunately we encountered a deadlock. This time it's deadlock between outer and inner loader. Because it's another kind of deadlock it explains why it shows much rarlier now. Please ignore fact that we use overriden implementation of GCL - MyGroovyClassLoader is only for extra security (restricted access only to dictionary packages) and does interfere with any synchronizations. Thread t@218 java.lang.Thread.State: BLOCKED at java.lang.ClassLoader.loadClass(ClassLoader.java:292) waiting to lock <158f69e2> (a palio.compiler.groovy.GroovyEngine$MyGroovyClassLoader) owned by Thread t@215 at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:732) at groovy.lang.GroovyClassLoader$InnerLoader.loadClass(GroovyClassLoader.java:466) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:835) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:169) at eway.ui.web.portal.widget.coupon.CouponWidgetController.class$(eway.ui.web.portal.widget.coupon.CouponWidgetController.groovy) at eway.ui.web.portal.widget.coupon.CouponWidgetController.$get$$class$palio$Groovy(eway.ui.web.portal.widget.coupon.CouponWidgetController.groovy) at eway.ui.web.portal.widget.coupon.CouponWidgetController.<clinit>(eway.ui.web.portal.widget.coupon.CouponWidgetController.groovy:16) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at java.lang.Class.newInstance0(Class.java:355) at java.lang.Class.newInstance(Class.java:308) at palio.pelements.PObject.executeAsRoot(PObject.java:273) at palio.pelements.PPage.executeBody(PPage.java:285) at palio.pelements.PPage.writeBody(PPage.java:236) at html.run.buildResponse(run.java:359) at html.run.service(run.java:443) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) Locked ownable synchronizers: None Thread t@215 java.lang.Thread.State: BLOCKED at java.lang.Class.forName0(Native Method) waiting to lock <1b234cfc> (a groovy.lang.GroovyClassLoader$InnerLoader) owned by Thread t@218 at java.lang.Class.forName(Class.java:247) at sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:95) at sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:107) at sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:31) at sun.reflect.annotation.AnnotationParser.parseSig(AnnotationParser.java:370) at sun.reflect.annotation.AnnotationParser.parseAnnotation(AnnotationParser.java:181) at sun.reflect.annotation.AnnotationParser.parseAnnotations2(AnnotationParser.java:69) at sun.reflect.annotation.AnnotationParser.parseAnnotations(AnnotationParser.java:52) at java.lang.reflect.Method.declaredAnnotations(Method.java:693) locked <1c207eea> (a java.lang.reflect.Method) at java.lang.reflect.Method.getDeclaredAnnotations(Method.java:686) at java.lang.reflect.AccessibleObject.getAnnotations(AccessibleObject.java:175) at org.codehaus.groovy.vmplugin.v5.Java5.configureClassNode(Java5.java:325) at org.codehaus.groovy.ast.ClassNode.lazyClassInit(ClassNode.java:263) locked <12a7a16f> (a java.lang.Object) at org.codehaus.groovy.ast.ClassNode.getInterfaces(ClassNode.java:341) at org.codehaus.groovy.ast.ClassNode.declaresInterface(ClassNode.java:929) at org.codehaus.groovy.ast.ClassNode.implementsInterface(ClassNode.java:909) at org.codehaus.groovy.ast.ClassNode.isDerivedFromGroovyObject(ClassNode.java:899) at org.codehaus.groovy.classgen.AsmClassGenerator.loadWrapper(AsmClassGenerator.java:3225) at org.codehaus.groovy.classgen.AsmClassGenerator.makeCallSite(AsmClassGenerator.java:2183) at org.codehaus.groovy.classgen.AsmClassGenerator.makeCall(AsmClassGenerator.java:2019) at org.codehaus.groovy.classgen.AsmClassGenerator.makeCall(AsmClassGenerator.java:2005) at org.codehaus.groovy.classgen.AsmClassGenerator.makeInvokeMethodCall(AsmClassGenerator.java:1990) at org.codehaus.groovy.classgen.AsmClassGenerator.visitMethodCallExpression(AsmClassGenerator.java:2342) at org.codehaus.groovy.ast.expr.MethodCallExpression.visit(MethodCallExpression.java:75) at org.codehaus.groovy.classgen.AsmClassGenerator.visitAndAutoboxBoolean(AsmClassGenerator.java:4122) at org.codehaus.groovy.classgen.AsmClassGenerator.evaluateExpression(AsmClassGenerator.java:1447) at org.codehaus.groovy.classgen.AsmClassGenerator.visitReturnStatement(AsmClassGenerator.java:1408) at org.codehaus.groovy.ast.stmt.ReturnStatement.visit(ReturnStatement.java:47) at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClassCodeContainer(ClassCodeVisitorSupport.java:101) at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitConstructorOrMethod(ClassCodeVisitorSupport.java:112) at org.codehaus.groovy.classgen.AsmClassGenerator.visitStdMethod(AsmClassGenerator.java:626) at org.codehaus.groovy.classgen.AsmClassGenerator.visitConstructorOrMethod(AsmClassGenerator.java:601) at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitMethod(ClassCodeVisitorSupport.java:123) at org.codehaus.groovy.classgen.AsmClassGenerator.visitMethod(AsmClassGenerator.java:696) at org.codehaus.groovy.ast.ClassNode.visitContents(ClassNode.java:1039) at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClass(ClassCodeVisitorSupport.java:50) at org.codehaus.groovy.classgen.AsmClassGenerator.visitClass(AsmClassGenerator.java:276) at org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:748) at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:942) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:519) at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:497) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:474) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:315) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:289) locked <3b5787fa> (a java.util.HashMap) at palio.compiler.groovy.GroovyEngine$MyGroovyClassLoader.parseClass(GroovyEngine.java:75) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:270) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:266) at palio.compiler.groovy.GroovyEngine$MyGroovyClassLoader.parseClass(GroovyEngine.java:69) at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:816) locked <3b5787fa> (a java.util.HashMap) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:773) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:835) at palio.compiler.groovy.GroovyEngine$MyGroovyClassLoader.loadClass(GroovyEngine.java:61) locked <158f69e2> (a palio.compiler.groovy.GroovyEngine$MyGroovyClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) at palio.compiler.groovy.GroovyEngine.getCompiledClass(GroovyEngine.java:236) at palio.pelements.PObject.precompile(PObject.java:308) at palio.pelements.PObject.executeAsRoot(PObject.java:268) at palio.pelements.PPage.executeBody(PPage.java:285) at palio.pelements.PPage.writeBody(PPage.java:236) at html.run.buildResponse(run.java:359) at html.run.service(run.java:443) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) Locked ownable synchronizers: None
Hide
Permalink
Szymon Kuklewicz added a comment -

Hi

Is there any progress in resolving this deadlock?
Stability of server is much greater but still deadlock happens once few days.

Show
Szymon Kuklewicz added a comment - Hi Is there any progress in resolving this deadlock? Stability of server is much greater but still deadlock happens once few days.
Hide
Permalink
blackdrag blackdrag added a comment -

If it is possible that first a lock on inner loader is acquired and then the thread tries to lock GCL, while another Thread is locking GCL and tries to acquire a lock on inner loader, then this is nothing that we can fix I believe. The locking is done through native code. It is different with Java7, which uses less locking. There this problem should not happen. Other than that, only a structural change can help.

How can this happen?

If you load a class from the inner loader using a class defined on inner loader, then it my cause for example through Class#forName0 a lock on that inner loader. If that class depends on a class from the outer loader, it may cause a similar lock on that loader. That's one direction.

The other direction is actually more difficult.. we start with a lock on GCL and then for example cause the init of a class defined in the innerLoader. This could happen through a compilation for example. The question is where this first lock should come from. GCL itself does not define classes, so any Class#forName0 way would start with the inner loader, same for bean property stuff and other reflection related locking.

Looking at your stack again makes me now realize, that there is:
at palio.compiler.groovy.GroovyEngine$MyGroovyClassLoader.loadClass(GroovyEngine.java:61)
locked <158f69e2> (a palio.compiler.groovy.GroovyEngine$MyGroovyClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at palio.compiler.groovy.GroovyEngine.getCompiledClass(GroovyEngine.java:236)

And I read it that ClassLoader#loadClass is calling a GCL#loadClass. After realizing this the reason becomes obvious to me... GCL is missing loadClass(String)! If GCL would override loadClass(String) without doing the locking, then this first lock cannot happen. If the first lock cannot happen, there will be no deadlock

Show
blackdrag blackdrag added a comment - If it is possible that first a lock on inner loader is acquired and then the thread tries to lock GCL, while another Thread is locking GCL and tries to acquire a lock on inner loader, then this is nothing that we can fix I believe. The locking is done through native code. It is different with Java7, which uses less locking. There this problem should not happen. Other than that, only a structural change can help. How can this happen? If you load a class from the inner loader using a class defined on inner loader, then it my cause for example through Class#forName0 a lock on that inner loader. If that class depends on a class from the outer loader, it may cause a similar lock on that loader. That's one direction. The other direction is actually more difficult.. we start with a lock on GCL and then for example cause the init of a class defined in the innerLoader. This could happen through a compilation for example. The question is where this first lock should come from. GCL itself does not define classes, so any Class#forName0 way would start with the inner loader, same for bean property stuff and other reflection related locking. Looking at your stack again makes me now realize, that there is: at palio.compiler.groovy.GroovyEngine$MyGroovyClassLoader.loadClass(GroovyEngine.java:61) locked <158f69e2> (a palio.compiler.groovy.GroovyEngine$MyGroovyClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) at palio.compiler.groovy.GroovyEngine.getCompiledClass(GroovyEngine.java:236) And I read it that ClassLoader#loadClass is calling a GCL#loadClass. After realizing this the reason becomes obvious to me... GCL is missing loadClass(String)! If GCL would override loadClass(String) without doing the locking, then this first lock cannot happen. If the first lock cannot happen, there will be no deadlock
Hide
Permalink
blackdrag blackdrag added a comment -

I made some changes to GroovyClassLoader that may or may not have improved the situation. Is it possible for someone here to test it?

Show
blackdrag blackdrag added a comment - I made some changes to GroovyClassLoader that may or may not have improved the situation. Is it possible for someone here to test it?

People

  • Assignee:
    blackdrag blackdrag
    Reporter:
    Szymon Kuklewicz
Vote (0)
Watch (5)

Dates

  • Created:
    Updated: