groovy

Concurrency problem when compiling scripts

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Critical Critical
  • Resolution: Fixed
  • Affects Version/s: 1.0-JSR-4
  • Fix Version/s: 1.0-JSR-5
  • Component/s: None
  • Labels:
    None
  • Environment:
    * Pentium IV 3.0Ghz HT, Windows XP, 1.5Gb Memory
    * Dual Pentium Xeon 3.0Ghz HT, Fedora 3 Core Linux, 4Gb Memory
  • Testcase included:
    yes
  • Number of attachments :
    1

Description

Apparently, there are a concurrency bug on GroovyShell that shows up when compiling scripts simultaneously on different threads.

I developed a test case that shows the problem.
Please, note that each thread creates its own GroovyShell instance and there is no shared data between threads. Even then, sometimes groovy shell throws and Exception.

I got 3 different exceptions already. The most common one is:

General error during class generation: java.util.ConcurrentModificationException

Less frequently, but also happens several times:

java.lang.NullPointerException

And an exception that I got only 3 times was:

java.lang.OutOfMemoryError

Concurrency problems, as you know are not deterministic by nature, so sometimes it happens, sometimes it doesn't. So, please execute the test case several times. Eventually it will raise one exception or another. Also, the computers where I'm running the test are all HyperThread, so they have (from an Operating System point of view) 2 and 4 processors, respectivelly. The problem happens both on Windows and Linux. Although, the OutOfMemoryError I got only in Linux.

Also, if I try to debug the test case, using Eclipse, the errors simply don't occur. As I don't know the groovy internals (I downloaded the source code), I was not able to advance much more on the problem analysis.

Bellow I paste the stack trace for the errors and I attached the test case.

This is affecting my production system using groovy. As a workaround I'm synchronizing all threads when parsing scripts, but this is critical for the application, so it is really not good to have to do this. This is why I cathegorized the issue as critical.

Let me know if you need any other information.

Thanks in advance.
Edson

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Edson Tirelli added a comment -

Sorry, forgot to attach the stack traces. Please note that sometimes it raises one Exception, sometimes the other, but I never received all at the same time.

------------------------------
java.lang.NullPointerException
at org.codehaus.groovy.ast.ClassNode.getMethods(ClassNode.java:549)
at org.codehaus.groovy.classgen.JSRVariableScopeCodeVisitor.checkMethodsForOverwritingFinal(JSRVariableScopeCodeVisitor.java:371)
at org.codehaus.groovy.classgen.JSRVariableScopeCodeVisitor.visitClass(JSRVariableScopeCodeVisitor.java:501)
at org.codehaus.groovy.control.CompilationUnit$5.call(CompilationUnit.java:680)
at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:974)
at org.codehaus.groovy.control.CompilationUnit.classgen(CompilationUnit.java:642)
at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:481)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:286)
at groovy.lang.GroovyShell.parseClass(GroovyShell.java:536)
at groovy.lang.GroovyShell.parse(GroovyShell.java:548)
at groovy.lang.GroovyShell.parse(GroovyShell.java:527)
at groovy.lang.GroovyShell.parse(GroovyShell.java:566)
at groovy.lang.GroovyShellConcurrentTest$ScriptCompiler.run(GroovyShellConcurrentTest.java:75)

-----------------
General error during class generation: java.util.ConcurrentModificationException

1 Error

at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:325)
at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1000)
at org.codehaus.groovy.control.CompilationUnit.classgen(CompilationUnit.java:642)
at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:481)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:286)
at groovy.lang.GroovyShell.parseClass(GroovyShell.java:536)
at groovy.lang.GroovyShell.parse(GroovyShell.java:548)
at groovy.lang.GroovyShell.parse(GroovyShell.java:527)
at groovy.lang.GroovyShell.parse(GroovyShell.java:566)
at groovy.lang.GroovyShellConcurrentTest$ScriptCompiler.run(GroovyShellConcurrentTest.java:75)

-----------------------------
Caused by: java.lang.OutOfMemoryError
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(ZipFile.java:204)
at java.util.jar.JarFile.<init>(JarFile.java:132)
at java.util.jar.JarFile.<init>(JarFile.java:97)
at groovy.lang.GroovyClassLoader.expandClassPath(GroovyClassLoader.java:500)
at groovy.lang.GroovyClassLoader.getClassPath(GroovyClassLoader.java:458)
at groovy.lang.GroovyClassLoader.getSourceFile(GroovyClassLoader.java:723)
at groovy.lang.GroovyClassLoader.access$000(GroovyClassLoader.java:85)
at groovy.lang.GroovyClassLoader$1.loadGroovySource(GroovyClassLoader.java:91)
at groovy.lang.GroovyClassLoader$3.run(GroovyClassLoader.java:662)
at java.security.AccessController.doPrivileged(Native Method)
at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:659)
at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:705)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.beans.Introspector.instantiate(Introspector.java:1425)
at java.beans.Introspector.findExplicitBeanInfo(Introspector.java:406)
at java.beans.Introspector.<init>(Introspector.java:355)
at java.beans.Introspector.getBeanInfo(Introspector.java:155)
at groovy.lang.MetaClassImpl$2.run(MetaClassImpl.java:143)
at java.security.AccessController.doPrivileged(Native Method)
at groovy.lang.MetaClassImpl.<init>(MetaClassImpl.java:141)
at groovy.lang.MetaClassRegistry.getMetaClassFor(MetaClassRegistry.java:277)
at groovy.lang.MetaClassRegistry.getMetaClass(MetaClassRegistry.java:144)
at org.codehaus.groovy.runtime.Invoker.getMetaClass(Invoker.java:104)
at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:87)
at groovy.lang.GroovyObjectSupport.<init>(GroovyObjectSupport.java:61)
at groovy.lang.Script.<init>(Script.java:65)
at Script5.<init>(Script5.groovy)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:494)
at java.lang.Class.newInstance0(Class.java:350)
at java.lang.Class.newInstance(Class.java:303)
at org.codehaus.groovy.runtime.InvokerHelper.createScript(InvokerHelper.java:529)
at groovy.lang.GroovyShell.parse(GroovyShell.java:548)
at groovy.lang.GroovyShell.parse(GroovyShell.java:527)
at groovy.lang.GroovyShell.parse(GroovyShell.java:566)

Show
Edson Tirelli added a comment - Sorry, forgot to attach the stack traces. Please note that sometimes it raises one Exception, sometimes the other, but I never received all at the same time. ------------------------------ java.lang.NullPointerException at org.codehaus.groovy.ast.ClassNode.getMethods(ClassNode.java:549) at org.codehaus.groovy.classgen.JSRVariableScopeCodeVisitor.checkMethodsForOverwritingFinal(JSRVariableScopeCodeVisitor.java:371) at org.codehaus.groovy.classgen.JSRVariableScopeCodeVisitor.visitClass(JSRVariableScopeCodeVisitor.java:501) at org.codehaus.groovy.control.CompilationUnit$5.call(CompilationUnit.java:680) at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:974) at org.codehaus.groovy.control.CompilationUnit.classgen(CompilationUnit.java:642) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:481) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:286) at groovy.lang.GroovyShell.parseClass(GroovyShell.java:536) at groovy.lang.GroovyShell.parse(GroovyShell.java:548) at groovy.lang.GroovyShell.parse(GroovyShell.java:527) at groovy.lang.GroovyShell.parse(GroovyShell.java:566) at groovy.lang.GroovyShellConcurrentTest$ScriptCompiler.run(GroovyShellConcurrentTest.java:75) ----------------- General error during class generation: java.util.ConcurrentModificationException 1 Error at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:325) at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1000) at org.codehaus.groovy.control.CompilationUnit.classgen(CompilationUnit.java:642) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:481) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:286) at groovy.lang.GroovyShell.parseClass(GroovyShell.java:536) at groovy.lang.GroovyShell.parse(GroovyShell.java:548) at groovy.lang.GroovyShell.parse(GroovyShell.java:527) at groovy.lang.GroovyShell.parse(GroovyShell.java:566) at groovy.lang.GroovyShellConcurrentTest$ScriptCompiler.run(GroovyShellConcurrentTest.java:75) ----------------------------- Caused by: java.lang.OutOfMemoryError at java.util.zip.ZipFile.open(Native Method) at java.util.zip.ZipFile.<init>(ZipFile.java:204) at java.util.jar.JarFile.<init>(JarFile.java:132) at java.util.jar.JarFile.<init>(JarFile.java:97) at groovy.lang.GroovyClassLoader.expandClassPath(GroovyClassLoader.java:500) at groovy.lang.GroovyClassLoader.getClassPath(GroovyClassLoader.java:458) at groovy.lang.GroovyClassLoader.getSourceFile(GroovyClassLoader.java:723) at groovy.lang.GroovyClassLoader.access$000(GroovyClassLoader.java:85) at groovy.lang.GroovyClassLoader$1.loadGroovySource(GroovyClassLoader.java:91) at groovy.lang.GroovyClassLoader$3.run(GroovyClassLoader.java:662) at java.security.AccessController.doPrivileged(Native Method) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:659) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:705) at java.lang.ClassLoader.loadClass(ClassLoader.java:251) at java.beans.Introspector.instantiate(Introspector.java:1425) at java.beans.Introspector.findExplicitBeanInfo(Introspector.java:406) at java.beans.Introspector.<init>(Introspector.java:355) at java.beans.Introspector.getBeanInfo(Introspector.java:155) at groovy.lang.MetaClassImpl$2.run(MetaClassImpl.java:143) at java.security.AccessController.doPrivileged(Native Method) at groovy.lang.MetaClassImpl.<init>(MetaClassImpl.java:141) at groovy.lang.MetaClassRegistry.getMetaClassFor(MetaClassRegistry.java:277) at groovy.lang.MetaClassRegistry.getMetaClass(MetaClassRegistry.java:144) at org.codehaus.groovy.runtime.Invoker.getMetaClass(Invoker.java:104) at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:87) at groovy.lang.GroovyObjectSupport.<init>(GroovyObjectSupport.java:61) at groovy.lang.Script.<init>(Script.java:65) at Script5.<init>(Script5.groovy) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:494) at java.lang.Class.newInstance0(Class.java:350) at java.lang.Class.newInstance(Class.java:303) at org.codehaus.groovy.runtime.InvokerHelper.createScript(InvokerHelper.java:529) at groovy.lang.GroovyShell.parse(GroovyShell.java:548) at groovy.lang.GroovyShell.parse(GroovyShell.java:527) at groovy.lang.GroovyShell.parse(GroovyShell.java:566)
Hide
Permalink
blackdrag blackdrag added a comment -

I guess the reason for thesse problems is the ClassHelper class in combination with the lazy init mechanism that I implemented. That part isn't thread save, I will think about how to change this.

Show
blackdrag blackdrag added a comment - I guess the reason for thesse problems is the ClassHelper class in combination with the lazy init mechanism that I implemented. That part isn't thread save, I will think about how to change this.
Hide
Permalink
blackdrag blackdrag added a comment -

ok, I committed a little fix basing on a guess. I don't like to such fixed but I don't have a multy processor engine as your HT system. This means that I will have only one active thread at the same time while you can have 2. That also means I can't test the code. it would be nice if you could download cvs head make a new build without the tests (htey are failing at the moment) using "maven rebuild" (note: maven 1.0.2 has to be used) and tell me if it is ok now.

Show
blackdrag blackdrag added a comment - ok, I committed a little fix basing on a guess. I don't like to such fixed but I don't have a multy processor engine as your HT system. This means that I will have only one active thread at the same time while you can have 2. That also means I can't test the code. it would be nice if you could download cvs head make a new build without the tests (htey are failing at the moment) using "maven rebuild" (note: maven 1.0.2 has to be used) and tell me if it is ok now.
Hide
Permalink
blackdrag blackdrag added a comment -

after getting a smp machine from alleon I wasn'r able to reproduce the problem with jsr-04. So best would really be if you could run the test again.

Show
blackdrag blackdrag added a comment - after getting a smp machine from alleon I wasn'r able to reproduce the problem with jsr-04. So best would really be if you could run the test again.
Hide
Permalink
Edson Tirelli added a comment -

Hi Jochen,

I just checked out CVS head, compiled and run the test several times in both machines and I was not able to reproduce the problem also. So, I think your fix worked.
Thank you for working so quick on that.

Regards,
Edson

Show
Edson Tirelli added a comment - Hi Jochen, I just checked out CVS head, compiled and run the test several times in both machines and I was not able to reproduce the problem also. So, I think your fix worked. Thank you for working so quick on that. Regards, Edson
Hide
Permalink
blackdrag blackdrag added a comment -

very nice to hear. If you do always such nice reprots I am sure the bugs you will find will be fixed very soon.

Show
blackdrag blackdrag added a comment - very nice to hear. If you do always such nice reprots I am sure the bugs you will find will be fixed very soon.

People

  • Assignee:
    blackdrag blackdrag
    Reporter:
    Edson Tirelli
Vote (0)
Watch (1)

Dates

  • Created:
    Updated:
    Resolved: