groovy
  1. groovy
  2. GROOVY-4154

TemplateServlet with default configuration leaks classes resulting in a PermGen error under Tomcat

    Details

    • Type: Bug Bug
    • Status: Open Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 1.7.2
    • Fix Version/s: None
    • Component/s: Groovlet / GSP
    • Labels:
      None
    • Number of attachments :
      3

      Description

      TemplateServlet by default uses SimpleTemplateEngine. SimpleTemplateEngine uses a cache to store loaded Template objects. These template objects contain a "script" instance variable that corresponds to the compiled gsp script, the classes are named SimpleTemplateScript[x] where x is a growing number. Problem with this cache is that it is a WeakHashMap. This means that when garbage collection occurs, templates will eventually be removed from the cache-map. It was probably intended so that the cache would empty itself in case heap is running out. Problem is that the ClassLoader that loaded the script classes inside the Template-objects is still kept in memory (by TemplateServlet.engine.groovyShell.loader reference) and this prevents the classes from being garbage collected.

      After the Template object has been removed from the cache by GC and someone request the corresponding gsp template again, TemplateServlet will have to generate a new Template object and because the script classes in templates are generated by passing in a string to the groovyShell, the GroovyClassLoader doesn't recognize that it has already loaded a similar class. In the end, PermGen fills up from the same template classes being compiled multiple times.

      The best fix I've found yet (also the easiest), just use the groovy.text.GStringTemplateEngine instead. It doesn't have this problem. My suggestion for "fixing" the issue would be just to change the default. Maybe even marking SimpleTemplateEngine as "deprecated", unless there are good reasons to use it.

      To reproduce the issue:
      1.download an empty Tomcat instance (I used apache-tomcat-5.5.27)
      2.empty webapps folder
      3.to speed up testing, configure tomcat with restricted heap and permgen with following JAVA_OPTS:
      JAVA_OPTS="
      -verbose:gc
      -Xmx64M
      -XX:MaxPermSize=30M
      -XX:+PrintGCDetails
      -XX:+PrintGCTimeStamps
      -XX:+HeapDumpOnOutOfMemoryError
      -Xloggc:gc.log
      -XX:HeapDumpPath=dump.hprof
      "
      4.copy the attached groovlets.war webapp to the webapps folder (expanded or not, no matter). The webapp contains the default groovlet setup with one index.groovy and one index.gsp that index.groovy dispatches to. Both groovy files are very simple they just print "Hello, groovy!" on a webpage many times.
      5.start tomcat, open jconsole on tomcat pid (optional).
      6.verify installation by calling http://localhost:8080/groovlets/index.groovy you should see "Hello, Groovy multiple times".
      7.view perm gen from jconsole.
      8.repeat cycle:
      a) with your favorite load tester, stress the script (fire multiple threads and multiple http gets).
      For example, with Apache Bench: ab -c 30 -n 10000 "http://localhost:8080/groovlets/index.groovy"
      b) trigger garbage collection from jconsole (Perform GC button)
      c) watch Perm Gen going up slowly

      After 5-10 cycles you either get PermGen error to catalina.out or it just doesn't answer anymore.

        Activity

        Hide
        Antti Pöyhönen added a comment -

        Looks like I jumped to conclusions too early. Switching to GStringTemplateEngine does prevent the PermGen problem, but I am still getting memory problems. I saw increased load on the server where I've deployed groovlets, heap summary gave me >99.999% of Old Generation in use and the load was coming from FullGC cycles happening. I looked at the heap dump with Eclipse's MAT tool, java.beans.introspector was first in the dominator tree, second was the WebAppClassloader, and inside that hashmaps that kept BeanInfo and some Closurex_Metaclass-objects. I think that with GStringTemplateEngine the classes are actually unloaded so PermGen doesn't fill up, but reflection and other metainformation that Groovy requires is left behind from the classes and fill the heap.

        Should it be possible to compile and execute the groovy script multiple (endless) times in a loop endlessly, always throwing away the produced class and it's unique classloader? For me this doesn't seem to work as the heap gets full from introspection caches on other objects containing meta information about the groovy scripts. I could use some help here from someone with more groovy knowledge.

        Show
        Antti Pöyhönen added a comment - Looks like I jumped to conclusions too early. Switching to GStringTemplateEngine does prevent the PermGen problem, but I am still getting memory problems. I saw increased load on the server where I've deployed groovlets, heap summary gave me >99.999% of Old Generation in use and the load was coming from FullGC cycles happening. I looked at the heap dump with Eclipse's MAT tool, java.beans.introspector was first in the dominator tree, second was the WebAppClassloader, and inside that hashmaps that kept BeanInfo and some Closurex_Metaclass-objects. I think that with GStringTemplateEngine the classes are actually unloaded so PermGen doesn't fill up, but reflection and other metainformation that Groovy requires is left behind from the classes and fill the heap. Should it be possible to compile and execute the groovy script multiple (endless) times in a loop endlessly, always throwing away the produced class and it's unique classloader? For me this doesn't seem to work as the heap gets full from introspection caches on other objects containing meta information about the groovy scripts. I could use some help here from someone with more groovy knowledge.
        Hide
        blackdrag blackdrag added a comment -

        BeanInfo of what is kept? of Groovy generated classes? That I can hardly imagine. I guess it is about classes used by the generated classes. Now what happens if a BeanInfo class is loaded? As with any other class, the class stays in memory till the classloader is unloaded. Since you don't run into a permgen problem anymore now, it means the laoder used for generated classes is unloaded, but the beaninfo stuff is loaded on a higher loader, which is not under Groovy control. In short, besids not loading the beaninfo classes there is nothing we can do about them.

        I wonder about the Closurex_Metaclass-objects you mention. I guess you mean ClosureMetaClass instead. This class is, unless you set it yourself, not hard referenced, but weak referenced. So if you take a look at memory it is natural to see this class around. But it does not mean they are not unloaded if memory is low.

        Have you ensured you are getting OutOfMemoryErrors still?

        Show
        blackdrag blackdrag added a comment - BeanInfo of what is kept? of Groovy generated classes? That I can hardly imagine. I guess it is about classes used by the generated classes. Now what happens if a BeanInfo class is loaded? As with any other class, the class stays in memory till the classloader is unloaded. Since you don't run into a permgen problem anymore now, it means the laoder used for generated classes is unloaded, but the beaninfo stuff is loaded on a higher loader, which is not under Groovy control. In short, besids not loading the beaninfo classes there is nothing we can do about them. I wonder about the Closurex_Metaclass-objects you mention. I guess you mean ClosureMetaClass instead. This class is, unless you set it yourself, not hard referenced, but weak referenced. So if you take a look at memory it is natural to see this class around. But it does not mean they are not unloaded if memory is low. Have you ensured you are getting OutOfMemoryErrors still?
        Hide
        Antti Pöyhönen added a comment - - edited

        I attached two screenshots from the heap dump's dominator tree. The java.beans.Instropector's static caches are not being cleared. The caches are under a WeakHashMap, so they should eventually be cleared if no one else is pointing at the items in the caches (although Introspector's javadoc does recommend to flush the caches manually if multiple classloaders are used).

        I think the reason why these classes cannot be cleared may be Tomcat's WebappClassLoader. I peeked at Tomcat's source and the HashMap that is visible in the heap dump screenshots is probably WebappClassLoader.resourceEntries. Javadoc describes it as "The cache of ResourceEntry for classes and resources we have loaded, keyed by resource name.".

        So the problem I am having may be the result of GSP templates being compiled multiple times and the way Tomcat caches webapplication resources. I first actually tried to reproduce this with Jetty but couldn't and then moved to Tomcat which was the one used on the problematic server.

        The server didn't actually throw OOME, but full gc cycles were happening and oldgen was staying at 99.999%. After that I restarted the server and deployed a modified version of Groovy where TemplateServlet.cache is a java.util.concurrent.ConcurrentHashMap instead of a WeakHashMap. Using a normal Map as the cache prevents the templates from being garbage collected so that gets around the problem. If I would now modify those templates repeatedly I would eventually run out of heap, but for me this is not a problem and I can accept such a limitation if I get a stable server in return.

        Show
        Antti Pöyhönen added a comment - - edited I attached two screenshots from the heap dump's dominator tree. The java.beans.Instropector's static caches are not being cleared. The caches are under a WeakHashMap, so they should eventually be cleared if no one else is pointing at the items in the caches (although Introspector's javadoc does recommend to flush the caches manually if multiple classloaders are used). I think the reason why these classes cannot be cleared may be Tomcat's WebappClassLoader. I peeked at Tomcat's source and the HashMap that is visible in the heap dump screenshots is probably WebappClassLoader.resourceEntries. Javadoc describes it as "The cache of ResourceEntry for classes and resources we have loaded, keyed by resource name.". So the problem I am having may be the result of GSP templates being compiled multiple times and the way Tomcat caches webapplication resources. I first actually tried to reproduce this with Jetty but couldn't and then moved to Tomcat which was the one used on the problematic server. The server didn't actually throw OOME, but full gc cycles were happening and oldgen was staying at 99.999%. After that I restarted the server and deployed a modified version of Groovy where TemplateServlet.cache is a java.util.concurrent.ConcurrentHashMap instead of a WeakHashMap. Using a normal Map as the cache prevents the templates from being garbage collected so that gets around the problem. If I would now modify those templates repeatedly I would eventually run out of heap, but for me this is not a problem and I can accept such a limitation if I get a stable server in return.
        Hide
        blackdrag blackdrag added a comment -

        About the BeanInfo stuff... why don't you do a clean of the bean info caches? As for the WebAppClasLoader... this looks awfully like the loader saves entries for classes it did not find. I can imagine that being an performance improvement in some cases, but in a case were many classes are created, this will cause the creation of many many class entries. I would either look for a way to replace that class loader or to turn that off. But in both cases this does not absolutely look like Groovy bugs.

        Show
        blackdrag blackdrag added a comment - About the BeanInfo stuff... why don't you do a clean of the bean info caches? As for the WebAppClasLoader... this looks awfully like the loader saves entries for classes it did not find. I can imagine that being an performance improvement in some cases, but in a case were many classes are created, this will cause the creation of many many class entries. I would either look for a way to replace that class loader or to turn that off. But in both cases this does not absolutely look like Groovy bugs.
        Hide
        Antti Pöyhönen added a comment -

        About the BeanInfo stuff, I think everone would prefer groovlets to work without explicit cleaning calls. But I admit that there is no evidence that the data in the Introspector caches is anything that should have been GC'd. And the WebappClassLoader thing, looks like it's a Tomcat bug, and I'm in luck. Looks like it's about to be fixed on the next release of Tomcat 5.5 series and also has already been fixed in 6 series: https://issues.apache.org/bugzilla/show_bug.cgi?id=47987

        So I agree, TemplateServlet with GStringTemplateEngine works with the latest tomcat versions. What about the original bug report though, if TemplateServlet is used with SimpleTemplateEngine?

        Show
        Antti Pöyhönen added a comment - About the BeanInfo stuff, I think everone would prefer groovlets to work without explicit cleaning calls. But I admit that there is no evidence that the data in the Introspector caches is anything that should have been GC'd. And the WebappClassLoader thing, looks like it's a Tomcat bug, and I'm in luck. Looks like it's about to be fixed on the next release of Tomcat 5.5 series and also has already been fixed in 6 series: https://issues.apache.org/bugzilla/show_bug.cgi?id=47987 So I agree, TemplateServlet with GStringTemplateEngine works with the latest tomcat versions. What about the original bug report though, if TemplateServlet is used with SimpleTemplateEngine?
        Hide
        blackdrag blackdrag added a comment -

        We don't clean the beaninfo caches, because it costs runtime performance and should be done only if it is really needed.

        As for the SimpleTemplateEngine, true, that should be fixed

        Show
        blackdrag blackdrag added a comment - We don't clean the beaninfo caches, because it costs runtime performance and should be done only if it is really needed. As for the SimpleTemplateEngine, true, that should be fixed
        Hide
        Deep Varma added a comment -

        Any more update on this issue

        Show
        Deep Varma added a comment - Any more update on this issue
        Hide
        Yoryos Valotasios added a comment -

        I would like to report the same problem using groovy v1.7.4 (I am using GroovyServlet thought) with tomcat 6 (testing with jetty had also the same result). My PermGen was growing bigger as the loaded classes. Trying to figure out what was going on I tried to list the loaded classes with jconsole and saw a lot of groovlets (the same ones!) being loaded continuously. Falling back to v1.7.3 made the problem disappear.

        As I am using a different version of tomcat, jetty and groovy and make use of the groovy.servlet.GroovyServlet I am not quiet sure if it is the same bug. Should I create a new issue?

        Show
        Yoryos Valotasios added a comment - I would like to report the same problem using groovy v1.7.4 (I am using GroovyServlet thought) with tomcat 6 (testing with jetty had also the same result). My PermGen was growing bigger as the loaded classes. Trying to figure out what was going on I tried to list the loaded classes with jconsole and saw a lot of groovlets (the same ones!) being loaded continuously. Falling back to v1.7.3 made the problem disappear. As I am using a different version of tomcat, jetty and groovy and make use of the groovy.servlet.GroovyServlet I am not quiet sure if it is the same bug. Should I create a new issue?

          People

          • Assignee:
            Unassigned
            Reporter:
            Antti Pöyhönen
          • Votes:
            3 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated: