JRuby (please use github issues at http://bugs.jruby.org)
  1. JRuby (please use github issues at http://bugs.jruby.org)
  2. JRUBY-5498

important JRuby 1.6 startup time regression launching JRuby script from JSR223 (12 sec -> 45 sec)

    Details

    • Type: Bug Bug
    • Status: Closed Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: JRuby 1.6
    • Fix Version/s: JRuby 1.6RC3
    • Component/s: Embedding
    • Labels:
      None
    • Environment:
      OS: Ubuntu Maverick 64 bits; Java: Java HotSpot(TM) 64-Bit Server VM (build 20.0-b06, mixed mode)
    • Testcase included:
      yes
    • Number of attachments :
      0

      Description

      This a serious performance regression. startup time to launch the exact same small JRuby program (see how to donwload it below)

      • along with jruby-complete 1.5.6: 12 seconds
      • along with jruby-complete trunk: 45 seconds!
        (trunk is from today, GIT id 254d3d0cde5b27eba920ff26b08b5adf68f9ab5e )

      The part of the program I'm testing here does something very simple in that test: basically, it's just the standard jruby-complete.jar executable with the following modifications:

      There is absolutely no code logic involved here from the 'ooor' bundled gem so just don't bother about what it does.

      testing it:
      here is the exectable jar in both configurations:

      just test the startup time by doing:
      java -jar jruby-ooor-xxx.jar

      I'm pretty sure this regression occurs those last 3/4 weeks as I remembered I tested it and it wasn't that slow.
      If that is of any help here is how the jar is assembled: https://github.com/rvalyi/terminatooor (build.xml and 'ant' task)

      Hope this helps. Don't hesitate if you have question, I'm aslo rvalyi on twitter for fast feedback.

        Activity

        Hide
        Charles Oliver Nutter added a comment -

        Ok, Tom gave me the idea to use JarFile for the top-level jars, which reduces the cache memory load immensely. I've pushed that change, and the "require 'openssl'" example now loads slightly faster, but uses only a couple MB more than the scanning version. I find that totally acceptable.

        I'm also going to look into ways to make JarFile work against in-memory resources (it normally only works against files), which would allow us to only cache JarFile.

        commit 2f624eeb4249507cab453cbd61b3665f935be78a
        Author: Charles Oliver Nutter <headius@headius.com>
        Date: Thu Mar 3 11:15:58 2011 -0600

        Improve jar-in-jar cache memory use by not caching contents of jars directly on the filesystem.

        Show
        Charles Oliver Nutter added a comment - Ok, Tom gave me the idea to use JarFile for the top-level jars, which reduces the cache memory load immensely. I've pushed that change, and the "require 'openssl'" example now loads slightly faster, but uses only a couple MB more than the scanning version. I find that totally acceptable. I'm also going to look into ways to make JarFile work against in-memory resources (it normally only works against files), which would allow us to only cache JarFile. commit 2f624eeb4249507cab453cbd61b3665f935be78a Author: Charles Oliver Nutter <headius@headius.com> Date: Thu Mar 3 11:15:58 2011 -0600 Improve jar-in-jar cache memory use by not caching contents of jars directly on the filesystem.
        Hide
        Nick Sieger added a comment -

        Unfortunate that making something work better exposes a performance issue, but nice that we can fix it as well. Patches look good so far, and looking forward to seeing this in action.

        Show
        Nick Sieger added a comment - Unfortunate that making something work better exposes a performance issue, but nice that we can fix it as well. Patches look good so far, and looking forward to seeing this in action.
        Hide
        Raphael Valyi added a comment -

        Charles,

        Regarding our use case, the only advantage of packaging my gems in jruby-complete is to provide a single click experience to the user. We only need SSL connections eventually from within the ETL so I just moved those SSL gems outside the jar and it works perfect, at least for us. Once again thanks a ton for your support!

        Show
        Raphael Valyi added a comment - Charles, Regarding our use case, the only advantage of packaging my gems in jruby-complete is to provide a single click experience to the user. We only need SSL connections eventually from within the ETL so I just moved those SSL gems outside the jar and it works perfect, at least for us. Once again thanks a ton for your support!
        Hide
        Charles Oliver Nutter added a comment -

        Ok, I think I'm going to call this one fixed.

        There's no way to use JarFile or ZipFile against in-memory resources, since they depend on zlib which only works against files. There are some other options that could perhaps be modified to work against in-memory resources (like Apache Harmony's java.util.zip stuff), but that's a much larger job than I want to take on.

        The modified jar-in-jar logic works like this:

        • For top-level file: jar files, open them directly and use JarFile to get entries. Nothing from top-level jar files is cached in memory.
        • For nested jar files, open once and cache all contents directly in memory. There's a memory impact from this, the uncompressed contents are held in memory forever. However, nested jars are generally not too large, and the memory tradeoff comes with a very solid improvement in lookup performance.

        With the new logic, only nested jars will have a memory impact equal to their uncompressed size, but the load time for resources when jars-in-jars are present is as fast as loading from filesystem jars.

        Show
        Charles Oliver Nutter added a comment - Ok, I think I'm going to call this one fixed. There's no way to use JarFile or ZipFile against in-memory resources, since they depend on zlib which only works against files. There are some other options that could perhaps be modified to work against in-memory resources (like Apache Harmony's java.util.zip stuff), but that's a much larger job than I want to take on. The modified jar-in-jar logic works like this: For top-level file: jar files, open them directly and use JarFile to get entries. Nothing from top-level jar files is cached in memory. For nested jar files, open once and cache all contents directly in memory. There's a memory impact from this, the uncompressed contents are held in memory forever. However, nested jars are generally not too large, and the memory tradeoff comes with a very solid improvement in lookup performance. With the new logic, only nested jars will have a memory impact equal to their uncompressed size, but the load time for resources when jars-in-jars are present is as fast as loading from filesystem jars.
        Hide
        Charles Oliver Nutter added a comment -

        Raphael: I think we were commenting at exactly the same time

        That sounds like a fine workaround, and if you want to move them back into the jar after RC3 that will work too. Because the main part of the memory hit was from the top-level jruby-complete jar itself, I've just switched over to using the caching logic only.

        Show
        Charles Oliver Nutter added a comment - Raphael: I think we were commenting at exactly the same time That sounds like a fine workaround, and if you want to move them back into the jar after RC3 that will work too. Because the main part of the memory hit was from the top-level jruby-complete jar itself, I've just switched over to using the caching logic only.

          People

          • Assignee:
            Charles Oliver Nutter
            Reporter:
            Raphael Valyi
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: