JRuby

Need to split jar for App Engine

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Blocker Blocker
  • Resolution: Fixed
  • Affects Version/s: JRuby 1.6RC2
  • Fix Version/s: JRuby 1.6.4
  • Component/s: None
  • Labels:
    None
  • Environment:
    google-appengine
  • Number of attachments :
    0

Description

The rc2 jruby-jars won't work on GAE.
The file size limit is 10 million bytes:

  • jruby-core-1.6.0.RC2.jar 10149262 # ERROR
  • jruby-core-1.5.6.jar 8190691 # WORKS

I was able to get the jar down to 9785738 bytes
by removing the jni directory. One solution would
be to have 3 jars, with a new one called:

  • jruby-native-1.6.0.RC3.jar

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Charles Oliver Nutter added a comment -

The jar is probably larger due to the addition of SnakeYAML and other 1.9 functionality.

On my system, jruby.jar is 9458740, but perhaps jruby-jars adds more stuff in?

Pulling out native libs into a separate jar is a pretty good idea too.

Marking for 1.6.

Show
Charles Oliver Nutter added a comment - The jar is probably larger due to the addition of SnakeYAML and other 1.9 functionality. On my system, jruby.jar is 9458740, but perhaps jruby-jars adds more stuff in? Pulling out native libs into a separate jar is a pretty good idea too. Marking for 1.6.
Hide
Permalink
kristian meier added a comment -

how does the whole thing work for a maven build (with using gwt-maven-plugin and/or maven-gae-plugin)? I can not find a jruby-core artifact nowhere. and jruby-complete is not working. the jruby artifact has no jruby.home so can not be used, unless there is a artifact which contains jruby.home - I have not saw such an artifact.

Show
kristian meier added a comment - how does the whole thing work for a maven build (with using gwt-maven-plugin and/or maven-gae-plugin)? I can not find a jruby-core artifact nowhere. and jruby-complete is not working. the jruby artifact has no jruby.home so can not be used, unless there is a artifact which contains jruby.home - I have not saw such an artifact.
Hide
Permalink
Charles Oliver Nutter added a comment -

kristian: Unfortunately the jruby-jars gem also has a "jruby-core" jar, but it's basically equivalent to the "jruby" artifact in Maven. I guess I goofed on picking the name for "jruby-core" maven artifact.

Woodie: I'm looking into this today.

Show
Charles Oliver Nutter added a comment - kristian: Unfortunately the jruby-jars gem also has a "jruby-core" jar, but it's basically equivalent to the "jruby" artifact in Maven. I guess I goofed on picking the name for "jruby-core" maven artifact. Woodie: I'm looking into this today.
Hide
Permalink
kristian meier added a comment -

in case I am a bit out of context, but I sent another set of pom.xml around which would include a jruby-stdlib.jar as maven artifact which would answer my above question maybe someone can look into that as well.

Show
kristian meier added a comment - in case I am a bit out of context, but I sent another set of pom.xml around which would include a jruby-stdlib.jar as maven artifact which would answer my above question maybe someone can look into that as well.
Hide
Permalink
Charles Oliver Nutter added a comment -

kristian: yeah I realized I had forgotten your update. I'll pull them in.

Woodie: I started to make the native split, but now I'm having second thoughts.

Warbler currently just pulls in core and stdlib jars from jruby-jars. Without an update, anyone using existing warbler versions will see a behavioral difference if we split out the jruby-native stuff into a separate jar. Specifically, anyone with webapps using FFI (e.g. nokogiri) or any system-level features (symlink, stat, etc) will see those features suddenly break when they update jruby-jars to 1.6 and do nothing else.

We also have so many release artifacts now that it's getting difficult to manage them. I think we should look at some other options for GAE JRuby. Maybe one of these:

  • A separate gem jruby-jars-appengine that's specifically tailored for GAE. It could be even more aggressive about stripping things out, like removing the whole FFI subsystem, jline/readline, and more.
  • Pull from Maven; all deps are there, and there's various ways we could automate pulling in dependencies. In addition, if we pulled dependency libs from Maven and deployed with GAE we'd also benefit from GAE's own caches of those jars.

If the Maven approach isn't appropriate, perhaps we should see about transitioning the appengine-specific jar stuff to a separate gem you guys can depend on? I'd be happy to spin up a project that we can both maintain that's structured specifically for GAE's limits on jar size. How does that sound?

Show
Charles Oliver Nutter added a comment - kristian: yeah I realized I had forgotten your update. I'll pull them in. Woodie: I started to make the native split, but now I'm having second thoughts. Warbler currently just pulls in core and stdlib jars from jruby-jars. Without an update, anyone using existing warbler versions will see a behavioral difference if we split out the jruby-native stuff into a separate jar. Specifically, anyone with webapps using FFI (e.g. nokogiri) or any system-level features (symlink, stat, etc) will see those features suddenly break when they update jruby-jars to 1.6 and do nothing else. We also have so many release artifacts now that it's getting difficult to manage them. I think we should look at some other options for GAE JRuby. Maybe one of these: A separate gem jruby-jars-appengine that's specifically tailored for GAE. It could be even more aggressive about stripping things out, like removing the whole FFI subsystem, jline/readline, and more. Pull from Maven; all deps are there, and there's various ways we could automate pulling in dependencies. In addition, if we pulled dependency libs from Maven and deployed with GAE we'd also benefit from GAE's own caches of those jars. If the Maven approach isn't appropriate, perhaps we should see about transitioning the appengine-specific jar stuff to a separate gem you guys can depend on? I'd be happy to spin up a project that we can both maintain that's structured specifically for GAE's limits on jar size. How does that sound?
Hide
Permalink
John-Mason P. Shackelford added a comment - - edited

If we pursue one of these options would you expect this project to require an update / release as JRuby releases versions? It would be nice if whatever approach we were to take wouldn't involve another piece to maintain.

As far as the Warbler issue is concerned, perhaps we make this change in JRuby 1.6.2+ after a new Warbler is released-- watching github I'm guessing it is nearing readiness for a major release-- and give Warbler have a bit of logic so that if it finds a jruby-core and jruby-stdlib in jruby-jars it assumes we bundle both, but if it finds jruby, jruby-stdlib, and jruby-native (new version), it bundles all three unless a we set a flag in the config or perhaps the new Warbler would just have a config item that is an Array of the jars to bundle and it defaults to all the jars in 1.6.2+ scheme. One can still use it with JRuby =< 1.5.5 by changing the property to jruby-core, jruby-stdlib-- a minor inconvenience that would create a an incentive to move up to newest JRuby with the newest Warbler. Downside is that users will have to upgrade to newest Warbler when they upgrade JRuby, but since jruby-core will be renamed to jruby, they'll fail fast instead of wondering what is going on with missing libraries.

Show
John-Mason P. Shackelford added a comment - - edited If we pursue one of these options would you expect this project to require an update / release as JRuby releases versions? It would be nice if whatever approach we were to take wouldn't involve another piece to maintain. As far as the Warbler issue is concerned, perhaps we make this change in JRuby 1.6.2+ after a new Warbler is released-- watching github I'm guessing it is nearing readiness for a major release-- and give Warbler have a bit of logic so that if it finds a jruby-core and jruby-stdlib in jruby-jars it assumes we bundle both, but if it finds jruby, jruby-stdlib, and jruby-native (new version), it bundles all three unless a we set a flag in the config or perhaps the new Warbler would just have a config item that is an Array of the jars to bundle and it defaults to all the jars in 1.6.2+ scheme. One can still use it with JRuby =< 1.5.5 by changing the property to jruby-core, jruby-stdlib-- a minor inconvenience that would create a an incentive to move up to newest JRuby with the newest Warbler. Downside is that users will have to upgrade to newest Warbler when they upgrade JRuby, but since jruby-core will be renamed to jruby, they'll fail fast instead of wondering what is going on with missing libraries.
Hide
Permalink
Charles Oliver Nutter added a comment -

Ok, so ultimately I was able to fix this by removing the "cext" dir from jruby-core jar, which brings the size back down. This is not a good long-term fix, since it could easily grow bigger again...but hopefully by the time 1.7 releases start coming out, GAE will allow >10MB jar files.

jruby-1_6: 5c37857

master:

commit 096b85498e6d1ce9e18c47576de7603f9a05eb1a
Author: Charles Oliver Nutter <headius@headius.com>
Date:   Tue Jul 12 16:02:56 2011 -0500

    Fix JRUBY-5496: Need to split jar for App Engine
    
    * Removing cext stuff from the jruby-core jar gets it below 10MB. This is not going to be a long-term fix, so hopefully GAE will start allowing >10MB jars soon.
Show
Charles Oliver Nutter added a comment - Ok, so ultimately I was able to fix this by removing the "cext" dir from jruby-core jar, which brings the size back down. This is not a good long-term fix, since it could easily grow bigger again...but hopefully by the time 1.7 releases start coming out, GAE will allow >10MB jar files. jruby-1_6: 5c37857 master: commit 096b85498e6d1ce9e18c47576de7603f9a05eb1a Author: Charles Oliver Nutter <headius@headius.com> Date: Tue Jul 12 16:02:56 2011 -0500 Fix JRUBY-5496: Need to split jar for App Engine * Removing cext stuff from the jruby-core jar gets it below 10MB. This is not going to be a long-term fix, so hopefully GAE will start allowing >10MB jars soon.

People

  • Assignee:
    Charles Oliver Nutter
    Reporter:
    John Woodell
Vote (1)
Watch (4)

Dates

  • Created:
    Updated:
    Resolved:

Time Tracking

Estimated:
4h
Original Estimate - 4 hours
Remaining:
4h
Remaining Estimate - 4 hours
Logged:
Not Specified
Time Spent - Not Specified