I wanted to document some of my work in this area.

RedCloth (4.1.1) is a gem quite similar to hpricot. It was originally written by Why and when building the gem it uses ragel to produce wither Java or C. Jason Garber has taken over development:

• http://github.com/jgarber/redcloth/tree/master

When building redcloth under JRuby the Java classes are archived into lib/redcloth_scan.jar which is included with the JRuby version of the gem.

I chose RedCloth to work on first because it uses a somewhat simpler (and to me hackable) Rakefile for building the gem based on echoe.

I'm experimenting to see if I can get a gem that normally uses a jar to use the .class files instead. If this works I could package the gem with .class files instead of the jar and then more easily include it in with jruby-complete.jar (avoiding the jar within a jar problem).

I modified the Rake task so the classes are copied to lib/ also:

[redcloth.git (nojar)]$ ls -l lib/
total 3360
-rw-r--r--  1 stephen  staff   24976 Jan 12 14:32 RedclothAttributes.class
-rw-r--r--  1 stephen  staff  560002 Jan 12 14:32 RedclothInline.class
-rw-r--r--  1 stephen  staff    9548 Jan 12 14:32 RedclothScanService$Base.class
-rw-r--r--  1 stephen  staff  597763 Jan 12 14:32 RedclothScanService$Transformer.class
-rw-r--r--  1 stephen  staff    5423 Jan 12 14:32 RedclothScanService.class
drwxr-xr-x  3 stephen  staff     102 Jul 27 21:23 case_sensitive_require
drwxr-xr-x  7 stephen  staff     238 Nov 20 17:59 redcloth
-rw-r--r--@ 1 stephen  staff    1522 Jan 12 18:39 redcloth.rb
-rw-r--r--  1 stephen  staff  506272 Jan 12 14:32 redcloth_scan.jar

RedclothScanService implements BasicLibraryService – it's adding methods to RedCloth::TextileDoc
You can see the RedclothScanService.java code ragel generated here: http://gist.github.com/46177

When JRuby requires a jar and there is a class in that jar that implements BasicLibraryService then it's basicLoad method get's called. It looks like it's called from: LoadService.smartLoad which calls the private method: tryLoadingLibraryOrScript.

If the classes are made available in lib/ then this statement:

require 'redcloth_scan'

can be replaced by these and the gem operates and passes all it tests:

require 'jruby'
$CLASSPATH <<  File.dirname(File.expand_path(__FILE__)) + '/'
Java::RedclothScanService.new.basicLoad(JRuby.runtime)

So at least for the redcloth gem which includes redcloth_scan.jar in it's lib/ dir – if you instead just put the class files in lib/ and replace: require 'redcloth_scan' with the three lines above it should work fine when embedded into a jar with the rest of JRuby.

It's not obvious to me yet how this could be adapted into a generalized solution which doesn't require changes to the gems.

One thing I noticed in the example that Nick provided is that the "require 'hpricot'", near the end of the session, returned false. I assume this is because JRuby is saying that it already loaded 'hpricot' when it processed the -rhpricot option. So one caveat of using the gem-in-a-jar feature is that the jar shouldn't be named the same as anything in the jar files. e.g. renaming hpricot.jar to hpricot-0.6.164.jar changes the IRB session to this:

>>> ./bin/jruby -rhpricot-0.6.164.jar -S jirb
irb(main):001:0> require 'hpricot'
LoadError: no such file to load -- hpricot
        from (irb):2:in `require'
        from (irb):2
irb(main):002:0>

My thoughts on resolving this issue start with JRubyClassLoader looking for anything that's added with a url protocol of "jar". From there, it seems like there are a couple of approaches:

• Preload everything in the jar when it's required / added to the class path.
• Add the jar-in-a-jar to a list of search paths, and override findClass(String) to search the jars-in-jars for the required class.
• Extract the jar-in-a-jar to a temp dir. Permissions and cleanup are obvious issues with this.

diff --git a/src/org/jruby/runtime/load/JarredScript.java b/src/org/jruby/runtime/load/JarredScript.java
index 3299b67..9383688 100644
--- a/src/org/jruby/runtime/load/JarredScript.java
+++ b/src/org/jruby/runtime/load/JarredScript.java
@@ -63,6 +63,7 @@ public class JarredScript implements Library {
 
         // Make Java class files in the jar reachable from Ruby
         runtime.getJRubyClassLoader().addURL(jarFile);
+        runtime.getLoadService().getLoadPathArray().append(runtime.newString(jarFile.toString()));
 
         try {
             JarInputStream in = new JarInputStream(new BufferedInputStream(jarFile.openStream()));

...I can get it to load two levels of jars, which might be enough for Hpricot (since the inner jar does get added to the classloader). But I couldn't get it to require additional files out of the innermost jar.

To test whether this is good enough for Hpricot, I tried creating a simple Java class and putting it in a jar-in-a-jar:

$ cat TestNested.java
public class TestNested {
  public String hello() { return "hello"; }
}
$ javac TestNested.java
$ jar cvf inner.jar TestNested.class
added manifest
adding: TestNested.class(in = 269) (out= 199)(deflated 26%)
$ jar cvf outer.jar inner.jar
added manifest
adding: inner.jar(in = 657) (out= 437)(deflated 33%)
$ jruby -rjava -e "require 'outer.jar'; require 'inner.jar'; puts Java::TestNested.new.hello"
hello

Huzzah! This appears to work reasonably well. So we can at least get a jar full of classes to work from within another jar.

However I'd like to make this a bit more general-purpose, so it can support any number of nested jars and load path and require will still work correctly for anything within the nested-most jar file. So...help?

• We need specs for various levels of nesting, so we know what we're working toward
• We need to make the jar-file loading and searching logic cleaner than it is currently and general-purpose enough that it can nest deeply without breaking.

So, team, anyone up for the challenge?

$ rm inner.jar
$ rm TestNested.class
$ jruby -rjava -e "require 'outer.jar'; require 'inner.jar'; puts Java::TestNested.new.hello"
Java::TestNestedNewHello

So I think we're close but not quite there.

Index: src/org/jruby/util/JRubyClassLoader.java
===================================================================
--- src/org/jruby/util/JRubyClassLoader.java    (revision 8786)
+++ src/org/jruby/util/JRubyClassLoader.java    (working copy)
@@ -3,21 +3,47 @@
 import java.net.URL;
 import java.net.URLClassLoader;
 import java.security.ProtectionDomain;
+import java.io.InputStream;
 
 public class JRubyClassLoader extends URLClassLoader {
     private final static ProtectionDomain DEFAULT_DOMAIN
             = JRubyClassLoader.class.getProtectionDomain();
 
+    private List embeddedJars;
+
     public JRubyClassLoader(ClassLoader parent) {
         super(new URL[0], parent);
+        embeddedJars = new List();
     }
 
     // Change visibility so others can see it
     @Override
     public void addURL(URL url) {
-        super.addURL(url);
+        if(url.getProtocol().equals("jar")) {
+            // TODO: something like http://one-jar.cvs.sourceforge.net/viewvc/one-jar/one-jar/src/com/simontuffs/onejar/JarClassLoader.java?revision=1.35&view=markup
+            // 1A. Add protocol to start of url to indicate another classloader.
+            // 2A. When classes are pulled from that classloader, crack open the Jar file and read them ?
+            // 1B. Add jar file to a list to scan when asked for a class.
+            // 1C. Define all classes RIGHT NOW.
+            System.out.println(" loading embedded jar file " + url + " manually.");
+//            InputStream jarStream = url.openStream();
+            synchronized (embeddedJars) {
+                embeddedJars.add(url);
+            }
+        } else {
+            super.addURL(url);
+        }
     }
 
+    @Override
+    public Class findClass(java.lang.String className)
+      throws java.lang.ClassNotFoundException
+    {
+        // See if the class is in any of the embedded jars, and load it from there.
+        System.out.println("findClass(" + className + ")");
+        return super.findClass(className);
+    }
+
     public Class<?> defineClass(String name, byte[] bytes) {
         return super.defineClass(name, bytes, 0, bytes.length, DEFAULT_DOMAIN);
      }

It's really rough, and obviously spews tons of crap to stdout, but I think it just needs a little more work to finish. My first attempt was going to be an implementation of findClass that searches the embeddedJars for the requested class. I'll do this as I find free time.

It seems that gems-in-a-jar can't access files packaged with that gem. For example if I get the gem's version from a file named VERSION in the root directory of my gem i would do something like this:

File.read(File.join(File.dirname(__FILE__), "..", "VERSION")).strip

This will fail with the following exception:

file:/Users/LacKac/Working/Lab/tmp/testgem/test/gems.jar!/gems/testgem-0.0.1/lib/testgem.rb:6:in `version': No such file or directory - File not found - file:/Users/LacKac/Working/Lab/tmp/testgem/test/gems.jar!/gems/testgem-0.0.1/lib/../VERSION (Errno::ENOENT)
	from test.rb:6

I attached the testgem gem and I used this for testing:

require 'rubygems'
gem 'testgem'
require 'testgem'

p TestGem.test
p TestGem.version

If I create a new JAR containing both the .rb and .class files from Hpricot, it works.

<target name="hpricot-jar" description="Create JAR for Hpricot">
  <jar destfile="${lib.dir}/hpricot-for-jruby.jar" compress="true" index="true">
    <fileset dir="${lib.dir}/ruby/gems/1.8/gems/hpricot-0.6.164-java/lib">
      <include name="**/*"/>
      <exclude name="**/*.jar"/>
    </fileset>
    <zipfileset src="${lib.dir}/ruby/gems/1.8/gems/hpricot-0.6.164-java/lib/universal-java1.6/hpricot_scan.jar"/>
    <zipfileset src="${lib.dir}/ruby/gems/1.8/gems/hpricot-0.6.164-java/lib/universal-java1.6/fast_xs.jar"/>
  </jar>
</target>