Provide more control for @Grapes to exclude transitives or adjust classloader

Attached patch covers the following cases:

Ex1 will 'grab' htmlunit but no transitive dependencies.
It fails after not finding 'httpclient'

@Grapes([
   @Grab(group='net.sourceforge.htmlunit', module='htmlunit', version='2.6', transitive=false),
])
import com.gargoylesoftware.htmlunit.WebClient

try {
   def client = new WebClient()
   def page = client.getPage('http://www.google.com')
   assert 'Google' == page.titleText
   println 'done without error'
} catch(java.lang.NoClassDefFoundError ncdfe) {
   assert ncdfe.message.contains('org/apache/commons/httpclient')
   println 'done with expected error'
}

Ex2 will 'grab' htmlunit and transitive dependencies but not xerces.
It fails after not finding 'xerces'. At the moment, the exclude is
kind of global. This might need some more work.

@Grapes([
   @Grab('net.sourceforge.htmlunit:htmlunit:2.6'),
   @GrabExclude('xerces:xercesImpl')
])
import com.gargoylesoftware.htmlunit.WebClient

def client = new WebClient()
try {
   def page = client.getPage('http://www.google.com')
   assert 'Google' == page.titleText
   println 'done without error'
} catch(java.lang.NoClassDefFoundError ncdfe) {
   assert ncdfe.message.contains('org/apache/xerces')
   println 'done with expected error'
}

Ex3 will 'grab' mysql jars and add via system classloader

@Grapes([
 @Grab('mysql:mysql-connector-java:5.1.6'),
 @GrabConfig(systemClassLoader=true)
])
import groovy.sql.Sql

def sql=Sql.newInstance("jdbc:mysql://localhost/test", "manager", "password", "com.mysql.jdbc.Driver")
println sql.firstRow('SELECT * FROM INFORMATION_SCHEMA.COLUMNS')

Ex4 will 'grab' xstream and xpp3 jars and add via system classloader
and also points the context class loader to the current script class loader.
(In some ways this last bit has nothing to do with Grapes but it still
occurs commonly enough to tack on a partial solution here I think).

@Grapes([
 @Grab('com.thoughtworks.xstream:xstream:1.3.1'),
 @Grab('xpp3:xpp3_min:1.1.4c'),
 @GrabConfig(systemClassLoader=true, initContextClassLoader=true)
])
import com.thoughtworks.xstream.*

class Staff {
   String firstname, lastname, position
}

def xstream = new XStream()
def john1 = new Staff(firstname:'John',
                    lastname:'Connor',
                    position:'Resistance Leader')

// write out to XML file
new File("john.xml").withOutputStream { out ->
   xstream.toXML(john1, out)
}

// now read back in
def john2
new File("john.xml").withInputStream { ins ->
   john2 = xstream.fromXML(ins)
}

println john2.dump() 

Updated patch also supporting 'conf'

hmm... the class loader logic worries me a bit. Setting the context class loader through the transform by adding code seems to be a fragile solution that will for sure cause other problems later. So it might work for this case, but not in general. How das grape use classloaders to load the classes?

Re: "class loader logic worries me a bit"

Yes, never fun to deal with. Grapes doesn't use context classloader at all. Perhaps that piece should be its own AST macro. It simply gives a way to make context classloader the same as script class loader which seems to be the expected thing for Java people/some Java library developers. It certainly won't work in all cases - but then just don't use this convenience and do manually as expected now.

For system class loader, Grapes has always had this but you could never get to it before from annotations. In fact, Grapes.grab is still more general since as well as using system class loader you can pass in any classloader - but from annotations, system class loader gives the next biggest win for what most people seem to want so they can deal with db drivers, xstream, and avoid those dreaded linkage errors about NamedNodeMap!

Updated patch now handles group#module;version[confs] syntax too

(copied from dev list)

I did a bit of testing in our real environment. We generally use a filesystem resolver with Ivy, but our master Ivy repo is in Svn and can be accessed with the IvySvn resolver.

I put this near the top of my script:

@Grab('com.oracle#ojdbc6;11.1.0.7.0')

This works fine:

new Sql(new oracle.jdbc.pool.OracleDataSource(URL: url))

This doesn't:

Sql.newInstance(url, driverClassName)

It gives:
Caught: java.sql.SQLException: No suitable driver found for jdbc:oracle:thin:...

It works if I change the @Grab() to

@Grapes([
@Grab('com.oracle#ojdbc6;11.1.0.7.0'),
@GrabConfig(systemClassLoader=true)
])

I guess the @GrabConfig(systemClassLoader=true) is rather important! It's a pity that it is so verbose ...

I tested this scenario with clearing out ~/.groovy/grapes, using the IvySvn resolver over svn+ssh. I also tested using a filesystem resolver (what we usually use).

I also tried:

Grape.setEnableGrapes(true)
Grape.grab('com.oracle#ojdbc6;11.1.0.7.0')
//...
new Sql(new oracle.jdbc.pool.OracleDataSource(URL: url))

But that gives:

unable to resolve class oracle.jdbc.pool.OracleDataSource

Perhaps you could give a hint about correct Grape.grab() usage?

Anything else you'd like to see tested?

Great! Yes, GrabConfig is a little verbose - have to think about that - but seems to work which is good.

The correct Grape.grab() syntax involves using the maps version of grab() with a config map.
Check out here (highlighted comment towards bottom):
http://jira.codehaus.org/browse/GROOVY-3583?focusedCommentId=189944

Anything else? A test of GrabExclude would be good. Quickest hack to test this is to Grab something
with dependencies but indicate that one of the dependencies is to be excluded. It should fail with
no class def found error mentioning that dependency.

I tried an example with an internal project having multiple publications. I was able to get the default configuration's publications using the normal syntax. I was able to get a selected configuration using 'group#module;version[conf]'. Great!

I tried some tests of @GrabExclude().

This works:

@Grab('org.apache.poi#poi;3.1-FINAL')

(brings POI jar and dependencies.)

This works the same:

@Grapes([
  @Grab('org.apache.poi#poi;3.1-FINAL')
])

When I try this:

@Grapes([
  @Grab('org.apache.poi#poi;3.1-FINAL'),
  @GrabExclude('org.apache.log4j:log4j:1.2.13')
])

I get this:

[jhurst@zappa Desktop]$groovy TryGrape.groovy
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
/home/jhurst/Desktop/TryGrape.groovy: 5: 'version'is not part of the annotation groovy.lang.GrabExclude -> groovy.lang.GrabExclude in @groovy.lang.GrabExclude
 @ line 5, column 3.
     @GrabExclude('org.apache.log4j:log4j:1.2.13')
     ^

/home/jhurst/Desktop/TryGrape.groovy: -1: Unexpected type java.lang.Object in @groovy.lang.GrabExclude
 @ line -1, column -1.
2 errors

Same result using the 'org.apache.log4j#log4j;1.2.13' syntax.

Something funny here. I could have sworn I got one variation of this working, but cannot, now. Am I doing it right?

Should it support a list in @GrabExclude, like this?

@Grapes([
  @Grab('org.apache.poi#poi;3.1-FINAL'),
  @GrabExclude(['org.apache.commons#commons-logging;1.1', 'org.apache.log4j#log4j;1.2.13'])
])

That would be nice. Alternatively, not too bad I guess to put:

@Grapes([
  @Grab('org.apache.poi#poi;3.1-FINAL'),
  @GrabExclude('org.apache.commons#commons-logging;1.1'),
  @GrabExclude('org.apache.log4j#log4j;1.2.13')
])

Also, should it support not specifying the version in @GrabExclude, like this?

@Grapes([
  @Grab('org.apache.poi#poi;3.1-FINAL'),
  @GrabExclude('org.apache.commons#commons-logging')
])

That would be nice.

I also noticed that if I don't have any body to the script, only the @Grab annotation,
I get:

  org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
  /home/jhurst/Desktop/TryGrape.groovy: 14: unexpected token:  @ line 14, column 1.
  1 error

I guess it's not valid to put nothing in a script apart from an annotation. (Annotations are supposed to annotate something. right?)

All your examples work as I currently expect. GrabExclude doesn't take a version - it will exclude all versions - that might need finessing.

The other short-hands you suggest are all good, I was trying to avoid having too many complicated options - that would perhaps be better done using a dependency DSL.

More thinking to be done no doubt. Thanks for your efforts.

I see several potential class loader problems here.

(1) this makes compilation environment and execution environment equal. They may not be.
(2) the context loader is set from outside, then this here might overwrite it.

Anything uses root loader, but this seems to use the system loader. which lead to:
(3) libs in the rootloader will shadow libs in the system loader loaded through grapes
(4) grapes libs requiring Groovy will have to load a different Groovy into the system loader

If you execute a script from the command line, then nothing of these is an actual issue. So are we to say, that grape can used only from the commandline?

Hi Jochen,

I think I understand most of where you are coming from but perhaps not all.
Perhaps I'll try to catch you on skype in the morning - I am about to sign off. In case I miss you, some further questions:

For (1), can you elaborate a little more on which part you are referring to?

For (2), wouldn't that just mean don't use that flag in those cases - indeed, if in doubt, leave it out. For cases where you know you need it, it is there to use.

For (3) and (4), Grape.grab() lets you set the class loader already. Hence we already have this problem. No? If I set up my classpath in conflict with what grapes I ask for do I not already have problems?

No doubt there are some tricky things here and plenty of scope for use to rush in a solution that may come back to bite us - on the other hand, the improvements were made specifically to address problems faced by the current user base - so we need to find some way to address them.

With regard to shorthanded versions, I don't know what I was thinking. The current scheme where @Grapes() takes a list of @GrabConfig(), @Grab()s, and @GrabExclude()s is simple, elegant and functional. Adding lists within @GrabExclude() would be detrimental, because it would be inconsistent and more difficult to keep straight in one's head.

Amazing the better perspective one gains from a night's sleep.

Sorry about that.

JH

(1) here I am refering to the fact that you provide up to two class loader for the compiler. The separation is not exactly new, but wasn't done so strictly in the past. But now we seperated that much more, because transforms need a different path potentially. running Groovy from the command line means of course that it is the same class loader, or at last has the same ancestor.

(2) "don't use that flag"... does it mean it is on or off by default? If it is off by default it is ok for me.

(3) If you use Grab.grab() it is up to the user to use the correct loader I would say. I am thinking about the annotation, where the user has not this control.

Added to trunk after further discussions. Could do with some more tests but that might have to wait.

Hmm. As I said in my comment above, I couldn't get @GrabExclude() to work.

Does it work for you?

Here's what I'm trying now, with Groovy trunk:

@Grapes([
  @Grab("org.apache.poi#poi;3.5-beta5"),
  @GrabExclude("logkit#logkit"),
  @GrabExclude("avalon-framework#avalon-framework")
])
void foo() {
}

with this result:

[jhurst@zappa Desktop]$groovy --version
Groovy Version: 1.7-beta-2-SNAPSHOT JVM: 1.6.0_14
[jhurst@zappa Desktop]$rm -rf ~/.groovy/grapes/*
[jhurst@zappa Desktop]$rm -rf ~/.ivy2/cache/*
[jhurst@zappa Desktop]$groovy TryGrapes.groovy 
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
/home/jhurst/Desktop/TryGrapes.groovy: 4: 'version'is not part of the annotation groovy.lang.GrabExclude -> groovy.lang.GrabExclude in @groovy.lang.GrabExclude
 @ line 4, column 3.
     @GrabExclude("logkit#logkit"),
     ^

/home/jhurst/Desktop/TryGrapes.groovy: -1: Unexpected type java.lang.Object in @groovy.lang.GrabExclude
 @ line -1, column -1.
/home/jhurst/Desktop/TryGrapes.groovy: 5: 'version'is not part of the annotation groovy.lang.GrabExclude -> groovy.lang.GrabExclude in @groovy.lang.GrabExclude
 @ line 5, column 3.
     @GrabExclude("avalon-framework#avalon-framework")
     ^

/home/jhurst/Desktop/TryGrapes.groovy: -1: Unexpected type java.lang.Object in @groovy.lang.GrabExclude
 @ line -1, column -1.
4 errors

I'm not clear on whether @GrabExclude is supposed to be working.

My bad, can you try again now (with trunk). The example below should work for example:

@Grapes([
  @Grab("org.apache.poi#poi;3.5-beta6"),
  @GrabExclude("logkit:logkit"),
  @GrabExclude("avalon-framework#avalon-framework")
])
import org.apache.poi.hssf.util.CellReference
assert new CellReference(0, 0, false, false).formatAsString() == 'A1'
assert new CellReference(1, 3).formatAsString() == '$D$2'

Thanks Paul, that's working now.

I like this mechanism!

JH