For integration test source directory, I think it must be defined as a plugin parameter because a project can have multiple type of integration tests (tests for war/ear, test for ui, test for performance...). Each test type are specific to a plugin.

src/itest/projects/project-a/ [pom.xml, etc]
src/itest/projects/project-b/ [pom.xml. etc]

Each project would be executed (an 'm2 install' forked somehow) in a new environment containing
the artifact[s] of the project (the one the src/itest dir is a part of).

This way plugins can be easily tested - you still need to specify that you want to use the plugin
in the project-a's pom, of course.

The same goes for 'normal' projects, like those with packaging war/ear. The nested test
project could use normal tests to test the artifact, for instance using cargo in a Suite to deploy
the artifact and then run tests.

The artifact here is from the project being integration-tested. It can be referenced by defining
a dependency in the nested test project, and a different ArtifactFactory will be used in the forked
lifecycle, so that the packaged (but not installed) artifact gets picked up from the target/ dir.

So basically, the integration-test could stay unmodified, as long as it's possible to fork an m2 lifecycle on
a different project (than a clone of the current one). This would mean exposing more of the lifecycle to plugins.

Even better, per packaging a different integration-test plugin/mojo could be used, because plugins / war / ear / ejb / etc
may need different test circumstances. But this could also be specified in the nested test projects.

We are also missing the case where you might have intensive tests that you don't want to run during development. I still think these belong before install, and can be enabled in an integration environment using profiles instead.

Can we think of better ways to specify test sets than this?

What I'm thinking is:

• we have unit-test, functional-test (both before package), and integration-test
• each has an identical lifecycle to main (generate/process-sources/resources, compile, process-classes)
• it is up to the user to use profiles to apply things they want in dev or not

I think the itest plugin IDEA above is good (could we use the verifier plugin?) However, I don't think we should have to rely on it being installed before it is tested. Maybe that's not a big deal, esp. since the reactor should be able to take care of it, however I feel there is some merit to keeping it all together, especially as IDEs might deal with it better that way if they have plugins for the testing framework. Of course it is still possible to have a separate test suite, using dependencies to get the code to test.

I'm still a little uncertain how we might do something like cactus in the same as m1. Perhaps it could fork it's own lifecycle to compile it's special sources, then package them up, deploy, and run tests? It could map flat to the integration-test-* stuff, but then if you have >1 the order is whacky and you also end up having a cactus:compile goal that has to extend AbstractCompilerMojo which is not ideal.

" how is providing integration-test phase dependencies any different than providing an <integrationTestSourceDirectory/>?

In other words, if we're leaving the source directory to be specified by cactus, etc. (because they may need different sources, or none), then why can we assume that all integration tests will have the same dependency needs?"

Something like:

<build>
<plugins>
<plugin>
<!-- g:a:v -->
<dependencies>
<dependency>
<groupId>com.myorg.project</groupId>
<artifactId>myproject-testharness</artifactId>
<version>1.0</version>
</dependency>
</dependencies>

<configuration/>
</plugin>
</plugins>
</build>

It's a little awkward, but at least it's immediately apparent where these deps will be used...

wanna keep the attribution straight here...

As long as we're being complete, I'll add another reason this is nice to have and some more food for thought:

The xdoclet plugin currently has to have dependencies on all plugins for xdoclet itself (container / environment specific),
because there's no way to add those to the plugin execution environment, except now using <extensions> which contaminates
the root classloader. So basically by allowing this you can have plugins for the plugins.

However, what we need to consider is the need to add some libraries to higher level than the plugin classloader, for instance to
a lifecycle phase. Currently artifacts with scope=provided aren't being resolved in the test scope:

(ScopeArtifactFilter.java:
else if ( DefaultArtifact.SCOPE_TEST.equals( scope ) )

)

This poses a problem for the eclipse plugin. Enabling this will fix that, but they stil aren't included in the plugin
classpath - the plugin will need to create it's own classloader for that. However, sometimes just an -api artifact is
depended upon, and during tests an -impl version should be provided - ideally this artifact is present in the classpath during all mojo
executions in a particular phase. (This may be a bad example since there' s only one test plugin - does anybody have more examples?)

Also, it would remain up to the plugin's underlying technology (xdoclet, f.e.) to actually discover the optional components contained in the extension artifacts...we'd just make them available.

I think it's reasonable to expect that integration- or functional- or ... tests will need their own set of dependencies, and that it's unreasonable to expect that these can be satisfied using <scope>test</scope>. The source dir is easier, although we'd still have to provide a utility to help these plugins compile it.

This got shot down once before, but I'll toss it out there again, just to see if it'll help generate some discussion/thinking:

what about allowing dependencies to be tied to a particular lifecycle phase?

While this would help by allowing <phase>integration-test</phase> in the dependency, it could conceivably muddy the waters of scope vs. bindings for dependencies. Calculating inheritedScope is easy on this...if it's not runtime || compile, it's not included transitively.

I think it might not be such a great thing to ship maven 2.0 with the disclaimer "We can't handle all of your testing needs yet." However, I'm not sure what the way forward is. I'd really like to get started on this issue, though, since it looks like the highest-risk issue left for -beta-1.

So I think test deps go to all tests, and anything specific to a certain type of test goes to that plugin.

• unit tests - run before package, want 100% success before moving on generally
• integration tests - require the artifact, so run after package but before install. generally want 100% before moving on. May use junit or other plugin like cactus, including deploying to a server
• acceptance/regression/functional tests - not run every time, probably a report and/or part of a profile. Regression tests (acceptance tests from previous milestones) should be at 100%, whereas acceptance (also called functional) tests only need to get to 100% at release time.
• performance/stress tests - another form of acceptance test, really.

• plugin integration tests (c.f. eclipse plugin)
• report integration tests (see Vincent S's work - htmlunit)
• cactus plugin
• geronimo itest plugin
• need to be able to compile additional sources, have own dependencies (basically reapply the whole test lifecycle again - given that this also matches the main one, perhaps that can be reusable/forked)

separate project is ideal
+ fits most things by dealing with them in the same way

• a bit unnatural, more strung out
• doesn't fit IDE well

While a separate project might be something we recommend, I think we need to support them within the same project.

I'll put together a solution tomorrow.

<plugin>
<artifactId>maven-it-plugin</artifactId>
<executions>
<execution>
<phase>integration-test</phase>
<goals>
<goal>fork</goal>
</goals>
</execution>
</executions>
</plugin>

Brett, can you see if this is a suitable solution for this issue? All setup/teardown/deployment/whatever
can now be done in the integration test projects. Test projects can depend on the parent project
(which could be added at runtime) so that they have access to the generated artifact (and test against it).

I pretty much need test types to exist in different directories so I can run them independently in eclipse. I don't want to use packages to segregate the tests, because I need the unit tests to exist the same directory as the CUT. Is seperate projects the only solution for maven2 v1.x? And how does this relate to http://jira.codehaus.org/browse/SUREFIRE-7?

then maven2 would have magically worked. alas not.

i'm not at work now so can access svn (hooray) - but can't find where to check out the sandbox plugins.

I was able to check out http://svn.apache.org/repos/asf/maven/components/trunk/ and build it fine, and some googling turned up http://svn.apache.org/repos/asf/maven/plugins/trunk/ too but i can't find, nor guess the sandbox url. could someone perhaps post that here, or even a url of a page detailing where to find such things.

cheers dave