Integration tests lifecycle

we chatted with jdcasey for add some new phases.

For integration test source directory, I think it must be defined as a plugin parameter because a project can have multiple type of integration tests (tests for war/ear, test for ui, test for performance...). Each test type are specific to a plugin.

Yes, the entire thing could be just a single plugin. I also talked to jdcasey, and we liked the following approach:

src/itest/projects/project-a/ [pom.xml, etc]
src/itest/projects/project-b/ [pom.xml. etc]

Each project would be executed (an 'm2 install' forked somehow) in a new environment containing
the artifact[s] of the project (the one the src/itest dir is a part of).

This way plugins can be easily tested - you still need to specify that you want to use the plugin
in the project-a's pom, of course.

The same goes for 'normal' projects, like those with packaging war/ear. The nested test
project could use normal tests to test the artifact, for instance using cargo in a Suite to deploy
the artifact and then run tests.

The artifact here is from the project being integration-tested. It can be referenced by defining
a dependency in the nested test project, and a different ArtifactFactory will be used in the forked
lifecycle, so that the packaged (but not installed) artifact gets picked up from the target/ dir.

So basically, the integration-test could stay unmodified, as long as it's possible to fork an m2 lifecycle on
a different project (than a clone of the current one). This would mean exposing more of the lifecycle to plugins.

Even better, per packaging a different integration-test plugin/mojo could be used, because plugins / war / ear / ejb / etc
may need different test circumstances. But this could also be specified in the nested test projects.

just trying to collect all itest-related stuff, and since I've got this one on my radar, this is as good of a focal point as any.

Currently, we have "test" which exists before package, and has its own source code and resources, but its classes are not processed like a normal build. Then we have integration-test and verify which occur after packaging, but before installation, and don't have any sources of their own.

We are also missing the case where you might have intensive tests that you don't want to run during development. I still think these belong before install, and can be enabled in an integration environment using profiles instead.

Can we think of better ways to specify test sets than this?

What I'm thinking is:

• we have unit-test, functional-test (both before package), and integration-test
• each has an identical lifecycle to main (generate/process-sources/resources, compile, process-classes)
• it is up to the user to use profiles to apply things they want in dev or not

I think the itest plugin IDEA above is good (could we use the verifier plugin?) However, I don't think we should have to rely on it being installed before it is tested. Maybe that's not a big deal, esp. since the reactor should be able to take care of it, however I feel there is some merit to keeping it all together, especially as IDEs might deal with it better that way if they have plugins for the testing framework. Of course it is still possible to have a separate test suite, using dependencies to get the code to test.

I'm still a little uncertain how we might do something like cactus in the same as m1. Perhaps it could fork it's own lifecycle to compile it's special sources, then package them up, deploy, and run tests? It could map flat to the integration-test-* stuff, but then if you have >1 the order is whacky and you also end up having a cactus:compile goal that has to extend AbstractCompilerMojo which is not ideal.

John's point in MNG-475 is relevant to the above:

" how is providing integration-test phase dependencies any different than providing an <integrationTestSourceDirectory/>?

In other words, if we're leaving the source directory to be specified by cactus, etc. (because they may need different sources, or none), then why can we assume that all integration tests will have the same dependency needs?"

maybe we need to add dependencies that are included in the execution context of a plugin ONLY...that way, we can have cactus-specific dependencies which are only relevant for our project (i.e. not needed for every cactus execution everywhere...they'd be plugin-deps in this case).

Something like:

<build>
<plugins>
<plugin>
<!-- g:a:v -->
<dependencies>
<dependency>
<groupId>com.myorg.project</groupId>
<artifactId>myproject-testharness</artifactId>
<version>1.0</version>
</dependency>
</dependencies>

<configuration/>
</plugin>
</plugins>
</build>

It's a little awkward, but at least it's immediately apparent where these deps will be used...

the previous comment was actually kenney's idea, I just thought we need to keep it in the discussion...

wanna keep the attribution straight here...

That really wasn't necessary! I should have filed it.

As long as we're being complete, I'll add another reason this is nice to have and some more food for thought:

The xdoclet plugin currently has to have dependencies on all plugins for xdoclet itself (container / environment specific),
because there's no way to add those to the plugin execution environment, except now using <extensions> which contaminates
the root classloader. So basically by allowing this you can have plugins for the plugins.

However, what we need to consider is the need to add some libraries to higher level than the plugin classloader, for instance to
a lifecycle phase. Currently artifacts with scope=provided aren't being resolved in the test scope:

(ScopeArtifactFilter.java:
else if ( DefaultArtifact.SCOPE_TEST.equals( scope ) )

)

This poses a problem for the eclipse plugin. Enabling this will fix that, but they stil aren't included in the plugin
classpath - the plugin will need to create it's own classloader for that. However, sometimes just an -api artifact is
depended upon, and during tests an -impl version should be provided - ideally this artifact is present in the classpath during all mojo
executions in a particular phase. (This may be a bad example since there' s only one test plugin - does anybody have more examples?)

in the xdoclet example above, would it make more sense to emulate the build-extension pattern and allow plugin-extensions that are loaded into the plugin container? I know this is a repackaging of the plugin-dependency argument, but I believe calling them plugin-extensions would be more "correct".

Also, it would remain up to the plugin's underlying technology (xdoclet, f.e.) to actually discover the optional components contained in the extension artifacts...we'd just make them available.

the problem with adding integration-test dependency scope and srcdir is that this is a slippery slope. Can we really define all of the types of tests that are out there, and if we do, do we really want <integrationTestSourceDirectory/> and <functionalTestSourceDirectory/> and ...?

I think it's reasonable to expect that integration- or functional- or ... tests will need their own set of dependencies, and that it's unreasonable to expect that these can be satisfied using <scope>test</scope>. The source dir is easier, although we'd still have to provide a utility to help these plugins compile it.

This got shot down once before, but I'll toss it out there again, just to see if it'll help generate some discussion/thinking:

what about allowing dependencies to be tied to a particular lifecycle phase?

While this would help by allowing <phase>integration-test</phase> in the dependency, it could conceivably muddy the waters of scope vs. bindings for dependencies. Calculating inheritedScope is easy on this...if it's not runtime || compile, it's not included transitively.

I think it might not be such a great thing to ship maven 2.0 with the disclaimer "We can't handle all of your testing needs yet." However, I'm not sure what the way forward is. I'd really like to get started on this issue, though, since it looks like the highest-risk issue left for -beta-1.

I'm taking a look at this. Just to answer about dependencies and scopes - I think per plugin dependencies are a better idea than going to phase over scope. I originally proposed phase instead of scope, but now see the relationships not mapping that way, as we discussed in March.

So I think test deps go to all tests, and anything specific to a certain type of test goes to that plugin.

types of tests:

• unit tests - run before package, want 100% success before moving on generally
• integration tests - require the artifact, so run after package but before install. generally want 100% before moving on. May use junit or other plugin like cactus, including deploying to a server
• acceptance/regression/functional tests - not run every time, probably a report and/or part of a profile. Regression tests (acceptance tests from previous milestones) should be at 100%, whereas acceptance (also called functional) tests only need to get to 100% at release time.
• performance/stress tests - another form of acceptance test, really.

we should have a setup and teardown phase around the actual test goal

I just committed sandbox/maven-it-plugin which has the ability to fork maven (sort of)
to run all projects found in src/it/. Use like this:

<plugin>
<artifactId>maven-it-plugin</artifactId>
<executions>
<execution>
<phase>integration-test</phase>
<goals>
<goal>fork</goal>
</goals>
</execution>
</executions>
</plugin>

Brett, can you see if this is a suitable solution for this issue? All setup/teardown/deployment/whatever
can now be done in the integration test projects. Test projects can depend on the parent project
(which could be added at runtime) so that they have access to the generated artifact (and test against it).

Kenney's plugin should be sufficient for 2.0

After reading all this, I'm a bit confused. What is the recommended approach in maven2 v1.x for handling different test types?

I pretty much need test types to exist in different directories so I can run them independently in eclipse. I don't want to use packages to segregate the tests, because I need the unit tests to exist the same directory as the CUT. Is seperate projects the only solution for maven2 v1.x? And how does this relate to http://jira.codehaus.org/browse/SUREFIRE-7?

I would like to split out some of my heavier tests as integration tests. i'd kinda hoped that if I put my src files into src/itest/java/.. and added a simple
<plugin>
<artifactId>maven-it-plugin</artifactId>
</plugin>

then maven2 would have magically worked. alas not.

i'm not at work now so can access svn (hooray) - but can't find where to check out the sandbox plugins.

I was able to check out http://svn.apache.org/repos/asf/maven/components/trunk/ and build it fine, and some googling turned up http://svn.apache.org/repos/asf/maven/plugins/trunk/ too but i can't find, nor guess the sandbox url. could someone perhaps post that here, or even a url of a page detailing where to find such things.

cheers dave

worth seeing Craig's use case in MNG-2344 also.

are we happy with IT's now, or do we need another proposal for this? Kenney?

The last comment in here was years ago, inviting the OP to comment if the work done was insufficient. There is no comment in all that time, so a close is overdue.