There should be a way to run unit tests from a dependency jar.

Basically the description at http://maven.apache.org/guides/mini/guide-attached-tests.html gives you the procedure to create the test.jar. However, the plugin does not execute the tests from the JAR, rendering the whole thing rather useless.

One way to solve the issue would be to have the surefire plugin check all dependencies of type "test-jar" (or classifier "tests") for tests, too.

Another way would be to provide additional dependencies for the surefire-plugin in the plugin's configuration.

As workaround simply unpack the jar with the dependency plugin into target/test-classes.

Yes, this is a work around. However, it means not just adding a test-dependency but also fiddling with the build, which is rather dangerous.

In the one case I can tell my teammates: Hey, I got this cool set if tests you can run your implementation against, just add this dependency with scope "test". In the other I'd have to introduce them to the mysteries of plugin configurations and build process customization. The main beauty of maven is that it just works for most cases without having to tell anyone how to get it to work.

One drawback of simply using dependency:unpack as a workaround is that the plugin doesn't work properly in conjunction with M2Eclipse's workspace resolution mechanism (cf. https://issues.sonatype.org/browse/MNGECLIPSE-1027). That being said, Thomas's argument should be convincing enough by itself.

This is possible if using TestNG. Define a suite XML file in the Surefire plugin configuration, then in the suite XML file include any TestNG test on the testing classpath.

Tim, that is correct.

But no decent tessuite just contains a few test classes, it contains hundreds if not thousands of classes for testing. Having "AllSuites" for them all and requiring users to subclass them all to ensure it actually gets run is an unnecessary cumbersome step IMO.

The inheriting test suite can extend TestSuite or TestCase (or implement Test itself), but the suite() providing class should not extend either, or it too will be discovered my the Maven Surefire plugin and your tests will be run twice.

@Max Andersen completely agree, I am amazed, in the light of http://maven.apache.org/guides/mini/guide-attached-tests.html at the existence of this bug.

I jarred my tests in 2008, assuming that this was the expected use case, then blocked until now.

I do not want to have to manually maintain the test suite, but that seems to be the only currently available mechanism.

I have a fix for this here: https://github.com/aslakknutsen/maven-surefire/commit/d6390d36fc491db10e7579af1535879e8b181aa0

It basically allow you to configure surefire to scan specific Dependencies for TestClasses via the 'dependenciesToScan' options.

comments?

There's almost always some controveresy regarding the use case for this issue, but I think it's time we add this feature. @Aslak; your patches are usually nice and I just skimmed it quickly on-line; will go through it in more detail later.

You will need to attach the patch to the issue here too, since we're not entirely in the clear on accepting pull requests yet. Things move slowly around here

git diff attached. Need some other format?

I'm not going to argue with your process, you do as you please, just mentioning that technically you could pull it directly from my repo via git without it being a pull request pr say.

git remote add aslak git://github.com/aslakknutsen/maven-surefire.git
git fetch aslak
git cherry-pick d6390d36fc491db10e7579af1535879e8b181aa0

I attempted to change as little existing code as possible to avoid any conflict. (Basically only extracted some of the DirectoryScanner 'util' methods into ScannerUtil for reuse in DependencyScanner).

One little challenge with this now tho, is that even tho DependencyScanner and DirectoryScanner use the same includes/excludes/test expressions, they don't share the actual matcher logic. The Surefire DirectoryScanner hands of the matching to shared.io.DirectoryScanner and I found no easy way to reuse that logic. Both DirectoryScanner and DependencyScanner in the end use MatchPatterns, but DirectoryScanner use it via shared.io.DirectoryScanner..

Another oddity is the 'merging' of the DependecyScanner and DirectoryScanner results. Didn't want to change DefaultScannerResult, but maybe add a method for merge/append or similar to DefaultScannerResult would be better.

List combined = new ArrayList();
combined.addAll(scan.getFiles());
combined.addAll(scanDeps.getFiles());
scan = new DefaultScanResult(combined);

Currently it's just for legal reasons, we need a record of submission that is in our systems; github is really not. We hope to get this sorted out "soon".

Things move a bit slowly here

I will look at the patch sometime soon; 2.14 is still a few weeks away.

Yea, assumed it was for legal reasons. You need a CLA or similar signed?

Any progress on this issue?

Having spent the best part of a day trying to workaround this, I can't emphasise enough how much this is needed. Being able to produce a jar of test classes without being able to run them in a separate project is basically just a bug.

The patch could do with an enhancement, which is to pickup all jar files of type "test-jar", rather than having to name each individually, but I'd take the basic patch right now.

If you search the web, you'll see lots of people want to be able to test outside the original project.
http://softwaremavens.blogspot.co.uk/2009/09/running-tests-from-maven-test-jar-in.html
http://dharshanaw.blogspot.co.uk/2012/10/how-to-execute-testng-tests-in-side.html
http://stackoverflow.com/questions/3543473/run-maven-tests-from-classpath
http://stackoverflow.com/questions/13859737/run-testng-on-a-maven-test-dependency
http://stackoverflow.com/questions/11331518/testng-execute-compiled-tests-in-jar-file

I wanted to specify which artifacts to scan basically because I have use cases where I depend on test-jars without wanting to run the tests in them.

I often operate with a 'shadow' test dependency tree to reuse Test APIs exposed by other modules.

Could be seen as 3 artifacts:
'module tests' depends on 'module test api' depends on 'module'

But 'module tests' and 'module test api' are merged into one artifact: test-jar

So 'reuse/depend on' and 'run' are two different stories.

Strictly speaking the filter expression should accept wildcards and 4-part specifications like for instance includes does on maven-shade-plugin; which would allow for specifications like G:*:*:test-jar
which would pick up all test-jars from a given group-id.

The artifact filtering could/should probably use maven-common-artifact-filters if it is to be extended any further; this module was extracted with filtering in mind.

I think the current patch is a nice starting point and I have applied it in c808eec805509f14160454829df8f496499f0f4e.

@Aslak; I appreciate you keeping the patch footprint moderate, it makes it easier to review and accept. On the other hand I think all the changes you describe should be done I would've ok'd all those changes and may choose do some of them myself !

Thanks for the patch!

And yes, surefire has pretty good test coverage. If your change does not break a test then it's likely you didn't break anything. So I'm generally not afraid of changes to existing code...

Thank you very much for applying the patch. And yes, that extra "standardised" filtering would be very nice!