Maven Surefire
  1. Maven Surefire
  2. SUREFIRE-569

There should be a way to run unit tests from a dependency jar.

    Details

    • Type: New Feature New Feature
    • Status: Closed Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.15
    • Component/s: Maven Surefire Plugin
    • Labels:
      None
    • Complexity:
      Intermediate
    • Number of attachments :
      1

      Description

      In some cases it would be useful to have a set of tests that run with various dependency configurations. One way to accomplish this would be to have a single project that contains the unit tests and generates a test jar. Several test configuration projects could then consume the unit tests and run them with different dependency sets. The problem is that there is no easy way to run tests in a dependency jar. The surefire plugin should have a configuration to allow me to run all or a set of unit tests contained in a dependency jar.

        Activity

        Hide
        Thomas Strecker added a comment -

        Basically the description at http://maven.apache.org/guides/mini/guide-attached-tests.html gives you the procedure to create the test.jar. However, the plugin does not execute the tests from the JAR, rendering the whole thing rather useless.

        One way to solve the issue would be to have the surefire plugin check all dependencies of type "test-jar" (or classifier "tests") for tests, too.

        Another way would be to provide additional dependencies for the surefire-plugin in the plugin's configuration.

        Show
        Thomas Strecker added a comment - Basically the description at http://maven.apache.org/guides/mini/guide-attached-tests.html gives you the procedure to create the test.jar. However, the plugin does not execute the tests from the JAR, rendering the whole thing rather useless. One way to solve the issue would be to have the surefire plugin check all dependencies of type "test-jar" (or classifier "tests") for tests, too. Another way would be to provide additional dependencies for the surefire-plugin in the plugin's configuration.
        Hide
        Jörg Schaible added a comment -

        As workaround simply unpack the jar with the dependency plugin into target/test-classes.

        Show
        Jörg Schaible added a comment - As workaround simply unpack the jar with the dependency plugin into target/test-classes.
        Hide
        Thomas Strecker added a comment -

        Yes, this is a work around. However, it means not just adding a test-dependency but also fiddling with the build, which is rather dangerous.

        In the one case I can tell my teammates: Hey, I got this cool set if tests you can run your implementation against, just add this dependency with scope "test". In the other I'd have to introduce them to the mysteries of plugin configurations and build process customization. The main beauty of maven is that it just works for most cases without having to tell anyone how to get it to work.

        Show
        Thomas Strecker added a comment - Yes, this is a work around. However, it means not just adding a test-dependency but also fiddling with the build, which is rather dangerous. In the one case I can tell my teammates: Hey, I got this cool set if tests you can run your implementation against, just add this dependency with scope "test". In the other I'd have to introduce them to the mysteries of plugin configurations and build process customization. The main beauty of maven is that it just works for most cases without having to tell anyone how to get it to work.
        Hide
        Andreas Sewe added a comment -

        One drawback of simply using dependency:unpack as a workaround is that the plugin doesn't work properly in conjunction with M2Eclipse's workspace resolution mechanism (cf. https://issues.sonatype.org/browse/MNGECLIPSE-1027). That being said, Thomas's argument should be convincing enough by itself.

        Show
        Andreas Sewe added a comment - One drawback of simply using dependency:unpack as a workaround is that the plugin doesn't work properly in conjunction with M2Eclipse's workspace resolution mechanism (cf. https://issues.sonatype.org/browse/MNGECLIPSE-1027 ). That being said, Thomas's argument should be convincing enough by itself.
        Hide
        Orien Madgwick added a comment -

        This is possible if using TestNG. Define a suite XML file in the Surefire plugin configuration, then in the suite XML file include any TestNG test on the testing classpath.

        Show
        Orien Madgwick added a comment - This is possible if using TestNG. Define a suite XML file in the Surefire plugin configuration, then in the suite XML file include any TestNG test on the testing classpath.
        Hide
        Tim Pizey added a comment -

        This can be done as follows (junit3):

        Ensure test jar contains a class which has a static suite() method

        import junit.framework.Test;
        import junit.framework.TestSuite;

        public class AllTests {

        public static Test suite()

        { TestSuite suite = new TestSuite( "All Tests"); suite.addTestSuite(TestOne.class); suite.addTestSuite(TestTwo.class); return suite; }

        }

        Then in the project using the test-jar dependency:
        create a TestCase:

        package org.melati.example.contacts;

        import org.melati.poem.AllExportedTests;

        import junit.framework.Test;
        import junit.framework.TestCase;

        public class PoemTest extends TestCase {

        public static Test suite()

        { return AllExportedTests.suite(); }

        }

        Now the tests will be found.

        Show
        Tim Pizey added a comment - This can be done as follows (junit3): Ensure test jar contains a class which has a static suite() method import junit.framework.Test; import junit.framework.TestSuite; public class AllTests { public static Test suite() { TestSuite suite = new TestSuite( "All Tests"); suite.addTestSuite(TestOne.class); suite.addTestSuite(TestTwo.class); return suite; } } Then in the project using the test-jar dependency: create a TestCase: package org.melati.example.contacts; import org.melati.poem.AllExportedTests; import junit.framework.Test; import junit.framework.TestCase; public class PoemTest extends TestCase { public static Test suite() { return AllExportedTests.suite(); } } Now the tests will be found.
        Hide
        Max Andersen added a comment -

        Tim, that is correct.

        But no decent tessuite just contains a few test classes, it contains hundreds if not thousands of classes for testing. Having "AllSuites" for them all and requiring users to subclass them all to ensure it actually gets run is an unnecessary cumbersome step IMO.

        Show
        Max Andersen added a comment - Tim, that is correct. But no decent tessuite just contains a few test classes, it contains hundreds if not thousands of classes for testing. Having "AllSuites" for them all and requiring users to subclass them all to ensure it actually gets run is an unnecessary cumbersome step IMO.
        Hide
        Tim Pizey added a comment -

        The inheriting test suite can extend TestSuite or TestCase (or implement Test itself), but the suite() providing class should not extend either, or it too will be discovered my the Maven Surefire plugin and your tests will be run twice.

        Show
        Tim Pizey added a comment - The inheriting test suite can extend TestSuite or TestCase (or implement Test itself), but the suite() providing class should not extend either, or it too will be discovered my the Maven Surefire plugin and your tests will be run twice.
        Hide
        Tim Pizey added a comment -

        @Max Andersen completely agree, I am amazed, in the light of http://maven.apache.org/guides/mini/guide-attached-tests.html at the existence of this bug.

        I jarred my tests in 2008, assuming that this was the expected use case, then blocked until now.

        I do not want to have to manually maintain the test suite, but that seems to be the only currently available mechanism.

        Show
        Tim Pizey added a comment - @Max Andersen completely agree, I am amazed, in the light of http://maven.apache.org/guides/mini/guide-attached-tests.html at the existence of this bug. I jarred my tests in 2008, assuming that this was the expected use case, then blocked until now. I do not want to have to manually maintain the test suite, but that seems to be the only currently available mechanism.
        Hide
        Aslak Knutsen added a comment -

        I have a fix for this here: https://github.com/aslakknutsen/maven-surefire/commit/d6390d36fc491db10e7579af1535879e8b181aa0

        It basically allow you to configure surefire to scan specific Dependencies for TestClasses via the 'dependenciesToScan' options.

        comments?

        Show
        Aslak Knutsen added a comment - I have a fix for this here: https://github.com/aslakknutsen/maven-surefire/commit/d6390d36fc491db10e7579af1535879e8b181aa0 It basically allow you to configure surefire to scan specific Dependencies for TestClasses via the 'dependenciesToScan' options. comments?
        Hide
        Kristian Rosenvold added a comment -

        There's almost always some controveresy regarding the use case for this issue, but I think it's time we add this feature. @Aslak; your patches are usually nice and I just skimmed it quickly on-line; will go through it in more detail later.

        You will need to attach the patch to the issue here too, since we're not entirely in the clear on accepting pull requests yet. Things move slowly around here

        Show
        Kristian Rosenvold added a comment - There's almost always some controveresy regarding the use case for this issue, but I think it's time we add this feature. @Aslak; your patches are usually nice and I just skimmed it quickly on-line; will go through it in more detail later. You will need to attach the patch to the issue here too, since we're not entirely in the clear on accepting pull requests yet. Things move slowly around here
        Hide
        Aslak Knutsen added a comment -

        git diff attached. Need some other format?

        I'm not going to argue with your process, you do as you please, just mentioning that technically you could pull it directly from my repo via git without it being a pull request pr say.

        git remote add aslak git://github.com/aslakknutsen/maven-surefire.git
        git fetch aslak
        git cherry-pick d6390d36fc491db10e7579af1535879e8b181aa0
        
        
        Show
        Aslak Knutsen added a comment - git diff attached. Need some other format? I'm not going to argue with your process, you do as you please, just mentioning that technically you could pull it directly from my repo via git without it being a pull request pr say. git remote add aslak git: //github.com/aslakknutsen/maven-surefire.git git fetch aslak git cherry-pick d6390d36fc491db10e7579af1535879e8b181aa0
        Hide
        Aslak Knutsen added a comment -

        I attempted to change as little existing code as possible to avoid any conflict. (Basically only extracted some of the DirectoryScanner 'util' methods into ScannerUtil for reuse in DependencyScanner).

        One little challenge with this now tho, is that even tho DependencyScanner and DirectoryScanner use the same includes/excludes/test expressions, they don't share the actual matcher logic. The Surefire DirectoryScanner hands of the matching to shared.io.DirectoryScanner and I found no easy way to reuse that logic. Both DirectoryScanner and DependencyScanner in the end use MatchPatterns, but DirectoryScanner use it via shared.io.DirectoryScanner..

        Another oddity is the 'merging' of the DependecyScanner and DirectoryScanner results. Didn't want to change DefaultScannerResult, but maybe add a method for merge/append or similar to DefaultScannerResult would be better.

        List combined = new ArrayList();
        combined.addAll(scan.getFiles());
        combined.addAll(scanDeps.getFiles());
        scan = new DefaultScanResult(combined);
        
        Show
        Aslak Knutsen added a comment - I attempted to change as little existing code as possible to avoid any conflict. (Basically only extracted some of the DirectoryScanner 'util' methods into ScannerUtil for reuse in DependencyScanner). One little challenge with this now tho, is that even tho DependencyScanner and DirectoryScanner use the same includes/excludes/test expressions, they don't share the actual matcher logic. The Surefire DirectoryScanner hands of the matching to shared.io.DirectoryScanner and I found no easy way to reuse that logic. Both DirectoryScanner and DependencyScanner in the end use MatchPatterns, but DirectoryScanner use it via shared.io.DirectoryScanner.. Another oddity is the 'merging' of the DependecyScanner and DirectoryScanner results. Didn't want to change DefaultScannerResult, but maybe add a method for merge/append or similar to DefaultScannerResult would be better. List combined = new ArrayList(); combined.addAll(scan.getFiles()); combined.addAll(scanDeps.getFiles()); scan = new DefaultScanResult(combined);
        Hide
        Kristian Rosenvold added a comment -

        Currently it's just for legal reasons, we need a record of submission that is in our systems; github is really not. We hope to get this sorted out "soon".

        Things move a bit slowly here

        I will look at the patch sometime soon; 2.14 is still a few weeks away.

        Show
        Kristian Rosenvold added a comment - Currently it's just for legal reasons, we need a record of submission that is in our systems; github is really not. We hope to get this sorted out "soon". Things move a bit slowly here I will look at the patch sometime soon; 2.14 is still a few weeks away.
        Hide
        Aslak Knutsen added a comment -

        Yea, assumed it was for legal reasons. You need a CLA or similar signed?

        Show
        Aslak Knutsen added a comment - Yea, assumed it was for legal reasons. You need a CLA or similar signed?
        Hide
        Cristiano Gavião added a comment -

        Any progress on this issue?

        Show
        Cristiano Gavião added a comment - Any progress on this issue?
        Hide
        Stephen Colebourne added a comment -

        Having spent the best part of a day trying to workaround this, I can't emphasise enough how much this is needed. Being able to produce a jar of test classes without being able to run them in a separate project is basically just a bug.

        The patch could do with an enhancement, which is to pickup all jar files of type "test-jar", rather than having to name each individually, but I'd take the basic patch right now.

        If you search the web, you'll see lots of people want to be able to test outside the original project.
        http://softwaremavens.blogspot.co.uk/2009/09/running-tests-from-maven-test-jar-in.html
        http://dharshanaw.blogspot.co.uk/2012/10/how-to-execute-testng-tests-in-side.html
        http://stackoverflow.com/questions/3543473/run-maven-tests-from-classpath
        http://stackoverflow.com/questions/13859737/run-testng-on-a-maven-test-dependency
        http://stackoverflow.com/questions/11331518/testng-execute-compiled-tests-in-jar-file

        Show
        Stephen Colebourne added a comment - Having spent the best part of a day trying to workaround this, I can't emphasise enough how much this is needed. Being able to produce a jar of test classes without being able to run them in a separate project is basically just a bug. The patch could do with an enhancement, which is to pickup all jar files of type "test-jar", rather than having to name each individually, but I'd take the basic patch right now. If you search the web, you'll see lots of people want to be able to test outside the original project. http://softwaremavens.blogspot.co.uk/2009/09/running-tests-from-maven-test-jar-in.html http://dharshanaw.blogspot.co.uk/2012/10/how-to-execute-testng-tests-in-side.html http://stackoverflow.com/questions/3543473/run-maven-tests-from-classpath http://stackoverflow.com/questions/13859737/run-testng-on-a-maven-test-dependency http://stackoverflow.com/questions/11331518/testng-execute-compiled-tests-in-jar-file
        Hide
        Aslak Knutsen added a comment -

        I wanted to specify which artifacts to scan basically because I have use cases where I depend on test-jars without wanting to run the tests in them.

        I often operate with a 'shadow' test dependency tree to reuse Test APIs exposed by other modules.

        Could be seen as 3 artifacts:
        'module tests' depends on 'module test api' depends on 'module'

        But 'module tests' and 'module test api' are merged into one artifact: test-jar

        So 'reuse/depend on' and 'run' are two different stories.

        Show
        Aslak Knutsen added a comment - I wanted to specify which artifacts to scan basically because I have use cases where I depend on test-jars without wanting to run the tests in them. I often operate with a 'shadow' test dependency tree to reuse Test APIs exposed by other modules. Could be seen as 3 artifacts: 'module tests' depends on 'module test api' depends on 'module' But 'module tests' and 'module test api' are merged into one artifact: test-jar So 'reuse/depend on' and 'run' are two different stories.
        Hide
        Kristian Rosenvold added a comment - - edited

        Strictly speaking the filter expression should accept wildcards and 4-part specifications like for instance includes does on maven-shade-plugin; which would allow for specifications like G:*:*:test-jar
        which would pick up all test-jars from a given group-id.

        The artifact filtering could/should probably use maven-common-artifact-filters if it is to be extended any further; this module was extracted with filtering in mind.

        I think the current patch is a nice starting point and I have applied it in c808eec805509f14160454829df8f496499f0f4e.

        @Aslak; I appreciate you keeping the patch footprint moderate, it makes it easier to review and accept. On the other hand I think all the changes you describe should be done I would've ok'd all those changes and may choose do some of them myself !

        Thanks for the patch!

        Show
        Kristian Rosenvold added a comment - - edited Strictly speaking the filter expression should accept wildcards and 4-part specifications like for instance includes does on maven-shade-plugin; which would allow for specifications like G:*:*:test-jar which would pick up all test-jars from a given group-id. The artifact filtering could/should probably use maven-common-artifact-filters if it is to be extended any further; this module was extracted with filtering in mind. I think the current patch is a nice starting point and I have applied it in c808eec805509f14160454829df8f496499f0f4e. @Aslak; I appreciate you keeping the patch footprint moderate, it makes it easier to review and accept. On the other hand I think all the changes you describe should be done I would've ok'd all those changes and may choose do some of them myself ! Thanks for the patch!
        Hide
        Kristian Rosenvold added a comment -

        And yes, surefire has pretty good test coverage. If your change does not break a test then it's likely you didn't break anything. So I'm generally not afraid of changes to existing code...

        Show
        Kristian Rosenvold added a comment - And yes, surefire has pretty good test coverage. If your change does not break a test then it's likely you didn't break anything. So I'm generally not afraid of changes to existing code...
        Hide
        Stephen Colebourne added a comment -

        Thank you very much for applying the patch. And yes, that extra "standardised" filtering would be very nice!

        Show
        Stephen Colebourne added a comment - Thank you very much for applying the patch. And yes, that extra "standardised" filtering would be very nice!
        Hide
        Stephen Colebourne added a comment -

        This worked perfectly for me in the 2.15 release. As the config isn't fully documented I looked at the patch and used that:

        <dependenciesToScan>
          <dependency>com.group.id:my-artifact</dependency>
          <dependency>com.group.id:my-other-artifact</dependency>
        </dependenciesToScan>
        
        Show
        Stephen Colebourne added a comment - This worked perfectly for me in the 2.15 release. As the config isn't fully documented I looked at the patch and used that: <dependenciesToScan> <dependency>com.group.id:my-artifact</dependency> <dependency>com.group.id:my-other-artifact</dependency> </dependenciesToScan>
        Hide
        Andreas Gudian added a comment -

        I've created an issue so that we don't forget implementing that - including updating the docs and examples: SUREFIRE-1004

        Show
        Andreas Gudian added a comment - I've created an issue so that we don't forget implementing that - including updating the docs and examples: SUREFIRE-1004
        Hide
        Torsten Liermann added a comment -

        The extension "dependenciesToScan" is not enough for jars of type test-jar. Please support classifier "tests". Thanks.

        Show
        Torsten Liermann added a comment - The extension "dependenciesToScan" is not enough for jars of type test-jar. Please support classifier "tests". Thanks.
        Hide
        Andreas Gudian added a comment -

        @Torsten: that's supposed to be part of SUREFIRE-1004. Patches are always welcome, if you want to make it more likely that it'll be part of Surefire 2.17

        Show
        Andreas Gudian added a comment - @Torsten: that's supposed to be part of SUREFIRE-1004 . Patches are always welcome, if you want to make it more likely that it'll be part of Surefire 2.17

          People

          • Assignee:
            Kristian Rosenvold
            Reporter:
            Paul Gier
          • Votes:
            19 Vote for this issue
            Watchers:
            20 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: