Details

    • Type: Improvement Improvement
    • Status: Closed Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Plugins and Lifecycle
    • Labels:
      None
    • Number of attachments :
      0

      Description

      I'm trying to do an integration test that depends on a war/ear to be deployed.

      What i'm missing is:

      • integration-test-compile stage and/or:
      • a way to specify an integrationTestSourceDirectory or multiple testSourceDirectories in the pom

      I can't put the test sources in src/test/java because then surefire will run them in the test stage, when
      there's no artifact to deploy yet.

      [Btw, I'm doing this while creating a cactus plugin, for the moment using cargo in the TestSuite itself to deploy.]

      The idea is that the integration test sources go in src/itest/*; that there be a integration-test-compile,
      integration-test-package and/or integration-test-appdeploy[or something] and that surefire
      is also bound to integration-test.

      Maybe something can be done using the src/test/project/some-project/.... approach seen in
      maven-javadoc-plugin, maven-site-plugin and maven-eclipse-plugin (i'd like to see some of that
      standardized anyway to allow plugin testing generally - which can also be seen as integration testing).

      Thoughts, comments, approaches?

        Issue Links

          Activity

          Hide
          Emmanuel Venisse added a comment -

          we chatted with jdcasey for add some new phases.

          For integration test source directory, I think it must be defined as a plugin parameter because a project can have multiple type of integration tests (tests for war/ear, test for ui, test for performance...). Each test type are specific to a plugin.

          Show
          Emmanuel Venisse added a comment - we chatted with jdcasey for add some new phases. For integration test source directory, I think it must be defined as a plugin parameter because a project can have multiple type of integration tests (tests for war/ear, test for ui, test for performance...). Each test type are specific to a plugin.
          Hide
          Kenney Westerhof added a comment -

          Yes, the entire thing could be just a single plugin. I also talked to jdcasey, and we liked the following approach:

          src/itest/projects/project-a/ [pom.xml, etc]
          src/itest/projects/project-b/ [pom.xml. etc]

          Each project would be executed (an 'm2 install' forked somehow) in a new environment containing
          the artifact[s] of the project (the one the src/itest dir is a part of).

          This way plugins can be easily tested - you still need to specify that you want to use the plugin
          in the project-a's pom, of course.

          The same goes for 'normal' projects, like those with packaging war/ear. The nested test
          project could use normal tests to test the artifact, for instance using cargo in a Suite to deploy
          the artifact and then run tests.

          The artifact here is from the project being integration-tested. It can be referenced by defining
          a dependency in the nested test project, and a different ArtifactFactory will be used in the forked
          lifecycle, so that the packaged (but not installed) artifact gets picked up from the target/ dir.

          So basically, the integration-test could stay unmodified, as long as it's possible to fork an m2 lifecycle on
          a different project (than a clone of the current one). This would mean exposing more of the lifecycle to plugins.

          Even better, per packaging a different integration-test plugin/mojo could be used, because plugins / war / ear / ejb / etc
          may need different test circumstances. But this could also be specified in the nested test projects.

          Show
          Kenney Westerhof added a comment - Yes, the entire thing could be just a single plugin. I also talked to jdcasey, and we liked the following approach: src/itest/projects/project-a/ [pom.xml, etc] src/itest/projects/project-b/ [pom.xml. etc] Each project would be executed (an 'm2 install' forked somehow) in a new environment containing the artifact [s] of the project (the one the src/itest dir is a part of). This way plugins can be easily tested - you still need to specify that you want to use the plugin in the project-a's pom, of course. The same goes for 'normal' projects, like those with packaging war/ear. The nested test project could use normal tests to test the artifact, for instance using cargo in a Suite to deploy the artifact and then run tests. The artifact here is from the project being integration-tested. It can be referenced by defining a dependency in the nested test project, and a different ArtifactFactory will be used in the forked lifecycle, so that the packaged (but not installed) artifact gets picked up from the target/ dir. So basically, the integration-test could stay unmodified, as long as it's possible to fork an m2 lifecycle on a different project (than a clone of the current one). This would mean exposing more of the lifecycle to plugins. Even better, per packaging a different integration-test plugin/mojo could be used, because plugins / war / ear / ejb / etc may need different test circumstances. But this could also be specified in the nested test projects.
          Hide
          John Casey added a comment -

          just trying to collect all itest-related stuff, and since I've got this one on my radar, this is as good of a focal point as any.

          Show
          John Casey added a comment - just trying to collect all itest-related stuff, and since I've got this one on my radar, this is as good of a focal point as any.
          Hide
          Brett Porter added a comment -

          Currently, we have "test" which exists before package, and has its own source code and resources, but its classes are not processed like a normal build. Then we have integration-test and verify which occur after packaging, but before installation, and don't have any sources of their own.

          We are also missing the case where you might have intensive tests that you don't want to run during development. I still think these belong before install, and can be enabled in an integration environment using profiles instead.

          Can we think of better ways to specify test sets than this?

          What I'm thinking is:

          • we have unit-test, functional-test (both before package), and integration-test
          • each has an identical lifecycle to main (generate/process-sources/resources, compile, process-classes)
          • it is up to the user to use profiles to apply things they want in dev or not

          I think the itest plugin IDEA above is good (could we use the verifier plugin?) However, I don't think we should have to rely on it being installed before it is tested. Maybe that's not a big deal, esp. since the reactor should be able to take care of it, however I feel there is some merit to keeping it all together, especially as IDEs might deal with it better that way if they have plugins for the testing framework. Of course it is still possible to have a separate test suite, using dependencies to get the code to test.

          I'm still a little uncertain how we might do something like cactus in the same as m1. Perhaps it could fork it's own lifecycle to compile it's special sources, then package them up, deploy, and run tests? It could map flat to the integration-test-* stuff, but then if you have >1 the order is whacky and you also end up having a cactus:compile goal that has to extend AbstractCompilerMojo which is not ideal.

          Show
          Brett Porter added a comment - Currently, we have "test" which exists before package, and has its own source code and resources, but its classes are not processed like a normal build. Then we have integration-test and verify which occur after packaging, but before installation, and don't have any sources of their own. We are also missing the case where you might have intensive tests that you don't want to run during development. I still think these belong before install, and can be enabled in an integration environment using profiles instead. Can we think of better ways to specify test sets than this? What I'm thinking is: we have unit-test, functional-test (both before package), and integration-test each has an identical lifecycle to main (generate/process-sources/resources, compile, process-classes) it is up to the user to use profiles to apply things they want in dev or not I think the itest plugin IDEA above is good (could we use the verifier plugin?) However, I don't think we should have to rely on it being installed before it is tested. Maybe that's not a big deal, esp. since the reactor should be able to take care of it, however I feel there is some merit to keeping it all together, especially as IDEs might deal with it better that way if they have plugins for the testing framework. Of course it is still possible to have a separate test suite, using dependencies to get the code to test. I'm still a little uncertain how we might do something like cactus in the same as m1. Perhaps it could fork it's own lifecycle to compile it's special sources, then package them up, deploy, and run tests? It could map flat to the integration-test-* stuff, but then if you have >1 the order is whacky and you also end up having a cactus:compile goal that has to extend AbstractCompilerMojo which is not ideal.
          Hide
          Brett Porter added a comment -

          John's point in MNG-475 is relevant to the above:

          " how is providing integration-test phase dependencies any different than providing an <integrationTestSourceDirectory/>?

          In other words, if we're leaving the source directory to be specified by cactus, etc. (because they may need different sources, or none), then why can we assume that all integration tests will have the same dependency needs?"

          Show
          Brett Porter added a comment - John's point in MNG-475 is relevant to the above: " how is providing integration-test phase dependencies any different than providing an <integrationTestSourceDirectory/>? In other words, if we're leaving the source directory to be specified by cactus, etc. (because they may need different sources, or none), then why can we assume that all integration tests will have the same dependency needs?"
          Hide
          John Casey added a comment -

          maybe we need to add dependencies that are included in the execution context of a plugin ONLY...that way, we can have cactus-specific dependencies which are only relevant for our project (i.e. not needed for every cactus execution everywhere...they'd be plugin-deps in this case).

          Something like:

          <build>
          <plugins>
          <plugin>
          <!-- g:a:v -->
          <dependencies>
          <dependency>
          <groupId>com.myorg.project</groupId>
          <artifactId>myproject-testharness</artifactId>
          <version>1.0</version>
          </dependency>
          </dependencies>

          <configuration/>
          </plugin>
          </plugins>
          </build>

          It's a little awkward, but at least it's immediately apparent where these deps will be used...

          Show
          John Casey added a comment - maybe we need to add dependencies that are included in the execution context of a plugin ONLY...that way, we can have cactus-specific dependencies which are only relevant for our project (i.e. not needed for every cactus execution everywhere...they'd be plugin-deps in this case). Something like: <build> <plugins> <plugin> <!-- g:a:v --> <dependencies> <dependency> <groupId>com.myorg.project</groupId> <artifactId>myproject-testharness</artifactId> <version>1.0</version> </dependency> </dependencies> <configuration/> </plugin> </plugins> </build> It's a little awkward, but at least it's immediately apparent where these deps will be used...
          Hide
          John Casey added a comment -

          the previous comment was actually kenney's idea, I just thought we need to keep it in the discussion...

          wanna keep the attribution straight here...

          Show
          John Casey added a comment - the previous comment was actually kenney's idea, I just thought we need to keep it in the discussion... wanna keep the attribution straight here...
          Hide
          Kenney Westerhof added a comment -

          That really wasn't necessary! I should have filed it.

          As long as we're being complete, I'll add another reason this is nice to have and some more food for thought:

          The xdoclet plugin currently has to have dependencies on all plugins for xdoclet itself (container / environment specific),
          because there's no way to add those to the plugin execution environment, except now using <extensions> which contaminates
          the root classloader. So basically by allowing this you can have plugins for the plugins.

          However, what we need to consider is the need to add some libraries to higher level than the plugin classloader, for instance to
          a lifecycle phase. Currently artifacts with scope=provided aren't being resolved in the test scope:

          (ScopeArtifactFilter.java:
          else if ( DefaultArtifact.SCOPE_TEST.equals( scope ) )

          { providedScope = false; compileScope = true; runtimeScope = true; testScope = true; }

          )

          This poses a problem for the eclipse plugin. Enabling this will fix that, but they stil aren't included in the plugin
          classpath - the plugin will need to create it's own classloader for that. However, sometimes just an -api artifact is
          depended upon, and during tests an -impl version should be provided - ideally this artifact is present in the classpath during all mojo
          executions in a particular phase. (This may be a bad example since there' s only one test plugin - does anybody have more examples?)

          Show
          Kenney Westerhof added a comment - That really wasn't necessary! I should have filed it. As long as we're being complete, I'll add another reason this is nice to have and some more food for thought: The xdoclet plugin currently has to have dependencies on all plugins for xdoclet itself (container / environment specific), because there's no way to add those to the plugin execution environment, except now using <extensions> which contaminates the root classloader. So basically by allowing this you can have plugins for the plugins. However, what we need to consider is the need to add some libraries to higher level than the plugin classloader, for instance to a lifecycle phase. Currently artifacts with scope=provided aren't being resolved in the test scope: (ScopeArtifactFilter.java: else if ( DefaultArtifact.SCOPE_TEST.equals( scope ) ) { providedScope = false; compileScope = true; runtimeScope = true; testScope = true; } ) This poses a problem for the eclipse plugin. Enabling this will fix that, but they stil aren't included in the plugin classpath - the plugin will need to create it's own classloader for that. However, sometimes just an -api artifact is depended upon, and during tests an -impl version should be provided - ideally this artifact is present in the classpath during all mojo executions in a particular phase. (This may be a bad example since there' s only one test plugin - does anybody have more examples?)
          Hide
          John Casey added a comment -

          in the xdoclet example above, would it make more sense to emulate the build-extension pattern and allow plugin-extensions that are loaded into the plugin container? I know this is a repackaging of the plugin-dependency argument, but I believe calling them plugin-extensions would be more "correct".

          Also, it would remain up to the plugin's underlying technology (xdoclet, f.e.) to actually discover the optional components contained in the extension artifacts...we'd just make them available.

          Show
          John Casey added a comment - in the xdoclet example above, would it make more sense to emulate the build-extension pattern and allow plugin-extensions that are loaded into the plugin container? I know this is a repackaging of the plugin-dependency argument, but I believe calling them plugin-extensions would be more "correct". Also, it would remain up to the plugin's underlying technology (xdoclet, f.e.) to actually discover the optional components contained in the extension artifacts...we'd just make them available.
          Hide
          John Casey added a comment -

          the problem with adding integration-test dependency scope and srcdir is that this is a slippery slope. Can we really define all of the types of tests that are out there, and if we do, do we really want <integrationTestSourceDirectory/> and <functionalTestSourceDirectory/> and ...?

          I think it's reasonable to expect that integration- or functional- or ... tests will need their own set of dependencies, and that it's unreasonable to expect that these can be satisfied using <scope>test</scope>. The source dir is easier, although we'd still have to provide a utility to help these plugins compile it.

          This got shot down once before, but I'll toss it out there again, just to see if it'll help generate some discussion/thinking:

          what about allowing dependencies to be tied to a particular lifecycle phase?

          While this would help by allowing <phase>integration-test</phase> in the dependency, it could conceivably muddy the waters of scope vs. bindings for dependencies. Calculating inheritedScope is easy on this...if it's not runtime || compile, it's not included transitively.

          I think it might not be such a great thing to ship maven 2.0 with the disclaimer "We can't handle all of your testing needs yet." However, I'm not sure what the way forward is. I'd really like to get started on this issue, though, since it looks like the highest-risk issue left for -beta-1.

          Show
          John Casey added a comment - the problem with adding integration-test dependency scope and srcdir is that this is a slippery slope. Can we really define all of the types of tests that are out there, and if we do, do we really want <integrationTestSourceDirectory/> and <functionalTestSourceDirectory/> and ...? I think it's reasonable to expect that integration- or functional- or ... tests will need their own set of dependencies, and that it's unreasonable to expect that these can be satisfied using <scope>test</scope>. The source dir is easier, although we'd still have to provide a utility to help these plugins compile it. This got shot down once before, but I'll toss it out there again, just to see if it'll help generate some discussion/thinking: what about allowing dependencies to be tied to a particular lifecycle phase? While this would help by allowing <phase>integration-test</phase> in the dependency, it could conceivably muddy the waters of scope vs. bindings for dependencies. Calculating inheritedScope is easy on this...if it's not runtime || compile, it's not included transitively. I think it might not be such a great thing to ship maven 2.0 with the disclaimer "We can't handle all of your testing needs yet." However, I'm not sure what the way forward is. I'd really like to get started on this issue, though, since it looks like the highest-risk issue left for -beta-1.
          Hide
          Brett Porter added a comment -

          I'm taking a look at this. Just to answer about dependencies and scopes - I think per plugin dependencies are a better idea than going to phase over scope. I originally proposed phase instead of scope, but now see the relationships not mapping that way, as we discussed in March.

          So I think test deps go to all tests, and anything specific to a certain type of test goes to that plugin.

          Show
          Brett Porter added a comment - I'm taking a look at this. Just to answer about dependencies and scopes - I think per plugin dependencies are a better idea than going to phase over scope. I originally proposed phase instead of scope, but now see the relationships not mapping that way, as we discussed in March. So I think test deps go to all tests, and anything specific to a certain type of test goes to that plugin.
          Hide
          Brett Porter added a comment -

          types of tests:

          • unit tests - run before package, want 100% success before moving on generally
          • integration tests - require the artifact, so run after package but before install. generally want 100% before moving on. May use junit or other plugin like cactus, including deploying to a server
          • acceptance/regression/functional tests - not run every time, probably a report and/or part of a profile. Regression tests (acceptance tests from previous milestones) should be at 100%, whereas acceptance (also called functional) tests only need to get to 100% at release time.
          • performance/stress tests - another form of acceptance test, really.
          Show
          Brett Porter added a comment - types of tests: unit tests - run before package, want 100% success before moving on generally integration tests - require the artifact, so run after package but before install. generally want 100% before moving on. May use junit or other plugin like cactus, including deploying to a server acceptance/regression/functional tests - not run every time, probably a report and/or part of a profile. Regression tests (acceptance tests from previous milestones) should be at 100%, whereas acceptance (also called functional) tests only need to get to 100% at release time. performance/stress tests - another form of acceptance test, really.
          Hide
          Brett Porter added a comment -

          various things we need:

          • plugin integration tests (c.f. eclipse plugin)
          • report integration tests (see Vincent S's work - htmlunit)
          • cactus plugin
          • geronimo itest plugin
          • need to be able to compile additional sources, have own dependencies (basically reapply the whole test lifecycle again - given that this also matches the main one, perhaps that can be reusable/forked)

          separate project is ideal
          + fits most things by dealing with them in the same way

          • a bit unnatural, more strung out
          • doesn't fit IDE well

          While a separate project might be something we recommend, I think we need to support them within the same project.

          I'll put together a solution tomorrow.

          Show
          Brett Porter added a comment - various things we need: plugin integration tests (c.f. eclipse plugin) report integration tests (see Vincent S's work - htmlunit) cactus plugin geronimo itest plugin need to be able to compile additional sources, have own dependencies (basically reapply the whole test lifecycle again - given that this also matches the main one, perhaps that can be reusable/forked) separate project is ideal + fits most things by dealing with them in the same way a bit unnatural, more strung out doesn't fit IDE well While a separate project might be something we recommend, I think we need to support them within the same project. I'll put together a solution tomorrow.
          Hide
          Brett Porter added a comment -

          we should have a setup and teardown phase around the actual test goal

          Show
          Brett Porter added a comment - we should have a setup and teardown phase around the actual test goal
          Hide
          Kenney Westerhof added a comment -

          I just committed sandbox/maven-it-plugin which has the ability to fork maven (sort of)
          to run all projects found in src/it/. Use like this:

          <plugin>
          <artifactId>maven-it-plugin</artifactId>
          <executions>
          <execution>
          <phase>integration-test</phase>
          <goals>
          <goal>fork</goal>
          </goals>
          </execution>
          </executions>
          </plugin>

          Brett, can you see if this is a suitable solution for this issue? All setup/teardown/deployment/whatever
          can now be done in the integration test projects. Test projects can depend on the parent project
          (which could be added at runtime) so that they have access to the generated artifact (and test against it).

          Show
          Kenney Westerhof added a comment - I just committed sandbox/maven-it-plugin which has the ability to fork maven (sort of) to run all projects found in src/it/. Use like this: <plugin> <artifactId>maven-it-plugin</artifactId> <executions> <execution> <phase>integration-test</phase> <goals> <goal>fork</goal> </goals> </execution> </executions> </plugin> Brett, can you see if this is a suitable solution for this issue? All setup/teardown/deployment/whatever can now be done in the integration test projects. Test projects can depend on the parent project (which could be added at runtime) so that they have access to the generated artifact (and test against it).
          Hide
          Brett Porter added a comment -

          Kenney's plugin should be sufficient for 2.0

          Show
          Brett Porter added a comment - Kenney's plugin should be sufficient for 2.0
          Hide
          Barry Kaplan added a comment -

          After reading all this, I'm a bit confused. What is the recommended approach in maven2 v1.x for handling different test types?

          I pretty much need test types to exist in different directories so I can run them independently in eclipse. I don't want to use packages to segregate the tests, because I need the unit tests to exist the same directory as the CUT. Is seperate projects the only solution for maven2 v1.x? And how does this relate to http://jira.codehaus.org/browse/SUREFIRE-7?

          Show
          Barry Kaplan added a comment - After reading all this, I'm a bit confused. What is the recommended approach in maven2 v1.x for handling different test types? I pretty much need test types to exist in different directories so I can run them independently in eclipse. I don't want to use packages to segregate the tests, because I need the unit tests to exist the same directory as the CUT. Is seperate projects the only solution for maven2 v1.x? And how does this relate to http://jira.codehaus.org/browse/SUREFIRE-7?
          Hide
          Dave Sag added a comment -

          I would like to split out some of my heavier tests as integration tests. i'd kinda hoped that if I put my src files into src/itest/java/.. and added a simple
          <plugin>
          <artifactId>maven-it-plugin</artifactId>
          </plugin>

          then maven2 would have magically worked. alas not.

          i'm not at work now so can access svn (hooray) - but can't find where to check out the sandbox plugins.

          I was able to check out http://svn.apache.org/repos/asf/maven/components/trunk/ and build it fine, and some googling turned up http://svn.apache.org/repos/asf/maven/plugins/trunk/ too but i can't find, nor guess the sandbox url. could someone perhaps post that here, or even a url of a page detailing where to find such things.

          cheers dave

          Show
          Dave Sag added a comment - I would like to split out some of my heavier tests as integration tests. i'd kinda hoped that if I put my src files into src/itest/java/.. and added a simple <plugin> <artifactId>maven-it-plugin</artifactId> </plugin> then maven2 would have magically worked. alas not. i'm not at work now so can access svn (hooray) - but can't find where to check out the sandbox plugins. I was able to check out http://svn.apache.org/repos/asf/maven/components/trunk/ and build it fine, and some googling turned up http://svn.apache.org/repos/asf/maven/plugins/trunk/ too but i can't find, nor guess the sandbox url. could someone perhaps post that here, or even a url of a page detailing where to find such things. cheers dave
          Hide
          Brett Porter added a comment -

          worth seeing Craig's use case in MNG-2344 also.

          Show
          Brett Porter added a comment - worth seeing Craig's use case in MNG-2344 also.
          Hide
          Brett Porter added a comment -

          are we happy with IT's now, or do we need another proposal for this? Kenney?

          Show
          Brett Porter added a comment - are we happy with IT's now, or do we need another proposal for this? Kenney?
          Hide
          Benson Margulies added a comment -

          The last comment in here was years ago, inviting the OP to comment if the work done was insufficient. There is no comment in all that time, so a close is overdue.

          Show
          Benson Margulies added a comment - The last comment in here was years ago, inviting the OP to comment if the work done was insufficient. There is no comment in all that time, so a close is overdue.

            People

            • Assignee:
              Benson Margulies
              Reporter:
              Kenney Westerhof
            • Votes:
              14 Vote for this issue
              Watchers:
              20 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 8 hours
                8h
                Remaining:
                Time Spent - 45 minutes Remaining Estimate - 7 hours, 15 minutes
                7h 15m
                Logged:
                Time Spent - 45 minutes Remaining Estimate - 7 hours, 15 minutes
                45m