Maven Surefire

Test list preprocessor support for tests to be run

Details

  • Type: New Feature New Feature
  • Status: Open Open
  • Priority: Major Major
  • Resolution: Unresolved
  • Affects Version/s: 2.9
  • Fix Version/s: 3.0
  • Component/s: None
  • Labels:
    None
  • Complexity:
    Intermediate
  • Testcase included:
    yes
  • Patch Submitted:
    Yes
  • Number of attachments :
    4

Description

This exposes an interface(named TestListPreprocessor, which has a method named preprocessTestClasses) that can be implemented by a user and injected into surefire plugin configuration to have it used for preprocessing list of tests to be run. Given the original test list, user's preprocessing algorithm can choose to prune or reorder the list and return it back from TestListPreprocessor#preprocessTestClasses, which is then used as the list of tests to be executed.

The patches attached expose a configuration element named "testPreprocessor" the default value of which is "none". User can choose to set it to a value matching the format "<fully-qualified-class-name>[<group-id>:<artifact-id>:<version>]" and have the class loaded and called with list of tests to be run. The list of tests returned by the call is then considered for execution.

Patch description:
The feature is done in 2 patches. Description follows:

0001-Adds-test-list-preprocessor-support-which-loads-an-o.patch
This patch actually adds the feature and unit tests. It adds the interface, an abstraction that encapsulates aforementioned configuration, booter serialization/deserialization, directory-scanner changes etc. The changes have effect in both forked and in-process execution mode.

0002-Integration-test-for-preprocessor.-2-modules-one-pro.patch
This patch adds an integration test for preprocessor feature. The integration test uses a 2 module setup, where the first module implements the TestListPreprocessor interface which selects only even indexed test classes from the list passed in. The second module(which depends on first) uses the artifact from first and uses the following configuration:
--------------------
<configuration>
<testPreprocessor>test.preprocessor.EvenTestOnlyPreprocessor[org.apache.maven.plugins.surefire:preprocessor-impl:1.0-SNAPSHOT]</testPreprocessor>
</configuration>
--------------------
The test then asserts that only 2(0th and 2nd) of the 4 test classes the second module has are executed.

The patches are created on a git-svn clone using format-patch, please use 'patch -p1 < 0001-Adds...'(while in the trunk directory) to apply the patches on svn repository. If using git-svn, git-am can be directly be invoked.

The patches are done over http://svn.apache.org/repos/asf/maven/surefire/trunk@1091357 (trunk HEAD) so should apply cleanly on any recent enough revision.

===============================================================================

Context:
http://stackoverflow.com/questions/5124823/reducing-the-build-time-hudson
http://code.google.com/p/tlb/issues/detail?id=1
http://test-load-balancer.github.com

Issue Links

is related to

New Feature - A new feature of the product, which has yet to be developed. SUREFIRE-701 showSuccess option for test reports: show reports only for failed tests

  • Major - Major loss of function.
  • Open - The issue is open and ready for the assignee to start work on it.

New Feature - A new feature of the product, which has yet to be developed. SUREFIRE-713 Allow setting of excludes and includes via text files

  • Major - Major loss of function.
  • Open - The issue is open and ready for the assignee to start work on it.

New Feature - A new feature of the product, which has yet to be developed. SUREFIRE-131 Excluding tests with command line pattern

  • Minor - Minor loss of function, or other problem where easy workaround is present.
  • Open - The issue is open and ready for the assignee to start work on it.

Improvement - An improvement or enhancement to an existing feature or task. SUREFIRE-526 Better support for other plugins to determine which tests are included/excluded and the order the tests get run

  • Major - Major loss of function.
  • Open - The issue is open and ready for the assignee to start work on it.

Improvement - An improvement or enhancement to an existing feature or task. SUREFIRE-528 Splitting tests names in two categories : success and failure

  • Major - Major loss of function.
  • Open - The issue is open and ready for the assignee to start work on it.

Improvement - An improvement or enhancement to an existing feature or task. SUREFIRE-530 Allow runtime ordering of tests to be specified

  • Major - Major loss of function.
  • Open - The issue is open and ready for the assignee to start work on it.

Improvement - An improvement or enhancement to an existing feature or task. SUREFIRE-632 add a test-failed goal, to only run the tests which failed last time

  • Major - Major loss of function.
  • Open - The issue is open and ready for the assignee to start work on it.

Improvement - An improvement or enhancement to an existing feature or task. SUREFIRE-561 after running test, when tests fail, it's hard to the find the failure reason

  • Major - Major loss of function.
  • Closed - The issue is considered finished, the resolution is correct. Issues which are not closed can be reopened.

Wish - General wishlist item SUREFIRE-582 1. Define "set of tests" that should be run at the begining. 2. Fail build if any error/failure occured in some specified "set of tests".

  • Minor - Minor loss of function, or other problem where easy workaround is present.
  • Open - The issue is open and ready for the assignee to start work on it.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Kristian Rosenvold added a comment - - edited

Nice strategy for solving this one, we should be thinking about viewing this feature request together with one or more of the linked issues, maybe especially SUREFIRE-530

Show
Kristian Rosenvold added a comment - - edited Nice strategy for solving this one, we should be thinking about viewing this feature request together with one or more of the linked issues, maybe especially SUREFIRE-530
Hide
Permalink
Janmejay Singh added a comment -

It can easily be enhanced to allow injection of testPreprocessor declaration via system-property.

As in, user has a maven plugin that hooks up in the lifecycle before test and adds a system-property(for instance, surefire-test-preprocessor which has value "test.preprocessor.EvenTestOnlyPreprocessor[org.apache.maven.plugins.surefire:preprocessor-impl:1.0-SNAPSHOT]") and the AbstractSurefireMojo#testPreprocessor can first check for configuration and if no config found, can check for the system-property and use first one found.

This will also make it more flexible, because user can choose the strategy before invoking the build(to choose a different implementation all the user has to do is -Dsurefire-test-preprocessor=test.preprocessor.EvenTestOnlyPreprocessor[org.apache.maven.plugins.surefire:preprocessor-impl:1.0-SNAPSHOT]).

Show
Janmejay Singh added a comment - It can easily be enhanced to allow injection of testPreprocessor declaration via system-property. As in, user has a maven plugin that hooks up in the lifecycle before test and adds a system-property(for instance, surefire-test-preprocessor which has value "test.preprocessor.EvenTestOnlyPreprocessor [org.apache.maven.plugins.surefire:preprocessor-impl:1.0-SNAPSHOT] ") and the AbstractSurefireMojo#testPreprocessor can first check for configuration and if no config found, can check for the system-property and use first one found. This will also make it more flexible, because user can choose the strategy before invoking the build(to choose a different implementation all the user has to do is -Dsurefire-test-preprocessor=test.preprocessor.EvenTestOnlyPreprocessor [org.apache.maven.plugins.surefire:preprocessor-impl:1.0-SNAPSHOT] ).
Hide
Permalink
Janmejay Singh added a comment -

added patch 0003 which applies atop 0002, and exposes the testPreprocessor as a property and adds an integration test for it. Property "maven.test.preprocessor" accepts the testPreprocessor configuration string and uses the same format as explicit configuration.

Show
Janmejay Singh added a comment - added patch 0003 which applies atop 0002, and exposes the testPreprocessor as a property and adds an integration test for it. Property "maven.test.preprocessor" accepts the testPreprocessor configuration string and uses the same format as explicit configuration.
Hide
Permalink
Janmejay Singh added a comment -

Patch 0004 applies atop 0003 and exposes testPreprocessor configuration as property "maven.test.preprocessor" for IntegrationTestMojo

Show
Janmejay Singh added a comment - Patch 0004 applies atop 0003 and exposes testPreprocessor configuration as property "maven.test.preprocessor" for IntegrationTestMojo
Hide
Permalink
Parameswaran Raman added a comment -

Hi,

I am looking to use TLB with Maven for my project. It would be great if you can push in this patch as quickly as possible as there is a significant improvement (in build time) that we are looking forward to.

–
Thanks,
Params
parameshr@gmail.com

Show
Parameswaran Raman added a comment - Hi, I am looking to use TLB with Maven for my project. It would be great if you can push in this patch as quickly as possible as there is a significant improvement (in build time) that we are looking forward to. – Thanks, Params parameshr@gmail.com
Hide
Permalink
Daniele Dellafiore added a comment -

Hi. I step into this issue lookig for a way to have a sort of global test setup, without having to deal with suites.
The purpose is to have a reset of the whole database and reimport of test data at the beginning of the tests, and not pay the reset/reimport on every single test case.
Maybe this is a nice way to address my problem, what do you think>

Show
Daniele Dellafiore added a comment - Hi. I step into this issue lookig for a way to have a sort of global test setup, without having to deal with suites. The purpose is to have a reset of the whole database and reimport of test data at the beginning of the tests, and not pay the reset/reimport on every single test case. Maybe this is a nice way to address my problem, what do you think>
Hide
Permalink
Janmejay Singh added a comment -

Yes, this can be used as a mechanism to do things before the whole suite executes.

Show
Janmejay Singh added a comment - Yes, this can be used as a mechanism to do things before the whole suite executes.
Hide
Permalink
Janmejay Singh added a comment -

Has the date for 3.0 been decided?

It so happens that im working with a lot of maven projects these days with very long builds, I can totally use this feature. If 3.0 is still far, is it possible to have this merged in one of the 12.2.x releases?

Show
Janmejay Singh added a comment - Has the date for 3.0 been decided? It so happens that im working with a lot of maven projects these days with very long builds, I can totally use this feature. If 3.0 is still far, is it possible to have this merged in one of the 12.2.x releases?

People

  • Assignee:
    Unassigned
    Reporter:
    Janmejay Singh
Vote (4)
Watch (5)

Dates

  • Created:
    Updated: