Maven Surefire

Exlude tests via an external exlude file

Details

  • Type: New Feature New Feature
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Won't Fix
  • Affects Version/s: None
  • Fix Version/s: None
  • Component/s: None
  • Labels:
    None
  • Complexity:
    Intermediate
  • Number of attachments :
    0

Description

Similar to Ant, Surefire should support excludes defined in an external file like this

<testExcludes>
<excludeFile>test-excludes-jboss501.txt</excludeFile>
</testExcludes>

The exclude file would look like this

  1. [JBWS-2135] WebServiceException: Undefined port type
    org/jboss/test/ws/jaxws/jbws1822/**
  1. [JBWS-2136] ClassCastException: com.sun.xml.ws.server.EndpointMessageContextImpl
    org/jboss/test/ws/jaxws/namespace/**
  1. [JBAS-5359] mapped-name is required for context of deployment
    org/jboss/test/ws/jaxws/jbws1611/**

Issue Links

This issue relates to:
MCOMPILER-72 Add support for specifying includes/excludes in an external file Major Open

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Dan Fabulich added a comment -

I don't think that's the feature you want; there's a better way to do it. JUnit 4 includes a @Ignore annotiation; TestNG allows you to have your tests throw a SkipException which does something similar. That will keep the tests in the report, but mark them as "yellow" or "skipped".

If we excluded known bugs entirely, they would simply disappear from the report.

Show
Dan Fabulich added a comment - I don't think that's the feature you want; there's a better way to do it. JUnit 4 includes a @Ignore annotiation; TestNG allows you to have your tests throw a SkipException which does something similar. That will keep the tests in the report, but mark them as "yellow" or "skipped". If we excluded known bugs entirely, they would simply disappear from the report.
Hide
Permalink
Thomas Diesler added a comment -

Binding the execution of a test to the test source code is not a good idea. Also your suggested solution would introduce a dependency to a particular testing framework/version

Suppose you have a test suite that runs against various environments. In environment A the test is expected to pass in environment B the test should be excluded. Switching the exclude file depending on environment is common practice in various TCKs (Test Compatibility Kit)

Please have a look at http://jbws.dyndns.org/mediawiki/index.php?title=WSF for more background on this.

Show
Thomas Diesler added a comment - Binding the execution of a test to the test source code is not a good idea. Also your suggested solution would introduce a dependency to a particular testing framework/version Suppose you have a test suite that runs against various environments. In environment A the test is expected to pass in environment B the test should be excluded. Switching the exclude file depending on environment is common practice in various TCKs (Test Compatibility Kit) Please have a look at http://jbws.dyndns.org/mediawiki/index.php?title=WSF for more background on this.
Hide
Permalink
Dan Fabulich added a comment -

That page doesn't have anything to do with testing, though I'm quite familiar with the idea that some tests may be expected to fail on some platforms but pass on other platforms.

TestNG allows you to specify an external suite file which configures which tests will be skipped. You can also use code to decide when to throw a skip exception. You can even dynamically query your bug DB to find out whether a bug is still unresolved, and skip only if the bug is open. [Of course, it may not be a good idea to make your tests that complex. ;-)]

Your remark about being "tied to a testing framework" is strange, because Surefire only runs tests that are tied to a testing framework: it will run JUnit tests (which are "tied" to JUnit) or it will run TestNG tests (which are "tied" to TestNG). The @Ignore/Skip feature is not specific to either framework; you can use it in either, but if you want to switch frameworks, you have to re-code all of your tests.

More generally, sometimes people file feature requests to Surefire that are appropriately targeted to the underlying test framework. ["I wish JUnit had feature X. But the JUnit guys do releases less than once a year! Maybe I could ask the Surefire guys to do it for me instead."]

This is one of those feature requests that not only can be solved by the underlying test framework, it already has been solved in both of the test frameworks we support.

Show
Dan Fabulich added a comment - That page doesn't have anything to do with testing, though I'm quite familiar with the idea that some tests may be expected to fail on some platforms but pass on other platforms. TestNG allows you to specify an external suite file which configures which tests will be skipped. You can also use code to decide when to throw a skip exception. You can even dynamically query your bug DB to find out whether a bug is still unresolved, and skip only if the bug is open. [Of course, it may not be a good idea to make your tests that complex. ;-)] Your remark about being "tied to a testing framework" is strange, because Surefire only runs tests that are tied to a testing framework: it will run JUnit tests (which are "tied" to JUnit) or it will run TestNG tests (which are "tied" to TestNG). The @Ignore/Skip feature is not specific to either framework; you can use it in either, but if you want to switch frameworks, you have to re-code all of your tests. More generally, sometimes people file feature requests to Surefire that are appropriately targeted to the underlying test framework. ["I wish JUnit had feature X. But the JUnit guys do releases less than once a year! Maybe I could ask the Surefire guys to do it for me instead."] This is one of those feature requests that not only can be solved by the underlying test framework, it already has been solved in both of the test frameworks we support.
Hide
Permalink
Thomas Diesler added a comment -

Well, we need this feature nevertheless. Folks that also do, please have a look at MCOMPILER-72

Show
Thomas Diesler added a comment - Well, we need this feature nevertheless. Folks that also do, please have a look at MCOMPILER-72

People

Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved:
Atlassian JIRA (v4.4.3#663-r165197) | Report a problem
Powered by a free Atlassian JIRA open source license for Codehaus. Try JIRA - bug tracking software for your team.