Reason for the Ignored/Skipped testcases is not reported

Details

  • Complexity:
    Intermediate

Description

Tests that are skipped with a @Ignore("Test doesn't work") annotation only appear in the report as an increment in the "Skipped" column. Name of the skipped testcases is also reported. However, the reason (i.e. "Test doesn't work") is not reported.

Quick look at the source code, it seems that it is not implemented:
In class: org.apache.maven.plugins.surefire.report.TestSuiteXmlParser

else if ( "skipped".equals( qName ) )
            {
                testCase.addFailure( "skipped", "skipped" ); // TODO extract real reasons
                currentSuite.setNumberOfSkipped( 1 + currentSuite.getNumberOfSkipped() );
            }

Issue Links

is related to

Improvement - An improvement or enhancement to an existing feature or task. SUREFIRE-303 Ignored/Skipped tests are not reported

  • Major - Major loss of function.
  • Closed - The issue is considered finished, the resolution is correct. Issues which are not closed can be reopened.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Herve Boutemy added a comment -

py.test seem to have implemented the feature in the XML report they produce by using message attribute to skipped element (like error or failure): see http://lists.idyll.org/pipermail/testing-in-python/2011-March/003930.html

this message does not seem to be supported by JENKINS, for example, see https://issues.jenkins-ci.org/browse/JENKINS-8713

but add such message attribute to actual "skipped" element generated by Surefire should not cause any harm

Show
Herve Boutemy added a comment - py.test seem to have implemented the feature in the XML report they produce by using message attribute to skipped element (like error or failure): see http://lists.idyll.org/pipermail/testing-in-python/2011-March/003930.html this message does not seem to be supported by JENKINS, for example, see https://issues.jenkins-ci.org/browse/JENKINS-8713 but add such message attribute to actual "skipped" element generated by Surefire should not cause any harm
Hide
Permalink
Kristian Rosenvold added a comment -

I think adding a message attribute sounds like a good solution

Show
Kristian Rosenvold added a comment - I think adding a message attribute sounds like a good solution
Hide
Permalink
Kristian Rosenvold added a comment -

Fixed in r1236882

Show
Kristian Rosenvold added a comment - Fixed in r1236882
Hide
Permalink
Harry Soehalim added a comment - - edited

I've tried maven-surefire-plugin version 2.12 (and JUnit 4.10), and I still don't see any of the "skipped" tests being reported. Perhaps, I'm missing something here?

output:
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.037 sec

Show
Harry Soehalim added a comment - - edited I've tried maven-surefire-plugin version 2.12 (and JUnit 4.10), and I still don't see any of the "skipped" tests being reported. Perhaps, I'm missing something here? output: Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.037 sec
Hide
Permalink
Kristian Rosenvold added a comment -

When running mvn -Dsurefire:version=2.12 on the test project at https://svn.apache.org/repos/asf/maven/surefire/trunk/surefire-integration-tests/src/test/resources/junit-ignore I certainly see ignored tests being reported as skipped. If you're missing something, please file a separate issue and be a lot more specific about what you're missing, maybe try to reproduce in context of the test project so you can file a specific bug.

Show
Kristian Rosenvold added a comment - When running mvn -Dsurefire:version=2.12 on the test project at https://svn.apache.org/repos/asf/maven/surefire/trunk/surefire-integration-tests/src/test/resources/junit-ignore I certainly see ignored tests being reported as skipped. If you're missing something, please file a separate issue and be a lot more specific about what you're missing, maybe try to reproduce in context of the test project so you can file a specific bug.
Hide
Permalink
Harry Soehalim added a comment -

@Kristian - Thanks, that helps. I missed adding @Test after @Ignore.

Show
Harry Soehalim added a comment - @Kristian - Thanks, that helps. I missed adding @Test after @Ignore.

People

  • Assignee:
    Kristian Rosenvold
    Reporter:
    Rakesh Arora
Vote (3)
Watch (6)

Dates

  • Created:
    Updated:
    Resolved: