Maven Surefire

With forkmode once, XML reports are cumulative

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: 2.0 (2.2 plugin)
  • Fix Version/s: 2.3
  • Component/s: None
  • Labels:
    None
  • Number of attachments :
    1

Description

With forkmode set to once, the resulting XML files seem to include the test results of all the tests in the previous suites as well as it's own.

pseudo example:
ATest is run, it has 2 test methods. The ATest.xml report says 2 tests run and passed
BTest is run, it has 2 test methods. The BTest.xml report says 4 tests run and passed.
CTest is run, it has 3 test methods. The CTest.xml report says 7 tests run and passed.

When we use cruisecontrol or other reporting tools, it then say 13 tests run and passed instead of 5. What's worse, if a test in ATest fails, it's listed as a failure in all the tests so you get 3 tests failed, not 1. Plus, it's harder to figure out which test suite reallly had the failure.

Issue Links

This issue is duplicated by:
SUREFIRE-256 Incoherent data between 'Package List ' and 'Test Cases' items in report Major Closed
SUREFIRE-226 XML Test Results files are not accurate Major Closed

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Gaz Moorst added a comment -

I get this problem regardless of the forkmode setting.
It only seems to have been doing this since I started using Maven 2.0.4, with maven-surefire-plugin 2.2 instead of maven 2.0.2 with pluginh 2.1.2.

Show
Gaz Moorst added a comment - I get this problem regardless of the forkmode setting. It only seems to have been doing this since I started using Maven 2.0.4, with maven-surefire-plugin 2.2 instead of maven 2.0.2 with pluginh 2.1.2.
Hide
Permalink
Patrice Bertrand added a comment -

I have the same issue. I use Maven 2.0.4 with the 2.2 plugin. I had to force Maven to use of version 2.1.3 to work around the problem.

Show
Patrice Bertrand added a comment - I have the same issue. I use Maven 2.0.4 with the 2.2 plugin. I had to force Maven to use of version 2.1.3 to work around the problem.
Hide
Permalink
Bernard Marshall added a comment -

I also have the problem using Maven 2.0.4 with the 2.2 plugin. If however I set <forkMode>always</forkMode> the results are not cumulative.

Show
Bernard Marshall added a comment - I also have the problem using Maven 2.0.4 with the 2.2 plugin. If however I set <forkMode>always</forkMode> the results are not cumulative.
Hide
Permalink
Mark Chaimungkalanont added a comment -

I also see this problem. It has definitely only started happening since 2.2. If I add

<plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.1.3</version>
      </plugin>

Things start working again. The linked issue MSUREFIRE-149 has a test project setup that seems to cause this problem consistently.

My experience re: the forkMode setting is that it happens less frequently with forkMode == always, but it does still seem to happen

Show
Mark Chaimungkalanont added a comment - I also see this problem. It has definitely only started happening since 2.2. If I add 
<plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.1.3</version>
      </plugin>
Things start working again. The linked issue MSUREFIRE-149 has a test project setup that seems to cause this problem consistently. My experience re: the forkMode setting is that it happens less frequently with forkMode == always, but it does still seem to happen
Hide
Permalink
stephane bouchet added a comment -

Hi,

Got the same problem. (maven 2.0.4, surefireplugin 2.3-SNAPSHOT)
this is very annoying to have duplicate entry to my reports.

Tracking a little bit shows that 2.1.3 plugin use surefire-booter 1.5.3 and 2.2 use surefire-booter 2.0 .

I think the problem comes from there.

Stéphane

Show
stephane bouchet added a comment - Hi, Got the same problem. (maven 2.0.4, surefireplugin 2.3-SNAPSHOT) this is very annoying to have duplicate entry to my reports. Tracking a little bit shows that 2.1.3 plugin use surefire-booter 1.5.3 and 2.2 use surefire-booter 2.0 . I think the problem comes from there. Stéphane
Hide
Permalink
Eugene Zhuravlev added a comment -

The fix for this is quite easy: need to override the reset() method for the XMLReporter and clear the list of tests there.

Show
Eugene Zhuravlev added a comment - The fix for this is quite easy: need to override the reset() method for the XMLReporter and clear the list of tests there.
Hide
Permalink
Eugene Zhuravlev added a comment -

Attached is the patch for the XMLReporter that fixes the problem

Show
Eugene Zhuravlev added a comment - Attached is the patch for the XMLReporter that fixes the problem
Hide
Permalink
Jason van Zyl added a comment -

Patch applied. Thanks!

Show
Jason van Zyl added a comment - Patch applied. Thanks!
Hide
Permalink
Robert Watkins added a comment -

For anyone else who stumbles upon this, the workaround above (using the 2.1.3 plugin) has its own set of issues.

The default values of a few of the configuration arguments for the surefire plugin changed between 2.1.3 and 2.2. The two I've come across were:

  • forkmode (defaults to 'once' in 2.2, but 'none' in 2.1.3)
  • childDelegation (defaults to 'false' in 2.2, but 'true' in 2.1.3)

The changes in these defaults can cause issues (I got OOM errors and problems with the XML classloaders) that are somewhat hard to diagnose, partially because the 2.2 plugin documentation doesn't indicate that the defaults had changed between versions. (For example, the only reason I noticed the forkmode problem is because I was dumping the running Java processes in order to diagnose the OOM).

Show
Robert Watkins added a comment - For anyone else who stumbles upon this, the workaround above (using the 2.1.3 plugin) has its own set of issues. The default values of a few of the configuration arguments for the surefire plugin changed between 2.1.3 and 2.2. The two I've come across were:
  • forkmode (defaults to 'once' in 2.2, but 'none' in 2.1.3)
  • childDelegation (defaults to 'false' in 2.2, but 'true' in 2.1.3)
The changes in these defaults can cause issues (I got OOM errors and problems with the XML classloaders) that are somewhat hard to diagnose, partially because the 2.2 plugin documentation doesn't indicate that the defaults had changed between versions. (For example, the only reason I noticed the forkmode problem is because I was dumping the running Java processes in order to diagnose the OOM).

People

Vote (18)
Watch (15)

Dates

  • Created:
    Updated:
    Resolved:
Atlassian JIRA (v4.4.3#663-r165197) | Report a problem
Powered by a free Atlassian JIRA open source license for Codehaus. Try JIRA - bug tracking software for your team.