[#SUREFIRE-122] With forkmode once, XML reports are cumulative

Maven Surefire

With forkmode once, XML reports are cumulative

Created: 17/May/06 10:02 AM Updated: 22/Jan/07 05:51 PM

Component/s:

None

Affects Version/s:

2.0 (2.2 plugin)

Fix Version/s:

2.3

Time Tracking:

Not Specified

File Attachments:

XMLReporter-patch.txt (0.3 kb)

Issue Links:

Duplicate

This issue is duplicated by:

~~SUREFIRE-226~~ XML Test Results files are not accurate

~~SUREFIRE-256~~ Incoherent data between 'Package List ' and 'Test Cases' items in report

Description

« Hide

With forkmode set to once, the resulting XML files seem to include the test results of all the tests in the previous suites as well as it's own.

pseudo example:
ATest is run, it has 2 test methods. The ATest.xml report says 2 tests run and passed
BTest is run, it has 2 test methods. The BTest.xml report says 4 tests run and passed.
CTest is run, it has 3 test methods. The CTest.xml report says 7 tests run and passed.

When we use cruisecontrol or other reporting tools, it then say 13 tests run and passed instead of 5. What's worse, if a test in ATest fails, it's listed as a failure in all the tests so you get 3 tests failed, not 1. Plus, it's harder to figure out which test suite reallly had the failure.

All

Comments

Work Log

Change History

Sort Order:

Ascending order - Click to sort in descending order

[ Permlink | « Hide ]

Gaz Moorst - 22/May/06 10:25 AM

I get this problem regardless of the forkmode setting.
It only seems to have been doing this since I started using Maven 2.0.4, with maven-surefire-plugin 2.2 instead of maven 2.0.2 with pluginh 2.1.2.

[ Permlink | « Hide ]

Patrice Bertrand - 02/Jun/06 03:01 AM

I have the same issue. I use Maven 2.0.4 with the 2.2 plugin. I had to force Maven to use of version 2.1.3 to work around the problem.

[ Permlink | « Hide ]

Bernard Marshall - 05/Jul/06 06:59 AM

I also have the problem using Maven 2.0.4 with the 2.2 plugin. If however I set <forkMode>always</forkMode> the results are not cumulative.

[ Permlink | « Hide ]

Mark Chaimungkalanont - 23/Jul/06 09:17 PM

I also see this problem. It has definitely only started happening since 2.2. If I add

<plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.1.3</version>
      </plugin>

Things start working again. The linked issue MSUREFIRE-149 has a test project setup that seems to cause this problem consistently.

My experience re: the forkMode setting is that it happens less frequently with forkMode == always, but it does still seem to happen

[ Permlink | « Hide ]

stephane bouchet - 26/Jul/06 05:04 AM

Hi,

Got the same problem. (maven 2.0.4, surefireplugin 2.3-SNAPSHOT)
this is very annoying to have duplicate entry to my reports.

Tracking a little bit shows that 2.1.3 plugin use surefire-booter 1.5.3 and 2.2 use surefire-booter 2.0 .

I think the problem comes from there.

Stéphane

[ Permlink | « Hide ]

Eugene Zhuravlev - 09/Aug/06 01:06 PM

The fix for this is quite easy: need to override the reset() method for the XMLReporter and clear the list of tests there.

[ Permlink | « Hide ]

Eugene Zhuravlev - 28/Aug/06 07:20 AM

Attached is the patch for the XMLReporter that fixes the problem

[ Permlink | « Hide ]

Jason van Zyl - 27/Nov/06 06:26 AM

Patch applied. Thanks!

[ Permlink | « Hide ]

Robert Watkins - 22/Jan/07 05:51 PM

For anyone else who stumbles upon this, the workaround above (using the 2.1.3 plugin) has its own set of issues.

The default values of a few of the configuration arguments for the surefire plugin changed between 2.1.3 and 2.2. The two I've come across were:

forkmode (defaults to 'once' in 2.2, but 'none' in 2.1.3)
childDelegation (defaults to 'false' in 2.2, but 'true' in 2.1.3)

The changes in these defaults can cause issues (I got OOM errors and problems with the XML classloaders) that are somewhat hard to diagnose, partially because the 2.2 plugin documentation doesn't indicate that the defaults had changed between versions. (For example, the only reason I noticed the forkmode problem is because I was dumping the running Java processes in order to diagnose the OOM).