Maven Surefire

Parallel method execution skips Tests

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: 2.8, 2.8.1
  • Fix Version/s: None
  • Component/s: Junit 4.7+ (parallel) support
  • Labels:
    None
  • Complexity:
    Intermediate
  • Number of attachments :
    0

Description

Here is a simple testcase.

POM file:

			<plugin>
				<artifactId>maven-surefire-plugin</artifactId>
				<version>2.8.2-SNAPSHOT</version>
				<configuration>
					<includes>
						<include>**/TestSuite.java</include>
					</includes>
					<parallel>methods</parallel>
				</configuration>
			</plugin>
		</plugins>

TestSuite.java

import org.junit.runner.RunWith;
import org.junit.runners.Suite;

@RunWith(Suite.class)
@Suite.SuiteClasses(
{
	ParallelTest.class,
	ParallelTest2.class
})
public class TestSuite
{
}

ParallelTest.java

public class ParallelTest
{
	public ParallelTest()
	{
		System.out.println("ParallelTest.constructor");
	}
	
	@Before
	public void setUp()
	{
		System.out.println("ParallelTest.setUp");
	}

	@After
	public void tearDown()
	{
		System.out.println("ParallelTest.tearDown");
	}

	@Test
	public void first() throws InterruptedException
	{
		System.out.println("begin ParallelTest.first");
		Thread.sleep(1000);
		System.out.println("end ParallelTest.first");
	}

	@Test
	public void second() throws InterruptedException
	{
		System.out.println("begin ParallelTest.second");
		Thread.sleep(1000);
		System.out.println("end ParallelTest.second");
	}

	@Test
	public void third() throws InterruptedException
	{
		System.out.println("begin ParallelTest.third");
		Thread.sleep(1000);
		System.out.println("end ParallelTest.third");
	}
}

ParallelTest2.java is identical except that log output contains ParallelTest2 instead of ParallelTest.

Actual Output:

Concurrency config is parallel='methods', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false
Running com.holdmyspot.health.server.ParallelTest
ParallelTest.constructor
ParallelTest.setUp
begin ParallelTest.first
end ParallelTest.first
ParallelTest.tearDown
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.014 sec
Running com.holdmyspot.health.server.ParallelTest2
ParallelTest2.constructor
ParallelTest2.setUp
begin ParallelTest2.first
end ParallelTest2.first
ParallelTest2.tearDown
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.014 sec

Results :

Tests run: 2, Failures: 0, Errors: 0, Skipped: 0

Problem: methods second() and third() are never run for either ParallelTest or ParallelTest2.

Issue Links

is related to

Bug - A problem which impairs or prevents the functions of the product. SUREFIRE-797 Parallel junit does not run in parallel when a Suite is used at the top level

  • Major - Major loss of function.
  • Open - The issue is open and ready for the assignee to start work on it.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Gili added a comment -

Also, the tests don't actually run in parallel. I believe this justifies raising the priority of this bug...

Show
Gili added a comment - Also, the tests don't actually run in parallel. I believe this justifies raising the priority of this bug...
Hide
Permalink
Kristian Rosenvold added a comment - - edited

It's the combination of the TestSuite and "methods" parallel that does not work. "methods" without the TestSuite works fine (Change include pattern to 

**/Parallel*.java

to verify this.

Show
Kristian Rosenvold added a comment - - edited It's the combination of the TestSuite and "methods" parallel that does not work. "methods" without the TestSuite works fine (Change include pattern to **/Parallel*.java to verify this.
Hide
Permalink
Gili added a comment -

Agreed but I've yet to see parallel methods work properly. When I remove the TestSuite and run it against my actual unit tests I am running into a whole bunch of problems involving slf4j on the one hand, and problems making HTTP requests on the other. Now, it could very well be that slf4j and my HTTP test framework are at fault but I've seen enough stuff broken in parallel methods to believe that the problem lies there. In fact, I remember running 2.8.1 (not 2.8.2-SNAPSHOT) and getting back a negative value for total test time. This seems to be fixed in 2.8.2-SNAPSHOT.

I'll try providing testcases for all these cases but you need to do some serious testing on your end as well. Something is fundamentally broken here.

Show
Gili added a comment - Agreed but I've yet to see parallel methods work properly. When I remove the TestSuite and run it against my actual unit tests I am running into a whole bunch of problems involving slf4j on the one hand, and problems making HTTP requests on the other. Now, it could very well be that slf4j and my HTTP test framework are at fault but I've seen enough stuff broken in parallel methods to believe that the problem lies there. In fact, I remember running 2.8.1 (not 2.8.2-SNAPSHOT) and getting back a negative value for total test time. This seems to be fixed in 2.8.2-SNAPSHOT. I'll try providing testcases for all these cases but you need to do some serious testing on your end as well. Something is fundamentally broken here.
Hide
Permalink
Kristian Rosenvold added a comment -

If you had bothered to check the surefire source code, which you obviously haven't, you'd see that it's fairly well tested code. The parallel stuff too. But you found a bug with Suite and methods parallel, which you incidentally reported quite well; thanks for the report; those two are simply not going to work in combination as-is (I'm unsure about all the parallel stuff and the suite, you're the first person I've seen trying it out).

Due to some design issues in the Http stack itself, http-clients are typically some of the first failure signs of non-threadsafe code. That is your code, which is probably the only thing "fundamentally broken" in this context.

But please be assured that any relevant failing test cases you supply will be examined promptly.

Show
Kristian Rosenvold added a comment - If you had bothered to check the surefire source code, which you obviously haven't, you'd see that it's fairly well tested code. The parallel stuff too. But you found a bug with Suite and methods parallel, which you incidentally reported quite well; thanks for the report; those two are simply not going to work in combination as-is (I'm unsure about all the parallel stuff and the suite, you're the first person I've seen trying it out). Due to some design issues in the Http stack itself, http-clients are typically some of the first failure signs of non-threadsafe code. That is your code, which is probably the only thing "fundamentally broken" in this context. But please be assured that any relevant failing test cases you supply will be examined promptly.
Hide
Permalink
Gili added a comment - - edited

Hi Kristan,

Sorry if I ruffled some feathers. I've tracked down two bugs so far:

1. slf4j's initialization was not thread-safe: http://bugzilla.slf4j.org/show_bug.cgi?id=176
2. guice's servlet module was not thread-safe: http://code.google.com/p/google-guice/issues/detail?id=635

There is at least one bug that seems to be Surefire-specific. "Time elapsed" is always wrong. Sometimes I get negative values:

"Tests run: 12, Failures: 5, Errors: 0, Skipped: 0, Time elapsed: -1,058,443.657 sec <<< FAILURE!"

Sometimes I get positive values that are too large:

"Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.522 sec"

I know for a fact the tests only ran for 2.2 seconds because I logged the time at the beginning and end of each test.

Show
Gili added a comment - - edited Hi Kristan, Sorry if I ruffled some feathers. I've tracked down two bugs so far: 1. slf4j's initialization was not thread-safe: http://bugzilla.slf4j.org/show_bug.cgi?id=176 2. guice's servlet module was not thread-safe: http://code.google.com/p/google-guice/issues/detail?id=635 There is at least one bug that seems to be Surefire-specific. "Time elapsed" is always wrong. Sometimes I get negative values: "Tests run: 12, Failures: 5, Errors: 0, Skipped: 0, Time elapsed: -1,058,443.657 sec <<< FAILURE!" Sometimes I get positive values that are too large: "Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.522 sec" I know for a fact the tests only ran for 2.2 seconds because I logged the time at the beginning and end of each test.
Hide
Permalink
Kristian Rosenvold added a comment -

I'm closing this issue, since I created a separate issue SUREFIRE-797 for parallel not working with suite, which may actually be a JUnit issue. The issue of reported run-time has been fixed in other issues.

When reporting issues, please bear in mind that someone will be fixing these issues, and the more un-related gripes you attach to each issue, the harder it becomes to actually resolve the issue. So if you report issues, generally try to report one distinct issue per problem.

Thank you for the report, I'd be very interested to hear if you can get "parallel" to work with a suite using just regular junit.

Show
Kristian Rosenvold added a comment - I'm closing this issue, since I created a separate issue SUREFIRE-797 for parallel not working with suite, which may actually be a JUnit issue. The issue of reported run-time has been fixed in other issues. When reporting issues, please bear in mind that someone will be fixing these issues, and the more un-related gripes you attach to each issue, the harder it becomes to actually resolve the issue. So if you report issues, generally try to report one distinct issue per problem. Thank you for the report, I'd be very interested to hear if you can get "parallel" to work with a suite using just regular junit.

People

  • Assignee:
    Kristian Rosenvold
    Reporter:
    Gili
Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved: