Maven Surefire
  1. Maven Surefire
  2. SUREFIRE-747

Parallel method execution skips Tests

    Details

    • Type: Bug Bug
    • Status: Closed Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.8, 2.8.1
    • Fix Version/s: None
    • Labels:
      None
    • Complexity:
      Intermediate
    • Number of attachments :
      0

      Description

      Here is a simple testcase.

      POM file:

      			<plugin>
      				<artifactId>maven-surefire-plugin</artifactId>
      				<version>2.8.2-SNAPSHOT</version>
      				<configuration>
      					<includes>
      						<include>**/TestSuite.java</include>
      					</includes>
      					<parallel>methods</parallel>
      				</configuration>
      			</plugin>
      		</plugins>
      

      TestSuite.java

      import org.junit.runner.RunWith;
      import org.junit.runners.Suite;
      
      @RunWith(Suite.class)
      @Suite.SuiteClasses(
      {
      	ParallelTest.class,
      	ParallelTest2.class
      })
      public class TestSuite
      {
      }
      

      ParallelTest.java

      public class ParallelTest
      {
      	public ParallelTest()
      	{
      		System.out.println("ParallelTest.constructor");
      	}
      	
      	@Before
      	public void setUp()
      	{
      		System.out.println("ParallelTest.setUp");
      	}
      
      	@After
      	public void tearDown()
      	{
      		System.out.println("ParallelTest.tearDown");
      	}
      
      	@Test
      	public void first() throws InterruptedException
      	{
      		System.out.println("begin ParallelTest.first");
      		Thread.sleep(1000);
      		System.out.println("end ParallelTest.first");
      	}
      
      	@Test
      	public void second() throws InterruptedException
      	{
      		System.out.println("begin ParallelTest.second");
      		Thread.sleep(1000);
      		System.out.println("end ParallelTest.second");
      	}
      
      	@Test
      	public void third() throws InterruptedException
      	{
      		System.out.println("begin ParallelTest.third");
      		Thread.sleep(1000);
      		System.out.println("end ParallelTest.third");
      	}
      }
      

      ParallelTest2.java is identical except that log output contains ParallelTest2 instead of ParallelTest.

      Actual Output:

      Concurrency config is parallel='methods', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false
      Running com.holdmyspot.health.server.ParallelTest
      ParallelTest.constructor
      ParallelTest.setUp
      begin ParallelTest.first
      end ParallelTest.first
      ParallelTest.tearDown
      Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.014 sec
      Running com.holdmyspot.health.server.ParallelTest2
      ParallelTest2.constructor
      ParallelTest2.setUp
      begin ParallelTest2.first
      end ParallelTest2.first
      ParallelTest2.tearDown
      Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.014 sec

      Results :

      Tests run: 2, Failures: 0, Errors: 0, Skipped: 0

      Problem: methods second() and third() are never run for either ParallelTest or ParallelTest2.

        Issue Links

          Activity

          Hide
          Gili added a comment -

          Also, the tests don't actually run in parallel. I believe this justifies raising the priority of this bug...

          Show
          Gili added a comment - Also, the tests don't actually run in parallel. I believe this justifies raising the priority of this bug...
          Hide
          Kristian Rosenvold added a comment - - edited

          It's the combination of the TestSuite and "methods" parallel that does not work. "methods" without the TestSuite works fine (Change include pattern to

          **/Parallel*.java

          to verify this.

          Show
          Kristian Rosenvold added a comment - - edited It's the combination of the TestSuite and "methods" parallel that does not work. "methods" without the TestSuite works fine (Change include pattern to **/Parallel*.java to verify this.
          Hide
          Gili added a comment -

          Agreed but I've yet to see parallel methods work properly. When I remove the TestSuite and run it against my actual unit tests I am running into a whole bunch of problems involving slf4j on the one hand, and problems making HTTP requests on the other. Now, it could very well be that slf4j and my HTTP test framework are at fault but I've seen enough stuff broken in parallel methods to believe that the problem lies there. In fact, I remember running 2.8.1 (not 2.8.2-SNAPSHOT) and getting back a negative value for total test time. This seems to be fixed in 2.8.2-SNAPSHOT.

          I'll try providing testcases for all these cases but you need to do some serious testing on your end as well. Something is fundamentally broken here.

          Show
          Gili added a comment - Agreed but I've yet to see parallel methods work properly. When I remove the TestSuite and run it against my actual unit tests I am running into a whole bunch of problems involving slf4j on the one hand, and problems making HTTP requests on the other. Now, it could very well be that slf4j and my HTTP test framework are at fault but I've seen enough stuff broken in parallel methods to believe that the problem lies there. In fact, I remember running 2.8.1 (not 2.8.2-SNAPSHOT) and getting back a negative value for total test time. This seems to be fixed in 2.8.2-SNAPSHOT. I'll try providing testcases for all these cases but you need to do some serious testing on your end as well. Something is fundamentally broken here.
          Hide
          Kristian Rosenvold added a comment -

          If you had bothered to check the surefire source code, which you obviously haven't, you'd see that it's fairly well tested code. The parallel stuff too. But you found a bug with Suite and methods parallel, which you incidentally reported quite well; thanks for the report; those two are simply not going to work in combination as-is (I'm unsure about all the parallel stuff and the suite, you're the first person I've seen trying it out).

          Due to some design issues in the Http stack itself, http-clients are typically some of the first failure signs of non-threadsafe code. That is your code, which is probably the only thing "fundamentally broken" in this context.

          But please be assured that any relevant failing test cases you supply will be examined promptly.

          Show
          Kristian Rosenvold added a comment - If you had bothered to check the surefire source code, which you obviously haven't, you'd see that it's fairly well tested code. The parallel stuff too. But you found a bug with Suite and methods parallel, which you incidentally reported quite well; thanks for the report; those two are simply not going to work in combination as-is (I'm unsure about all the parallel stuff and the suite, you're the first person I've seen trying it out). Due to some design issues in the Http stack itself, http-clients are typically some of the first failure signs of non-threadsafe code. That is your code, which is probably the only thing "fundamentally broken" in this context. But please be assured that any relevant failing test cases you supply will be examined promptly.
          Hide
          Gili added a comment - - edited

          Hi Kristan,

          Sorry if I ruffled some feathers. I've tracked down two bugs so far:

          1. slf4j's initialization was not thread-safe: http://bugzilla.slf4j.org/show_bug.cgi?id=176
          2. guice's servlet module was not thread-safe: http://code.google.com/p/google-guice/issues/detail?id=635

          There is at least one bug that seems to be Surefire-specific. "Time elapsed" is always wrong. Sometimes I get negative values:

          "Tests run: 12, Failures: 5, Errors: 0, Skipped: 0, Time elapsed: -1,058,443.657 sec <<< FAILURE!"

          Sometimes I get positive values that are too large:

          "Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.522 sec"

          I know for a fact the tests only ran for 2.2 seconds because I logged the time at the beginning and end of each test.

          Show
          Gili added a comment - - edited Hi Kristan, Sorry if I ruffled some feathers. I've tracked down two bugs so far: 1. slf4j's initialization was not thread-safe: http://bugzilla.slf4j.org/show_bug.cgi?id=176 2. guice's servlet module was not thread-safe: http://code.google.com/p/google-guice/issues/detail?id=635 There is at least one bug that seems to be Surefire-specific. "Time elapsed" is always wrong. Sometimes I get negative values: "Tests run: 12, Failures: 5, Errors: 0, Skipped: 0, Time elapsed: -1,058,443.657 sec <<< FAILURE!" Sometimes I get positive values that are too large: "Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.522 sec" I know for a fact the tests only ran for 2.2 seconds because I logged the time at the beginning and end of each test.
          Hide
          Kristian Rosenvold added a comment -

          I'm closing this issue, since I created a separate issue SUREFIRE-797 for parallel not working with suite, which may actually be a JUnit issue. The issue of reported run-time has been fixed in other issues.

          When reporting issues, please bear in mind that someone will be fixing these issues, and the more un-related gripes you attach to each issue, the harder it becomes to actually resolve the issue. So if you report issues, generally try to report one distinct issue per problem.

          Thank you for the report, I'd be very interested to hear if you can get "parallel" to work with a suite using just regular junit.

          Show
          Kristian Rosenvold added a comment - I'm closing this issue, since I created a separate issue SUREFIRE-797 for parallel not working with suite, which may actually be a JUnit issue. The issue of reported run-time has been fixed in other issues. When reporting issues, please bear in mind that someone will be fixing these issues, and the more un-related gripes you attach to each issue, the harder it becomes to actually resolve the issue. So if you report issues, generally try to report one distinct issue per problem. Thank you for the report, I'd be very interested to hear if you can get "parallel" to work with a suite using just regular junit.

            People

            • Assignee:
              Kristian Rosenvold
              Reporter:
              Gili
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: