When tests timeout, report files on disk are incorrect

Details

  • Complexity:
    Intermediate

Description

When forkmode is always/prtest (probably that could be true for the last test and forkmode once) when one test hangs and timeout occurs, est suite execution stops and report file for the offending test not generated. That could mislead somebody to think all tests passed if all tests before the offending one passed.

AFAICT that should be synchronized between one of these:

1. CommandLineUtils.executeCommandLine()
2. SurefireBooter.fork()
3. SurefireBooter.run()
4. SurefirePlugin.execute()

Probably fork must detect a timeout. Then the timeout be gracefully handled by generating a report file for the test. Then continue execution of remaining tests.

Issue Links

relates to

Bug - A problem which impairs or prevents the functions of the product. SUREFIRE-705 Setting both "forkedProcessTimeoutInSeconds" and "parallel" fails with an exception

  • Major - Major loss of function.
  • Closed - The issue is considered finished, the resolution is correct. Issues which are not closed can be reopened.

Improvement - An improvement or enhancement to an existing feature or task. PLXUTILS-67 CommandLineUtils.executeCommandLine() to throw TimeoutException on timeout

  • Major - Major loss of function.
  • Closed - The issue is considered finished, the resolution is correct. Issues which are not closed can be reopened.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
A added a comment -

PLXUTILS-67 is a feature request for the cli tools to throw TimeoutException in case of timeout.

Show
A added a comment - PLXUTILS-67 is a feature request for the cli tools to throw TimeoutException in case of timeout.
Hide
Permalink
Dan Fabulich added a comment -

When I run the "timeout-forked" integration test, it fails the Maven build, which is correct behavior.

However, it seems to always generate the TXT file, and sometimes generates the XML file and sometimes not. (Seems like it usually doesn't generate XML when using "mvn clean test", but usually does generate XML when using "mvn clean && mvn test".)

In all cases, all of the files remaining on disk seem to suggest that the test passed. For example, the TXT file says "Tests run: 1, Failures: 0," the XML file says <testsuite failures="0" tests="1">, etc.

Show
Dan Fabulich added a comment - When I run the "timeout-forked" integration test, it fails the Maven build, which is correct behavior. However, it seems to always generate the TXT file, and sometimes generates the XML file and sometimes not. (Seems like it usually doesn't generate XML when using "mvn clean test", but usually does generate XML when using "mvn clean && mvn test".) In all cases, all of the files remaining on disk seem to suggest that the test passed. For example, the TXT file says "Tests run: 1, Failures: 0," the XML file says <testsuite failures="0" tests="1">, etc.
Hide
Permalink
A added a comment -

It should not stop execution of subsequent tests if "testFailureIgnore" system property is set. Sorry , I've forgot to include that in the issue description.

So the issues are really two:
1. report generation
2. not all tests executed even when "testFailureIgnore" system property is set.

Show
A added a comment - It should not stop execution of subsequent tests if "testFailureIgnore" system property is set. Sorry , I've forgot to include that in the issue description. So the issues are really two: 1. report generation 2. not all tests executed even when "testFailureIgnore" system property is set.
Hide
Permalink
Stephen Connolly added a comment -

Moving to the backlog as it's not a regression and it is more pressing to get 2.6 out first

Show
Stephen Connolly added a comment - Moving to the backlog as it's not a regression and it is more pressing to get 2.6 out first
Hide
Permalink
Kristian Rosenvold added a comment -

Fixed in r1062022, updated IT

Show
Kristian Rosenvold added a comment - Fixed in r1062022, updated IT
Hide
Permalink
Kristian Rosenvold added a comment -

Testing SUREFIRE-705 reveals that this patch does not work well enough to stay in. Handling this issue must be done "differently".

Show
Kristian Rosenvold added a comment - Testing SUREFIRE-705 reveals that this patch does not work well enough to stay in. Handling this issue must be done "differently".

People

  • Assignee:
    Kristian Rosenvold
    Reporter:
    A
Vote (3)
Watch (2)

Dates

  • Created:
    Updated: