Maven Surefire (moved to ASF)
  1. Maven Surefire (moved to ASF)
  2. SUREFIRE-745

-Dtest supports multiple test classes but not multiple test methods

    Details

    • Type: Improvement Improvement
    • Status: Closed Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.12.1, 2.19
    • Component/s: None
    • Labels:
      None
    • Environment:
    • Complexity:
      Intermediate
    • Number of attachments :
      6

      Description

      The -Dtest parameter is very handy for running a specific test class or test method. It also supports running multiple test classes. Unfortunately, it does not permit specifying running multiple test methods. It would be great if this were possible.

      The examples below are from the Apache Commons project.

      WORKS: Run multiple test classes:
      mvn test -Dtest=ImmutablePairTest,StopWatchTest

      WORKS: Run a specific test method:
      mvn test -Dtest=ImmutablePairTest#testBasic

      DOES NOT WORK:
      mvn test -Dtest=StopWatchTest#testStopWatchSimple,StopWatchTest#testStopWatchSimpleGet
      mvn test -Dtest=ImmutablePairTest#testBasic,StopWatchTest#testLang315

      1. multipleMethods.patch
        10 kB
        rainLee
      2. multipleMethods-v2.patch
        5 kB
        rainLee
      3. multipleMethods-v3.patch
        14 kB
        rainLee
      4. multipleMethods-v4.patch
        14 kB
        rainLee
      5. SUREFIRE-745.patch
        18 kB
        rainLee
      6. SUREFIRE-745-v2.patch
        22 kB
        rainLee

        Activity

        Hide
        Jianfeng Sun added a comment - - edited

        @Tibor

        Thanks for the response.

        By using 2.18-SNAPSHOT for both surefire and failsafe,
        1) if I use -Dit.test=class1#method1,class2#method2
        I got this:
        main:
        [INFO] Executed tasks
        [INFO]
        [INFO] — maven-surefire-plugin:2.18-SNAPSHOT:test (default-test) @ selenium —
        [INFO]
        [INFO] — maven-jar-plugin:2.4:jar (default-jar) @ selenium —
        [INFO]
        [INFO] — maven-failsafe-plugin:2.18-SNAPSHOT:integration-test (run-test) @ selenium —
        [INFO] Failsafe report directory: C:\Automation testing tool\workspace\fox.emf.selenium.Oct5\target\surefire-reports\10-15-2014-03-55-PM
        [INFO] Using configured provider org.apache.maven.surefire.junitcore.JUnitCoreProvider
        [INFO] parallel='none', perCoreThreadCount=true, threadCount=0, useUnlimitedThreads=false, threadCountSuites=0, threadCountClasses=0, threadCountMethods=0, parallelOptimized=true

        -------------------------------------------------------
        T E S T S
        -------------------------------------------------------

        -------------------------------------------------------
        T E S T S
        -------------------------------------------------------

        Results :

        Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

        [WARNING] File encoding has not been set, using platform encoding Cp1252, i.e. build is platform dependent!
        [INFO]
        [INFO] — properties-maven-plugin:1.0-alpha-2:read-project-properties (default) @ selenium —
        [INFO]
        [INFO] — maven-postman-plugin:0.1.6:send-mail (send_email) @ selenium —

        2) if I use -Dtest=class1#method1,class2#method2
        I got Failed tests:
        class1.method1:21 ? NullPointer
        class1.method2:16 ? NullPointer
        class2.method1:21 ? NullPointer
        class2.method2:17 ? NullPointer

        the java code for NullPointer is "System.getProperty("user.dir")+"/testflow/Spot_TC07_Export Functionality.xls""
        I guess the NullPorinter is from system property user dir.
        I'm sure it will work when we only run one test at a time.

        Show
        Jianfeng Sun added a comment - - edited @Tibor Thanks for the response. By using 2.18-SNAPSHOT for both surefire and failsafe, 1) if I use -Dit.test=class1#method1,class2#method2 I got this: main: [INFO] Executed tasks [INFO] [INFO] — maven-surefire-plugin:2.18-SNAPSHOT:test (default-test) @ selenium — [INFO] [INFO] — maven-jar-plugin:2.4:jar (default-jar) @ selenium — [INFO] [INFO] — maven-failsafe-plugin:2.18-SNAPSHOT:integration-test (run-test) @ selenium — [INFO] Failsafe report directory: C:\Automation testing tool\workspace\fox.emf.selenium.Oct5\target\surefire-reports\10-15-2014-03-55-PM [INFO] Using configured provider org.apache.maven.surefire.junitcore.JUnitCoreProvider [INFO] parallel='none', perCoreThreadCount=true, threadCount=0, useUnlimitedThreads=false, threadCountSuites=0, threadCountClasses=0, threadCountMethods=0, parallelOptimized=true ------------------------------------------------------- T E S T S ------------------------------------------------------- ------------------------------------------------------- T E S T S ------------------------------------------------------- Results : Tests run: 0, Failures: 0, Errors: 0, Skipped: 0 [WARNING] File encoding has not been set, using platform encoding Cp1252, i.e. build is platform dependent! [INFO] [INFO] — properties-maven-plugin:1.0-alpha-2:read-project-properties (default) @ selenium — [INFO] [INFO] — maven-postman-plugin:0.1.6:send-mail (send_email) @ selenium — 2) if I use -Dtest=class1#method1,class2#method2 I got Failed tests: class1.method1:21 ? NullPointer class1.method2:16 ? NullPointer class2.method1:21 ? NullPointer class2.method2:17 ? NullPointer the java code for NullPointer is "System.getProperty("user.dir")+"/testflow/Spot_TC07_Export Functionality.xls"" I guess the NullPorinter is from system property user dir. I'm sure it will work when we only run one test at a time.
        Hide
        Tibor Digana added a comment -

        @Jianfeng Sun
        Regarding filters (classes & methods) and groups my plan is to build IT tests with surefire-junit47 in 2.18.
        If they fail I will report and fix a bug so we will have chance to get stable surefire plugin.
        Did you combine "parallel=..." with surefire in the above tests ?

        Show
        Tibor Digana added a comment - @Jianfeng Sun Regarding filters (classes & methods) and groups my plan is to build IT tests with surefire-junit47 in 2.18. If they fail I will report and fix a bug so we will have chance to get stable surefire plugin. Did you combine "parallel=..." with surefire in the above tests ?
        Hide
        Jianfeng Sun added a comment - - edited

        @Tibo Digana
        Filters and groups will be great features for surefire. Thank you for the work.
        I didn't use parallel parameter in this test. So I believe by default it should be set as "false".

        And I just tried

        <configuration>
        <parallel>methods</parallel>
        <threadCount>10</threadCount>
        </configuration>

        see the same result.

        Show
        Jianfeng Sun added a comment - - edited @Tibo Digana Filters and groups will be great features for surefire. Thank you for the work. I didn't use parallel parameter in this test. So I believe by default it should be set as "false". And I just tried <configuration> <parallel>methods</parallel> <threadCount>10</threadCount> </configuration> see the same result.
        Hide
        Tibor Digana added a comment -

        Multiple Classes and Methods, regex, non-regex support for JUnit4, JUnit47+, TestNG providers.
        https://github.com/apache/maven-surefire/pull/83

        Show
        Tibor Digana added a comment - Multiple Classes and Methods, regex, non-regex support for JUnit4, JUnit47+, TestNG providers. https://github.com/apache/maven-surefire/pull/83
        Hide
        Tibor Digana added a comment -

        commit 866a535b66229e9bab3f09cbd42732d70ca9504f
        commit 947abde38a101b028deabf0316d5efa44dd074b7
        commit ca9eb3d1a344813457a959481c3ff73bde074721
        commit 2824f9408e751b383770a77d260a07d9f1d7e544

        Show
        Tibor Digana added a comment - commit 866a535b66229e9bab3f09cbd42732d70ca9504f commit 947abde38a101b028deabf0316d5efa44dd074b7 commit ca9eb3d1a344813457a959481c3ff73bde074721 commit 2824f9408e751b383770a77d260a07d9f1d7e544

          People

          • Assignee:
            Tibor Digana
            Reporter:
            reid holmes

            Dates

            • Created:
              Updated:
              Resolved: