Maven
  1. Maven
  2. MNG-3004

Allow build lifecycle to execute projects in parallel

    Details

    • Complexity:
      Intermediate
    • Patch Submitted:
      Yes
    • Number of attachments :
      6

      Description

      One of the great advantages with maven over scripted build environments is that it can calculate the dependencies of the build, and it could execute items that are independent of each other in parallel.

      Unfortunately it currently doesn't do this, which would be a big win over tools such as 'ant'. It also means that multicore machines have lots of idle capacity when running a serial build that could be utilised.

      I had a quick shot at seeing what might be required. Bear in mind this is the first time I have looked at maven internally, and I was just trying to feel my way around and build a POC. I got some of the way there, but my build threads don't seem to have the correct classpath - I think this is something to do with plexus / classworlds - but I don't know enough.

      It'd be great to get this feature in a future version, or a way of running my hack (figuring out why in a thread has not the plexus stuff) in the interim.

      1. MNG-3004.increased-testability.patch
        24 kB
        Kristian Rosenvold
      2. mng3004.patch
        7 kB
        Kristian Rosenvold
      3. MNG-3004-Resurrected-StringSearchModelInterpolatorTest.patch
        30 kB
        Kristian Rosenvold
      4. MNG3004-SSMI.patch
        8 kB
        Kristian Rosenvold
      5. mng3004v2_rev2.patch
        423 kB
        Kristian Rosenvold
      6. parallel-builds.patch
        39 kB
        Nigel Magnay

        Issue Links

          Activity

          Hide
          Arnaud Heritier added a comment -

          Nothing new since december ?

          Show
          Arnaud Heritier added a comment - Nothing new since december ?
          Hide
          Kristian Rosenvold added a comment -

          r931884:

          MNG-3004 Concurrent building of multimodule projects

          This commit consists of a modularization of the DefaultLifecycleExecutor into smaller
          components. This is primarily done with the intention of extracting execution strategy
          somewhat from the rest of the logic.

          It also contains three differenent execution strategies:
          A) Linear ("Classic")
          B) Parallel (Based on Dan's original implemementation)
          C) Weave (Still experimental)

          The command line -T option activates parallel running, and the argument
          is in the form -T <[nThreads, a float][C][W]|auto>

          Notes:

          • The output demultiplexer is disabled in this commit, and output will appear intermingled.
          • The modello plugin has known problems
          • Surefire must be run in forkmode at least once.
          Show
          Kristian Rosenvold added a comment - r931884: MNG-3004 Concurrent building of multimodule projects This commit consists of a modularization of the DefaultLifecycleExecutor into smaller components. This is primarily done with the intention of extracting execution strategy somewhat from the rest of the logic. It also contains three differenent execution strategies: A) Linear ("Classic") B) Parallel (Based on Dan's original implemementation) C) Weave (Still experimental) The command line -T option activates parallel running, and the argument is in the form -T < [nThreads, a float] [C] [W] |auto> Notes: The output demultiplexer is disabled in this commit, and output will appear intermingled. The modello plugin has known problems Surefire must be run in forkmode at least once.
          Hide
          Paul Benedict added a comment -

          With concurrency, is the "downloading dependency" progress still readable? If X downloads are occurring, I hope the output isn't difficult to track.

          Show
          Paul Benedict added a comment - With concurrency, is the "downloading dependency" progress still readable? If X downloads are occurring, I hope the output isn't difficult to track.
          Hide
          Christian Gleissner added a comment -

          This feature works very well and resulted in a significant build time reduction, depending on the number of cores. Why was the output de-multiplexer commented for Maven 3 Beta 1 and when will this very useful feature be re-enabled?

          Also, when performing a parallel build, TeamCity 5's automatic collection of Maven Surefire results no longer finds all results. We are using an external invocation of Maven 3. This may be related to how TeamCity checks for the completion of modules by scanning the log file and could go away once the output de-multiplexing will be re-enabled.

          Show
          Christian Gleissner added a comment - This feature works very well and resulted in a significant build time reduction, depending on the number of cores. Why was the output de-multiplexer commented for Maven 3 Beta 1 and when will this very useful feature be re-enabled? Also, when performing a parallel build, TeamCity 5's automatic collection of Maven Surefire results no longer finds all results. We are using an external invocation of Maven 3. This may be related to how TeamCity checks for the completion of modules by scanning the log file and could go away once the output de-multiplexing will be re-enabled.
          Hide
          Kristian Rosenvold added a comment -

          Thank you, Christian, nice with some feedback.

          The output demultiplexer that is present in the code base was only intended as a stop-gap, and is architecturally not the proper solution. MNG-2727 is targeted at beta-2 and we are awaiting its completion. In the event that MNG-2727 does not resolve demultiplexing of the output in parallel build, there will be another issue (with a likely target of beta-2) to fix this.

          Regarding your team-city issue I will not be surprised if they're parsing the output somehow, so I can imagine it being slightly confused.

          Please also note that there are several other issues being fixed in the maven ecosystem related to stable running of parallel tests, and I'll try to keep this updated in the "relates to section" of this issue. The workaround until updated plugins are released will normally involve adding dependencies to newer library versions to your plugins at the moment.

          Show
          Kristian Rosenvold added a comment - Thank you, Christian, nice with some feedback. The output demultiplexer that is present in the code base was only intended as a stop-gap, and is architecturally not the proper solution. MNG-2727 is targeted at beta-2 and we are awaiting its completion. In the event that MNG-2727 does not resolve demultiplexing of the output in parallel build, there will be another issue (with a likely target of beta-2) to fix this. Regarding your team-city issue I will not be surprised if they're parsing the output somehow, so I can imagine it being slightly confused. Please also note that there are several other issues being fixed in the maven ecosystem related to stable running of parallel tests, and I'll try to keep this updated in the "relates to section" of this issue. The workaround until updated plugins are released will normally involve adding dependencies to newer library versions to your plugins at the moment.

            People

            • Assignee:
              Kristian Rosenvold
              Reporter:
              Nigel Magnay
            • Votes:
              44 Vote for this issue
              Watchers:
              40 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: