Maven
  1. Maven
  2. MNG-3379

Parallel resolution of artifacts

    Details

    • Type: New Feature New Feature
    • Status: Closed Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.8
    • Fix Version/s: 2.1.0
    • Labels:
      None
    • Patch Submitted:
      Yes
    • Number of attachments :
      3

      Description

      Artifacts should be resolved in parallel, grouped by group id's to get around the lack of synchronization in the local repository. The patch does the following:

      • Use a ThreadPoolExecutor to parallelize artifact resolution, but takes care not to resolve multiple artifacts from the same group id simultaneously. (requires Java 5)
      • Makes the http wagon the default instead of the poor performing http-client

      Disadvantages:

      • Requires Java 5, but the backport jars could be substituted pretty easily
      • Breaks some plugins due to commons-logging being in the Maven uber jar (required by commons-httpclient), notably the apt plugin (maybe more should use the isolatedRealm setting?)
      • Screws up the progress monitor as multiple threads are updating it

      Advantages:

      • Much faster when combined with the http wagon (WAGON-98). I was seeing 40% improvement on some test builds.
      1. parallel-resolution.diff
        8 kB
        Don Brown
      2. parallel-resolution-2.diff
        9 kB
        Don Brown
      3. parallel-resolution-3.diff
        9 kB
        Don Brown

        Issue Links

          Activity

          Hide
          Brett Porter added a comment -

          I was able to reproduce this once, I'm looking into ways to test it more reliably, and also checking all the code that gets touched for sync issues.

          Show
          Brett Porter added a comment - I was able to reproduce this once, I'm looking into ways to test it more reliably, and also checking all the code that gets touched for sync issues.
          Hide
          Brett Porter added a comment -

          reviewing coverage details for artifact resolution, I've narrowed it down to just DefaultWagonManager that is not thread safe (but is expected to be)

          Show
          Brett Porter added a comment - reviewing coverage details for artifact resolution, I've narrowed it down to just DefaultWagonManager that is not thread safe (but is expected to be)
          Hide
          Brett Porter added a comment -

          the only potential problem I could find under current usage is the one triggered here, which is a bug in Plexus but can be externally synchronized. This wouldn't have tripped Don up as his branch was taken before the additional component configuration step was taken (for setting the HTTP headers).

          I'm working on a test case in Maven for this and will then synchronize it properly.

          Other potential problem spots: a number of configuration maps in DefaultWagonManager that are only modified on initialization (if they were modified during a multithreaded scenario there'd be a problem). Also, the repository list passed in to the resolution must not be modifiable as it isn't cloned.

          Show
          Brett Porter added a comment - the only potential problem I could find under current usage is the one triggered here, which is a bug in Plexus but can be externally synchronized. This wouldn't have tripped Don up as his branch was taken before the additional component configuration step was taken (for setting the HTTP headers). I'm working on a test case in Maven for this and will then synchronize it properly. Other potential problem spots: a number of configuration maps in DefaultWagonManager that are only modified on initialization (if they were modified during a multithreaded scenario there'd be a problem). Also, the repository list passed in to the resolution must not be modifiable as it isn't cloned.
          Hide
          Brett Porter added a comment -

          sync issue fixed. Added a test case that can reproduce in most scenarios though it does rely on timing as Plexus is not testable in that fashion at the moment

          Show
          Brett Porter added a comment - sync issue fixed. Added a test case that can reproduce in most scenarios though it does rely on timing as Plexus is not testable in that fashion at the moment
          Hide
          Brett Porter added a comment -

          Benjamin has identified an issue where exceptions in the resolution can cause the build to hang instead of fail correctly.

          I'll work on a test case and fix for this tomorrow.

          Show
          Brett Porter added a comment - Benjamin has identified an issue where exceptions in the resolution can cause the build to hang instead of fail correctly. I'll work on a test case and fix for this tomorrow.

            People

            • Assignee:
              Brett Porter
              Reporter:
              Don Brown
            • Votes:
              71 Vote for this issue
              Watchers:
              38 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: