Maven
  1. Maven
  2. MNG-2802

Concurrent-safe access to local Maven repository

    Details

    • Complexity:
      Intermediate
    • Number of attachments :
      0

      Description

      It seems that access to local Maven repository is not concurrent-safe that is multiple Mavens running in parallel may damage contents of local Maven repository. It would be a nice improvement, because sharing of local repository will lower the need for contacting any other repository. I know that Maven proxy can be used, but that adds another layer which may unnecessarily stress the machine it runs on.

        Issue Links

          Activity

          Hide
          brianfox brianfox added a comment -

          John, you were recently fixing this in 2.1 i believe

          Show
          brianfox brianfox added a comment - John, you were recently fixing this in 2.1 i believe
          Hide
          John Casey added a comment -

          To go any further than just ensuring the locking of resolver-status files (which are a sort of proto-metadata for the resolver to use that helps track unfound artifacts and adhere to an updateInterval for those), we'll need to modify the various Wagons, I think. Since these are file-oriented, they manage opening, writing to, and closing the files internally. One thing I might be able to do is try to manage a file lock on the main artifact file instead of the temp file that the wagon is using as its destination (which may not exist, so that could complicate things even further). Yet another approach might involve creating a lockfile, and managing a lock on that...

          I'll look into a couple of these strategies. I don't think it's optimal to bury the locking logic in the various wagons, as we're likely to run into concurrency problems again and again with each marginal wagon implementation.

          Show
          John Casey added a comment - To go any further than just ensuring the locking of resolver-status files (which are a sort of proto-metadata for the resolver to use that helps track unfound artifacts and adhere to an updateInterval for those), we'll need to modify the various Wagons, I think. Since these are file-oriented, they manage opening, writing to, and closing the files internally. One thing I might be able to do is try to manage a file lock on the main artifact file instead of the temp file that the wagon is using as its destination (which may not exist, so that could complicate things even further). Yet another approach might involve creating a lockfile, and managing a lock on that... I'll look into a couple of these strategies. I don't think it's optimal to bury the locking logic in the various wagons, as we're likely to run into concurrency problems again and again with each marginal wagon implementation.
          Hide
          Christian Gruber added a comment -

          Frankly, one could mitigate this substantially by downloading artifacts into temporary files, then once the artifact is down with all its pieces, moving the temporary files into the permanent location. There is still a race condition but it's down to the order of magnitude of milliseconds during the move operation, rather than the many seconds during downloading.

          Show
          Christian Gruber added a comment - Frankly, one could mitigate this substantially by downloading artifacts into temporary files, then once the artifact is down with all its pieces, moving the temporary files into the permanent location. There is still a race condition but it's down to the order of magnitude of milliseconds during the move operation, rather than the many seconds during downloading.
          Hide
          Brett Porter added a comment -
          Show
          Brett Porter added a comment - This is also discussed in http://docs.codehaus.org/display/MAVEN/Local+repository+separation , along with MNG-3655
          Hide
          Brett Porter added a comment -

          see also linked issues for other use cases

          Show
          Brett Porter added a comment - see also linked issues for other use cases
          Hide
          Paul Benedict added a comment -

          Giving Maven an exclusive lock on an artifact's local metadata should be a sure way of ensuring this.

          Show
          Paul Benedict added a comment - Giving Maven an exclusive lock on an artifact's local metadata should be a sure way of ensuring this.
          Hide
          John Lonergan added a comment -

          This issue really impacts our ability to run concurrent continuous integration builds making out CI take several times longer to cycle than we'd like. We have compensated by having each CI task have it's own private local repo but this massively expands the disk requirements and has the affect of slowing everything down due to all the duplicate file downloading going on. But the really painful thing is that when we have separate local repo's the various threads cannot share the snapshots via the local repo.

          Please try to get a fix in place for this.

          Show
          John Lonergan added a comment - This issue really impacts our ability to run concurrent continuous integration builds making out CI take several times longer to cycle than we'd like. We have compensated by having each CI task have it's own private local repo but this massively expands the disk requirements and has the affect of slowing everything down due to all the duplicate file downloading going on. But the really painful thing is that when we have separate local repo's the various threads cannot share the snapshots via the local repo. Please try to get a fix in place for this.
          Hide
          John Lonergan added a comment -

          Or possibly someone could advise where the piece of code that maintains the local repo lies and I'll have look.

          Things that hurt seem mostly to be that the repo metadata gets corrupted.

          Show
          John Lonergan added a comment - Or possibly someone could advise where the piece of code that maintains the local repo lies and I'll have look. Things that hurt seem mostly to be that the repo metadata gets corrupted.
          Hide
          cforce added a comment -

          This issue hits us most on a Jenkins buidl server, where all Maven builds use same repo folder. Having many dupes of local repos will use a lot of local hard disk space. Maybe there is a Maven Jenkins Plugin for Jenkins which coordinates the Concurrent access?

          Show
          cforce added a comment - This issue hits us most on a Jenkins buidl server, where all Maven builds use same repo folder. Having many dupes of local repos will use a lot of local hard disk space. Maybe there is a Maven Jenkins Plugin for Jenkins which coordinates the Concurrent access?
          Hide
          Eric Pabst added a comment -

          It's very frustrating that there is no apparent progress on this. It was reported years ago and is still hurting productivity significantly.

          Show
          Eric Pabst added a comment - It's very frustrating that there is no apparent progress on this. It was reported years ago and is still hurting productivity significantly.
          Hide
          Jason van Zyl added a comment -

          You can drop this implementation into your $

          {M2_HOME}

          /lib/ext:

          http://search.maven.org/#artifactdetails%7Cio.tesla%7Ctesla-concurrent-localrepo%7C0.0.2%7Cjar

          I will be integrating this with the Aether version once Maven core is updated to Aether 0.9.0.M1

          Show
          Jason van Zyl added a comment - You can drop this implementation into your $ {M2_HOME} /lib/ext: http://search.maven.org/#artifactdetails%7Cio.tesla%7Ctesla-concurrent-localrepo%7C0.0.2%7Cjar I will be integrating this with the Aether version once Maven core is updated to Aether 0.9.0.M1
          Hide
          luke call added a comment -

          Thanks Jason!!

          Show
          luke call added a comment - Thanks Jason!!
          Hide
          Adam Downer added a comment -

          I have read that the aether that's used in maven 3.0.3 is 1.11. Is that correct. Does this mean we could have this fix?

          Show
          Adam Downer added a comment - I have read that the aether that's used in maven 3.0.3 is 1.11. Is that correct. Does this mean we could have this fix?
          Hide
          Marc Guenther added a comment -

          Is this still something we need to do manually, or has this been incorporated into an official maven release?

          Can someone confirm that this actually works, or does this have other side effects that prevent this from being released?

          Show
          Marc Guenther added a comment - Is this still something we need to do manually, or has this been incorporated into an official maven release? Can someone confirm that this actually works, or does this have other side effects that prevent this from being released?
          Hide
          Marc Guenther added a comment -

          Is this still needed for Maven 3.0.5 or 3.1.0?

          Also I found a newer version of tesla-concurrent-localrepo (0.0.4), is that the one we should use?

          Show
          Marc Guenther added a comment - Is this still needed for Maven 3.0.5 or 3.1.0? Also I found a newer version of tesla-concurrent-localrepo (0.0.4), is that the one we should use?
          Hide
          Hannes Kogler added a comment -

          this is really very time consuming and makes Maven unusable for productive build environments after all.

          Is it fixed in Maven 3.1.1?

          Show
          Hannes Kogler added a comment - this is really very time consuming and makes Maven unusable for productive build environments after all. Is it fixed in Maven 3.1.1?

            People

            • Assignee:
              John Casey
              Reporter:
              Stepan Roh
            • Votes:
              49 Vote for this issue
              Watchers:
              50 Start watching this issue

              Dates

              • Created:
                Updated: