Maven 2 & 3

Concurrent-safe access to local Maven repository

Details

  • Complexity:
    Intermediate
  • Number of attachments :
    0

Description

It seems that access to local Maven repository is not concurrent-safe that is multiple Mavens running in parallel may damage contents of local Maven repository. It would be a nice improvement, because sharing of local repository will lower the need for contacting any other repository. I know that Maven proxy can be used, but that adds another layer which may unnecessarily stress the machine it runs on.

Issue Links

is depended upon by

New Feature - A new feature of the product, which has yet to be developed. CONTINUUM-265 Concurrent builds

  • Major - Major loss of function.
  • Closed - The issue is considered finished, the resolution is correct. Issues which are not closed can be reopened.

Improvement - An improvement or enhancement to an existing feature or task. MNG-3004 Allow build lifecycle to execute projects in parallel

  • Major - Major loss of function.
  • Closed - The issue is considered finished, the resolution is correct. Issues which are not closed can be reopened.

Improvement - An improvement or enhancement to an existing feature or task. MNG-3204 Parallel dependency fetching

  • Major - Major loss of function.
  • Closed - The issue is considered finished, the resolution is correct. Issues which are not closed can be reopened.
is duplicated by

New Feature - A new feature of the product, which has yet to be developed. MNG-3655 Allow multiple local repositories

  • Major - Major loss of function.
  • Reopened - This issue was once resolved, but the resolution was deemed incorrect. From here issues are either marked assigned or resolved.

Wish - General wishlist item MNG-4071 Support additional read-only local repositories

  • Minor - Minor loss of function, or other problem where easy workaround is present.
  • Closed - The issue is considered finished, the resolution is correct. Issues which are not closed can be reopened.
relates to

Bug - A problem which impairs or prevents the functions of the product. MNG-4706 Multithreaded building can create bad files for downloaded artifacts in local repository

  • Major - Major loss of function.
  • Open - The issue is open and ready for the assignee to start work on it.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Brian Fox added a comment -

John, you were recently fixing this in 2.1 i believe

Show
Brian Fox added a comment - John, you were recently fixing this in 2.1 i believe
Hide
Permalink
John Casey added a comment -

To go any further than just ensuring the locking of resolver-status files (which are a sort of proto-metadata for the resolver to use that helps track unfound artifacts and adhere to an updateInterval for those), we'll need to modify the various Wagons, I think. Since these are file-oriented, they manage opening, writing to, and closing the files internally. One thing I might be able to do is try to manage a file lock on the main artifact file instead of the temp file that the wagon is using as its destination (which may not exist, so that could complicate things even further). Yet another approach might involve creating a lockfile, and managing a lock on that...

I'll look into a couple of these strategies. I don't think it's optimal to bury the locking logic in the various wagons, as we're likely to run into concurrency problems again and again with each marginal wagon implementation.

Show
John Casey added a comment - To go any further than just ensuring the locking of resolver-status files (which are a sort of proto-metadata for the resolver to use that helps track unfound artifacts and adhere to an updateInterval for those), we'll need to modify the various Wagons, I think. Since these are file-oriented, they manage opening, writing to, and closing the files internally. One thing I might be able to do is try to manage a file lock on the main artifact file instead of the temp file that the wagon is using as its destination (which may not exist, so that could complicate things even further). Yet another approach might involve creating a lockfile, and managing a lock on that... I'll look into a couple of these strategies. I don't think it's optimal to bury the locking logic in the various wagons, as we're likely to run into concurrency problems again and again with each marginal wagon implementation.
Hide
Permalink
Christian Gruber added a comment -

Frankly, one could mitigate this substantially by downloading artifacts into temporary files, then once the artifact is down with all its pieces, moving the temporary files into the permanent location. There is still a race condition but it's down to the order of magnitude of milliseconds during the move operation, rather than the many seconds during downloading.

Show
Christian Gruber added a comment - Frankly, one could mitigate this substantially by downloading artifacts into temporary files, then once the artifact is down with all its pieces, moving the temporary files into the permanent location. There is still a race condition but it's down to the order of magnitude of milliseconds during the move operation, rather than the many seconds during downloading.
Hide
Permalink
Brett Porter added a comment -

This is also discussed in http://docs.codehaus.org/display/MAVEN/Local+repository+separation, along with MNG-3655

Show
Brett Porter added a comment - This is also discussed in http://docs.codehaus.org/display/MAVEN/Local+repository+separation , along with MNG-3655
Hide
Permalink
Brett Porter added a comment -

see also linked issues for other use cases

Show
Brett Porter added a comment - see also linked issues for other use cases
Hide
Permalink
Paul Benedict added a comment -

Giving Maven an exclusive lock on an artifact's local metadata should be a sure way of ensuring this.

Show
Paul Benedict added a comment - Giving Maven an exclusive lock on an artifact's local metadata should be a sure way of ensuring this.
Hide
Permalink
John Lonergan added a comment -

This issue really impacts our ability to run concurrent continuous integration builds making out CI take several times longer to cycle than we'd like. We have compensated by having each CI task have it's own private local repo but this massively expands the disk requirements and has the affect of slowing everything down due to all the duplicate file downloading going on. But the really painful thing is that when we have separate local repo's the various threads cannot share the snapshots via the local repo.

Please try to get a fix in place for this.

Show
John Lonergan added a comment - This issue really impacts our ability to run concurrent continuous integration builds making out CI take several times longer to cycle than we'd like. We have compensated by having each CI task have it's own private local repo but this massively expands the disk requirements and has the affect of slowing everything down due to all the duplicate file downloading going on. But the really painful thing is that when we have separate local repo's the various threads cannot share the snapshots via the local repo. Please try to get a fix in place for this.
Hide
Permalink
John Lonergan added a comment -

Or possibly someone could advise where the piece of code that maintains the local repo lies and I'll have look.

Things that hurt seem mostly to be that the repo metadata gets corrupted.

Show
John Lonergan added a comment - Or possibly someone could advise where the piece of code that maintains the local repo lies and I'll have look. Things that hurt seem mostly to be that the repo metadata gets corrupted.
Hide
Permalink
cforce added a comment -

This issue hits us most on a Jenkins buidl server, where all Maven builds use same repo folder. Having many dupes of local repos will use a lot of local hard disk space. Maybe there is a Maven Jenkins Plugin for Jenkins which coordinates the Concurrent access?

Show
cforce added a comment - This issue hits us most on a Jenkins buidl server, where all Maven builds use same repo folder. Having many dupes of local repos will use a lot of local hard disk space. Maybe there is a Maven Jenkins Plugin for Jenkins which coordinates the Concurrent access?
Hide
Permalink
Eric Pabst added a comment -

It's very frustrating that there is no apparent progress on this. It was reported years ago and is still hurting productivity significantly.

Show
Eric Pabst added a comment - It's very frustrating that there is no apparent progress on this. It was reported years ago and is still hurting productivity significantly.
Hide
Permalink
Jason van Zyl added a comment -

You can drop this implementation into your $

{M2_HOME}

/lib/ext:

http://search.maven.org/#artifactdetails%7Cio.tesla%7Ctesla-concurrent-localrepo%7C0.0.2%7Cjar

I will be integrating this with the Aether version once Maven core is updated to Aether 0.9.0.M1

Show
Jason van Zyl added a comment - You can drop this implementation into your $ {M2_HOME} /lib/ext: http://search.maven.org/#artifactdetails%7Cio.tesla%7Ctesla-concurrent-localrepo%7C0.0.2%7Cjar I will be integrating this with the Aether version once Maven core is updated to Aether 0.9.0.M1
Hide
Permalink
luke call added a comment -

Thanks Jason!!

Show
luke call added a comment - Thanks Jason!!
Hide
Permalink
Adam Downer added a comment -

I have read that the aether that's used in maven 3.0.3 is 1.11. Is that correct. Does this mean we could have this fix?

Show
Adam Downer added a comment - I have read that the aether that's used in maven 3.0.3 is 1.11. Is that correct. Does this mean we could have this fix?
Hide
Permalink
Marc Guenther added a comment -

Is this still something we need to do manually, or has this been incorporated into an official maven release?

Can someone confirm that this actually works, or does this have other side effects that prevent this from being released?

Show
Marc Guenther added a comment - Is this still something we need to do manually, or has this been incorporated into an official maven release? Can someone confirm that this actually works, or does this have other side effects that prevent this from being released?

People

  • Assignee:
    John Casey
    Reporter:
    Stepan Roh
Vote (43)
Watch (42)

Dates

  • Created:
    Updated: