Maven 2 & 3

Concurrent-safe access to local Maven repository

Details

  • Complexity:
    Intermediate
  • Number of attachments :
    0

Description

It seems that access to local Maven repository is not concurrent-safe that is multiple Mavens running in parallel may damage contents of local Maven repository. It would be a nice improvement, because sharing of local repository will lower the need for contacting any other repository. I know that Maven proxy can be used, but that adds another layer which may unnecessarily stress the machine it runs on.

Issue Links

This issue is duplicated by:
MNG-3655 Allow multiple local repositories Major Reopened
MNG-4071 Support additional read-only local repositories Minor Closed
This issue relates to:
MNG-4706 Multithreaded building can create bad files for downloaded artifacts in local repository Major Open
This issue is depended upon by:
CONTINUUM-265 Concurrent builds Major Closed
MNG-3204 Parallel dependency fetching Major Closed
MNG-3004 Allow build lifecycle to execute projects in parallel Major Closed

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Brian Fox added a comment -

John, you were recently fixing this in 2.1 i believe

Show
Brian Fox added a comment - John, you were recently fixing this in 2.1 i believe
Hide
Permalink
John Casey added a comment -

To go any further than just ensuring the locking of resolver-status files (which are a sort of proto-metadata for the resolver to use that helps track unfound artifacts and adhere to an updateInterval for those), we'll need to modify the various Wagons, I think. Since these are file-oriented, they manage opening, writing to, and closing the files internally. One thing I might be able to do is try to manage a file lock on the main artifact file instead of the temp file that the wagon is using as its destination (which may not exist, so that could complicate things even further). Yet another approach might involve creating a lockfile, and managing a lock on that...

I'll look into a couple of these strategies. I don't think it's optimal to bury the locking logic in the various wagons, as we're likely to run into concurrency problems again and again with each marginal wagon implementation.

Show
John Casey added a comment - To go any further than just ensuring the locking of resolver-status files (which are a sort of proto-metadata for the resolver to use that helps track unfound artifacts and adhere to an updateInterval for those), we'll need to modify the various Wagons, I think. Since these are file-oriented, they manage opening, writing to, and closing the files internally. One thing I might be able to do is try to manage a file lock on the main artifact file instead of the temp file that the wagon is using as its destination (which may not exist, so that could complicate things even further). Yet another approach might involve creating a lockfile, and managing a lock on that... I'll look into a couple of these strategies. I don't think it's optimal to bury the locking logic in the various wagons, as we're likely to run into concurrency problems again and again with each marginal wagon implementation.
Hide
Permalink
Christian Gruber added a comment -

Frankly, one could mitigate this substantially by downloading artifacts into temporary files, then once the artifact is down with all its pieces, moving the temporary files into the permanent location. There is still a race condition but it's down to the order of magnitude of milliseconds during the move operation, rather than the many seconds during downloading.

Show
Christian Gruber added a comment - Frankly, one could mitigate this substantially by downloading artifacts into temporary files, then once the artifact is down with all its pieces, moving the temporary files into the permanent location. There is still a race condition but it's down to the order of magnitude of milliseconds during the move operation, rather than the many seconds during downloading.
Hide
Permalink
Brett Porter added a comment -

This is also discussed in http://docs.codehaus.org/display/MAVEN/Local+repository+separation, along with MNG-3655

Show
Brett Porter added a comment - This is also discussed in http://docs.codehaus.org/display/MAVEN/Local+repository+separation, along with MNG-3655
Hide
Permalink
Brett Porter added a comment -

see also linked issues for other use cases

Show
Brett Porter added a comment - see also linked issues for other use cases
Hide
Permalink
Paul Benedict added a comment -

Giving Maven an exclusive lock on an artifact's local metadata should be a sure way of ensuring this.

Show
Paul Benedict added a comment - Giving Maven an exclusive lock on an artifact's local metadata should be a sure way of ensuring this.
Hide
Permalink
John Lonergan added a comment -

This issue really impacts our ability to run concurrent continuous integration builds making out CI take several times longer to cycle than we'd like. We have compensated by having each CI task have it's own private local repo but this massively expands the disk requirements and has the affect of slowing everything down due to all the duplicate file downloading going on. But the really painful thing is that when we have separate local repo's the various threads cannot share the snapshots via the local repo.

Please try to get a fix in place for this.

Show
John Lonergan added a comment - This issue really impacts our ability to run concurrent continuous integration builds making out CI take several times longer to cycle than we'd like. We have compensated by having each CI task have it's own private local repo but this massively expands the disk requirements and has the affect of slowing everything down due to all the duplicate file downloading going on. But the really painful thing is that when we have separate local repo's the various threads cannot share the snapshots via the local repo. Please try to get a fix in place for this.
Hide
Permalink
John Lonergan added a comment -

Or possibly someone could advise where the piece of code that maintains the local repo lies and I'll have look.

Things that hurt seem mostly to be that the repo metadata gets corrupted.

Show
John Lonergan added a comment - Or possibly someone could advise where the piece of code that maintains the local repo lies and I'll have look. Things that hurt seem mostly to be that the repo metadata gets corrupted.
Hide
Permalink
cforce added a comment -

This issue hits us most on a Jenkins buidl server, where all Maven builds use same repo folder. Having many dupes of local repos will use a lot of local hard disk space. Maybe there is a Maven Jenkins Plugin for Jenkins which coordinates the Concurrent access?

Show
cforce added a comment - This issue hits us most on a Jenkins buidl server, where all Maven builds use same repo folder. Having many dupes of local repos will use a lot of local hard disk space. Maybe there is a Maven Jenkins Plugin for Jenkins which coordinates the Concurrent access?

People

Vote (30)
Watch (27)

Dates

  • Created:
    Updated:
Atlassian JIRA (v4.4.3#663-r165197) | Report a problem
Powered by a free Atlassian JIRA open source license for Codehaus. Try JIRA - bug tracking software for your team.