Concurrent-safe access to local Maven repository

John, you were recently fixing this in 2.1 i believe

To go any further than just ensuring the locking of resolver-status files (which are a sort of proto-metadata for the resolver to use that helps track unfound artifacts and adhere to an updateInterval for those), we'll need to modify the various Wagons, I think. Since these are file-oriented, they manage opening, writing to, and closing the files internally. One thing I might be able to do is try to manage a file lock on the main artifact file instead of the temp file that the wagon is using as its destination (which may not exist, so that could complicate things even further). Yet another approach might involve creating a lockfile, and managing a lock on that...

I'll look into a couple of these strategies. I don't think it's optimal to bury the locking logic in the various wagons, as we're likely to run into concurrency problems again and again with each marginal wagon implementation.

Frankly, one could mitigate this substantially by downloading artifacts into temporary files, then once the artifact is down with all its pieces, moving the temporary files into the permanent location. There is still a race condition but it's down to the order of magnitude of milliseconds during the move operation, rather than the many seconds during downloading.

This is also discussed in http://docs.codehaus.org/display/MAVEN/Local+repository+separation, along with MNG-3655

see also linked issues for other use cases

Giving Maven an exclusive lock on an artifact's local metadata should be a sure way of ensuring this.

This issue really impacts our ability to run concurrent continuous integration builds making out CI take several times longer to cycle than we'd like. We have compensated by having each CI task have it's own private local repo but this massively expands the disk requirements and has the affect of slowing everything down due to all the duplicate file downloading going on. But the really painful thing is that when we have separate local repo's the various threads cannot share the snapshots via the local repo.

Please try to get a fix in place for this.

Or possibly someone could advise where the piece of code that maintains the local repo lies and I'll have look.

Things that hurt seem mostly to be that the repo metadata gets corrupted.

This issue hits us most on a Jenkins buidl server, where all Maven builds use same repo folder. Having many dupes of local repos will use a lot of local hard disk space. Maybe there is a Maven Jenkins Plugin for Jenkins which coordinates the Concurrent access?

It's very frustrating that there is no apparent progress on this. It was reported years ago and is still hurting productivity significantly.

You can drop this implementation into your $

/lib/ext:

http://search.maven.org/#artifactdetails%7Cio.tesla%7Ctesla-concurrent-localrepo%7C0.0.2%7Cjar

I will be integrating this with the Aether version once Maven core is updated to Aether 0.9.0.M1

Thanks Jason!!

I have read that the aether that's used in maven 3.0.3 is 1.11. Is that correct. Does this mean we could have this fix?

Is this still something we need to do manually, or has this been incorporated into an official maven release?

Can someone confirm that this actually works, or does this have other side effects that prevent this from being released?