Maven 2 & 3

Compile dependency resolution is slow

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Cannot Reproduce
  • Affects Version/s: 2.0.4
  • Fix Version/s: None
  • Component/s: Dependencies
  • Labels:
    None
  • Number of attachments :
    0

Description

Compile dependency resolution is slow. I just wrote an empty module (only test sources that were not compiled), and it took Maven more than five minutes to build it, without any compilation/generation/testing/whatsoever. I just took literally minutes to resolve a couple of hundred compile time dependencies for the compiler plugin, which was then executed twice.

To me, this sounds like a problem with the algorithmic complexity of the dependency resolution.

Activity

Hide
Brett Porter added a comment -

please provide the sample project and anything relevant from settings.xml

Show
Brett Porter added a comment - please provide the sample project and anything relevant from settings.xml
Hide
Stefan Behnel added a comment -

Since the report, it has become clear that the problem is the number of snapshots our repository server stores. The number of snapshots in the local repository does not seem to have a major impact. We found this by accident as increasing the version number of our software cut the build time from about 4 hours down to 40 minutes.

I think some repository caching would really help here.

Show
Stefan Behnel added a comment - Since the report, it has become clear that the problem is the number of snapshots our repository server stores. The number of snapshots in the local repository does not seem to have a major impact. We found this by accident as increasing the version number of our software cut the build time from about 4 hours down to 40 minutes. I think some repository caching would really help here.
Hide
Stefan Behnel added a comment -

This is an issue.

Show
Stefan Behnel added a comment - This is an issue.
Hide
Brett Porter added a comment -

this is entirely controllable by configuration

  • you can configure <repository> definitions to check for snapshot updates less frequently, or not at all (ie, on demand)
  • if the problem is external network traffic, installation of a repository manager (eg Archiva, Artifactory, Proximity) will assist

let us know how this goes as to whether this issue can be re-closed - if it is network traffic configuration is the only thing that will help. If it is cpu time in calculating the dependencies then it is an issue, but more details will be needed.

Show
Brett Porter added a comment - this is entirely controllable by configuration
  • you can configure <repository> definitions to check for snapshot updates less frequently, or not at all (ie, on demand)
  • if the problem is external network traffic, installation of a repository manager (eg Archiva, Artifactory, Proximity) will assist
let us know how this goes as to whether this issue can be re-closed - if it is network traffic configuration is the only thing that will help. If it is cpu time in calculating the dependencies then it is an issue, but more details will be needed.
Hide
Stefan Behnel added a comment -

We are using an in-house repository server via HTTP. It pings in less than 4ms, so I don't think latency is an issue here. For the last version, we have about 200 snapshots lying around per module. That makes some 700-1000 files per module. Serving a directory listing for them shouldn't be a problem for a web server (especially since I would expect the whole directory tree to be cached in RAM pretty soon), so I wouldn't expect the problem to be on the server side.

Our modules have up to 250 transitive dependencies, while I would expect the average to be much lower. So let's say we have some 100 directory listings per module. A listing page is between 70K and 130K and wget retrieves it in less than 0.3 seconds. That would make it at most 30 seconds for the listings or about one minute if we count 250 dependencies. Compile time resolution takes about 3 minutes in total for 250 dependencies, that means we are already coming somewhat close to the times we see here, iff we really access the server. It doesn't explain the remaining difference of 2/3 of the time, though. BTW, keeping the HTTP connection open during the build (possibly only for directory listings) would likely drop the access time quite a bit, I don't know if Maven does this.

However, snapshot resolution happens once per day by default. But even if I re-run the build multiple times, I do not see that the times drop by a major extent. So I'm still convinced that the problem is not external to Maven.

Show
Stefan Behnel added a comment - We are using an in-house repository server via HTTP. It pings in less than 4ms, so I don't think latency is an issue here. For the last version, we have about 200 snapshots lying around per module. That makes some 700-1000 files per module. Serving a directory listing for them shouldn't be a problem for a web server (especially since I would expect the whole directory tree to be cached in RAM pretty soon), so I wouldn't expect the problem to be on the server side. Our modules have up to 250 transitive dependencies, while I would expect the average to be much lower. So let's say we have some 100 directory listings per module. A listing page is between 70K and 130K and wget retrieves it in less than 0.3 seconds. That would make it at most 30 seconds for the listings or about one minute if we count 250 dependencies. Compile time resolution takes about 3 minutes in total for 250 dependencies, that means we are already coming somewhat close to the times we see here, iff we really access the server. It doesn't explain the remaining difference of 2/3 of the time, though. BTW, keeping the HTTP connection open during the build (possibly only for directory listings) would likely drop the access time quite a bit, I don't know if Maven does this. However, snapshot resolution happens once per day by default. But even if I re-run the build multiple times, I do not see that the times drop by a major extent. So I'm still convinced that the problem is not external to Maven.
Hide
Mark Crocker added a comment -

A posting on Nabble suggested using:

<dependencyLocationsEnabled>false</dependencyLocationsEnabled>

as a work-around.

See: http://www.nabble.com/Dependency-Report-Taking-Forever-And-A-Day-td19416718.html

Show
Mark Crocker added a comment - A posting on Nabble suggested using: <dependencyLocationsEnabled>false</dependencyLocationsEnabled> as a work-around. See: http://www.nabble.com/Dependency-Report-Taking-Forever-And-A-Day-td19416718.html
Hide
Jason van Zyl added a comment -

Caching is a lot smarter with 3.x. Please reopen if the speed is still a problem.

Show
Jason van Zyl added a comment - Caching is a lot smarter with 3.x. Please reopen if the speed is still a problem.

People

Vote (1)
Watch (3)

Dates

  • Created:
    Updated:
    Resolved: