increase concurrency of parallel downloads

Details

  • Complexity:
    Intermediate

Description

this started out in 2.1.0. To avoid file locking issues parallelism was limited to artifacts in different groups. However, I'm now of the opinion this is not necessary and we should investigate allowing jars to be downloaded in parallel regardless of group.

Don has also suggested some improvements to the synchronization that might make the code cleaner and faster, which I'll look for him to add here.

Issue Links

relates to

Improvement - An improvement or enhancement to an existing feature or task. MNG-4432 reimplement parallel artifacts download

  • Major - Major loss of function.
  • Closed - The issue is considered finished, the resolution is correct. Issues which are not closed can be reopened.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
John Casey added a comment -

What else do we need to do to get this resolved? Should we be looking at pushing this to the next release, if it's going to entail more than simply increasing concurrency?

Show
John Casey added a comment - What else do we need to do to get this resolved? Should we be looking at pushing this to the next release, if it's going to entail more than simply increasing concurrency?
Hide
Permalink
Brett Porter added a comment -

I think we can remove the grouping of the downloads and allow them to be as parallel as possible since the way the tree works should prevent sequencing/concurrency problems.

However, that requires more testing than a simple bugfix - no problem bumping it if you're ready to release

Show
Brett Porter added a comment - I think we can remove the grouping of the downloads and allow them to be as parallel as possible since the way the tree works should prevent sequencing/concurrency problems. However, that requires more testing than a simple bugfix - no problem bumping it if you're ready to release
Hide
Permalink
John Casey added a comment -

Not quite ready yet...I'll revisit in a bit, when I find a way to tidy up these other issues...

Show
John Casey added a comment - Not quite ready yet...I'll revisit in a bit, when I find a way to tidy up these other issues...
Hide
Permalink
John Casey added a comment -

Do you have anything specific in mind for testing this once we remove the per-groupId restriction from parallel resolution?

Show
John Casey added a comment - Do you have anything specific in mind for testing this once we remove the per-groupId restriction from parallel resolution?
Hide
Permalink
Brett Porter added a comment -

given how quickly we found the problems on a clean repo before, smoke testing will probably do the most. I think we still have an IT in place that deliberately grabs multiple artifacts that might need to be revised to ensure they are from different groups.

Show
Brett Porter added a comment - given how quickly we found the problems on a clean repo before, smoke testing will probably do the most. I think we still have an IT in place that deliberately grabs multiple artifacts that might need to be revised to ensure they are from different groups.
Hide
Permalink
John Casey added a comment -

Let's push this until the next release, since we haven't made a start on it yet, and we're otherwise ready to start the release process.

Show
John Casey added a comment - Let's push this until the next release, since we haven't made a start on it yet, and we're otherwise ready to start the release process.

People

  • Assignee:
    Benjamin Bentmann
    Reporter:
    Brett Porter
Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved: