Archiva

Migrate repository proxy to the new repository API

Details

  • Type: Task Task
  • Status: Open Open
  • Priority: Major Major
  • Resolution: Unresolved
  • Affects Version/s: None
  • Fix Version/s: 1.4
  • Component/s: remote proxy
  • Labels:
    None
  • Number of attachments :
    0

Issue Links

depends upon

Improvement - An improvement or enhancement to an existing feature or task. MRM-1391 Move out components specific to repo proxying in webdav module to a separate module to the point where it can be turned on and off as a module

  • Major - Major loss of function.
  • Open - The issue is open and ready for the assignee to start work on it.

Improvement - An improvement or enhancement to an existing feature or task. MRM-1392 Separate webdav unit tests and integration tests

  • Major - Major loss of function.
  • Open - The issue is open and ready for the assignee to start work on it.
is depended upon by

Task - A task that needs to be done. MRM-1381 revise webdav module in light of new repository API

  • Major - Major loss of function.
  • Open - The issue is open and ready for the assignee to start work on it.

Task - A task that needs to be done. MRM-1383 remove archiva-repository-layer and archiva-model module

  • Major - Major loss of function.
  • Open - The issue is open and ready for the assignee to start work on it.
is related to

Bug - A problem which impairs or prevents the functions of the product. MRM-782 archiva attempts to get remote version metadata for snapshots, even if policy disables it

  • Major - Major loss of function.
  • Open - The issue is open and ready for the assignee to start work on it.

Bug - A problem which impairs or prevents the functions of the product. MRM-993 Metadata update should honor proxy connector blacklisting

  • Major - Major loss of function.
  • Open - The issue is open and ready for the assignee to start work on it.

Bug - A problem which impairs or prevents the functions of the product. MRM-1149 White/Blacklist pattern matching doesn't work for Maven-1-repos

  • Major - Major loss of function.
  • Open - The issue is open and ready for the assignee to start work on it.

Bug - A problem which impairs or prevents the functions of the product. MRM-1374 Temporary directories created during remote proxy (when build was suddenly suspended in the middle of downloading artifacts) are not cleaned up

  • Major - Major loss of function.
  • Open - The issue is open and ready for the assignee to start work on it.

Bug - A problem which impairs or prevents the functions of the product. MRM-684 Clients timeout due to Archiva blocking when downloading large files

  • Major - Major loss of function.
  • Reopened - This issue was once resolved, but the resolution was deemed incorrect. From here issues are either marked assigned or resolved.

Bug - A problem which impairs or prevents the functions of the product. MRM-1205 Should only create tmp directory for managed repository if the repository has remote repositories

  • Major - Major loss of function.
  • Closed - The issue is considered finished, the resolution is correct. Issues which are not closed can be reopened.

Bug - A problem which impairs or prevents the functions of the product. MRM-1035 Remote, legacy (Maven 1) repositories appear to be consulted for all artifacts, even if no appropriate proxy connector has been defined.

  • Minor - Minor loss of function, or other problem where easy workaround is present.
  • Closed - The issue is considered finished, the resolution is correct. Issues which are not closed can be reopened.

Improvement - An improvement or enhancement to an existing feature or task. MRM-798 blacklisting unavailable remote reps

  • Major - Major loss of function.
  • Open - The issue is open and ready for the assignee to start work on it.

Task - A task that needs to be done. MRM-588 proxy logging is not always effective for diagnosing issues

  • Major - Major loss of function.
  • Open - The issue is open and ready for the assignee to start work on it.
relates to

Bug - A problem which impairs or prevents the functions of the product. MRM-830 Archiva is very slow when remote repository is unreachable

  • Critical - Crashes, loss of data, severe memory leak.
  • Open - The issue is open and ready for the assignee to start work on it.

Bug - A problem which impairs or prevents the functions of the product. MRM-813 Audit log is reporting "Modify File (proxied)" when no proxy connectors exist and the file has not changed

  • Major - Major loss of function.
  • Open - The issue is open and ready for the assignee to start work on it.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Brett Porter added a comment -

note that we probably want this to plug into the resolution mechanism, not just be a front-end. For example, requesting a project in browse where the parent POM is not already in the repository should resolve it correctly. This isn't something that it is capable of doing today.

Also, this should be linked to some related issues that can be addressed at the same time - for example, by ensuring we use a streaming API we can address the issues about requests blocking.

Show
Brett Porter added a comment - note that we probably want this to plug into the resolution mechanism, not just be a front-end. For example, requesting a project in browse where the parent POM is not already in the repository should resolve it correctly. This isn't something that it is capable of doing today. Also, this should be linked to some related issues that can be addressed at the same time - for example, by ensuring we use a streaming API we can address the issues about requests blocking.
Hide
Permalink
Maria Odea Ching added a comment -

From what I understand, these are the things that need to be changed for the repository proxy:

  1. for artifact requests, resolve artifact first if it exists in the managed repository through the metadata resolver
  2. if the artifact is not found:
    1. resolve the artifact from the remote repository first
    2. if it is resolved from the remote repository, then proxy the artifact
    3. update metadata of proxied artifact
  3. otherwise, if the artifact is found:
    1. get the actual artifact file from the storage and return it

For 2.1, a new metadata resolver for remote repositories need to be written. I just have a clarification on this, a remote repository will have it's own metadata repository?

Show
Maria Odea Ching added a comment - From what I understand, these are the things that need to be changed for the repository proxy: for artifact requests, resolve artifact first if it exists in the managed repository through the metadata resolver if the artifact is not found: resolve the artifact from the remote repository first if it is resolved from the remote repository, then proxy the artifact update metadata of proxied artifact otherwise, if the artifact is found: get the actual artifact file from the storage and return it For 2.1, a new metadata resolver for remote repositories need to be written. I just have a clarification on this, a remote repository will have it's own metadata repository?
Hide
Permalink
Brett Porter added a comment -

There's a few things to clarify here.

Firstly, I don't think the logic for the proxy needs to change at all, and there's quite a few rules in there about how that does things. There are plenty of tests too, though you might want to shuffle them around to get a better separation of the unit tests and the "integration tests" that go via webdav. What we're looking to achieve is improve the way it is structured in the code (to the point where it could be turned on or off as a module, rather than being intertwined into the webdav stuff).

The key to the new architecture is that you want to obtain the metadata first, and only obtain the artifact when that is requested. This can get a bit confusing in Maven since it has its metadata, POM metadata, and then there is other metadata for plain artifact files The proxy should become a little "dumber" - it shouldn't know anything about repository storage or remote repo formats - but basically jumping in between trying to get metadata / artifacts from storage and going remote if necessary, though it will need to do a number of filtering operations (convert paths for the remote repo, whitelist/blacklist, search multiple remotes, error handling, determine if it needs to update something already in storage).

Not sure if that's making sense - we should sketch this out on a wiki page some more.

Show
Brett Porter added a comment - There's a few things to clarify here. Firstly, I don't think the logic for the proxy needs to change at all, and there's quite a few rules in there about how that does things. There are plenty of tests too, though you might want to shuffle them around to get a better separation of the unit tests and the "integration tests" that go via webdav. What we're looking to achieve is improve the way it is structured in the code (to the point where it could be turned on or off as a module, rather than being intertwined into the webdav stuff). The key to the new architecture is that you want to obtain the metadata first, and only obtain the artifact when that is requested. This can get a bit confusing in Maven since it has its metadata, POM metadata, and then there is other metadata for plain artifact files The proxy should become a little "dumber" - it shouldn't know anything about repository storage or remote repo formats - but basically jumping in between trying to get metadata / artifacts from storage and going remote if necessary, though it will need to do a number of filtering operations (convert paths for the remote repo, whitelist/blacklist, search multiple remotes, error handling, determine if it needs to update something already in storage). Not sure if that's making sense - we should sketch this out on a wiki page some more.
Hide
Permalink
Brett Porter added a comment -

BTW, here are the references I found to the model and repository layer that we'd be aiming to remove. It's probably not a one-for-one substitution in this case due to the redesign.

Keys (for debug logging) and RepositoryURL in DRPC; ArtifactReference in DRPC, RPC; RepositoryConnectors from repo-layer could be folded into this module in some way; MetadataTools (and others in that package) used for merging in DRPC can be pushed back into the maven2 repository implementation, ManagedRepositoryContent and RepositoryContentFactory are used quite a bit for path calculation in DRPC which would move to the storage (means rethinking the tmpdir calculation and how to get the connectors), RemoteRepositoryContent is something that needs to be rethought and perhaps added to the metadata repository API.

Show
Brett Porter added a comment - BTW, here are the references I found to the model and repository layer that we'd be aiming to remove. It's probably not a one-for-one substitution in this case due to the redesign. Keys (for debug logging) and RepositoryURL in DRPC; ArtifactReference in DRPC, RPC; RepositoryConnectors from repo-layer could be folded into this module in some way; MetadataTools (and others in that package) used for merging in DRPC can be pushed back into the maven2 repository implementation, ManagedRepositoryContent and RepositoryContentFactory are used quite a bit for path calculation in DRPC which would move to the storage (means rethinking the tmpdir calculation and how to get the connectors), RemoteRepositoryContent is something that needs to be rethought and perhaps added to the metadata repository API.

People

  • Assignee:
    Unassigned
    Reporter:
    Maria Odea Ching
Vote (0)
Watch (0)

Dates

  • Created:
    Updated: