Ability to merge repositories

SourceArtifact class is responsible for retuning artifact metadata to the caller of a given repo id to the caller. here we can get the artifact meta data of the source artifacts in the repository that we are going to merge. caller has to first get the Artifacts object list and then parse artifacts to get the Artifact metadata list

Proposal for this is in:

https://cwiki.apache.org/confluence/display/ARCHIVA/Staging+Repository+Merge

ongoing stage-repo-merging module

path for stage repo merge

Comments on the patch:

• the log4j.xml config in src/main/resources can already be removed, just use sl4fj logger for logging instead
• remove dependency to archiva-xmlrpc-api, you shouldn't be using the Artifact class in that module since it's specifically for web services only. I think you should use the new repository API instead.
• remove the commented out dependencies in the stage-repository-merge module
• apply codestyle

For the SourceArtifactTest, extend PlexusInSpringTestCase instead of JUnit's TestCase so you can lookup the components. You also need to lookup the components you're using that's why you're getting the NPE when running the tests. See DependencyTreeBuilderTest as an example.

Also, the test() method in SourceArtifactsTest class is wrong. It should be as follows:

  metadataResolverControl.expectAndReturn( metadataResolver.getRootNamespaces(TEST_REPO_ID), (ArrayList<Artifact>)getRootNameSpace() );     
  // put the rest of the expected calls to metadataResolver here

  metadataResolverControl.replay();
  
  sa.getSourceArtifactList();

  metadataResolverControl.verify();

patch with the test fixed test cases. here i have hard coded some values in the SourceArtifacts class.
when i creating ArchivaArtifact object.(classifier and the type).

here i have commented out one line in SourceArtifacts class when it uses facet to get classifier and the type. Still could not figure out the issue. sorry for attaching a patch which is not passing the test cases.

here i ckecked the condition for the facet like wise in ShowArtifactAction class. now the test cases are passing. so i need to know what should i do when the facet is null.(in order to assign type and the classifier to the ArchivaArtifact objects)

Well, you can probably set "jar" as a default value for null type. As for the classifier, it should be fine to set it to null since it's optional..

I've broken down the rest of the items that needs to be done and set some target dates (relative to the proposed timeline for GSoC):

1. repository merge API (June 30)
   • repo merger
   • unit tests
2. UI for repository merge (July 11)
   • button to merge repo
   • page showing conflicting artifacts
   • confirmation page
   • integration with repo merge API
3. webapp tests (July 19)
4. security (July 23)
5. audit logging (July 31)

Eshan, are you ok with the target dates?

ok. i will try my best to achieve those task by target dates.

changes are done according to the comments. (when facet == null set the default type as jar and null clasifier)

For the patch so far, per my comments on dev@, I think this could be pushed into the MetadataRepository interface instead of here. The merge should provide something like a RepositoryMerger or RepositoryArtifactCopier interface instead.

Some more general comments on the patch:

• there's no need to use the ArchivaArtifact class - that's something that we're in the process of removing in favour of the content model
• while either is fine, you might prefer to use Mockito for mocking objects for tests
• the POM here has more dependencies than necessary - watch this carefully

ArtifactsMerger class

artifact merger class updated one.

Hi Eshan, here are my comments on the ArtifactsMerger class you've attached:

• the name of the method isArtifactAvailableIntagerRepo(ArchivaArtifact artifact) should be fixed to isArtifactAvailableInTargetRepo(..)
• you'll have a problem in line 187 & 188, POM files are deployed as *.pom in the repo
• avoid using UPPER CASE for the first letter of a variable name (convention is using small case for the first letter/word for variable names). See below:

  File SourceArtifactFile = new File(sourceRepoPath , artifactPath); 	//line 168
  
  File TargetArtifactFile = new File(targetRepoPath , artifactPath);	//line 170
  

• in the method createFolderStructure(ArchivaArtifact artifact), you shouldn't be accessing the metadata.xml files because the merger shouldn't know about the layout/type of the repository. See Brett's comment regarding that in the dev list thread with subj: [GSoC] Staging repositories updates

here i have fixed issue regarding copying the POM files and the coding convention problem. And also updating project maven metadata is done using repository api. but i am still confusing why we cannot just copy version metadata xml (if it exists in the source artifact in the source repository ). since project meta data xml is depends on available versions i used API to update it.

updated version of ArtifactsMerger class.

updated version of artifactsmerger with test cases.

patch with test repositories

test repositories to be added in to src/test

sample artifacts were added to the test repository.

test case is fixed(ArtifactsMerger)

Hi Eshan, applying the latest patch (the one for adding additional artifacts to test-repository module) causes test failures in maven2 repository support module. I added the 1.2.2 version in the expected versions resolved in Maven2MetadataResolverTest to fix the test failure.

Also, the jar files weren't included in the diff file so I just added them locally. I've committed these to the branch in -r960320. Thanks!

getArtifactsFrom(String repo)method id added to the MetadataRepository and its implementations with fixed test cases.

meta data updating and getArtifact() method is moved from merge module.

unfortunately this doesn't build due to the circular dependency introduced. I don't think any of the additions to the POMs are needed - the API should be using the new model only (ie, not ArchivaArtifact, but ArtifactMetadata - see the other getArtifactBy* methods for comparison).

Can you provide the patch for this method separately to your other changes?

While I don't generally like subtasks - perhaps it might be good to split this issue into subtasks for individual pieces of work (like adding that method) that we can see completed and move on to the next ones. What do you think?

getArtifacts(...) method is added to repository api.(with fixed test cases)

here i have attached the patch for RepositoryMerger ind it's implementation with the test case.Although the test passes there is an issue. I think it is because the test-repository we unpacked does not contains archiva metadata model. (i tried it for a real repo ). So then test throw an exception(because mock expectations a not defined yet).

I already applied the latest patch to the branch in -r962756. Btw, I fixed the formatting of some of the classes. I noticed that there were some unused imports and variables declared in some of the classes. I didn't remove them for now since you might be using it in the next patches. Also, one of your tests is still using the test resources in src/ and not in target/

Thanks!

i have attached the latest version of merging module with fixed test case. and also now i am using target directory for the test case. for the the time being i have commented out the test case of the ArtifactsMerger. (here SourceArtifacts class , ArtifactsMerger class with there test cases should be removed).

thanks.

Can you create the patch from the latest in branch? I already applied your previous patch and the last one has some of the same changes so I'm not able to apply it. Thanks!

here the changes are done after get a svn up. so this can be applied with out conflicts. And is is it possible to delete ArtifactsMerger , ArtifactsMergerTest , SourceArtifacts , SourceArtifactsTest. (since they are not needed). I did not remove them because it may be caused to conflicts when you are applying the patch.

Also in the test case now i am using target directory.

Thanks.

Latest patch committed to branch -r962977. Thanks!

removed unwanted classes.

Patch to remove old classes applied to branch -r962979. Thanks Eshan!

in this patch i have done following things.

• when the user create a new repo there is a tick box to add a staging repo. For now the path of staging repo is not configurable and it creates the repository in the same directory level as the original repo.
  eg : original repo name = test then staging repo name = test-stage

• when we go to see repositories in right hand side of each managed repositories there is a url to do merge. it will merge the repository with its stage repository.

eg : lets say we are going to merge test repo. then it tries to merge test with test-merge repository.if it cannot find a repo called test-merge it returns a error.(not handle yet.)

here only the happy flow is working.
thanks

to get the conflicts artifacts list here i have added a method in maven2RepositpryMerger class with the test cases. (this will be useful in the web apps)

Please review this patch. (since previous one contains some commented lines). sorry for that.

Thanks.

getConflictArtifacts() method is modified with the fixed test case.

Thanks.

Hi Eshan, the patch seems to be missing the MergeAction class. I'm getting an error when running the webapp. Can you update the patch to include it? Thanks!

mergeAction class is added.

merge with skipping conflict artifacts is added.

in previous patch merge with skipping conflicts method is not correct and here i have fixed it.

Thanks.

merge all and merge with skipping functionalities are now only allowed to newly created managed repositories.Merging function is not allowed for the Internal repository and the snapshots repository.
And staging repository is not displaying on the repositories.jsp.

Thanks.

in this patch i have validate the repository id when the user creates a new repository and if the id contains word "stage" i just suggest him to go for a repo id with out the word "stage".

Thanks.

test case for the merge artifacts by skipping conflicts is added.

Thanks.

please add testng-5.8-jdk15.jar file to the src/test/resources/test-repository-with-conflict-artifacts
in merging module. this jar file is required to pass the test case.
I tried to take a fresh copy of branch and apply that patch.It works fine just after adding that jar file.

Thanks.

Hi Eshan, can you attach the test artifacts as a zip file? They don't get included when you do diff, so they should be added separately I'm getting a test failure without them. Thanks!

this should be added in to test/resources

Thanks

Applied latest patch to MRM-980 branch in -r965157.

I noted a few things that need to be fixed:

1. The repositories page is broken. See attached screenshot MRM-980-broken-repos-page.png
2. Display/show the url of the attached staging repository in the repositories page so it would be easier to know when deploying from command line
3. Maybe we can only have "Merge" in the Merge Actions in repositories page, then have the "Merge All" and "Merge with Skip" in the following page where the list of artifacts that are conflicting are listed.
4. What about in the artifact web browse? After the merge is done, should the repository ID of the artifact be changed to the managed repo instead of the staging one?
5. The merging doesn't seem to be working. I uploaded an artifact (from Upload Artifact in webapp) to the staging repo then clicked "Merge All". When I checked the managed repo, the artifact I uploaded in the staging repo wasn't merged.

Thanks!

following modifications are done

1). in repositories page show only one button called merge. If conflicts are available show the conflicts and allow skipp conflicts and merge all functions. If no conflicts are available only allow merge function.

2) repositories page is fixed.(broken one )

Thanks.

added following.

• staging repository path is added to repositories page.
• change the repository id from staging one to managed one after the merging is done.

removed unused actions in struts.xml and gave proper name for methods in MergeAction class.

Thanks.

changes are done on repositories page .
( when user gives gives repository location with a name which is different to the repo id ) previously in that case staging repo location in not in repositories page

Audit logs are added when the user complete the merging.

Thanks.

Patch submitted on 07/26 committed to MRM-980 branch in -r979534. Thanks Eshan!

Hi Eshan, please see the issues I've filed that this one depends upon. These need to be fixed as well. Thanks!

MRM 1406, MRM 1404, MRM 1400, MRM 1397 are fixed.
Thanks.

MRM 1405 is fixed.
Thanks.

Hi Eshan, I'm getting test failures in archiva-webapp after applying the patch you submitted on 7/27 8:25pm. The failures are specifically in DeleteManagedRepositoryActionTest and EditManagedRepositoryActionTest.

Also, would it be possible to submit each fix separately and attach it to their respective issues (e.g. 1 patch per issue fixed)? So it would be easier to track/isolate each fix. Thanks!

MRM 1406 fixed.

MRM 1400 is fixed.

I moved the merge jsps inside the admin/ dir in webapp in MRM-980-branch -r981149.

added web app test for repository merging.
Thanks.

Patch for selenium tests submitted to MRM-980 branch in -r985612.

The only thing missing now is the documentation then we can merge this to trunk already.

Closing as fixed. All issues it depends on has been fixed, branch also merged to trunk in -r1002059. Thanks again for implementing this feature Eshan!