Continuum

Continuum fills our server disk with SNAPSHOTs.

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: None
  • Fix Version/s: 1.2
  • Component/s: None
  • Labels:
    None
  • Complexity:
    Intermediate
  • Number of attachments :
    0

Description

Our Deployment Repository Directory fills our server disk with SNAPSHOTs after running hourly builds for a couple of weeks.

We have more or less a hundred applications configured in Continuum. A lot of them create assemblies with all their dependencies embedded, so each snapshot deployed eats a couple of megabytes. We also need hourly builds.

We use 'maven clean deploy' to deploy these snapshots to a dedicated Archiva snapshot repository. In Archiva, these snapshots get purged after a week, only keeping the last 2 or 3 snapshot versions.

However, continuum always installs to the local repository first, before uploading them to Archiva. This repository never gets purged. I was thinking to have Continuum use Archiva's repository directly as internal repository, but I don't know if that's safe or not. Archiva won't know these changes before it scans the repository again.

I was also looking at maven-dependency-plugin:purge-local-repository, but that's not exactly what we want. We just want to purge old snapshot versions when new snapshot versions are installed. Maybe that could be a general Maven feature, but it is especially important for Continuum.

Issue Links

is related to

New Feature - A new feature of the product, which has yet to be developed. CONTINUUM-1253 Allow to deploy artifact without timestamps

  • Major - Major loss of function.
  • Closed - The issue is considered finished, the resolution is correct. Issues which are not closed can be reopened.

New Feature - A new feature of the product, which has yet to be developed. MNG-233 ability to clean up local repository

  • Trivial - Cosmetic problem like misspelt words or misaligned text.
  • Closed - The issue is considered finished, the resolution is correct. Issues which are not closed can be reopened.

Improvement - An improvement or enhancement to an existing feature or task. MPREPO-1 delete old timestamped snapshot artifacts

  • Major - Major loss of function.
  • Open - The issue is open and ready for the assignee to start work on it.

Improvement - An improvement or enhancement to an existing feature or task. MNG-3472 configuration possibilities to limit size of local repository

  • Major - Major loss of function.
  • Closed - The issue is considered finished, the resolution is correct. Issues which are not closed can be reopened.
relates to

New Feature - A new feature of the product, which has yet to be developed. CONTINUUM-782 Add a feature to allow cleaning the m2 local repo once every N days

  • Major - Major loss of function.
  • Closed - The issue is considered finished, the resolution is correct. Issues which are not closed can be reopened.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Geert Pante added a comment -

This never made it into a release...

Show
Geert Pante added a comment - This never made it into a release...
Hide
Permalink
Wendy Smoak added a comment -

What version of Continuum are you using? The 'Affects Version' was not specified.

The summary mentions the deployment repository directory, but the description talks about the local repository. Which one are you concerned about?

The 'Deployment Repository Directory' (third item on the configuration page) is optional. If you have a separate remote repository, you probably want to leave that field blank.

The local repository is not optional, though you could clear it out on whatever schedule suits you. This is the same as a developer occasionally deleting his local repository.

A tool to purge the local repository the same way Archiva does for its managed repos would be nice, but is not specific to Continuum.

Show
Wendy Smoak added a comment - What version of Continuum are you using? The 'Affects Version' was not specified. The summary mentions the deployment repository directory, but the description talks about the local repository. Which one are you concerned about? The 'Deployment Repository Directory' (third item on the configuration page) is optional. If you have a separate remote repository, you probably want to leave that field blank. The local repository is not optional, though you could clear it out on whatever schedule suits you. This is the same as a developer occasionally deleting his local repository. A tool to purge the local repository the same way Archiva does for its managed repos would be nice, but is not specific to Continuum.
Hide
Permalink
Geert Pante added a comment -

This is Continuum 1.1.

As I understand it, continuum uses the 'Deployment Repository Directory' to override the local repository. If it is blank, I guess it uses the current user's default repository (~/.m2/repository). The documentation is not clear for this. Can you explain what you mean by 'The local repository is not optional'?

I agree that a solution is not specific for Continuum, however using Continuum intensively makes the problem a lot worse.

Show
Geert Pante added a comment - This is Continuum 1.1. As I understand it, continuum uses the 'Deployment Repository Directory' to override the local repository. If it is blank, I guess it uses the current user's default repository (~/.m2/repository). The documentation is not clear for this. Can you explain what you mean by 'The local repository is not optional'? I agree that a solution is not specific for Continuum, however using Continuum intensively makes the problem a lot worse.
Hide
Permalink
Wendy Smoak added a comment -

> As I understand it, continuum uses the 'Deployment Repository Directory' to override the local repository.

That's not my experience. I find that there is always a local repository (~/.m2/repository, or as configured in settings.xml,) and that the 'deployment repository' is an optional remote repository.

Show
Wendy Smoak added a comment - > As I understand it, continuum uses the 'Deployment Repository Directory' to override the local repository. That's not my experience. I find that there is always a local repository (~/.m2/repository, or as configured in settings.xml,) and that the 'deployment repository' is an optional remote repository.
Hide
Permalink
Wendy Smoak added a comment -

Edited the summary a bit, let's focus on the local repository cleanup with this issue.

Show
Wendy Smoak added a comment - Edited the summary a bit, let's focus on the local repository cleanup with this issue.
Hide
Permalink
Baptiste MATHUS added a comment -

Well, I fear that the dependency:purge-local-repository is not completely what is needed here: this mojo only iterate through the dependencies of the current project pom being used (if I'm not wrong).

So I guess we'd need a mojo to browse the whole localRepository and delete artefacts based on parameters, like:

  • old snapshots
  • artifact not accessed since x days
  • regexp
    and so on.

Cheers

Show
Baptiste MATHUS added a comment - Well, I fear that the dependency:purge-local-repository is not completely what is needed here: this mojo only iterate through the dependencies of the current project pom being used (if I'm not wrong). So I guess we'd need a mojo to browse the whole localRepository and delete artefacts based on parameters, like: old snapshots artifact not accessed since x days regexp and so on. Cheers
Hide
Permalink
Wendy Smoak added a comment -

Also consider that there could be more than one local repository.

You an use -Dmaven.repo.local on the command line to point at a different one, each "Installation" of Maven could have settings.xml configured to default to a different location, etc.

This may be more of an issue if/when Continuum supports concurrent builds.

Show
Wendy Smoak added a comment - Also consider that there could be more than one local repository. You an use -Dmaven.repo.local on the command line to point at a different one, each "Installation" of Maven could have settings.xml configured to default to a different location, etc. This may be more of an issue if/when Continuum supports concurrent builds.
Hide
Permalink
Felix Knecht added a comment -

Not sure, but looks like your using maven2.

You can deploy the snapshot file with name 'SNAPSHOT' instead of deploying the snapshot named with timestamp. Thus an already existing snapshot file gets overwritten when deploying the next snapshot.

See http://maven.apache.org/plugins/maven-deploy-plugin/deploy-file-mojo.html and search for uniqueVersion.
Maybe it helps.

Show
Felix Knecht added a comment - Not sure, but looks like your using maven2. You can deploy the snapshot file with name 'SNAPSHOT' instead of deploying the snapshot named with timestamp. Thus an already existing snapshot file gets overwritten when deploying the next snapshot. See http://maven.apache.org/plugins/maven-deploy-plugin/deploy-file-mojo.html and search for uniqueVersion. Maybe it helps.
Hide
Permalink
Geert Pante added a comment -

As far as I know, <uniqueVersion> has effect on 'remote' deployment. But our remote repository is archiva, and since 1.0.2, it cleans up old snapshots quite nicely.

This issue is about keeping the 'local' repository in shape. And according to CONTINUUM-1253, this isn't affected by <uniqueVersion>. CONTINUUM-1253 is, unfortunately, a 'Won't Fix'

Show
Geert Pante added a comment - As far as I know, <uniqueVersion> has effect on 'remote' deployment. But our remote repository is archiva, and since 1.0.2, it cleans up old snapshots quite nicely. This issue is about keeping the 'local' repository in shape. And according to CONTINUUM-1253 , this isn't affected by <uniqueVersion>. CONTINUUM-1253 is, unfortunately, a 'Won't Fix'
Hide
Permalink
Wendy Smoak added a comment -

CONTINUUM-1253 doesn't have anything to do with the local repository. It is related to the deployment repository, use of which is optional.

At Codehaus Mojo, someone is working on a 'local repository purge' plugin using concepts (if not code) from Archiva: http://www.nabble.com/local-repository-purge-plugin-td16937047.html

Show
Wendy Smoak added a comment - CONTINUUM-1253 doesn't have anything to do with the local repository. It is related to the deployment repository, use of which is optional. At Codehaus Mojo, someone is working on a 'local repository purge' plugin using concepts (if not code) from Archiva: http://www.nabble.com/local-repository-purge-plugin-td16937047.html
Hide
Permalink
Marc Lustig added a comment -

This task is already realized by Archiva.
Archiva offers options "Repository Purge By Days Older Than" and "Repository Purge By Retention Count".
And for non-managed repos it's definately a Maven issue.
I suggest to close this ticket.

Show
Marc Lustig added a comment - This task is already realized by Archiva. Archiva offers options "Repository Purge By Days Older Than" and "Repository Purge By Retention Count". And for non-managed repos it's definately a Maven issue. I suggest to close this ticket.
Hide
Permalink
Geert Pante added a comment -

This is indeed an issue that could be solved by a Maven feature, but if Maven doesn't implement it, Continuum should have a workaround.

When you use maven in normal development, it is acceptable that your local repositories aren't cleaned up automatically.
If you use maven in a Continuous Integration environment with a lot of big projects, this is no longer acceptable, and it is a serious threat for large deployments. As a workaround, we have a seperate crontab job to do the cleanup (with 'rm -rf'), and we make sure there is no schedule running at that time.

So, please do not close this issue without having a proper workaround, or before CONTINUUM-782 is implemented.

Show
Geert Pante added a comment - This is indeed an issue that could be solved by a Maven feature, but if Maven doesn't implement it, Continuum should have a workaround. When you use maven in normal development, it is acceptable that your local repositories aren't cleaned up automatically. If you use maven in a Continuous Integration environment with a lot of big projects, this is no longer acceptable, and it is a serious threat for large deployments. As a workaround, we have a seperate crontab job to do the cleanup (with 'rm -rf'), and we make sure there is no schedule running at that time. So, please do not close this issue without having a proper workaround, or before CONTINUUM-782 is implemented.
Hide
Permalink
Olivier Lamy added a comment -

fixed with CONTINUUM-782

Show
Olivier Lamy added a comment - fixed with CONTINUUM-782

People

  • Assignee:
    Olivier Lamy
    Reporter:
    Geert Pante
Vote (1)
Watch (3)

Dates

  • Created:
    Updated:
    Resolved: