Maven 2.x Clean Plugin

Out of Memory Error cleaning target directory with *large* number of sub-directories

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Critical Critical
  • Resolution: Fixed
  • Affects Version/s: 2.2
  • Fix Version/s: 2.4
  • Component/s: None
  • Labels:
    None
  • Environment:
    maven-2.0.8, solaris, jdk 1.6.0_05
  • Number of attachments :
    1

Description

running mvn clean on a target directory with a large number of sub directories causes an OutOfMemoryError. The plexus DirectoryScanner seems to be the major culprit from the stacktrace.

Issue Links

This issue relates to:
CONTINUUM-2119 Cleaning up large working directories fails with OutOfMemoryError Major Reopened

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Vincent Siveton added a comment -

What is your MAVEN_OPTS? Did you try to increase the memory?

Show
Vincent Siveton added a comment - What is your MAVEN_OPTS? Did you try to increase the memory?
Hide
Permalink
Ryan Sonnek added a comment -

We're running with 512M for the JVM heap size which should be more than enough for this operation.

The question is why is the directory scanner caching references to files so aggressively? This seems very inefficient.

Show
Ryan Sonnek added a comment - We're running with 512M for the JVM heap size which should be more than enough for this operation. The question is why is the directory scanner caching references to files so aggressively? This seems very inefficient.
Hide
Permalink
Vincent Siveton added a comment -

I created a dir in target with 100 sub dirs in 3 levels so 100 * 100 *100 sub dirs. The plugin is able to clean the target. with MAVEN_OPTS=-Xms256m -Xmx256m
So what is your large number of sub dir? Could you send us a test case with the env to reproduce it?

Show
Vincent Siveton added a comment - I created a dir in target with 100 sub dirs in 3 levels so 100 * 100 *100 sub dirs. The plugin is able to clean the target. with MAVEN_OPTS=-Xms256m -Xmx256m So what is your large number of sub dir? Could you send us a test case with the env to reproduce it?
Hide
Permalink
Bouiaw added a comment -

It seems to be linked with number AND the size of the data to clean. I don't know if it load in memory all the content to delete, but try to copy a large amount of data in target, and run mvn clean ...

Show
Bouiaw added a comment - It seems to be linked with number AND the size of the data to clean. I don't know if it load in memory all the content to delete, but try to copy a large amount of data in target, and run mvn clean ...
Hide
Permalink
Benjamin Bentmann added a comment -

Fixed in r897682.

On a test structure that resembles a binary tree of depth 14 with two files at each level besides the two child directories, I observed the following improvements:

[INFO] --- maven-clean-plugin:2.3:clean (default-clean) @ test ---
[INFO] Deleting file set: M:\test\. (included: [], excluded: [pom.xml])
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3:50.797s
[INFO] Finished at: Sun Jan 10 18:34:49 CET 2010
[INFO] Final Memory: 1M/42M
[INFO] ------------------------------------------------------------------------

[INFO] --- maven-clean-plugin:2.4-SNAPSHOT:clean (default-clean) @ test ---
[INFO] Deleting M:\test (includes = [], excludes = [pom.xml])
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 41.938s
[INFO] Finished at: Sun Jan 10 18:39:39 CET 2010
[INFO] Final Memory: 1M/4M
[INFO] ------------------------------------------------------------------------

i.e. the memory used dropped from 42 MB to 4MB.

Show
Benjamin Bentmann added a comment - Fixed in r897682. On a test structure that resembles a binary tree of depth 14 with two files at each level besides the two child directories, I observed the following improvements: 
[INFO] --- maven-clean-plugin:2.3:clean (default-clean) @ test ---
[INFO] Deleting file set: M:\test\. (included: [], excluded: [pom.xml])
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3:50.797s
[INFO] Finished at: Sun Jan 10 18:34:49 CET 2010
[INFO] Final Memory: 1M/42M
[INFO] ------------------------------------------------------------------------

[INFO] --- maven-clean-plugin:2.4-SNAPSHOT:clean (default-clean) @ test ---
[INFO] Deleting M:\test (includes = [], excludes = [pom.xml])
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 41.938s
[INFO] Finished at: Sun Jan 10 18:39:39 CET 2010
[INFO] Final Memory: 1M/4M
[INFO] ------------------------------------------------------------------------
i.e. the memory used dropped from 42 MB to 4MB.
Hide
Permalink
Benjamin Bentmann added a comment -

Also, my testing indicates that followSymlinks=false requires more memory than followSymlinks=true. So using the latter option value for projects with huge directories but knowingly no symlinks could also help.

Show
Benjamin Bentmann added a comment - Also, my testing indicates that followSymlinks=false requires more memory than followSymlinks=true. So using the latter option value for projects with huge directories but knowingly no symlinks could also help.

People

Vote (6)
Watch (6)

Dates

  • Created:
    Updated:
    Resolved:
Atlassian JIRA (v4.4.3#663-r165197) | Report a problem
Powered by a free Atlassian JIRA open source license for Codehaus. Try JIRA - bug tracking software for your team.