MAVEN_OPTS is not being set correctly

PermGen space is a different option, not -Xmx
Try MAVEN_OPTS=-XX:MaxPermSize=512m

I've tried that too(from command line only, %MAVEN_HOME%/bin/maven file still has MAVEN_OPTS=" -Xmx1024m -Xms1024m"),
here is the part of output

this happens when jcoverage is trying to run (after junit-report, which goes quite well)

[copy] Copying 52 files to C:\project\javasrc\media\target\j

java:jar-resources:

test:prepare-filesystem:

test:test-resources:

test:compile:

test:test:
[junit] Running media.client.ClientCommandTest
[junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
[junit] Exception in thread "Thread-2" java.lang.StackOverflowError
[junit] at com.jcoverage.coverage.InstrumentationFactory.newInstrumentation(InstrumentationFactory.java:88)
[junit] at com.ge.security.media.server.webservices.GenesisAppender.append(GenesisAppender.java:65)
[junit] at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:221)
[junit] at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:57)
[junit] at org.apache.log4j.Category.callAppenders(Category.java:187)
[junit] at org.apache.log4j.Category.forcedLog(Category.java:372)
[junit] at org.apache.log4j.Category.debug(Category.java:241)
[junit] at com.jcoverage.coverage.InstrumentationFactory.newInstrumentation(InstrumentationFactory.java:89)
................................
very long ...........
...................................

[junit] at com.jcoverage.coverage.InstrumentationFactory.newInstrumentation(InstrumentationFactory.java:89)
[junit] at com.ge.security.media.server.webservices.GenesisAppender.append(GenesisAppender.java:65)
[junit] at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:221)
[junit] at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:57)
[junit] at org.apache.log4j.Category.callAppenders(Category.java:187)
[junit] at org.apache.log4j.Category.forcedLog(Category.java:372)
[junit] at org.apache.log4j.Category.debug(Category.java:241)
[junit] at com.jcoverage.coverage.InstrumentationFactory.newInstrumentation(InstrumentationFactory.java:89)
[junit] at com.ge.security.media.server.webservices.GenesisAppender.append(GenesisAppender.java:65)
[junit] [ERROR] Test media.client.ClientCommandTest FAILED
[junit] Running media.client.ClientHandlerTest (this is the next test case)
..................................
.........................................
and all of them generate the same kind of output...

the JVM memory useage is just under 209m
however, if I remove the maven-junit-report-plugin from reports the build is successful(jcoverage is still in the list)

-Tigran

What do you mean whith %MAVEN_HOME%/bin/maven file still has MAVEN_OPTS=" -Xmx1024m -Xms1024m" ?

call this before running maven
set MAVEN_OPTS=" -Xmx1024m -Xms1024m -XX:MaxPermSize=512m"

I tried to set this from commond line before running maven, and when I tried to run maven I've got an error saying -Xms1024m is not excpected here.
I changed the value of MAVEN_OPTS in %MAVEN_HOME%/bin/maven file to be set up (if it's not set up yet) to be "-Xmx1024m -Xms1024m -XX:MaxPermSize=1024m" (after trying 512m for last part) and opened new cmd, when I run maven multiproject:site this time, I had the following output:

(note: this is while junit-report plugin is running)
..............................
..............................
all unit tests pased before this point
................................................................
[junit] Running media.BaseDataSourceTest
[junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
[junit] [ERROR] Test media.BaseDataSourceTest FAILED
[junit] Running media.BaseDataStreamTest
[junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
[junit] [ERROR] Test media.BaseDataStreamTest FAILED

Exception in thread "Thread-79" java.lang.OutOfMemoryError: PermGen space
java.lang.OutOfMemoryError: PermGen space
java.lang.reflect.InvocationTargetException
Exception in thread "main" java.lang.OutOfMemoryError: PermGen space

please note that all of the unit tests are passing when I just run test:test goal

-Tigran

what is your setting for maven.junit.fork?

Look at the junit plugin preperties and jcoverage ones
http://maven.apache.org/maven-1.x/reference/plugins/jcoverage/properties.html
http://maven.apache.org/maven-1.x/reference/plugins/test/properties.html

and set the fork and take a look to the jvmargs property

Thank you for you comments on plugin properties, this did help in a way that JVM memeory usage is not as high as without forking,
However I'm still having a problem with jcoverage (even if I run it all by itself as maven maven-jcoverage-plugin:report ), still getting the same error :

jcoverage:on:
instrumenting the class-files...
[instrument] jcoverage 1.0.5 copyright (c)2003 jcoverage ltd. http://jcoverage.com/
[instrument] jcoverage is licensed under the GNU General Public License
[instrument] jcoverage comes with ABSOLUTELY NO WARRANTY
[instrument] instrumenting 16 classes to C:\project\javasrc\media\target\jcoverage\classes

java:jar-resources:

test:prepare-filesystem:

test:test-resources:

test:compile:

test:test:
[junit] Running media.client.ClientCommandTest
[junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
[junit] Exception in thread "Thread-2" java.lang.StackOverflowError
[junit] at com.jcoverage.coverage.InstrumentationFactory.newInstrumentation(InstrumentationFactory.java:88)
[junit] at com.ge.security.media.server.webservices.GenesisAppender.append(GenesisAppender.java:65)
[junit] at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:221)
[junit] at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:57)
[junit] at org.apache.log4j.Category.callAppenders(Category.java:187)
[junit] at org.apache.log4j.Category.forcedLog(Category.java:372)
[junit] at org.apache.log4j.Category.debug(Category.java:241)
[junit] at com.jcoverage.coverage.InstrumentationFactory.newInstrumentation(InstrumentationFactory.java:89)
[junit] at com.ge.security.media.server.webservices.GenesisAppender.append(GenesisAppender.java:65)
------------------------------------
-------------- about 300 lines of simular log for each test class -------------
---------------------------------------

This may be jcoverage issue as well... since the JVM memory usage is much less now ( about 40m )

Well, you need to show the test reports to see where the problem comes from

I had a similar problem today.

A possible workaround could be to set both, -XX:PermSize=256m and -XX:MaxPermSize=256m (see http://www.unixville.com/~moazam/stories/2004/05/17/maxpermsizeAndHowItRelatesToTheOverallHeap.html)