Maven3 parallel build fails occasionally for classpath problems.

Now that I've studied maven architecture a bit more, I suppose this is not a problem in the maven-compiler-plugin because the "classPathElements" is fed to it via the plug-in framework.

-I could not yet find out which maven/other component is setting up the wrong "classPathElements"
-Anyone have any idea, where to address this fault?

Apparently this kind of issues have been handled in Maven 2 & 3: "Plugins and Lifecycle". Could somebody with enough rights move this issue there?

The same problem occur with maven 3.0.2 when building massive parallel. In my situation (> 200 modules) the problem occur within the compiler plugin for test classes. Sometimes the junit 4.8 classes can't be found and sometimes the mockito classes can't be found. I've never seen this error in non test classes compile. So maybe it has something to do with the surefire plugin, but I don't know.

When I run the build in serial or with only two parallel Threads (-T 2) it works always but if I increase the parallel threads, I always get this test class compilation errors. The errors does not occur always on the same module. Sometimes a module will build well and a later module will fail and on an other build the module fails which was successfully run on a previously build. But with more than -T 2 I never had a successfully complete build over the whole project.

Maybe this helps a bit for further analyzing.

Can you try to set the system property -Dmaven.artifact.threads=1 and see if the problem goes away ?

I uplodaded a snapshot version of 3.0.3 to https://repository.apache.org/content/repositories/snapshots/org/apache/maven/apache-maven/3.0.3-SNAPSHOT/apache-maven-3.0.3-20110201.153244-1-bin.zip that has a fix.

I would be very happy if you could test this.

Thanks Kristian. I tried both, the fixed version and also the -Dmaven.artifact.threads=1. I still get the same error.

I wonder which part of maven(or some plugin) is setting up&handing out the classpath?

I tried them both separately but not together. Is that relevant?

Thanks a lot, that's two very good pieces of information.

Artifact resolution (and classpath building) is a fairly long-winded process in maven core. MavenProject#getCompileClasspathElements is probably a nice place to start looking. (MaveProject line 419).

I would be very interested in knowing if the build fails if you add the following few lines:

public List<String> getCompileClasspathElements()
throws DependencyResolutionRequiredException {
List<String> list = new ArrayList<String>(getArtifacts().size() + 1);

list.add(getBuild().getOutputDirectory());

if (resolvedArtifacts.size() == 0)

... rest of method ...
}

(Check out from https://svn.apache.org/repos/asf/maven/maven-3/trunk, mvn clean install, zip file is in apache-maven/target/)

So if we could determine if resolvedArtifacts.size is 0 at the time you crash, we would effectively split the
entire code base in 2. Binary search is a powerful tool with these things. Make sure the patch works non-threaded first

I am assuming these are all non-public builds that I don't have access to ?

Uhhh...now this got to the level of my incompetence. Checked out succesfully, but my internal repository does not contain all the artifacts needed for building and I'm not supposed(capable? don't know how...) to use external repositories(maven central)...

(Yes, you figured it, it's unfortunately closed source.)

It seems I've forgotten to specifically mention that the jbi-maven-plugin is not yet marked as thread-safe and is causing a lot of warnings in the build log. I probably thought that it's not causing the problem because (as far as I know) it's mostly just binding the "basic" maven plugins to different phases. But actually, I really don't know. And actually I don't know which versions of the "basic" maven plugins(depensurefire/ it uses... maybe it's binding some specific(older) versions and not using the ones defined in our pom.xmls...... hmmmm... actually I propably thought the pluginArtifactMap there would reveal this...

I observed a similar behaviour in a build containing some 140 modules.
The build made heavy use of the antrun plugin to trigger custom generators and I observed two kinds of effects:

• java tasks executed via antrun did not find their main class
• compiles did not find any libraries

These effects appeared rather randomly revealing the kinds of broken classpaths described by Kari (classpath containing only the module's output directory).

Now I seem to have resolved the problem by supplying explicit versions of both the maven-ejb-plugin and the maven-source-plugin (which had been forgotten before).
The astonishing thing about this is that I had defined exactly those (threadsafe) versions of the plugins which in the broken builds maven had figured out to use by itself. Moreover I observed that the modules whose builds had been broken by the missing versions did not have to use these plugins (at least not both of them, but I think they always used at least the maven-source-plugin).
And it appears thet the effect cannot be reproduced by simply leaving off one of the versions. I need to remove both version numbers to see the build broken (but maybe this simply increases the probability).

So my impression is that not the usage of a non-threadsafe plugin causes the problem but rather the absence of version definitions.

At least my build seems stable now

OK.
I've been mistaken about this. Sometimes our build still fails as desrcibed above, although the probability seems to have been reduced.
Strange.

The problem is still there with the version 2.10 of the plugin. As a (bad) workaround, we now set the environment variable JAVA_TOOL_OPTIONS=-Xbootclasspath/p:<path to junit.jar>:<path to mockito.jar> before starting maven this works also with massive parallel build's (e.g. -T 20). As I told, not nice but it works.

Frank

I've tried to do further debugging with this problem (in my situation always surefire testing classpath problems). Til now, it looks like a problem in the maven itself when running massive parallel (-T 20) and not in the surefire plugin. I found the following:

Scenario:
Building with -T 20 the build fails after some minutes always with a ClassNotFound exception in the SurefirePlugin for classes which are defined as dependency projects with scope test. Often, the junit classes itself could not be found. The error occure randomly in different projects, but always while surefire executes.

Analyze:
The Method org.apache.maven.project.MavenProject.setResolvedArtifacts(...) is called from org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies(...) to set the actual resolved artifacts. The surefire plugin request this artifacts (later) when executing with a call to org.apache.maven.project.MavenProject.getArtifacts() to build the classpath for executing the testing.

I put some printouts with timestamp to the maven (3.0.3) code and found, that in case of this error, the setResolvedArtifacts() method in the MavenProject object is called in the wrong order or at the wrong time. First the setResolvedArtifacts() is called with all dependencies for the scopes "compile,provided,runtime,system,test", some milliseconds later this setter is called again for all dependencies for only the scopes "compile,runtime". After that, the surefireplugin starts and calls the getArtifacts() method on the MavenProject an gets the wrong (too short) list of artifacts and all test dependencies are missing to build the surefire test classpath. This results in the ClassNotFoundExecption of the surefire plugin.

Maybe the call to setResolvedArtifacts() is made too early before starting the surefire test, because in between it's called again without the test dependencies and before the surefire starts and calls the getArtifacts() method.

Looking a bit deeper I found, that the callstack for the setResolvedArtifacts() is:

org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies(LifecycleDependencyResolver.java:107)
org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved(MojoExecutor.java:259)
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:202)
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:155)
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:147)
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
org.apache.maven.lifecycle.internal.LifecycleThreadedBuilder$1.call(LifecycleThreadedBuilder.java:167)
org.apache.maven.lifecycle.internal.LifecycleThreadedBuilder$1.call(LifecycleThreadedBuilder.java:163)
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
java.util.concurrent.FutureTask.run(FutureTask.java:138)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
java.util.concurrent.FutureTask.run(FutureTask.java:138)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)

I don't really understand the deeper parallel and synchronization mechanisms in maven when using parallel builds, so I don't know what I could do for further analyzing. Maybe someone could give me some information how to dig deeper in this problem. I can reproduce the problem always for each run with -T 20 but it occures often in different sub-projects but always with the described problem.

Frank

We had a similar issue with parallel builds where test scoped dependencies would disappear and fail test-compile step randomly.

Turns out that during parallel builds, an ill-configured aggregator module can overwrite the project dependencies with scopes [compile/runtime] while the project is in process of being built and after dependencies have been resolved with [compile/runtime/provided/test] scopes.

Due to this, the test classpath in our case did not have any test scoped dependencies on the classpath, hence failing the compilation.
To fix this issue, we removed the aggregator plugin from a child module.

Maven should not execute aggregator plugins concurrently or at least produce a warning that helps users figure out the root cause of the compilation failure.

The problematic code is in MojoExecutor.ensureDependenciesAreResolved():
if ( aggregating )
{
Collection<String> scopesToCollect = toScopes( mojoDescriptor.getDependencyCollectionRequired() );
Collection<String> scopesToResolve = toScopes( mojoDescriptor.getDependencyResolutionRequired() );

if ( dependencyContext.isResolutionRequiredForAggregatedProjects( scopesToCollect, scopesToResolve ) )
{
for ( MavenProject aggregatedProject : session.getProjects() )
{
if ( aggregatedProject != project )

}
}
}

This calls aggregatedProject.setResolvedArtifacts() and overwrites the previously resolved dependencies.