Support *.external entries in NBMs

At least reporting this as an error in revision 13781.

It is EXTREMELY bad idea to let build FAIL because of this problem. Much better would be to produce WARNING only.

If there is complicated classpath in the project with many foreign modules, the only way to successfully run functional tests is in runtime scope (in a special testing profile). In that case your "correctness" kills the build while incorrectly created junit does not make any harm to the test because control is already given to NbModuleSuite.

For me the only workaround is fall back to pre-13781.

If you have a self-contained example of a project which incorrectly fails as a result of revision 13781, file a report blocking this one. Why would test dependencies ever be in runtime scope?

Sorry, I can't provide self-contained example because I am not able to locate exactly source of the problem. In general, functional tests of the project run, of course, in test scope, but test modules dependencies, including org-netbeans-libs-junit4, have to be declared as normal dependencies (without test scope) and included in regular clusters in order to be enabled at the right moment. Here is more detailed description:

The project under question has complicated dependency tree with several hundreds of modules including external and native libraries. Order of activation of modules seems to be important for proper operation of the application. Approx. 1 year ago, after several weeks of unsuccessful attempts to set-up functional testing, this setting proved itself as working and stable:

    public static Test suite() {
        NbModuleSuite.Configuration conf = NbModuleSuite.emptyConfiguration().enableClasspathModules(false);
        conf = conf.honorAutoloadEager(false);
        conf = conf.clusters(".*");
        conf = conf.enableModules("appcluster", ".*");
        conf = conf.enableModules("org.netbeans.libs.junit4");
        conf = conf.enableModules("org.netbeans.modules.nbjunit");
        conf = conf.enableModules("harness", "org.netbeans.modules.jemmy");
        conf = conf.enableModules("ide", "org.netbeans.modules.jellytools.ide");
        conf = conf.enableModules("harness", "org.netbeans.modules.jellytools.platform");
        conf = conf.addTest(
                MyTest.class, "test1", "test2",
                "test3");
        return conf.suite();
}

To make this work, test application has to be built with dependencies on testing libraries, i.e. Jemmy, jellytools, nbjunit and junit4. Test scope does not work because enableModules can't locate and enable the modules.

Any other settings, e.g. with enableClasspathModules(true) and test scope dependencies did not work for this project. Even if a test was started successfully, test run was unstable with unpredictable class loading errors depending among others e.g. on the order of activation of different factory methods in the application.

Functional tests created this way are stable and reliable on all our computers (both Linux and Windows) and can be run from Hudson (Tomcat) as well. The only disadvantage is that functional test application needs special build with modified dependencies (in a specialized profile). Now, with 13781 "improvement", my test application build fails with "not yet implemented" error. With previous version of MNB plugin tests are working fine.

Pseudomodules like jellytools.ide should not be enabled as modules in this way. Not sure what the "unpredictable class loading errors" you encountered were, but these are likely what you really needed to be solving.

Anyway I hope to implement this feature in the next plugin version so revision 13781 would of course be rewritten to actually unpack the correct files. Would be helpful for http://netbeans.org/bugzilla/show_bug.cgi?id=202041 for example.

Anything not killing the build will help.

Unpredictable class loading errors mean that if factories are enabled (i.e. windows are opened or various functions of the program are started) in one order, everything works while when the same windows/functions are opened in reverse order, failures like "Cannot load class from both x and y" or similar arrive.

Most probably, the source of problems is that some unneeded classes are loaded when enableClasspathModules is true while only correct modules are loaded when it is false. Starting the test suite way presented above works fine for this particular project. If you can propose other way how to make things working (i.e. correctly load pseudomodules with enableClasspathModules false), it would be welcome, of course. Nevertheless, if you will not kill my builds, I am OK with presented setting.

I cannot propose any fix without a test case to reproduce - would need to study the root cause of the class loading errors you encountered. Would be a separate issue of course.

Three parts to this:

1. CreateClusterAppMojo: unpacking external files when encountered in NBMs.
2. CreateNetBeansFileStructure: creating NBMs with external files where appropriate. Tricky since this is only appropriate when the artifact is present in Central; it seems this can be detected in Maven 3.
3. PopulateRepositoryMojo: translating netbeans.org NBMs to use external files where possible, as requested in #202041. Not clear what to do with signed NBMs - just drop the signature?

Committed revision 16055 for parts 1 & 2.

Great, revision 16055 returns my project back to normal, i.e. the same behavior as pre-13781. Functional tests are working for the project when pseudomodules are enabled using the way described above.

Thank you!

For the record, this is also blocking a project of mine (blueMarine). My use case is that I have some "acceptance tests" that are packaged as a plugin of my application. They are run by installing e.g. a release candidate, then you go to the update center and install them, then run them by means of a special menu integrated with the application. It's a very valuable feature since you can easily have end users (e.g. beta testers) run tests without needing any software development knowledge, and you're able to run tests on a very wide target of installations.

I'm going to drop JUnit in favour of TestNG in future, so sooner or later this won't affect me any longer, but it's not going to happen very soon. So if the bug is fixed, I'd be happy.

The parts of this bug relevant to Frantisek and Fabrizio are already fixed in trunk sources (16055) but not in the latest release (3.7). As a convenience I have deployed a snapshot of the plugin to tide you over until 3.8 is released (probably after 7.2 beta with MNBMODULE-164). Use repo https://nexus.codehaus.org/content/repositories/snapshots/ for this purpose.

Filed separate issue for part #3.

Parts 1 & 2 fixed for 3.8.

there is a bug in the generated .external file. There's no newline separator between CRC and SIZE fields, generating error. It was reported by a user testing 3.8..

that's easily fixable. However it led me to the CreateNetbeansFileStructure.trackedInCentral() method which is completely flawed.
1. relies on metadata explicitly denoted to be private and open for change in future releases.
2. it doesn't work with mirrors (that's why I never encountered with my my private local nexus)

I will remove the .external file generating without replacement in current version. In future we will have to come up with additional configuration elements.

Lack of newline: right; introduced with add-on fix matching https://netbeans.org/bugzilla/show_bug.cgi?id=212068 I guess. Fixed in revision 17359.

Regarding use of *-mirrors masking central: indeed this does break trackedInCentral. Still I think it is better to have this feature work when you are not using such a mirror than not to work at all. I am not sure if it possible for the plugin to automatically detect whether an artifact retrieved via mirror is in fact available in Central or not, so perhaps follow this algorithm:

1. If a plugin configuration element claims this artifact is in Central, then create the *.external; if it claims it is not, do not.
2. Otherwise (no such configuration) if _maven.repositories exists and says >central then create but issue a warning recommending the configuration in #1.
3. Otherwise (using Maven 2, or claims to be in another repo) try to make an online connection to Central to check for existence of this artifact; if present, create but again issue a warning.
4. Otherwise (not in Central, or offline or cannot reach Central) do not create but issue a warning.

and why is turning everything to external important? is is really so important to introduce such complicated, fragile scheme?

Still I think it is better to have this feature work when you are not using such a mirror than not to work at all.

Well, it makes the build different based on local configuration, maven version and existing metadata in local repository. That's unpredictability I would not expect from a build tool.

True. Maybe introduce the configuration element, always false (use plain JAR) by default, but when unconfigured issue a warning for each relevant artifact asking you to specify whether you want to package it as external or inline, showing example configuration to do so. (If _maven.repositories exists, default the value in the example config to true iff it says >central or the specified repo is currently configured to mirror *.)

That would make build results for existing projects fully reproducible yet guide users to using the feature wherever possible.

we would need another parameter similar to Libraries field in the module descriptor, which would list the external items (matched by groupid/artifactid)
The hardest part is explaining what it does and what are the consequences at platform/IDE runtime.

http://fisheye.codehaus.org/changelog/mojo/?cs=17363 explicit configuration element to mark artifacts as external. removed the check for private maven metadata

"the user has to be online and be capable of reaching [C]entral using [M]aven" - or direct HTTP.

Should also make it clearer that these restrictions on the user only apply to installing the NBM via e.g. Plugin Manager into an existing application, at which time the user is presumably already online; if this NBM is just being included in an app distributed in some other form (e.g. a ZIP file), there is no downside, just the upside of making the NBM considerably smaller.