When JUnit and TestNG tests are in same project, only one set gets run

IMO, this problem is probably too hard to be worth solving. See the mailing list archive thread here:

http://mail-archives.apache.org/mod_mbox/maven-surefire-dev/200711.mbox/%3cPine.WNT.4.64.0711191635230.2484@sfeng02.rf.lan%3e
http://tinyurl.com/32745g

Alex suggested a good way to tell the different between JUnit and TestNG runs, but trying to dynamically construct multiple test runs from two arrays of classes that don't clobber each other when you go to run them is harder than it has any right to be.

I'm going to revert my workspace to blow away my current work on this; in the meantime, here's the current check-in. It sort-of works, but the TestNG XML output is clobbered when you run TestNG twice; I don't see a good way to fix that.

Fixed in revision 598205. Basically just the patch as was, but we drop testng results in separate directories if we need to run both junit and testng results at once.

The fix does not work if tests are written using junit 4.4

Perhaps with additional check it would work:
...
if (junit.framework.Test.class.isAssignableFrom( c ) || isJunit4(c)) {
junitTestClasses.add( c );
} else {
testNgTestClasses.add( c );
}
...

public boolean isJunit4(Class c) {
Class<? extends Annotation> testAnnotation = null;
try { testAnnotation = (Class<? extends Annotation>) Class.forName("org.junit.Test"); } catch (ClassNotFoundException e) { return false; }
Method[] methods = c.getMethods();
for (int i = 0; i < methods.length; i++) {
if (methods[i].isAnnotationPresent(testAnnotation)) { return true; }
}
return false;
}

I agree with Pawel

When adopting TestNG into an existing project with lots of existing JUnit 4.4 tests, it would be nice if both types of tests could be run at once.

If Pawel's additional check would fix this, is it possible to add this to the next SNAPSHOT version?

I think I just got bit with this one, too.
Is this still broken with latest versions then? [i.e. junit 4.4 aren't run at all when packaged with testng classes]?
In my instance it was one module [junit] depending on another [testng] and thus the firsts tests aren't being run. Sniff.
-=r

In my case it might have been being caused by:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.1</version>
</plugin>

in my pom.xml [note the old version]

Hi,

I'm afraid there is more to be done to allow both JUnit and TestNG tests. Changing surefire plugin is the easy part. Surefire uses testng runner to run JUnit tests, which doesn't support JUnit4 tests. Therefore TestNG team would have to change their runner to support JUnit4

Actually, I've made changes to both libraries. If there is someone from surefire team willing to apply my changes I can generate a patch. I've also contacted TestNG team, but they are not 100% sure they want to do junit4 support.

Hi,
I am trying to run both JUnit 4.4 and TestNG 5.8 tests in the same suite. I seem to have come accross this problem in surefire 2.4.3 where as soon as I add testNG as a dependency, the TestNG runner is used. The problem is that this runner cannot handle JUnit 4.4 tests: I get:

Running TestSuite
org.testng.TestNGException:
Failure in JUnit mode for class com.sampleTest: could not create/run JUnit test suite:
cannot retrieve JUnit method

Which is resolved if I extend TestCase in com.sampleTest as in JUnit 3.x

A sample of the pom.xml I use is:

<dependencies>
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>5.8</version>
<scope>test</scope>
<classifier>jdk15</classifier>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.4</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.4.3</version>
<configuration>
<properties>
<property>
<name>junit</name>
<value>true</value>
</property>
</properties>
</configuration>
</plugin>
</plugins>
</build>

Note, at present I would be happy to run all JUnit 4.4 tests in one execution and then use a profile to run all TestNG tests.

Please can you tell me whether you know of a work around for this?

Cheers,
James

Well, I've found a work around which is not too painful, just a little awkward.

Because it is the TestNG dependency that triggers surefire to use the TestNG runner to execute JUnit tests, I've moved this dependency out of the main project scope. In order to compile and run all JUnit tests, I exclude the TestNG tests from the compiler and surefire plugins.

Then in a profile, I add my TestNG dependency and adjust the compiler and surefire plugins to include the TestNG tests and the test suite file.

To run JUnit tests I use: mvn test
To run testNG tests I use: mvn test -P testng-profile

Simple when you know how. Any better solutions appreciated since I see problems with this as my number of tests increases.

James

Shouldn't this bug's resolution should be "won't resolve" or something. If it's fixed can someone please help me to understand how it is fixed(because fixing this bug means both Junitand TestNG tests should run in one set, by some auto discovery mechanism internally)

Udai, check my comment above. Surefire uses TestNG library to run junit tests and TestNG doesn't support running junit4 (only junit3). That's why junit4 tests are not being run.

Thanks Pawel, I guess I might need a Junit 4 to Junit3 migration tool

I have a better solution.

The idea is to create two executions of the maven-surefire-plugin, one for JUnit, one for TestNG. You can disable one of JUnit/TestNG per execution by specifying nonexisting junitArtifactName or testNGArtifactName:

<plugin>
	<groupId>org.apache.maven.plugins</groupId>
	<artifactId>maven-surefire-plugin</artifactId>
	<executions>
		<execution>
			<phase>test</phase>
			<goals>
				<goal>test</goal>
			</goals>
			<configuration>	
				<testNGArtifactName>none:none</testNGArtifactName>
			</configuration>
		</execution>
		<execution>
			<id>test-testng</id>
			<phase>test</phase>
			<goals>
				<goal>test</goal>
			</goals>
			<configuration>	
				<junitArtifactName>none:none</junitArtifactName>
			</configuration>
		</execution>
	</executions>
</plugin>

Aleksei's fix works well, thanks! I just had to mess about with the default execution, so here is a variation using Aleksei's solution.

<plugin>
	<groupId>org.apache.maven.plugins</groupId>
	<artifactId>maven-surefire-plugin</artifactId>
	<version>2.4.3</version>
	<configuration>	          
                <testNGArtifactName>none:none</testNGArtifactName>   
		<excludes>
			<exclude>**/integration/*.java</exclude>
		</excludes> 
        </configuration>
	<executions>
		<execution>
			<id>integration-test</id>
			<goals>
				<goal>test</goal>
			</goals>
			<phase>integration-test</phase>
			<configuration>
				<junitArtifactName>none:none</junitArtifactName>
				<testNGArtifactName>org.testng:testng</testNGArtifactName>
				<suiteXmlFiles>
					<suiteXmlFile>
                                           src/test/resourcestestng.xml
                                        </suiteXmlFile>
				</suiteXmlFiles>
				<excludes>
					<exclude>none</exclude>
				</excludes>
				<includes>
					<include>**/integration/*.java</include>
				</includes>
			</configuration>
		</execution>
	</executions>
</plugin>

@Aleksei: thank you, a year from your comment I used your solution verbatim (+ version) and it worked. For the reference, this is the pom.

Thanks all for the solution. Note: I ran into an issue when using it with sonar. See SONAR-2408