Maven Surefire

Surefire finds test classes but ignores test methods and configuration methods with TestNG and includes tag

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: 2.0 (2.2 plugin)
  • Fix Version/s: 2.3
  • Component/s: None
  • Labels:
    None
  • Environment:
    Maven 2.0.4
    Surefire Plugin 2.2
    TestNG 4.7 jdk15 (using annotations)
    Sun JDK 1.5_06
    Windows XP SP 2
  • Testcase included:
    yes
  • Number of attachments :
    3

Description

Attachment contains 2 projects which are identical except for how the test cases are referenced. In one project they are referenced via a suite xml file. In the second they are referenced using the includes tag. I believe they should produce identical results.

Both projects find all 6 test classes, but only the one using a suite xml file actually executes the configuration methods and test methods. The project using an includes tag completely ignores the configuratiion methods and test methods.

Issue Links

This issue is duplicated by:
SUREFIRE-205 TestNG tests are not run by Maven Blocker Closed
SUREFIRE-208 Surefire doesn't work if testng is in the classpath and a suitexml isn't being used Major Closed

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Paul Russell added a comment -

I'm seeing the same thing when using a 'zero-config' Surefire plugin. It's finding the classes, but then not executing any of the methods. Thankfully I'm at the beginning of my project at the moment, so I can make do with JUnit for the time being, but I'd rather be using NG.

Show
Paul Russell added a comment - I'm seeing the same thing when using a 'zero-config' Surefire plugin. It's finding the classes, but then not executing any of the methods. Thankfully I'm at the beginning of my project at the moment, so I can make do with JUnit for the time being, but I'd rather be using NG.
Hide
Permalink
Howard M. Lewis Ship added a comment -

I've hit the same problems and am using a testng.xml configuration file to get my tests to execute.

Show
Howard M. Lewis Ship added a comment - I've hit the same problems and am using a testng.xml configuration file to get my tests to execute.
Hide
Permalink
Paul Russell added a comment -

Have twigged how to role back to the previous version of the surefire plugin (which seems to work okay for TestNG. Just in case anyone else hits this issue and hasn't twigged the workaround, my root project contains:

<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.1</version>
</plugin>
</plugins>
</build>

Show
Paul Russell added a comment - Have twigged how to role back to the previous version of the surefire plugin (which seems to work okay for TestNG. Just in case anyone else hits this issue and hasn't twigged the workaround, my root project contains: <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>2.1</version> </plugin> </plugins> </build>
Hide
Permalink
John Mazzitelli added a comment -

134 and 117 look like similar, if not the same, problems. 134 provides a workaround but 117 appears to have a patch that actually fixes the code.

Show
John Mazzitelli added a comment - 134 and 117 look like similar, if not the same, problems. 134 provides a workaround but 117 appears to have a patch that actually fixes the code.
Hide
Permalink
Paul Russell added a comment -

Chaps, to clarify, my comment above is incorrect. The workaround 'works' for me because I happen to be following JUnit style naming conventions despite my use of NG. This is a pretty critical bug imho, and would benefit from your full attention. Don't want you to be labouring under the misapprehension that my workaround above is solving the problem.

The inevitable question: and ETA for a patch for this?

Show
Paul Russell added a comment - Chaps, to clarify, my comment above is incorrect. The workaround 'works' for me because I happen to be following JUnit style naming conventions despite my use of NG. This is a pretty critical bug imho, and would benefit from your full attention. Don't want you to be labouring under the misapprehension that my workaround above is solving the problem. The inevitable question: and ETA for a patch for this?
Hide
Permalink
Jesse Kuhnert added a comment -

Damn my luck. I seem to be hitting the same wall. I have debugged the problem extensively and find that it is probably the result of something weird happening in the classpath.

The issue is that TestNG will do a Class.getAnnotation(<class annotation type>) on the classes to find methods/config/etc...For whatever reason the classes loaded via maven don't return annotations when this method is called.

I have however been able to list the annotations via Class.getDeclaredAnnotations()...They are actually listed fine but won't match via normal equality checking because they are only proxy classes for some reason. If you try to take one of the annotations in generic form and cast it to the actual class you get a runtime error of "$Proxy4" (or whatever it is in your jvm ) can't be cast to blah blah.

I'll have to see if anyone else can help me debug, but all of the annotation based unit tests (well mostly all) for jdk1.5 in TestNG are passing and working normally.

Show
Jesse Kuhnert added a comment - Damn my luck. I seem to be hitting the same wall. I have debugged the problem extensively and find that it is probably the result of something weird happening in the classpath. The issue is that TestNG will do a Class.getAnnotation(<class annotation type>) on the classes to find methods/config/etc...For whatever reason the classes loaded via maven don't return annotations when this method is called. I have however been able to list the annotations via Class.getDeclaredAnnotations()...They are actually listed fine but won't match via normal equality checking because they are only proxy classes for some reason. If you try to take one of the annotations in generic form and cast it to the actual class you get a runtime error of "$Proxy4" (or whatever it is in your jvm ) can't be cast to blah blah. I'll have to see if anyone else can help me debug, but all of the annotation based unit tests (well mostly all) for jdk1.5 in TestNG are passing and working normally.
Hide
Permalink
Jesse Kuhnert added a comment -

P.S. The patch referenced in MSUREFIRE-117 isn't an actual fix for the problem.

Show
Jesse Kuhnert added a comment - P.S. The patch referenced in MSUREFIRE-117 isn't an actual fix for the problem.
Hide
Permalink
Jesse Kuhnert added a comment -

Attaching patch that fixes the problem. May very well cause other problems but after reading a great deal of annotation related jvm source code and other items it starts to make more sense..

I think it simply isn't able to properly read the annotation meta data from the class file because it can't find the concrete annotation classes in the system classloader...This is probably why proxy classes are loaded in their place.

Show
Jesse Kuhnert added a comment - Attaching patch that fixes the problem. May very well cause other problems but after reading a great deal of annotation related jvm source code and other items it starts to make more sense.. I think it simply isn't able to properly read the annotation meta data from the class file because it can't find the concrete annotation classes in the system classloader...This is probably why proxy classes are loaded in their place.
Hide
Permalink
Jason Chaffee added a comment -

MSUREFIRE-117 IS the same as this bug...or at least they are caused by the same issue. Jesse is correct that patch in MSUREFIRE-117 does not fix the problem...except in very specific enviornment where everything is built and run with java 1.5.

Show
Jason Chaffee added a comment - MSUREFIRE-117 IS the same as this bug...or at least they are caused by the same issue. Jesse is correct that patch in MSUREFIRE-117 does not fix the problem...except in very specific enviornment where everything is built and run with java 1.5.
Hide
Permalink
Jesse Kuhnert added a comment -

Almost.....Your tests also have to be setup exactly like junit tests because chances are very likely that none of the annotation parameters will be found. If anything runs at all it's a lucky coincidence.

Show
Jesse Kuhnert added a comment - Almost.....Your tests also have to be setup exactly like junit tests because chances are very likely that none of the annotation parameters will be found. If anything runs at all it's a lucky coincidence.
Hide
Permalink
Jason Chaffee added a comment -

If you do that TestNG will run them as JUnit tests and not honor any of the annotations. Anyway, this is a pretty nasty bug. I have been working on it ever since this version was released, but I haven't found an adequate fix yet.

Show
Jason Chaffee added a comment - If you do that TestNG will run them as JUnit tests and not honor any of the annotations. Anyway, this is a pretty nasty bug. I have been working on it ever since this version was released, but I haven't found an adequate fix yet.
Hide
Permalink
Jesse Kuhnert added a comment -

The patch file I've attached to this issue fixes it pretty definitively. One line code change.

Hope that helps..

Show
Jesse Kuhnert added a comment - The patch file I've attached to this issue fixes it pretty definitively. One line code change. Hope that helps..
Hide
Permalink
Jason Chaffee added a comment -

It fixes TestNG, but it breaks JUnit tests that are being run by TestNG....they no longer run. This a problem because we have both types of tests.

Show
Jason Chaffee added a comment - It fixes TestNG, but it breaks JUnit tests that are being run by TestNG....they no longer run. This a problem because we have both types of tests.
Hide
Permalink
Jesse Kuhnert added a comment -

I have a few junit tests that are being run by TestNG with the patch supplied and they are working fine....Is there anything special about your environment that might act differently?

The plugin should already be doing a check to determine if the test extends the junit test base class and has junit run it in that case.

Show
Jesse Kuhnert added a comment - I have a few junit tests that are being run by TestNG with the patch supplied and they are working fine....Is there anything special about your environment that might act differently? The plugin should already be doing a check to determine if the test extends the junit test base class and has junit run it in that case.
Hide
Permalink
Jason Chaffee added a comment -

I am not sure...my tests that were being run with a testng.xml file are not being run now either.

Show
Jason Chaffee added a comment - I am not sure...my tests that were being run with a testng.xml file are not being run now either.
Hide
Permalink
Jason Chaffee added a comment -

I should add that I started with a clean repository, applied the patch...built surefire and the surfire plugin. Is there a snapshot available somewhere?

Show
Jason Chaffee added a comment - I should add that I started with a clean repository, applied the patch...built surefire and the surfire plugin. Is there a snapshot available somewhere?
Hide
Permalink
Jesse Kuhnert added a comment -

God...Really? That sucks...In fact this whole thing sucks. There were unit tests to verify these things before...Pretty ironic that unit tests would have told us that a change broke our unit test plugin .

Show
Jesse Kuhnert added a comment - God...Really? That sucks...In fact this whole thing sucks. There were unit tests to verify these things before...Pretty ironic that unit tests would have told us that a change broke our unit test plugin .
Hide
Permalink
Jason Chaffee added a comment -

Can you try JUnit tests with testng and a testng.xml? I would like confirm that it doesn't work or that there is something screwy on my machine. I have encountered strange anomalies with maven 2 where it works one way on one machine and differently on another.

Show
Jason Chaffee added a comment - Can you try JUnit tests with testng and a testng.xml? I would like confirm that it doesn't work or that there is something screwy on my machine. I have encountered strange anomalies with maven 2 where it works one way on one machine and differently on another.
Hide
Permalink
Jesse Kuhnert added a comment -

Yeah sure....I'll create a unit test or two and attach them as a patch tomorrow. Guess I have no choice.

Show
Jesse Kuhnert added a comment - Yeah sure....I'll create a unit test or two and attach them as a patch tomorrow. Guess I have no choice.
Hide
Permalink
Jesse Kuhnert added a comment -

The fix now requires applying both patches attached to this issue. The surefire-patch.txt file needs to be applied to the maven/surefire/trunk code directly, while the other should be applied to the maven-surefire-plugin dirs.

All Junit/annotation/xml suites/etc functionality should be working now.

Show
Jesse Kuhnert added a comment - The fix now requires applying both patches attached to this issue. The surefire-patch.txt file needs to be applied to the maven/surefire/trunk code directly, while the other should be applied to the maven-surefire-plugin dirs. All Junit/annotation/xml suites/etc functionality should be working now.
Hide
Permalink
Jesse Kuhnert added a comment -

Jason,

Can you report back on this ticket if the changes work for you?

Show
Jesse Kuhnert added a comment - Jason, Can you report back on this ticket if the changes work for you?
Hide
Permalink
Jason Chaffee added a comment -

Jesse,

It looks pretty good. However, this is one issue with the testng.xml...it works if I specify classes, but it does not work if I specify packages. Everything else is working correctly though. Thanks!

Show
Jason Chaffee added a comment - Jesse, It looks pretty good. However, this is one issue with the testng.xml...it works if I specify classes, but it does not work if I specify packages. Everything else is working correctly though. Thanks!
Hide
Permalink
Jesse Kuhnert added a comment -

Hi Jason,

The package resolution issue looks like another classloading problem that will only be solved by a new testng release. This effectively makes the patches created so far all that is required to get surefire working for us lowly TestNG users again.

I've found/made the required change in TestNG to get the package stuff working in maven but am waiting for approval before committing and trying to do a new ibiblio release.

Thanks for hanging in there with us!

Show
Jesse Kuhnert added a comment - Hi Jason, The package resolution issue looks like another classloading problem that will only be solved by a new testng release. This effectively makes the patches created so far all that is required to get surefire working for us lowly TestNG users again. I've found/made the required change in TestNG to get the package stuff working in maven but am waiting for approval before committing and trying to do a new ibiblio release. Thanks for hanging in there with us!
Hide
Permalink
Jesse Kuhnert added a comment -

The required testng changes have been made to resolve this package issue. I don't know when they will be released, but if you just want to get it working you can check it out from cvs and build your own version.

Show
Jesse Kuhnert added a comment - The required testng changes have been made to resolve this package issue. I don't know when they will be released, but if you just want to get it working you can check it out from cvs and build your own version.
Hide
Permalink
Jason Chaffee added a comment -

Thanks so much Jesse, you are a life saver.

Show
Jason Chaffee added a comment - Thanks so much Jesse, you are a life saver.
Hide
Permalink
fabrizio giustina added a comment -

patches committed to surefire and maven-surefire-plugin, thanks!

Show
fabrizio giustina added a comment - patches committed to surefire and maven-surefire-plugin, thanks!
Hide
Permalink
Roger Pack added a comment -

http://jira.codehaus.org/browse/SUREFIRE-377 seems related

Show
Roger Pack added a comment - http://jira.codehaus.org/browse/SUREFIRE-377 seems related

People

Vote (10)
Watch (13)

Dates

  • Created:
    Updated:
    Resolved:
Atlassian JIRA (v4.4.3#663-r165197) | Report a problem
Powered by a free Atlassian JIRA open source license for Codehaus. Try JIRA - bug tracking software for your team.