Hudson Plugin to import JBehave reports

The simplest way to create a test plug-in is take advantage of the xUnit plugin (http://wiki.hudson-ci.org/display/HUDSON/xUnit+Plugin). xUnit is used by several plugins (CppUnit, Gallio, Parasoft C++Test tool, JSUnit, AUnit, etc...) to provide test information to hudson.

It achieves this by using XSLT to transforms native test XML test reports into JUnit test reports and then passes this to Hudsons existing JUnit support.

The JUnit report features can map quite nicely onto the JBehave terms e.g.

• Story -> Test suite
• Scenario -> test case
• Scenarios containing pending steps and no failed steps can marked as a JUnit ignored test case

This would provide all the functionality required, except using JUnit terms in the Hudson UI instead of JBehave terms.

This is quite simple and I already have some code working. I'm quite happy to carry working on the xUnit approach unless anyone think s a full Hudson plugin would be a better option?

Hi Alan,
I think you should decide empirically.
Give it a try with xUnit.
I would gladly test your solution if you would expose it.

Thanks Ryszard, feedback will be most welcome.

I've gone down the xUnit route and it seems to be working well. Attached is my latest version that I have been running on my production server for the last 24 hours.

I think I have covered most of the common testing scenarios and it works well for my projects own stories but there might be the odd edge case I have missed.

First - this is great, thanks for the effort here - I was about to rip out the jbehave-maven plugin in favor of just using surefire but would rather not do that.

I've built the attachment and loaded it to a Hudson 1.377 instance using xUnit 1.12

To test it I'm building and running the tests from https://github.com/jbehave/jbehave-core.git

I am getting the .xml files in target\jbehave but they don't appear to be converting properly. Even the ones that do not error don't seem to have the info Hudson would need to report on the testcase. (I'll try and attach an example)

foot of Hudson console out is below:

[xUnit] [INFO] - Starting to record.
[xUnit] [INFO] - Processing JBehave-3.x
[xUnit] [INFO] - [JBehave-3.x] - 17 test report file(s) were found with the pattern '**\trader\target\jbehave*.xml' relative to 'C:\.hudson\jobs\Test Job\workspace' for the testing framework 'JBehave-3.x'.
[xUnit] [INFO] - Converting 'C:\.hudson\jobs\Test Job\workspace\examples\trader\target\jbehave\org.jbehave.examples.trader.stories.and_step.xml' .
[xUnit] [INFO] - Converting 'C:\.hudson\jobs\Test Job\workspace\examples\trader\target\jbehave\org.jbehave.examples.trader.stories.claims_with_null_calendar.xml' .
[xUnit] [INFO] - Converting 'C:\.hudson\jobs\Test Job\workspace\examples\trader\target\jbehave\org.jbehave.examples.trader.stories.examples_table_loaded_from_classpath.xml' .
[xUnit] [INFO] - Converting 'C:\.hudson\jobs\Test Job\workspace\examples\trader\target\jbehave\org.jbehave.examples.trader.stories.failing_before_scenario.xml' .
[xUnit] [INFO] - Converting 'C:\.hudson\jobs\Test Job\workspace\examples\trader\target\jbehave\org.jbehave.examples.trader.stories.failure_followed_by_given_stories.xml' .
[xUnit] [ERROR] - The plugin hasn't been performed correctly: Conversion error Error to convert - A file not found
Finished: FAILURE

org.jbehave.examples.trader.stories.claims_with_null_calendar.xml = jbehave xml output
TEST--778446379.xml = JBehave-3.x plugin conversion

Thanks Seth

I have recreated the problem and it is caused by the XML report from JBehave being invalid. In this case "failure_followed_by_given_stories.xml" is missing a closing story tag at the end of the file. I have reported this as JBEHAVE-401

I have made an update to the plugin to fix a bug I have found and moved the code to github.
https://github.com/aparkinson/jbehave-hudson-plugin/

Excellent. Just my luck
I'll get set up with the plugin from github and rerun.

Hi Alan, if you are happy to contribute the plugin the JBehave project we can include it 3.2 release.

I'm happy to contribute the hudson plugin to the JBehave project

Alan, I've been trying to incorporate the hudson plugin in the core build, but I've been running into issues. These stem from the fact that I'm trying to extend the jbehave parent rather than the hudson one.

I'm attaching the version of the your contribution, with only the pom.xml changed. Could perhaps have a look and see if you spot the problem? It needs to be unzipped in the master of the jbehave core build. It's failing on the hpi plugin with error message "Unable to find a plugin class. Did you put @plugin in javadoc?".

jbehave-hudson-plugin incorporated in maven core build.

Mauro, The problem was caused by the version of the maven-hpi-plugin and I have updated it to 1.51 in the attached POM.

I have also changed the Hudson dependency version from the latest release (1.389) to 1.355 as this is the minimum Hudson version for the required xUnit plugin dependency and we don't take advantage any new features. This will also avoid forcing many Hudson users to upgrade their installations (Hudson is released once a week)

I'm happy to help with any other problems

Thanks Alan, that fixed the problems. I've added to hudson plugin to core build.

Any chance the 2 repository definitions can be removed from the hudson plugin POM? According to this article from Sonatype, http://www.sonatype.com/people/2010/03/why-external-repos-are-being-phased-out-of-central/, they are phasing out external repos from artifacts hosted on central, IE won't host them.

I'm also not happy with maven now checking a repository on the internet I know nothing about, when I have a perfectly good local repository that is proxying central, etc.

Yes, I agree. We should provide a settings.xml with the list of repositories required to build. Then folks can build out of the box, while those using repository managers will need to create proxies appropriately.

Actually, wouldn't the hudson plugin be better off as a separate project rather than as a module within jbehave-core? This would allow the hudson plugin to put out a quick release if a newer version of hudson broke it, without impacting jbehave releases at all.

If that happens we can quickly put out a bugfix release. There is always a balance to be struck when creating separate projects, and having one for such a small codebase is not recommended. Also, the hudson plugin is very tied to the core reporting functionality, so it does make sense to keep them in sync.

Is it necessary to have a maven plugin and a hudson plugin? Is it not possible to solve this with a JUnit Runner and just run a story as a junit test? This way you have nice output in your IDE and in Hudson with the surefire plugin. I made a small prototype. If you run the stories in eclipse you can see nicely the results of the stories. In hudson you also get the results from the scenarios as described above.

Alan, I've upgraded the dependency on the xUnit plugin to latest 1.13. Are you aware of any counter-indications to using latest version? It's unsavory that one needs to have the plugin already installed and the is no transitive dependency. If the xUnit plugin is not installed, the JBehave plugin fails without a meaningful error message in the UI (there is more detailed message in the log stacktrace). I've updated the description to make more visible and apparent the dependency requirement, but I want to provide better online documentation.

Do you know how to configure in a Hudson plugin the clickable URL that appears on the plugins list? Currently defaults to http://jbehave.org/jbehave-hudson-plugin, but I'd like to be able to configure to something like http://jbehave.org/reference/stable/hudson-plugin.

Bart, you may choose to run JBehave stories as JUnit test. You have that option, but there are significant use cases for having multiple execution entry points and report integration views with different tools. As always, JBehave likes to support multiple ways of doing the same thing and letting users decide what best suits their needs.

Mauro,

I don't know of any problems with moving to the latest version of xUnit. I have been running the plug-in against xUnit 1.13 on a production instance for the whole of January.

Could you explain how the plug-in failure appears to the user? From my understanding Hudson should disable the JBehave plug-in if the xUnit plug-in is not installed. FYI Hudson plug-in dependencies are calculated using the dependencies specified in the POM

You can specify the URL for the plugin hyperlink in the Hudson update center by adding the "URL" element too the plug-in POM. see https://github.com/aparkinson/jbehave-core/commit/2342d0e4fafc14d5da33043c873c9a5d5d721ca7

Is there anything else on the plug-in development I can assist with?

Hi Alan, thanks for the pointer on the URL, the installation instructions and dependencies are now in on the doc page.

The issue is that if the xUnit dependency is not installed, Hudson does not install the JBehave plugin, without giving any significant reason on the UI.

BTW, given that then name will be soon changed to Jenkins, would it be worth to adopt that from the word go in place of Hudson?

Hi Mauro,

The install failure on a missing dependency looks like the norm

I was going to propose the name change when the new maven group and artifact ids are announced but I seem no harm in changing the name now. It could be a while before we have to change our dependencies to the Jenkins fork as we don't use any bleeding edge features.

I have signed up to the developer mailing list to keep track of Jenkins related changes and I will be following the fork closely.

Yes, not the most user-friendly way to notify of a missing dependency.

On the question of the name, I'm in two minds. But perhaps it's best (and easiest) to follow the trend rather than anticipate it. At the moment, we have a CI server call Hudson (of which, the releases will remain for future use) and for this we'll have the Hudson plugins. When the Jenkins fork will be released, we'll follow suite and release accordingly the renamed plugins. Being a simple rename, the two names will coexist for sometime anyway.

Sounds like a good strategy.

Is there anything outstanding for the first version of the plug-in?

I think it would be good to use the Hudson/Jenkins update center to publish/install the plug-in in the long term. I think this should be under a separate ticket as there are infrastructure changes taking place on both projects.

So, one outstanding thing to verify is how the user accesses the reports. Once the plugin is configured (e.g. to run the JBehave Core examples), I can see in the logs:

Other similar lines follow, but I can't see the test reports in the UI Test Results.

Also, it shows

http://localhost:8080/job/jbehave/3/testReport/junit/org_jbehave_examples_trader_stories_failing_before_scenario/story/We_want_to_show_that_failures_in__BeforeScenario_methods_will_not_prevent_rest_of_scenario_steps_to_run__marked_as_NOT_PERFORMED___Also__the__Before_AfterScenario_failure_messages_should_be_displayed_in_the_output_reports_/

which is blank (showing error 404).

I would just like to clean up these issues.

I agree that publishing it to the Central Plugin repo is a good thing, which we'll do post release after the dust has settled on the infrastructure.

I have managed to recreate the issues you have identified, it only occurs when a "maven job" aggregates test results from surefire/failsafe and jbehave executions. If you were to run the build as a "freestyle job" or don't generate junit reports in the "maven job" the aggregation works perfectly.

I'm note sure if this a bug in the JBehave plugin, xunit and or the Hudson maven plugin but I will continue to investigate.

I'm able to see the problem with the plugin configured yesterday. Its the one from the aparksinson repo, but it is not the problem for us, JBehave is. Or its invalid XML output.

Here's one that I guess loaded just fine into the hudson plugin:

<story path="foo.story" title="">
<meta>
<property keyword="@" name="channel" value="aaa"/>
</meta>
<scenario keyword="Scenario:" title="ttt">
<step outcome="failed" keyword="FAILED">Given I am a human with a booked fff<failure>blort</failure></step>
<step outcome="notPerformed" keyword="NOT PERFORMED">When I cancel the fff</step>
<step outcome="notPerformed" keyword="NOT PERFORMED">Then I receive an fff is now canceled</step>
</scenario>
</story>

Here is one that failed:

<story path="bar.story" title="">
<meta>
<property keyword="@" name="channel" value="aaa"/>
</meta>
<filter>+channel aaa -draft</filter>

Note there was no close of <story>, nor was there a <scenario> open or close.
It is like the stream was closed before it was flushed. Incidentally the second story file (the one that breaks in XML form) has a meta tag that has no value: @draft. This works fine from the command line of course - and should cause the story to be ignored for the Hudson run in question.

The @draft idea is a red herring. I moved that tag up the list of tags for the real (non obfuscated) story in question, an it was still corrupt XML in the way listed above.

https://github.com/jbehave/jbehave-core/commit/4952568886319b53b7314513ecf2eb5022524f2f is my attempt to fix the issue. Note I did not find a pre-existing .close() for the print stream in question.

At this stage though the issue (for me) appears in the hudson-plugin phase, I think it is a core JBehave issue, that nobody has noticed before.

Hi Alan,

I'm suspecting the problem may lie in the presence of the <filter> element which may not agree with the Hudson XSL.

I noticed the unit test cases do not contemplate the presence of filters. Could you try to enhance the test (and probably the XSL) to allow filters?

Thanks

I think it is filter related too. However, I can see corrupted XML output from a regular command line build (using filters) without Hudson in the mix. Even after my commit above.

I'm also seeing corrupted XML files being read by the Hudson plugin before any processing occurs. When reading a file xUnit will validate the XML file and then print "[xUnit] [ERROR] - The plugin hasn't been performed correctly: Conversion error Error to convert - A file not found" in the hudson build console if the file is not valid XML.

To rule out filters causing problem I will add some unit tests but I don't think this will be a problem because the XSL is only looking for elements that decide a test success, fail or skip.

I can now confirm Hudson "Maven2/3" jobs that generate surefire reports are the cause of missing jbehave reports in the test UI. The other hudson jobs type run ok) and you can run the examples (without XML corruptions) using maven within a "freestyle" hudson job and the tests will appear in the UI correctly.

There are related two causes for this problem:
1) Hudson can only process one test report in the UI. Mauro, you might haven seen two reports in the UI showing the same results and one having the links to failured test giving a 404.
2) xUnit aggregates test results from other tools to avoid duplicating test reports. It doesn't do this with the Maven2/3 job as it overrides Hudsons testing model and adds very Maven specific code. I have been through the xUnit and Hudson code and I haven't found away of injecting or aggregating the test results in a "Maven2/3" job to avoid issue 1.

The only way I can see of supporting the "Maven2/3" job within the Hudson plugin is to write some Hudson Maven specific code which could be complicated. My suggestion is to document the limitation of job type in the Hudson plugin and develop a solution for a future release.

Folks. Refer to JBEHAVE-420 (nothing to do with Hudson)

I may have fixed the broken XML issue. I am not sure if JBEHAVE-420 relates to JBEHAVE-394 or not.

Proposed documentation change: https://github.com/aparkinson/jbehave-core/commit/d5bc77a06743ba91ed598efe7bfce19b677ba79b

I can confirm that the fix for JBEHAVE-420 now allows for the XML to be consumed into Hudson results. At least it does for me in respect of my stories/scenarios.

I've added some extra testing for filters and some small XSL changes
https://github.com/aparkinson/jbehave-core/commit/fc6b0fa9fae5d43ca2e0dd88b50a3086e99fa858