JBehave
  1. JBehave
  2. JBEHAVE-394

Hudson Plugin to import JBehave reports

    Details

    • Type: New Feature New Feature
    • Status: Resolved Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.2
    • Component/s: Hudson Support
    • Labels:
      None
    • Environment:
      Hudsdon
    • Number of attachments :
      8

      Description

      As a developer I want to see a Hudson build test result showing failed scenarios so that I can see what broke my build

      As a developer I want scenarios containing pending steps and no failed steps to appear in build test result not as a fail or pass so that I can see what is missing from a story

      As a Manager I want to see individual scenarios in the build graph so that I can monitor progress of stories

      As a Manager I want to see Hudson build trend graph containing failed, pending and successful scenarios so that I can monitor the stability of my build

      1. org.jbehave.examples.trader.stories.claims_with_null_calendar.xml
        1 kB
        Seth Carter
      2. pom.xml
        3 kB
        Alan Parkinson
      3. TEST--778446379.xml
        0.4 kB
        Seth Carter
      1. eclipse-jbehave-runner.png
        110 kB
      2. hudson-jbehave-runner.png
        75 kB

        Activity

        Hide
        Alan Parkinson added a comment -

        The simplest way to create a test plug-in is take advantage of the xUnit plugin (http://wiki.hudson-ci.org/display/HUDSON/xUnit+Plugin). xUnit is used by several plugins (CppUnit, Gallio, Parasoft C++Test tool, JSUnit, AUnit, etc...) to provide test information to hudson.

        It achieves this by using XSLT to transforms native test XML test reports into JUnit test reports and then passes this to Hudsons existing JUnit support.

        The JUnit report features can map quite nicely onto the JBehave terms e.g.

        • Story -> Test suite
        • Scenario -> test case
        • Scenarios containing pending steps and no failed steps can marked as a JUnit ignored test case

        This would provide all the functionality required, except using JUnit terms in the Hudson UI instead of JBehave terms.

        This is quite simple and I already have some code working. I'm quite happy to carry working on the xUnit approach unless anyone think s a full Hudson plugin would be a better option?

        Show
        Alan Parkinson added a comment - The simplest way to create a test plug-in is take advantage of the xUnit plugin ( http://wiki.hudson-ci.org/display/HUDSON/xUnit+Plugin ). xUnit is used by several plugins (CppUnit, Gallio, Parasoft C++Test tool, JSUnit, AUnit, etc...) to provide test information to hudson. It achieves this by using XSLT to transforms native test XML test reports into JUnit test reports and then passes this to Hudsons existing JUnit support. The JUnit report features can map quite nicely onto the JBehave terms e.g. Story -> Test suite Scenario -> test case Scenarios containing pending steps and no failed steps can marked as a JUnit ignored test case This would provide all the functionality required, except using JUnit terms in the Hudson UI instead of JBehave terms. This is quite simple and I already have some code working. I'm quite happy to carry working on the xUnit approach unless anyone think s a full Hudson plugin would be a better option?
        Hide
        Ryszard Perkowski added a comment -

        Hi Alan,
        I think you should decide empirically.
        Give it a try with xUnit.
        I would gladly test your solution if you would expose it.

        Show
        Ryszard Perkowski added a comment - Hi Alan, I think you should decide empirically. Give it a try with xUnit. I would gladly test your solution if you would expose it.
        Hide
        Alan Parkinson added a comment -

        Thanks Ryszard, feedback will be most welcome.

        I've gone down the xUnit route and it seems to be working well. Attached is my latest version that I have been running on my production server for the last 24 hours.

        I think I have covered most of the common testing scenarios and it works well for my projects own stories but there might be the odd edge case I have missed.

        Show
        Alan Parkinson added a comment - Thanks Ryszard, feedback will be most welcome. I've gone down the xUnit route and it seems to be working well. Attached is my latest version that I have been running on my production server for the last 24 hours. I think I have covered most of the common testing scenarios and it works well for my projects own stories but there might be the odd edge case I have missed.
        Hide
        Seth Carter added a comment -

        First - this is great, thanks for the effort here - I was about to rip out the jbehave-maven plugin in favor of just using surefire but would rather not do that.

        I've built the attachment and loaded it to a Hudson 1.377 instance using xUnit 1.12

        To test it I'm building and running the tests from https://github.com/jbehave/jbehave-core.git

        I am getting the .xml files in target\jbehave but they don't appear to be converting properly. Even the ones that do not error don't seem to have the info Hudson would need to report on the testcase. (I'll try and attach an example)

        foot of Hudson console out is below:

        [xUnit] [INFO] - Starting to record.
        [xUnit] [INFO] - Processing JBehave-3.x
        [xUnit] [INFO] - [JBehave-3.x] - 17 test report file(s) were found with the pattern '**\trader\target\jbehave*.xml' relative to 'C:\.hudson\jobs\Test Job\workspace' for the testing framework 'JBehave-3.x'.
        [xUnit] [INFO] - Converting 'C:\.hudson\jobs\Test Job\workspace\examples\trader\target\jbehave\org.jbehave.examples.trader.stories.and_step.xml' .
        [xUnit] [INFO] - Converting 'C:\.hudson\jobs\Test Job\workspace\examples\trader\target\jbehave\org.jbehave.examples.trader.stories.claims_with_null_calendar.xml' .
        [xUnit] [INFO] - Converting 'C:\.hudson\jobs\Test Job\workspace\examples\trader\target\jbehave\org.jbehave.examples.trader.stories.examples_table_loaded_from_classpath.xml' .
        [xUnit] [INFO] - Converting 'C:\.hudson\jobs\Test Job\workspace\examples\trader\target\jbehave\org.jbehave.examples.trader.stories.failing_before_scenario.xml' .
        [xUnit] [INFO] - Converting 'C:\.hudson\jobs\Test Job\workspace\examples\trader\target\jbehave\org.jbehave.examples.trader.stories.failure_followed_by_given_stories.xml' .
        [xUnit] [ERROR] - The plugin hasn't been performed correctly: Conversion error Error to convert - A file not found
        Finished: FAILURE

        Show
        Seth Carter added a comment - First - this is great, thanks for the effort here - I was about to rip out the jbehave-maven plugin in favor of just using surefire but would rather not do that. I've built the attachment and loaded it to a Hudson 1.377 instance using xUnit 1.12 To test it I'm building and running the tests from https://github.com/jbehave/jbehave-core.git I am getting the .xml files in target\jbehave but they don't appear to be converting properly. Even the ones that do not error don't seem to have the info Hudson would need to report on the testcase. (I'll try and attach an example) foot of Hudson console out is below: [xUnit] [INFO] - Starting to record. [xUnit] [INFO] - Processing JBehave-3.x [xUnit] [INFO] - [JBehave-3.x] - 17 test report file(s) were found with the pattern '**\trader\target\jbehave*.xml' relative to 'C:\.hudson\jobs\Test Job\workspace' for the testing framework 'JBehave-3.x'. [xUnit] [INFO] - Converting 'C:\.hudson\jobs\Test Job\workspace\examples\trader\target\jbehave\org.jbehave.examples.trader.stories.and_step.xml' . [xUnit] [INFO] - Converting 'C:\.hudson\jobs\Test Job\workspace\examples\trader\target\jbehave\org.jbehave.examples.trader.stories.claims_with_null_calendar.xml' . [xUnit] [INFO] - Converting 'C:\.hudson\jobs\Test Job\workspace\examples\trader\target\jbehave\org.jbehave.examples.trader.stories.examples_table_loaded_from_classpath.xml' . [xUnit] [INFO] - Converting 'C:\.hudson\jobs\Test Job\workspace\examples\trader\target\jbehave\org.jbehave.examples.trader.stories.failing_before_scenario.xml' . [xUnit] [INFO] - Converting 'C:\.hudson\jobs\Test Job\workspace\examples\trader\target\jbehave\org.jbehave.examples.trader.stories.failure_followed_by_given_stories.xml' . [xUnit] [ERROR] - The plugin hasn't been performed correctly: Conversion error Error to convert - A file not found Finished: FAILURE
        Hide
        Seth Carter added a comment -

        org.jbehave.examples.trader.stories.claims_with_null_calendar.xml = jbehave xml output
        TEST--778446379.xml = JBehave-3.x plugin conversion

        Show
        Seth Carter added a comment - org.jbehave.examples.trader.stories.claims_with_null_calendar.xml = jbehave xml output TEST--778446379.xml = JBehave-3.x plugin conversion
        Hide
        Alan Parkinson added a comment -

        Thanks Seth

        I have recreated the problem and it is caused by the XML report from JBehave being invalid. In this case "failure_followed_by_given_stories.xml" is missing a closing story tag at the end of the file. I have reported this as JBEHAVE-401

        I have made an update to the plugin to fix a bug I have found and moved the code to github.
        https://github.com/aparkinson/jbehave-hudson-plugin/

        Show
        Alan Parkinson added a comment - Thanks Seth I have recreated the problem and it is caused by the XML report from JBehave being invalid. In this case "failure_followed_by_given_stories.xml" is missing a closing story tag at the end of the file. I have reported this as JBEHAVE-401 I have made an update to the plugin to fix a bug I have found and moved the code to github. https://github.com/aparkinson/jbehave-hudson-plugin/
        Hide
        Seth Carter added a comment -

        Excellent. Just my luck
        I'll get set up with the plugin from github and rerun.

        Show
        Seth Carter added a comment - Excellent. Just my luck I'll get set up with the plugin from github and rerun.
        Hide
        Mauro Talevi added a comment -

        Hi Alan, if you are happy to contribute the plugin the JBehave project we can include it 3.2 release.

        Show
        Mauro Talevi added a comment - Hi Alan, if you are happy to contribute the plugin the JBehave project we can include it 3.2 release.
        Hide
        Alan Parkinson added a comment -

        I'm happy to contribute the hudson plugin to the JBehave project

        Show
        Alan Parkinson added a comment - I'm happy to contribute the hudson plugin to the JBehave project
        Hide
        Mauro Talevi added a comment -

        Alan, I've been trying to incorporate the hudson plugin in the core build, but I've been running into issues. These stem from the fact that I'm trying to extend the jbehave parent rather than the hudson one.

        I'm attaching the version of the your contribution, with only the pom.xml changed. Could perhaps have a look and see if you spot the problem? It needs to be unzipped in the master of the jbehave core build. It's failing on the hpi plugin with error message "Unable to find a plugin class. Did you put @plugin in javadoc?".

        Show
        Mauro Talevi added a comment - Alan, I've been trying to incorporate the hudson plugin in the core build, but I've been running into issues. These stem from the fact that I'm trying to extend the jbehave parent rather than the hudson one. I'm attaching the version of the your contribution, with only the pom.xml changed. Could perhaps have a look and see if you spot the problem? It needs to be unzipped in the master of the jbehave core build. It's failing on the hpi plugin with error message "Unable to find a plugin class. Did you put @plugin in javadoc?".
        Hide
        Mauro Talevi added a comment -

        jbehave-hudson-plugin incorporated in maven core build.

        Show
        Mauro Talevi added a comment - jbehave-hudson-plugin incorporated in maven core build.
        Hide
        Alan Parkinson added a comment -

        Mauro, The problem was caused by the version of the maven-hpi-plugin and I have updated it to 1.51 in the attached POM.

        I have also changed the Hudson dependency version from the latest release (1.389) to 1.355 as this is the minimum Hudson version for the required xUnit plugin dependency and we don't take advantage any new features. This will also avoid forcing many Hudson users to upgrade their installations (Hudson is released once a week)

        I'm happy to help with any other problems

        Show
        Alan Parkinson added a comment - Mauro, The problem was caused by the version of the maven-hpi-plugin and I have updated it to 1.51 in the attached POM. I have also changed the Hudson dependency version from the latest release (1.389) to 1.355 as this is the minimum Hudson version for the required xUnit plugin dependency and we don't take advantage any new features. This will also avoid forcing many Hudson users to upgrade their installations (Hudson is released once a week) I'm happy to help with any other problems
        Hide
        Mauro Talevi added a comment -

        Thanks Alan, that fixed the problems. I've added to hudson plugin to core build.

        Show
        Mauro Talevi added a comment - Thanks Alan, that fixed the problems. I've added to hudson plugin to core build.
        Hide
        Daniel Godfrey added a comment -

        Any chance the 2 repository definitions can be removed from the hudson plugin POM? According to this article from Sonatype, http://www.sonatype.com/people/2010/03/why-external-repos-are-being-phased-out-of-central/, they are phasing out external repos from artifacts hosted on central, IE won't host them.

        I'm also not happy with maven now checking a repository on the internet I know nothing about, when I have a perfectly good local repository that is proxying central, etc.

        Show
        Daniel Godfrey added a comment - Any chance the 2 repository definitions can be removed from the hudson plugin POM? According to this article from Sonatype, http://www.sonatype.com/people/2010/03/why-external-repos-are-being-phased-out-of-central/ , they are phasing out external repos from artifacts hosted on central, IE won't host them. I'm also not happy with maven now checking a repository on the internet I know nothing about, when I have a perfectly good local repository that is proxying central, etc.
        Hide
        Mauro Talevi added a comment -

        Yes, I agree. We should provide a settings.xml with the list of repositories required to build. Then folks can build out of the box, while those using repository managers will need to create proxies appropriately.

        Show
        Mauro Talevi added a comment - Yes, I agree. We should provide a settings.xml with the list of repositories required to build. Then folks can build out of the box, while those using repository managers will need to create proxies appropriately.
        Hide
        Daniel Godfrey added a comment -

        Actually, wouldn't the hudson plugin be better off as a separate project rather than as a module within jbehave-core? This would allow the hudson plugin to put out a quick release if a newer version of hudson broke it, without impacting jbehave releases at all.

        Show
        Daniel Godfrey added a comment - Actually, wouldn't the hudson plugin be better off as a separate project rather than as a module within jbehave-core? This would allow the hudson plugin to put out a quick release if a newer version of hudson broke it, without impacting jbehave releases at all.
        Hide
        Mauro Talevi added a comment -

        If that happens we can quickly put out a bugfix release. There is always a balance to be struck when creating separate projects, and having one for such a small codebase is not recommended. Also, the hudson plugin is very tied to the core reporting functionality, so it does make sense to keep them in sync.

        Show
        Mauro Talevi added a comment - If that happens we can quickly put out a bugfix release. There is always a balance to be struck when creating separate projects, and having one for such a small codebase is not recommended. Also, the hudson plugin is very tied to the core reporting functionality, so it does make sense to keep them in sync.
        Hide
        Bart De Neuter added a comment -

        Is it necessary to have a maven plugin and a hudson plugin? Is it not possible to solve this with a JUnit Runner and just run a story as a junit test? This way you have nice output in your IDE and in Hudson with the surefire plugin. I made a small prototype. If you run the stories in eclipse you can see nicely the results of the stories. In hudson you also get the results from the scenarios as described above.

        Show
        Bart De Neuter added a comment - Is it necessary to have a maven plugin and a hudson plugin? Is it not possible to solve this with a JUnit Runner and just run a story as a junit test? This way you have nice output in your IDE and in Hudson with the surefire plugin. I made a small prototype. If you run the stories in eclipse you can see nicely the results of the stories. In hudson you also get the results from the scenarios as described above.
        Hide
        Alan Parkinson added a comment -

        Unfortunately I don't use JUnit for running my acceptance tests otherwise this would be one way of achieving the required functionality. Having a Hudson plug-in is a universal solution as it consumes the xml reports and doesn't care how the tests were executed.

        Show
        Alan Parkinson added a comment - Unfortunately I don't use JUnit for running my acceptance tests otherwise this would be one way of achieving the required functionality. Having a Hudson plug-in is a universal solution as it consumes the xml reports and doesn't care how the tests were executed.
        Hide
        Mauro Talevi added a comment -

        Alan, I've upgraded the dependency on the xUnit plugin to latest 1.13. Are you aware of any counter-indications to using latest version? It's unsavory that one needs to have the plugin already installed and the is no transitive dependency. If the xUnit plugin is not installed, the JBehave plugin fails without a meaningful error message in the UI (there is more detailed message in the log stacktrace). I've updated the description to make more visible and apparent the dependency requirement, but I want to provide better online documentation.

        Do you know how to configure in a Hudson plugin the clickable URL that appears on the plugins list? Currently defaults to http://jbehave.org/jbehave-hudson-plugin, but I'd like to be able to configure to something like http://jbehave.org/reference/stable/hudson-plugin.

        Show
        Mauro Talevi added a comment - Alan, I've upgraded the dependency on the xUnit plugin to latest 1.13. Are you aware of any counter-indications to using latest version? It's unsavory that one needs to have the plugin already installed and the is no transitive dependency. If the xUnit plugin is not installed, the JBehave plugin fails without a meaningful error message in the UI (there is more detailed message in the log stacktrace). I've updated the description to make more visible and apparent the dependency requirement, but I want to provide better online documentation. Do you know how to configure in a Hudson plugin the clickable URL that appears on the plugins list? Currently defaults to http://jbehave.org/jbehave-hudson-plugin , but I'd like to be able to configure to something like http://jbehave.org/reference/stable/hudson-plugin .
        Hide
        Mauro Talevi added a comment -

        Bart, you may choose to run JBehave stories as JUnit test. You have that option, but there are significant use cases for having multiple execution entry points and report integration views with different tools. As always, JBehave likes to support multiple ways of doing the same thing and letting users decide what best suits their needs.

        Show
        Mauro Talevi added a comment - Bart, you may choose to run JBehave stories as JUnit test. You have that option, but there are significant use cases for having multiple execution entry points and report integration views with different tools. As always, JBehave likes to support multiple ways of doing the same thing and letting users decide what best suits their needs.
        Hide
        Alan Parkinson added a comment -

        Mauro,

        I don't know of any problems with moving to the latest version of xUnit. I have been running the plug-in against xUnit 1.13 on a production instance for the whole of January.

        Could you explain how the plug-in failure appears to the user? From my understanding Hudson should disable the JBehave plug-in if the xUnit plug-in is not installed. FYI Hudson plug-in dependencies are calculated using the dependencies specified in the POM

        You can specify the URL for the plugin hyperlink in the Hudson update center by adding the "URL" element too the plug-in POM. see https://github.com/aparkinson/jbehave-core/commit/2342d0e4fafc14d5da33043c873c9a5d5d721ca7

        Is there anything else on the plug-in development I can assist with?

        Show
        Alan Parkinson added a comment - Mauro, I don't know of any problems with moving to the latest version of xUnit. I have been running the plug-in against xUnit 1.13 on a production instance for the whole of January. Could you explain how the plug-in failure appears to the user? From my understanding Hudson should disable the JBehave plug-in if the xUnit plug-in is not installed. FYI Hudson plug-in dependencies are calculated using the dependencies specified in the POM You can specify the URL for the plugin hyperlink in the Hudson update center by adding the "URL" element too the plug-in POM. see https://github.com/aparkinson/jbehave-core/commit/2342d0e4fafc14d5da33043c873c9a5d5d721ca7 Is there anything else on the plug-in development I can assist with?
        Hide
        Mauro Talevi added a comment -

        Hi Alan, thanks for the pointer on the URL, the installation instructions and dependencies are now in on the doc page.

        The issue is that if the xUnit dependency is not installed, Hudson does not install the JBehave plugin, without giving any significant reason on the UI.

        BTW, given that then name will be soon changed to Jenkins, would it be worth to adopt that from the word go in place of Hudson?

        Show
        Mauro Talevi added a comment - Hi Alan, thanks for the pointer on the URL, the installation instructions and dependencies are now in on the doc page. The issue is that if the xUnit dependency is not installed, Hudson does not install the JBehave plugin, without giving any significant reason on the UI. BTW, given that then name will be soon changed to Jenkins, would it be worth to adopt that from the word go in place of Hudson?
        Hide
        Alan Parkinson added a comment -

        Hi Mauro,

        The install failure on a missing dependency looks like the norm

        I was going to propose the name change when the new maven group and artifact ids are announced but I seem no harm in changing the name now. It could be a while before we have to change our dependencies to the Jenkins fork as we don't use any bleeding edge features.

        I have signed up to the developer mailing list to keep track of Jenkins related changes and I will be following the fork closely.

        Show
        Alan Parkinson added a comment - Hi Mauro, The install failure on a missing dependency looks like the norm I was going to propose the name change when the new maven group and artifact ids are announced but I seem no harm in changing the name now. It could be a while before we have to change our dependencies to the Jenkins fork as we don't use any bleeding edge features. I have signed up to the developer mailing list to keep track of Jenkins related changes and I will be following the fork closely.
        Hide
        Mauro Talevi added a comment -

        Yes, not the most user-friendly way to notify of a missing dependency.

        On the question of the name, I'm in two minds. But perhaps it's best (and easiest) to follow the trend rather than anticipate it. At the moment, we have a CI server call Hudson (of which, the releases will remain for future use) and for this we'll have the Hudson plugins. When the Jenkins fork will be released, we'll follow suite and release accordingly the renamed plugins. Being a simple rename, the two names will coexist for sometime anyway.

        Show
        Mauro Talevi added a comment - Yes, not the most user-friendly way to notify of a missing dependency. On the question of the name, I'm in two minds. But perhaps it's best (and easiest) to follow the trend rather than anticipate it. At the moment, we have a CI server call Hudson (of which, the releases will remain for future use) and for this we'll have the Hudson plugins. When the Jenkins fork will be released, we'll follow suite and release accordingly the renamed plugins. Being a simple rename, the two names will coexist for sometime anyway.
        Hide
        Alan Parkinson added a comment -

        Sounds like a good strategy.

        Is there anything outstanding for the first version of the plug-in?

        I think it would be good to use the Hudson/Jenkins update center to publish/install the plug-in in the long term. I think this should be under a separate ticket as there are infrastructure changes taking place on both projects.

        Show
        Alan Parkinson added a comment - Sounds like a good strategy. Is there anything outstanding for the first version of the plug-in? I think it would be good to use the Hudson/Jenkins update center to publish/install the plug-in in the long term. I think this should be under a separate ticket as there are infrastructure changes taking place on both projects.
        Hide
        Mauro Talevi added a comment -

        So, one outstanding thing to verify is how the user accesses the reports. Once the plugin is configured (e.g. to run the JBehave Core examples), I can see in the logs:

        {log}
        [xUnit] [INFO] - Processing JBehave-3.x
        [xUnit] [WARNING] - Can't create the path /Users/mauro/.hudson/jobs/jbehave/workspace/generatedJUnitFiles. Maybe the directory already exists.
        [xUnit] [INFO] - [JBehave-3.x] - 19 test report file(s) were found with the pattern '*/jbehave/org.jbehave.examples..xml' relative to '/Users/mauro/.hudson/jobs/jbehave/workspace' for the testing framework 'JBehave-3.x'.
        [xUnit] [INFO] - Converting '/Users/mauro/.hudson/jobs/jbehave/workspace/examples/trader/target/jbehave/org.jbehave.examples.trader.stories.after_scenario_outcome.xml' .{log}

        Other similar lines follow, but I can't see the test reports in the UI Test Results.

        Also, it shows

        http://localhost:8080/job/jbehave/3/testReport/junit/org_jbehave_examples_trader_stories_failing_before_scenario/story/We_want_to_show_that_failures_in__BeforeScenario_methods_will_not_prevent_rest_of_scenario_steps_to_run__marked_as_NOT_PERFORMED___Also__the__Before_AfterScenario_failure_messages_should_be_displayed_in_the_output_reports_/

        which is blank (showing error 404).

        I would just like to clean up these issues.

        I agree that publishing it to the Central Plugin repo is a good thing, which we'll do post release after the dust has settled on the infrastructure.

        Show
        Mauro Talevi added a comment - So, one outstanding thing to verify is how the user accesses the reports. Once the plugin is configured (e.g. to run the JBehave Core examples), I can see in the logs: {log} [xUnit] [INFO] - Processing JBehave-3.x [xUnit] [WARNING] - Can't create the path /Users/mauro/.hudson/jobs/jbehave/workspace/generatedJUnitFiles. Maybe the directory already exists. [xUnit] [INFO] - [JBehave-3.x] - 19 test report file(s) were found with the pattern '* /jbehave/org.jbehave.examples. .xml' relative to '/Users/mauro/.hudson/jobs/jbehave/workspace' for the testing framework 'JBehave-3.x'. [xUnit] [INFO] - Converting '/Users/mauro/.hudson/jobs/jbehave/workspace/examples/trader/target/jbehave/org.jbehave.examples.trader.stories.after_scenario_outcome.xml' .{log} Other similar lines follow, but I can't see the test reports in the UI Test Results. Also, it shows http://localhost:8080/job/jbehave/3/testReport/junit/org_jbehave_examples_trader_stories_failing_before_scenario/story/We_want_to_show_that_failures_in__BeforeScenario_methods_will_not_prevent_rest_of_scenario_steps_to_run__marked_as_NOT_PERFORMED___Also__the__Before_AfterScenario_failure_messages_should_be_displayed_in_the_output_reports_/ which is blank (showing error 404). I would just like to clean up these issues. I agree that publishing it to the Central Plugin repo is a good thing, which we'll do post release after the dust has settled on the infrastructure.
        Hide
        Alan Parkinson added a comment -

        I have managed to recreate the issues you have identified, it only occurs when a "maven job" aggregates test results from surefire/failsafe and jbehave executions. If you were to run the build as a "freestyle job" or don't generate junit reports in the "maven job" the aggregation works perfectly.

        I'm note sure if this a bug in the JBehave plugin, xunit and or the Hudson maven plugin but I will continue to investigate.

        Show
        Alan Parkinson added a comment - I have managed to recreate the issues you have identified, it only occurs when a "maven job" aggregates test results from surefire/failsafe and jbehave executions. If you were to run the build as a "freestyle job" or don't generate junit reports in the "maven job" the aggregation works perfectly. I'm note sure if this a bug in the JBehave plugin, xunit and or the Hudson maven plugin but I will continue to investigate.
        Hide
        Paul Hammant added a comment -

        I'm able to see the problem with the plugin configured yesterday. Its the one from the aparksinson repo, but it is not the problem for us, JBehave is. Or its invalid XML output.

        Here's one that I guess loaded just fine into the hudson plugin:

        <story path="foo.story" title="">
        <meta>
        <property keyword="@" name="channel" value="aaa"/>
        </meta>
        <scenario keyword="Scenario:" title="ttt">
        <step outcome="failed" keyword="FAILED">Given I am a human with a booked fff<failure>blort</failure></step>
        <step outcome="notPerformed" keyword="NOT PERFORMED">When I cancel the fff</step>
        <step outcome="notPerformed" keyword="NOT PERFORMED">Then I receive an fff is now canceled</step>
        </scenario>
        </story>

        Here is one that failed:

        <story path="bar.story" title="">
        <meta>
        <property keyword="@" name="channel" value="aaa"/>
        </meta>
        <filter>+channel aaa -draft</filter>

        Note there was no close of <story>, nor was there a <scenario> open or close.
        It is like the stream was closed before it was flushed. Incidentally the second story file (the one that breaks in XML form) has a meta tag that has no value: @draft. This works fine from the command line of course - and should cause the story to be ignored for the Hudson run in question.

        Show
        Paul Hammant added a comment - I'm able to see the problem with the plugin configured yesterday. Its the one from the aparksinson repo, but it is not the problem for us, JBehave is. Or its invalid XML output. Here's one that I guess loaded just fine into the hudson plugin: <story path="foo.story" title=""> <meta> <property keyword="@" name="channel" value="aaa"/> </meta> <scenario keyword="Scenario:" title="ttt"> <step outcome="failed" keyword="FAILED">Given I am a human with a booked fff<failure>blort</failure></step> <step outcome="notPerformed" keyword="NOT PERFORMED">When I cancel the fff</step> <step outcome="notPerformed" keyword="NOT PERFORMED">Then I receive an fff is now canceled</step> </scenario> </story> Here is one that failed: <story path="bar.story" title=""> <meta> <property keyword="@" name="channel" value="aaa"/> </meta> <filter>+channel aaa -draft</filter> Note there was no close of <story>, nor was there a <scenario> open or close. It is like the stream was closed before it was flushed. Incidentally the second story file (the one that breaks in XML form) has a meta tag that has no value: @draft. This works fine from the command line of course - and should cause the story to be ignored for the Hudson run in question.
        Hide
        Paul Hammant added a comment -

        The @draft idea is a red herring. I moved that tag up the list of tags for the real (non obfuscated) story in question, an it was still corrupt XML in the way listed above.

        Show
        Paul Hammant added a comment - The @draft idea is a red herring. I moved that tag up the list of tags for the real (non obfuscated) story in question, an it was still corrupt XML in the way listed above.
        Hide
        Paul Hammant added a comment -

        https://github.com/jbehave/jbehave-core/commit/4952568886319b53b7314513ecf2eb5022524f2f is my attempt to fix the issue. Note I did not find a pre-existing .close() for the print stream in question.

        At this stage though the issue (for me) appears in the hudson-plugin phase, I think it is a core JBehave issue, that nobody has noticed before.

        Show
        Paul Hammant added a comment - https://github.com/jbehave/jbehave-core/commit/4952568886319b53b7314513ecf2eb5022524f2f is my attempt to fix the issue. Note I did not find a pre-existing .close() for the print stream in question. At this stage though the issue (for me) appears in the hudson-plugin phase, I think it is a core JBehave issue, that nobody has noticed before.
        Hide
        Mauro Talevi added a comment -

        Hi Alan,

        I'm suspecting the problem may lie in the presence of the <filter> element which may not agree with the Hudson XSL.

        I noticed the unit test cases do not contemplate the presence of filters. Could you try to enhance the test (and probably the XSL) to allow filters?

        Thanks

        Show
        Mauro Talevi added a comment - Hi Alan, I'm suspecting the problem may lie in the presence of the <filter> element which may not agree with the Hudson XSL. I noticed the unit test cases do not contemplate the presence of filters. Could you try to enhance the test (and probably the XSL) to allow filters? Thanks
        Hide
        Paul Hammant added a comment -

        I think it is filter related too. However, I can see corrupted XML output from a regular command line build (using filters) without Hudson in the mix. Even after my commit above.

        Show
        Paul Hammant added a comment - I think it is filter related too. However, I can see corrupted XML output from a regular command line build (using filters) without Hudson in the mix. Even after my commit above.
        Hide
        Alan Parkinson added a comment -

        I'm also seeing corrupted XML files being read by the Hudson plugin before any processing occurs. When reading a file xUnit will validate the XML file and then print "[xUnit] [ERROR] - The plugin hasn't been performed correctly: Conversion error Error to convert - A file not found" in the hudson build console if the file is not valid XML.

        To rule out filters causing problem I will add some unit tests but I don't think this will be a problem because the XSL is only looking for elements that decide a test success, fail or skip.

        I can now confirm Hudson "Maven2/3" jobs that generate surefire reports are the cause of missing jbehave reports in the test UI. The other hudson jobs type run ok) and you can run the examples (without XML corruptions) using maven within a "freestyle" hudson job and the tests will appear in the UI correctly.

        There are related two causes for this problem:
        1) Hudson can only process one test report in the UI. Mauro, you might haven seen two reports in the UI showing the same results and one having the links to failured test giving a 404.
        2) xUnit aggregates test results from other tools to avoid duplicating test reports. It doesn't do this with the Maven2/3 job as it overrides Hudsons testing model and adds very Maven specific code. I have been through the xUnit and Hudson code and I haven't found away of injecting or aggregating the test results in a "Maven2/3" job to avoid issue 1.

        The only way I can see of supporting the "Maven2/3" job within the Hudson plugin is to write some Hudson Maven specific code which could be complicated. My suggestion is to document the limitation of job type in the Hudson plugin and develop a solution for a future release.

        Show
        Alan Parkinson added a comment - I'm also seeing corrupted XML files being read by the Hudson plugin before any processing occurs. When reading a file xUnit will validate the XML file and then print " [xUnit] [ERROR] - The plugin hasn't been performed correctly: Conversion error Error to convert - A file not found" in the hudson build console if the file is not valid XML. To rule out filters causing problem I will add some unit tests but I don't think this will be a problem because the XSL is only looking for elements that decide a test success, fail or skip. I can now confirm Hudson "Maven2/3" jobs that generate surefire reports are the cause of missing jbehave reports in the test UI. The other hudson jobs type run ok) and you can run the examples (without XML corruptions) using maven within a "freestyle" hudson job and the tests will appear in the UI correctly. There are related two causes for this problem: 1) Hudson can only process one test report in the UI. Mauro, you might haven seen two reports in the UI showing the same results and one having the links to failured test giving a 404. 2) xUnit aggregates test results from other tools to avoid duplicating test reports. It doesn't do this with the Maven2/3 job as it overrides Hudsons testing model and adds very Maven specific code. I have been through the xUnit and Hudson code and I haven't found away of injecting or aggregating the test results in a "Maven2/3" job to avoid issue 1. The only way I can see of supporting the "Maven2/3" job within the Hudson plugin is to write some Hudson Maven specific code which could be complicated. My suggestion is to document the limitation of job type in the Hudson plugin and develop a solution for a future release.
        Hide
        Paul Hammant added a comment -

        Folks. Refer to JBEHAVE-420 (nothing to do with Hudson)

        I may have fixed the broken XML issue. I am not sure if JBEHAVE-420 relates to JBEHAVE-394 or not.

        Show
        Paul Hammant added a comment - Folks. Refer to JBEHAVE-420 (nothing to do with Hudson) I may have fixed the broken XML issue. I am not sure if JBEHAVE-420 relates to JBEHAVE-394 or not.
        Show
        Alan Parkinson added a comment - Proposed documentation change: https://github.com/aparkinson/jbehave-core/commit/d5bc77a06743ba91ed598efe7bfce19b677ba79b
        Hide
        Paul Hammant added a comment -

        I can confirm that the fix for JBEHAVE-420 now allows for the XML to be consumed into Hudson results. At least it does for me in respect of my stories/scenarios.

        Show
        Paul Hammant added a comment - I can confirm that the fix for JBEHAVE-420 now allows for the XML to be consumed into Hudson results. At least it does for me in respect of my stories/scenarios.
        Hide
        Alan Parkinson added a comment -

        I've added some extra testing for filters and some small XSL changes
        https://github.com/aparkinson/jbehave-core/commit/fc6b0fa9fae5d43ca2e0dd88b50a3086e99fa858

        Show
        Alan Parkinson added a comment - I've added some extra testing for filters and some small XSL changes https://github.com/aparkinson/jbehave-core/commit/fc6b0fa9fae5d43ca2e0dd88b50a3086e99fa858

          People

          • Assignee:
            Mauro Talevi
            Reporter:
            Alan Parkinson
          • Votes:
            2 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: