JBehave
  1. JBehave
  2. JBEHAVE-551

Better integration with JUnit (Separate results for each story/scenario/example)

    Details

    • Type: Improvement Improvement
    • Status: Open Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 3.4.5
    • Fix Version/s: 3.x
    • Component/s: Core
    • Labels:
      None
    • Number of attachments :
      3

      Description

      I would like each example in each scenario in each story to show up as distinct result in the JUnit test results within my IDE and the JUnit reports. This has the advantage that assertion failures are linked to example failures and can be clicked on in the IDE to jump to source code of the offending failure. Currently I get a single failure in my IDE and have to look through log files or look at the JBehave report.

      The current AnnotatedPathRunner implementation extends org.junit.runners.BlockJUnit4ClassRunner and therefore uses the methods with a @Test annotation to drive the JUnit results. In most jbehave tests, there is only one such method which means that there is only a single result in the JUnit reports (IDE and HTML).

      I have started to implement a new org.junit.runner.Runner implementation which extends org.junit.runners.ParentRunner and can use the stories as the basis for generating the JUnit test results rather than the Test case itself.

      I have attached a screenshot and source code for running a single test (jbehave.ListTest) in Eclipse. The test case runs a single story (list.story). You can see from the results that there is a hierarchy in the JUnit results for story, scenario and example.

      1. eclipse-screenshot.jpg
        235 kB

        Activity

        Hide
        Lance added a comment -

        I have updated the code which includes the following:
        Maven pom
        JBehaveTestRunner - runs JBehaveTests
        SpringJBehaveTestRunner - runs JBehave / spring tests (supports spring dependency injection and TestExecutionListeners)
        ListTest - simple jbehave test
        MapSpringTest - spring test demonstrating @DirtiesContext
        MultiStoryTest - example running 2 stories

        Show
        Lance added a comment - I have updated the code which includes the following: Maven pom JBehaveTestRunner - runs JBehaveTests SpringJBehaveTestRunner - runs JBehave / spring tests (supports spring dependency injection and TestExecutionListeners) ListTest - simple jbehave test MapSpringTest - spring test demonstrating @DirtiesContext MultiStoryTest - example running 2 stories
        Hide
        Lance added a comment -

        I feel that my JBehaveRunner implementation has more similar behaviour to standard JUnit testing because:
        1. You get a new instance of the test case (and the steps) for every test execution.
        2. Methods annotated with @Before (org.junit.Before) will run before every test execution
        3. Methods annotated with @After (org.junit.After) will run after every test execution

        My current JBehaveTestRunner implementation supports all of the JBehave annotations and also supports two new annotations (@BeforeExample and @AfterExample)

        There are a couple of issues with these annotations with my JBehaveRunner implementation
        4. A test instance is required to retrieve the first steps collection
        5. A test instance is required for each story (@BeforeStory and @AfterStory)
        6. A test instance is required for each scenario (@BeforeScenario and @AfterScenario [and @Steps, @Before and @After if there are no examples])
        7. A test instance is required for each example (@BeforeExample, @AfterExample, @Steps, @Before, @After)

        Currently 4, 5, 6 and 7 generate a separate test instance (which is shared between the annotations mentioned).

        Suggestions:
        8. Any methods annotated with @BeforeStory and @AfterStorymust be static (similar to @BeforeClass and @AfterClass) therefore no test instance required.
        9. For scenarios with no examples, any methods annotated with @BeforeScenario and @AfterScenario must be static.
        10. Or we could get rid of all of the JBehave annotations and assume that @BeforeClass, @AfterClass and @Before and @After are sufficient

        Show
        Lance added a comment - I feel that my JBehaveRunner implementation has more similar behaviour to standard JUnit testing because: 1. You get a new instance of the test case (and the steps) for every test execution. 2. Methods annotated with @Before (org.junit.Before) will run before every test execution 3. Methods annotated with @After (org.junit.After) will run after every test execution My current JBehaveTestRunner implementation supports all of the JBehave annotations and also supports two new annotations (@BeforeExample and @AfterExample) There are a couple of issues with these annotations with my JBehaveRunner implementation 4. A test instance is required to retrieve the first steps collection 5. A test instance is required for each story (@BeforeStory and @AfterStory) 6. A test instance is required for each scenario (@BeforeScenario and @AfterScenario [and @Steps, @Before and @After if there are no examples] ) 7. A test instance is required for each example (@BeforeExample, @AfterExample, @Steps, @Before, @After) Currently 4, 5, 6 and 7 generate a separate test instance (which is shared between the annotations mentioned). Suggestions: 8. Any methods annotated with @BeforeStory and @AfterStorymust be static (similar to @BeforeClass and @AfterClass) therefore no test instance required. 9. For scenarios with no examples, any methods annotated with @BeforeScenario and @AfterScenario must be static. 10. Or we could get rid of all of the JBehave annotations and assume that @BeforeClass, @AfterClass and @Before and @After are sufficient
        Hide
        kevin tung added a comment -

        any idea on when this could possibly make it into a release?

        Show
        kevin tung added a comment - any idea on when this could possibly make it into a release?
        Hide
        Andreas Ebbert-Karroum added a comment -

        Hi,

        we released an open-source-project jbehave-junit-runner that does exactly that.

        Until JBehave supports it natively, You can use that.

        Show
        Andreas Ebbert-Karroum added a comment - Hi, we released an open-source-project jbehave-junit-runner that does exactly that. Until JBehave supports it natively, You can use that.
        Hide
        Sergey added a comment -

        Is anyone aware of such integration for TestNG?

        Show
        Sergey added a comment - Is anyone aware of such integration for TestNG?
        Hide
        Joachim Nilsson added a comment -

        jbehave-junit-runner is really awesome addon to JBehave! Please merge into JBehave if possible! And then add support to click on a JUnit result to get to the story file.

        Show
        Joachim Nilsson added a comment - jbehave-junit-runner is really awesome addon to JBehave! Please merge into JBehave if possible! And then add support to click on a JUnit result to get to the story file.
        Hide
        Joachim Nilsson added a comment -

        I have been experimenting with data driven tests in JBehave, TestNG integration would solve that. So that is a vote from me.

        Show
        Joachim Nilsson added a comment - I have been experimenting with data driven tests in JBehave, TestNG integration would solve that. So that is a vote from me.
        Hide
        Andreas Ebbert-Karroum added a comment -

        Joachim, thanks for Your support of jbehave-junit-runner. I'd love to implement the jump to the story file, when you click on the JUnit result, but to the best of my knowledge this is not possible. This limitation is a combination of JUnit and usual IDE integrations, like in Eclipse. In JUnit you can only remember, which method and class a test belongs to. The IDE integration then allows you to jump to that method in a class. If you have any idea, how to circumvent these challenges, I welcome your ideas

        Show
        Andreas Ebbert-Karroum added a comment - Joachim, thanks for Your support of jbehave-junit-runner. I'd love to implement the jump to the story file, when you click on the JUnit result, but to the best of my knowledge this is not possible. This limitation is a combination of JUnit and usual IDE integrations, like in Eclipse. In JUnit you can only remember, which method and class a test belongs to. The IDE integration then allows you to jump to that method in a class. If you have any idea, how to circumvent these challenges, I welcome your ideas
        Hide
        Sebastien Tardif added a comment -

        The same for TestNG would be useful.

        Show
        Sebastien Tardif added a comment - The same for TestNG would be useful.
        Hide
        Simon Zambrovski added a comment -

        Any progress on this? With API changes in 3.9 I had to create the patch for that change, but essentially it should be a part of JBehave, not an Add-On project. I would implement the merge on JBehave site and push it...

        Show
        Simon Zambrovski added a comment - Any progress on this? With API changes in 3.9 I had to create the patch for that change, but essentially it should be a part of JBehave, not an Add-On project. I would implement the merge on JBehave site and push it...
        Hide
        Simon Zambrovski added a comment - - edited

        Pull request added: https://github.com/jbehave/jbehave-core/pull/58.

        I skipped the additional methods for @Before and @After annotated stuff. Just signal if you think it is a good idea to add those...

        Show
        Simon Zambrovski added a comment - - edited Pull request added: https://github.com/jbehave/jbehave-core/pull/58 . I skipped the additional methods for @Before and @After annotated stuff. Just signal if you think it is a good idea to add those...
        Hide
        Mauro Talevi added a comment -

        Hi Simon,

        thanks for the contribution. It looks like a pretty good start. Things that I would add:

        • failed stories/scenarios are not apparent in the results layout. This is a must IMO.
        • colour-coding, say in red for failures and yellow for pending, or other equivalent decoration. Nice to have.

        Can you also tell me if you've tested it with jbehave-annotations-example CoreAnnotatedPathRunner so we can verify the same behaviour.

        Cheers

        Show
        Mauro Talevi added a comment - Hi Simon, thanks for the contribution. It looks like a pretty good start. Things that I would add: failed stories/scenarios are not apparent in the results layout. This is a must IMO. colour-coding, say in red for failures and yellow for pending, or other equivalent decoration. Nice to have. Can you also tell me if you've tested it with jbehave-annotations-example CoreAnnotatedPathRunner so we can verify the same behaviour. Cheers
        Hide
        Mauro Talevi added a comment -

        PS: Please remember to put the JIRA issue in your commits:

        http://jbehave.org/reference/stable/how-to-contribute.html

        Show
        Mauro Talevi added a comment - PS: Please remember to put the JIRA issue in your commits: http://jbehave.org/reference/stable/how-to-contribute.html

          People

          • Assignee:
            Mauro Talevi
            Reporter:
            Lance
          • Votes:
            15 Vote for this issue
            Watchers:
            18 Start watching this issue

            Dates

            • Created:
              Updated: