SonarQube
  1. SonarQube
  2. SONAR-2392

Combining code coverage results of integration and unit tests for a module

    Details

    • Type: Improvement Improvement
    • Status: Closed Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.3
    • Component/s: Code Coverage
    • Labels:
      None
    • Environment:
      N/a - happens on all environments
    • Number of attachments :
      0

      Description

      It would be excellent to have the integration tests for a component included along with the unit tests in the overall code coverage report for a module.

      For example, a typical Maven 2 project may be

      mycomponent
      mycomponent-integration-tests

      Sonar could generate the combined report for the mycomponent module in one of two ways:

      1. Run Cobertura on mycomponent unit tests with Cobertura - this will produce a cobertura.ser file in the mycomponent/target/cobertura directory.
      2. Run the mycomponent-integration-tests integration tests with Cobertura- this will produce a cobertura.ser file in the mycomponent-integration-tests/target/cobertura directory.
      3. Amalgamate the two cobertura.ser files
      4.Generate the code coverage report for the mycomponent module from the amalgamated cobertura.ser file.

      OR

      1. Run Cobertura on mycomponent unit tests with Cobertura - this will produce a cobertura.ser file in the mycomponent/target/cobertura directory.
      2. Run the mycomponent-integration-tests integration tests with Cobertura, having the results being written to the cobertura.ser file in mycomponent/target/cobertura
      3. Generate the code coverage report for the mycomponent module from the cobertura.ser in mycomponent/target/cobertura.

        Issue Links

          Activity

          Show
          Simon Brandhof added a comment - Here are some information : MCOBERTURA-86 MCOBERTURA-33 http://www.mail-archive.com/users@maven.apache.org/msg81665.html
          Hide
          Mik Quinlan added a comment -

          Hi Simon

          Thanks very much for the links. I will check them out today.

          I'm pleased you've decided to add this feature into version 1.6 of Sonar. I think this, in itself, will be an excellent addition to the functionality, particularly given that this issue is only included as part of a paid product (i.e. Clover). I haven't tested getting entire coverage with Emma through the use of multiple metadata files.

          I look forward to this feature being released!

          Mik

          Show
          Mik Quinlan added a comment - Hi Simon Thanks very much for the links. I will check them out today. I'm pleased you've decided to add this feature into version 1.6 of Sonar. I think this, in itself, will be an excellent addition to the functionality, particularly given that this issue is only included as part of a paid product (i.e. Clover). I haven't tested getting entire coverage with Emma through the use of multiple metadata files. I look forward to this feature being released! Mik
          Hide
          azgard added a comment - - edited

          Any news or plans about this issue? 20 votes so far!

          Show
          azgard added a comment - - edited Any news or plans about this issue? 20 votes so far!
          Hide
          Conny Kreyssel added a comment -

          36 votes - no news?

          Show
          Conny Kreyssel added a comment - 36 votes - no news?
          Hide
          Mik Quinlan added a comment - - edited

          For those of you watching this issue, since submitting this improvement request, I've partially solved the problem. It does, however, require purchasing Clover. Sonar's clover plugin executes in the test and integration-test phase of Maven and produces a combined code coverage report. The limitation is that the unit tests for a module produce code coverage for that module only. So in a multi-module project, if you have a separate module called xxx-acceptance-tests that are for blackbox automated acceptance tests, you will not get code coverage for, say, your web module or service modules.

          To get this code coverage to work and have tests run in the right phases, I have done the following:

          1. Make all unit tests have the suffix UnitTest.java
          2. Make all integration tests have the suffix IntegrationTest.java
          3. Configure the Surefire plugin to run those tests in the right phase (*UnitTest.java in the test phase, *IntegrationTest.java in the integration-test phase) by doing the following:

          <build>
                  <pluginManagement>
                      <plugins>
                          <plugin>
                              <groupId>org.apache.maven.plugins</groupId>
                              <artifactId>maven-surefire-plugin</artifactId>
                              <configuration>
                                  <excludes>
                                      <exclude>**/*IntegrationTest.java</exclude>
                                  </excludes>
                              </configuration>
                              <executions>
                                  <execution>
                                      <id>integration-test</id>
                                      <goals>
                                          <goal>test</goal>
                                      </goals>
                                      <phase>integration-test</phase>
                                      <configuration>
                                          <excludes>
                                              <exclude>none</exclude>
                                          </excludes>
                                          <includes>
                                              <include>**/*IntegrationTest.java</include>
                                          </includes>
                                      </configuration>
                                  </execution>
                              </executions>
                          </plugin>
                       </plugins>
                  </pluginManagement>
          </build>
          

          Hope that is of help.

          Cobertura, of course, does not do combine the tests from the test phase and the integration-test phase hence the existence of this improvement.

          This issue should be kept open for those of us seeking a solution that does not require non-free software.

          Show
          Mik Quinlan added a comment - - edited For those of you watching this issue, since submitting this improvement request, I've partially solved the problem. It does, however, require purchasing Clover. Sonar's clover plugin executes in the test and integration-test phase of Maven and produces a combined code coverage report. The limitation is that the unit tests for a module produce code coverage for that module only. So in a multi-module project, if you have a separate module called xxx-acceptance-tests that are for blackbox automated acceptance tests, you will not get code coverage for, say, your web module or service modules. To get this code coverage to work and have tests run in the right phases, I have done the following: 1. Make all unit tests have the suffix UnitTest.java 2. Make all integration tests have the suffix IntegrationTest.java 3. Configure the Surefire plugin to run those tests in the right phase (*UnitTest.java in the test phase, *IntegrationTest.java in the integration-test phase) by doing the following: <build> <pluginManagement> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <configuration> <excludes> <exclude>**/*IntegrationTest.java</exclude> </excludes> </configuration> <executions> <execution> <id>integration-test</id> <goals> <goal>test</goal> </goals> <phase>integration-test</phase> <configuration> <excludes> <exclude>none</exclude> </excludes> <includes> <include>**/*IntegrationTest.java</include> </includes> </configuration> </execution> </executions> </plugin> </plugins> </pluginManagement> </build> Hope that is of help. Cobertura, of course, does not do combine the tests from the test phase and the integration-test phase hence the existence of this improvement. This issue should be kept open for those of us seeking a solution that does not require non-free software.
          Hide
          Conny Kreyssel added a comment -

          Thanks, Mik.

          Today i have tried to enable clover in sonar, but i think the aggragation is not called. In Atlassian Clover documentation it is described "The command clover2:aggregate goal is used for merging coverage data generated by multi-module projects".

          See http://confluence.atlassian.com/display/CLOVER/Clover-for-Maven+2+Installation+Guide

          But sonar doesnt call in my case the clover goal?

          Any hints?

          Show
          Conny Kreyssel added a comment - Thanks, Mik. Today i have tried to enable clover in sonar, but i think the aggragation is not called. In Atlassian Clover documentation it is described "The command clover2:aggregate goal is used for merging coverage data generated by multi-module projects". See http://confluence.atlassian.com/display/CLOVER/Clover-for-Maven+2+Installation+Guide But sonar doesnt call in my case the clover goal? Any hints?
          Hide
          Freddy Mallet added a comment -

          You're right Conny, the Sonar Clover plugin doesn't call the aggregate goal. It's very simple do this but then far more complex to change the coverage information on a maven module which has already been analysed by Sonar.

          Show
          Freddy Mallet added a comment - You're right Conny, the Sonar Clover plugin doesn't call the aggregate goal. It's very simple do this but then far more complex to change the coverage information on a maven module which has already been analysed by Sonar.
          Hide
          Mik Quinlan added a comment -

          Connie - yes, I have found the same thing. In fact, I have a web service multi-module project that has an acceptance-tests module. If I use clover2:aggregate outside of Sonar with Clover I get 90% code coverage. Within Sonar, only 42%.

          I think it's important that the code coverage from black box acceptance tests is also included.

          Perhaps we should open a new issue to have the Sonar and Clover combination also have the clover2:aggregate feature? Currently, Sonar with Clover has less functionality than Clover on its own.

          I love Sonar, BTW. It's made my life very easy. I always seem to have an issue with the code coverage for some reason, though, and that, for me, is the most important feature!

          Show
          Mik Quinlan added a comment - Connie - yes, I have found the same thing. In fact, I have a web service multi-module project that has an acceptance-tests module. If I use clover2:aggregate outside of Sonar with Clover I get 90% code coverage. Within Sonar, only 42%. I think it's important that the code coverage from black box acceptance tests is also included. Perhaps we should open a new issue to have the Sonar and Clover combination also have the clover2:aggregate feature? Currently, Sonar with Clover has less functionality than Clover on its own. I love Sonar, BTW. It's made my life very easy. I always seem to have an issue with the code coverage for some reason, though, and that, for me, is the most important feature!
          Hide
          Conny Kreyssel added a comment -

          Thanks, Mik.

          Same kind of project here. We have Hibernate domain classes in a separate module and this module has 0% test coverage, but its fully tested in integration tests.

          @Freddy
          Can you more explain the problem to add all module coverage results in sonar?

          Show
          Conny Kreyssel added a comment - Thanks, Mik. Same kind of project here. We have Hibernate domain classes in a separate module and this module has 0% test coverage, but its fully tested in integration tests. @Freddy Can you more explain the problem to add all module coverage results in sonar?
          Hide
          Mik Quinlan added a comment -

          Connie - I'll create the new issue in the next couple of days. As a side, the instructions I gave, above, negate the need for integration tests in a separate module. You can put them in the same module and get the entire code coverage. This would solve your issue.

          Freddy - yes, I'd be interested to know what the issue is. Perhaps it's a design issue and the Sonar metrics are collated as each module is built and analysed as opposed to storing the data in a temporary area and reading it all at the end of the analysis (which would allow aggregate coverage)? I'm sure it's not as simple as that so your input is appreciated.

          Show
          Mik Quinlan added a comment - Connie - I'll create the new issue in the next couple of days. As a side, the instructions I gave, above, negate the need for integration tests in a separate module. You can put them in the same module and get the entire code coverage. This would solve your issue. Freddy - yes, I'd be interested to know what the issue is. Perhaps it's a design issue and the Sonar metrics are collated as each module is built and analysed as opposed to storing the data in a temporary area and reading it all at the end of the analysis (which would allow aggregate coverage)? I'm sure it's not as simple as that so your input is appreciated.
          Hide
          Freddy Mallet added a comment -

          Mik, your understanding is perfectly correct : once a module has been analyzed all measures at package, file, class and method levels are dumped in the DB and deleted from memory to prevent using too much memory.
          Let's say for instance that a Maven project contains two modules : module A and module B :

          • Module A contains somes sources
          • Module B depends on Module A and contains some integration tests

          Sonar starts to analyse Module A and store all measure in the DB. Then it analyses Module B, but as all resources from Module A have been purged from Memory there is no way (at that time) to change any measures on those resources (in fact, there is no way to know that a given resource is part of Module A).

          Show
          Freddy Mallet added a comment - Mik, your understanding is perfectly correct : once a module has been analyzed all measures at package, file, class and method levels are dumped in the DB and deleted from memory to prevent using too much memory. Let's say for instance that a Maven project contains two modules : module A and module B : Module A contains somes sources Module B depends on Module A and contains some integration tests Sonar starts to analyse Module A and store all measure in the DB. Then it analyses Module B, but as all resources from Module A have been purged from Memory there is no way (at that time) to change any measures on those resources (in fact, there is no way to know that a given resource is part of Module A).
          Hide
          Mik Quinlan added a comment -

          I have created the related issue with respect to Clover here: http://jira.codehaus.org/browse/SONAR-1613

          Show
          Mik Quinlan added a comment - I have created the related issue with respect to Clover here: http://jira.codehaus.org/browse/SONAR-1613
          Hide
          Dennis Homann added a comment -

          My project uses the surefire-setup as described above by Mik: integration tests are located in the same module as unit tests, but run in the "integration-test" phase. Confusingly, Sonar picks up the integration test results, but does not include the integration tests in the coverage report.

          We use Cobertura, not Clover. I believe, with the plain Cobertura maven plug-in it would be possible to merge coverage data from two runs. Do you have any plans to support this scenario in Sonar as well (either aggregate coverage data or side-by-side with unit tests)?

          Show
          Dennis Homann added a comment - My project uses the surefire-setup as described above by Mik: integration tests are located in the same module as unit tests, but run in the "integration-test" phase. Confusingly, Sonar picks up the integration test results, but does not include the integration tests in the coverage report. We use Cobertura, not Clover. I believe, with the plain Cobertura maven plug-in it would be possible to merge coverage data from two runs. Do you have any plans to support this scenario in Sonar as well (either aggregate coverage data or side-by-side with unit tests)?
          Hide
          manuel aldana added a comment -

          Same here, we have convention that integration tests sit in the same module but in a different package (starts with itest). They are triggered by passing a environment variable -Dskip.integration.test=false. These tests are run in 'integration-test' phase.

          We use clover.

          Show
          manuel aldana added a comment - Same here, we have convention that integration tests sit in the same module but in a different package (starts with itest). They are triggered by passing a environment variable -Dskip.integration.test=false. These tests are run in 'integration-test' phase. We use clover.
          Hide
          Evgeny Mandrikov added a comment -

          Hi all who interested in this issue,

          Just FYI : now possible to get coverage information in Sonar for integration tests using JaCoCo Plugin.

          Show
          Evgeny Mandrikov added a comment - Hi all who interested in this issue, Just FYI : now possible to get coverage information in Sonar for integration tests using JaCoCo Plugin .
          Hide
          elmandour omar added a comment - - edited

          As Evgeny Mandrikov mentionned, Jacoco made the trick for me & is plugin agnostic.
          I have use it on a real entreprise & multi-module maven projects :

          • A-core
          • B-dao
          • C-web

          Each of them have unit tests and C-web have integration test when the war is deployed.
          Those integration-test are composed of Jwebunit and SoapUi.

          For each module let say we have a 10% coverage.

          But obviously C is using indirectly A & B so it will be nice to report the coverage done during the integration-test to the respective modules (A & B)

          The trick is to not seperate unit & integration test and generate the jacoco.exec in one place.
          Each module will append his result to the same file as long as the tests are run sequentially.

          So the sonar maven plugin will analyse 3 times the same file for each modules.

          And after, the coverage results could be A(30%), B(30%) and C (10% unchange of course)

          Of course if you are really interested to seperate IT report coverage and Unit report coverage, use the same trick with two files for each need.

          The difficult part is to agregate report file with tcp stuff, produced in a distant J2ee server ;(.
          For this purpose of merge, there is a ant task to merge files.

          Omar

          Show
          elmandour omar added a comment - - edited As Evgeny Mandrikov mentionned, Jacoco made the trick for me & is plugin agnostic. I have use it on a real entreprise & multi-module maven projects : A-core B-dao C-web Each of them have unit tests and C-web have integration test when the war is deployed. Those integration-test are composed of Jwebunit and SoapUi. For each module let say we have a 10% coverage. But obviously C is using indirectly A & B so it will be nice to report the coverage done during the integration-test to the respective modules (A & B) The trick is to not seperate unit & integration test and generate the jacoco.exec in one place. Each module will append his result to the same file as long as the tests are run sequentially. So the sonar maven plugin will analyse 3 times the same file for each modules. And after, the coverage results could be A(30%), B(30%) and C (10% unchange of course) Of course if you are really interested to seperate IT report coverage and Unit report coverage, use the same trick with two files for each need. The difficult part is to agregate report file with tcp stuff, produced in a distant J2ee server ;(. For this purpose of merge, there is a ant task to merge files. Omar
          Hide
          Bartosz Radaczynski added a comment -

          @Evgeny is there any way to see the combined coverage of unit and integration tests?

          Show
          Bartosz Radaczynski added a comment - @Evgeny is there any way to see the combined coverage of unit and integration tests?
          Hide
          Evgeny Mandrikov added a comment -

          Not yet.

          Show
          Evgeny Mandrikov added a comment - Not yet.
          Hide
          Reinhold Erler added a comment -

          Hi Omar,

          Your solution is very interesting! Please could you tell us your steps in more detail?
          Did you:

          • mvn sonar:sonar on A-core
          • mvn sonar:sonar on B-dao
          • ant task for merging the 2 jacoco.execs
          • mvn sonar:sonar on C-web

          And how did you manage to not overwrite the merged file when you called C-web?

          Thank you very much for clarification!

          statler

          Show
          Reinhold Erler added a comment - Hi Omar, Your solution is very interesting! Please could you tell us your steps in more detail? Did you: mvn sonar:sonar on A-core mvn sonar:sonar on B-dao ant task for merging the 2 jacoco.execs mvn sonar:sonar on C-web And how did you manage to not overwrite the merged file when you called C-web? Thank you very much for clarification! statler
          Hide
          Jorge Costa added a comment -

          Hi,

          I have a bit of a different scenario from the ones described above, we have a c++ code base and use Bullseye c++ coverage and before actually running sonar all the data (unit test and integration test coverage) is available.

          So when running sonar, both coverage data is feed into sonar in two different xml reports and finally when looking at source code tab for a project than we see both coverage in separate lists.

          This is fine, but it would be nice that what ever solution found for feeding data into sonar, when looking into the source code tab we could see have a Checkbox that merges both.

          Hope you get this one there.

          Thanks

          JC

          Show
          Jorge Costa added a comment - Hi, I have a bit of a different scenario from the ones described above, we have a c++ code base and use Bullseye c++ coverage and before actually running sonar all the data (unit test and integration test coverage) is available. So when running sonar, both coverage data is feed into sonar in two different xml reports and finally when looking at source code tab for a project than we see both coverage in separate lists. This is fine, but it would be nice that what ever solution found for feeding data into sonar, when looking into the source code tab we could see have a Checkbox that merges both. Hope you get this one there. Thanks JC
          Hide
          Freddy Mallet added a comment -

          Indeed Jorge and that will be the case.

          Show
          Freddy Mallet added a comment - Indeed Jorge and that will be the case.
          Hide
          Jorge Costa added a comment -

          Excellent, really forward to get my hands on this one.

          Cheers

          JC

          Show
          Jorge Costa added a comment - Excellent, really forward to get my hands on this one. Cheers JC
          Hide
          Mik Quinlan added a comment -

          So glad you guys scheduled this! :-D

          Show
          Mik Quinlan added a comment - So glad you guys scheduled this! :-D
          Hide
          Mik Quinlan added a comment -

          Um, what happened to the fix version?

          Show
          Mik Quinlan added a comment - Um, what happened to the fix version?
          Hide
          Freddy Mallet added a comment -

          In version 3.3, we'll fix SONAR-2804 which is very close to this ticket except that SONAR-2804 is about Jacoco and SONAR-2392 about Cobertura.

          Show
          Freddy Mallet added a comment - In version 3.3, we'll fix SONAR-2804 which is very close to this ticket except that SONAR-2804 is about Jacoco and SONAR-2392 about Cobertura.
          Hide
          Freddy Mallet added a comment -

          I'm closing this ticket as in fact it duplicates SONAR-2804. This ability to get and play with three different code coverage : unit tests, integration tests, overall code coverage will be available only when using the Sonar Jacoco plugin.

          Show
          Freddy Mallet added a comment - I'm closing this ticket as in fact it duplicates SONAR-2804 . This ability to get and play with three different code coverage : unit tests, integration tests, overall code coverage will be available only when using the Sonar Jacoco plugin.
          Hide
          Daniel Holmes added a comment -

          Just to ask, why do these features have to be coverage tool specific? What is the difficulty in supporting other than JaCoCo?

          Show
          Daniel Holmes added a comment - Just to ask, why do these features have to be coverage tool specific? What is the difficulty in supporting other than JaCoCo?
          Hide
          Freddy Mallet added a comment -

          Hi Daniel, This is coverage tool specific as soon as we want to also merge coverage by branch. Indeed the information contained into Sonar only allows to aggregate the coverage by lines of code. Moreover, computing a code coverage by integration tests is far more straightforward with Jacoco than it is with Cobertura. As in the first case you just need to plug the jacoco agent to the JVM.

          Show
          Freddy Mallet added a comment - Hi Daniel, This is coverage tool specific as soon as we want to also merge coverage by branch. Indeed the information contained into Sonar only allows to aggregate the coverage by lines of code. Moreover, computing a code coverage by integration tests is far more straightforward with Jacoco than it is with Cobertura. As in the first case you just need to plug the jacoco agent to the JVM.

            People

            • Assignee:
              Freddy Mallet
              Reporter:
              Mik Quinlan
            • Votes:
              96 Vote for this issue
              Watchers:
              77 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: