Combining code coverage results of integration and unit tests for a module

Here are some information :

• MCOBERTURA-86
• MCOBERTURA-33
• http://www.mail-archive.com/users@maven.apache.org/msg81665.html

Hi Simon

Thanks very much for the links. I will check them out today.

I'm pleased you've decided to add this feature into version 1.6 of Sonar. I think this, in itself, will be an excellent addition to the functionality, particularly given that this issue is only included as part of a paid product (i.e. Clover). I haven't tested getting entire coverage with Emma through the use of multiple metadata files.

I look forward to this feature being released!

Mik

Any news or plans about this issue? 20 votes so far!

36 votes - no news?

For those of you watching this issue, since submitting this improvement request, I've partially solved the problem. It does, however, require purchasing Clover. Sonar's clover plugin executes in the test and integration-test phase of Maven and produces a combined code coverage report. The limitation is that the unit tests for a module produce code coverage for that module only. So in a multi-module project, if you have a separate module called xxx-acceptance-tests that are for blackbox automated acceptance tests, you will not get code coverage for, say, your web module or service modules.

To get this code coverage to work and have tests run in the right phases, I have done the following:

1. Make all unit tests have the suffix UnitTest.java
2. Make all integration tests have the suffix IntegrationTest.java
3. Configure the Surefire plugin to run those tests in the right phase (*UnitTest.java in the test phase, *IntegrationTest.java in the integration-test phase) by doing the following:

<build>
        <pluginManagement>
            <plugins>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <configuration>
                        <excludes>
                            <exclude>**/*IntegrationTest.java</exclude>
                        </excludes>
                    </configuration>
                    <executions>
                        <execution>
                            <id>integration-test</id>
                            <goals>
                                <goal>test</goal>
                            </goals>
                            <phase>integration-test</phase>
                            <configuration>
                                <excludes>
                                    <exclude>none</exclude>
                                </excludes>
                                <includes>
                                    <include>**/*IntegrationTest.java</include>
                                </includes>
                            </configuration>
                        </execution>
                    </executions>
                </plugin>
             </plugins>
        </pluginManagement>
</build>

Hope that is of help.

Cobertura, of course, does not do combine the tests from the test phase and the integration-test phase hence the existence of this improvement.

This issue should be kept open for those of us seeking a solution that does not require non-free software.

You're right Conny, the Sonar Clover plugin doesn't call the aggregate goal. It's very simple do this but then far more complex to change the coverage information on a maven module which has already been analysed by Sonar.

Connie - yes, I have found the same thing. In fact, I have a web service multi-module project that has an acceptance-tests module. If I use clover2:aggregate outside of Sonar with Clover I get 90% code coverage. Within Sonar, only 42%.

I think it's important that the code coverage from black box acceptance tests is also included.

Perhaps we should open a new issue to have the Sonar and Clover combination also have the clover2:aggregate feature? Currently, Sonar with Clover has less functionality than Clover on its own.

I love Sonar, BTW. It's made my life very easy. I always seem to have an issue with the code coverage for some reason, though, and that, for me, is the most important feature!

Thanks, Mik.

Same kind of project here. We have Hibernate domain classes in a separate module and this module has 0% test coverage, but its fully tested in integration tests.

@Freddy
Can you more explain the problem to add all module coverage results in sonar?

Connie - I'll create the new issue in the next couple of days. As a side, the instructions I gave, above, negate the need for integration tests in a separate module. You can put them in the same module and get the entire code coverage. This would solve your issue.

Freddy - yes, I'd be interested to know what the issue is. Perhaps it's a design issue and the Sonar metrics are collated as each module is built and analysed as opposed to storing the data in a temporary area and reading it all at the end of the analysis (which would allow aggregate coverage)? I'm sure it's not as simple as that so your input is appreciated.

Mik, your understanding is perfectly correct : once a module has been analyzed all measures at package, file, class and method levels are dumped in the DB and deleted from memory to prevent using too much memory.
Let's say for instance that a Maven project contains two modules : module A and module B :

• Module A contains somes sources
• Module B depends on Module A and contains some integration tests

Sonar starts to analyse Module A and store all measure in the DB. Then it analyses Module B, but as all resources from Module A have been purged from Memory there is no way (at that time) to change any measures on those resources (in fact, there is no way to know that a given resource is part of Module A).

I have created the related issue with respect to Clover here: http://jira.codehaus.org/browse/SONAR-1613

My project uses the surefire-setup as described above by Mik: integration tests are located in the same module as unit tests, but run in the "integration-test" phase. Confusingly, Sonar picks up the integration test results, but does not include the integration tests in the coverage report.

We use Cobertura, not Clover. I believe, with the plain Cobertura maven plug-in it would be possible to merge coverage data from two runs. Do you have any plans to support this scenario in Sonar as well (either aggregate coverage data or side-by-side with unit tests)?

Same here, we have convention that integration tests sit in the same module but in a different package (starts with itest). They are triggered by passing a environment variable -Dskip.integration.test=false. These tests are run in 'integration-test' phase.

We use clover.

Hi all who interested in this issue,

Just FYI : now possible to get coverage information in Sonar for integration tests using JaCoCo Plugin.

As Evgeny Mandrikov mentionned, Jacoco made the trick for me & is plugin agnostic.
I have use it on a real entreprise & multi-module maven projects :

• A-core
• B-dao
• C-web

Each of them have unit tests and C-web have integration test when the war is deployed.
Those integration-test are composed of Jwebunit and SoapUi.

For each module let say we have a 10% coverage.

But obviously C is using indirectly A & B so it will be nice to report the coverage done during the integration-test to the respective modules (A & B)

The trick is to not seperate unit & integration test and generate the jacoco.exec in one place.
Each module will append his result to the same file as long as the tests are run sequentially.

So the sonar maven plugin will analyse 3 times the same file for each modules.

And after, the coverage results could be A(30%), B(30%) and C (10% unchange of course)

Of course if you are really interested to seperate IT report coverage and Unit report coverage, use the same trick with two files for each need.

The difficult part is to agregate report file with tcp stuff, produced in a distant J2ee server ;(.
For this purpose of merge, there is a ant task to merge files.

Omar

@Evgeny is there any way to see the combined coverage of unit and integration tests?

Not yet.

Hi Omar,

Your solution is very interesting! Please could you tell us your steps in more detail?
Did you:

• mvn sonar:sonar on A-core
• mvn sonar:sonar on B-dao
• ant task for merging the 2 jacoco.execs
• mvn sonar:sonar on C-web

And how did you manage to not overwrite the merged file when you called C-web?

Thank you very much for clarification!

statler

Hi,

I have a bit of a different scenario from the ones described above, we have a c++ code base and use Bullseye c++ coverage and before actually running sonar all the data (unit test and integration test coverage) is available.

So when running sonar, both coverage data is feed into sonar in two different xml reports and finally when looking at source code tab for a project than we see both coverage in separate lists.

This is fine, but it would be nice that what ever solution found for feeding data into sonar, when looking into the source code tab we could see have a Checkbox that merges both.

Hope you get this one there.

Thanks

JC

Indeed Jorge and that will be the case.

Excellent, really forward to get my hands on this one.

Cheers

JC

So glad you guys scheduled this! :-D

Um, what happened to the fix version?

In version 3.3, we'll fix SONAR-2804 which is very close to this ticket except that SONAR-2804 is about Jacoco and SONAR-2392 about Cobertura.

I'm closing this ticket as in fact it duplicates SONAR-2804. This ability to get and play with three different code coverage : unit tests, integration tests, overall code coverage will be available only when using the Sonar Jacoco plugin.

Just to ask, why do these features have to be coverage tool specific? What is the difficulty in supporting other than JaCoCo?

Hi Daniel, This is coverage tool specific as soon as we want to also merge coverage by branch. Indeed the information contained into Sonar only allows to aggregate the coverage by lines of code. Moreover, computing a code coverage by integration tests is far more straightforward with Jacoco than it is with Cobertura. As in the first case you just need to plug the jacoco agent to the JVM.