JRuby 1.9 coverage library reports different results than ruby 1.9

I too am having this problem, would be great to see a fix. I can help test it.
Cheers

This is still a problem but I'm not sure how MRI is determining that it's a valid uncovered line versus an invalid line. We do not have this sort of information available in our representation of Ruby code. I'll poke around for a moment and see if I can find anything.

After looking this over I'm not sure it's something we can easily support.

Ruby 1.9 appears to manage this behavior by actually mutating its instruction sequence when there's an event hook installed, such as for coverage reporting. For each instruction, there is a "last_coverable_line", which appears to be used to indicate that there are other lines associated with this instruction that should get "0" for coverage data when encountered.

#define ADD_TRACE(seq, line, event) \
  do { \
      if ((event) == RUBY_EVENT_LINE && iseq->coverage && \
	  (line) != iseq->compile_data->last_coverable_line) { \
	  RARRAY_PTR(iseq->coverage)[(line) - 1] = INT2FIX(0); \
	  iseq->compile_data->last_coverable_line = (line); \
	  ADD_INSN1((seq), (line), trace, INT2FIX(RUBY_EVENT_COVERAGE)); \
      } \
      if (iseq->compile_data->option->trace_instruction) { \
	  ADD_INSN1((seq), (line), trace, INT2FIX(event)); \
      } \
  }while(0);

We have no such information in our AST, nor do we have any way to track that information in the Java compiled code.

Note that the 'coverage' library explicitly states that it is "experimental" and it may change in the future. I don't like that it might be implementation specific, but the way it is implemented right now requires that other implementations be able to know which lines are associated with the current line and be able to update the coverage information to say "oh, these other lines haven't been hit, but they are valid."

Charles, are there any effective code coverage tools that you know and use under JRuby at the moment?

FWIW, this appears to still be an issue as on jruby 1.6.7 (ruby-1.9.2-p312) (2012-02-22 3e82bc8) (Java HotSpot(TM) 64-Bit Server VM 1.6.0_31) [darwin-x86_64-java]

jruby --debug --1.9 -e "require 'coverage'; Coverage.start; require './test_cov.rb'; p Coverage.result"
{"/Library/Frameworks/JRuby.framework/Versions/1.6.7/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb"=>[nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, 1, 1], "/Users/jasonlunn/./test_cov.rb"=>[nil, nil, 1, 1, 1, nil, nil, nil, nil, nil, 1, nil, nil, nil, nil, nil, 1, nil, nil, nil, nil, nil, nil, nil, nil, nil, 1, 1], "-e"=>[2]}

Still an issue in 1.7.0...

jruby 1.7.0.preview1 (ruby-1.9.3-p203) (2012-05-19 00c8c98) (Java HotSpot(TM) 64-Bit Server VM 1.6.0_33) [darwin-x86_64-java]

"/Users/jasonlunn/./test_cov.rb"=>[nil, nil, 1, 1, 1, nil, nil, nil, nil, nil, 1, nil, nil, nil, nil, nil, 1, nil, nil, nil, nil, nil, nil, nil, nil, nil, 1, 1], "-e"=>[2]}

Has anyone had a chance to think more about this?

Is there a different approach that could be followed for doing a JRuby code coverage tool?

I think it would be fabulous to have a fix or a workaround for this issue. Understanding which code and how much code is covered by our testing is an important part of our processes. Not being able to get coverage results reduces the utility of JRuby.

Well as far as a different approach, the rcov gem should be 100% still. There's no major difference in JRuby between 1.8 and 1.9 mode as far as execution goes, and rcov has been kept working for a long time now.

I will poke at it a bit right now.

Ok, I think I'm getting a picture of how MRI does this.

Coverage support is turned on by loading the 'coverage' extension. This sets a global "rb_coverages" to non-nil, as a Hash.

At parse time, files parsed are added to the coverages hash as having 'nil' for all lines. Files parsed before the 'coverage' ext is loaded are not added to this hash. This is the first behavioral difference in JRuby, since we're just adding lines to coverage for any file+line encountered, regardless of whether the file was loaded before or after 'coverage' ext loaded. See parse.y, "coverage" function, called from yycompile0 to set up the array for that file.

At compile time (after parse) code lines get reset to zeros whenever that line would potentially trigger a "line" event in trace hooks. This is in compile.c in the ADD_TRACE macro. Essentially, compile.c adds the tracing logic as it goes only to the lines that should trigger a line event, and at the same time zeros out that line in the coverages array.

The determination of a "code line" happens in compile.c in the iseq_compile_each function, right at the top, where it determines the line is a newline.

    if (node->flags & NODE_FL_NEWLINE) {
	ADD_TRACE(ret, nd_line(node), RUBY_EVENT_LINE);
    }

So for every actual NEWLINE node in the AST, the according line in the coverages array is reset to zero.

Assuming our set of newline nodes matches MRI's (questionable), it should basically mean we just need to do the same zeroing of the according line when we construct a NewlineNode in our parser.

It is likely, however, that we do not have newlines in all the same places that MRI does, and as a result we might miss some lines that should be zero in coverage.

I will attempt some patching to see if I can get our logic to match MRI's a bit more closely, and we'll see what we get.

Well the initial experiment seems to be a success!

system ~/projects/jruby $ jruby --debug --1.9 -e "require 'coverage'; Coverage.start; require './test_cov.rb'; p Coverage.result"
{"/Users/headius/projects/jruby/./test_cov.rb"=>[nil, nil, 1, 1, 1, nil, nil, nil, nil, nil, 1, nil, 0, nil, nil, nil, 1, nil, nil, nil, nil, nil, nil, nil, nil, nil, 1, 1]}

system ~/projects/jruby $ ruby-1.9.3 -e "require 'coverage'; Coverage.start; require './test_cov.rb'; p Coverage.result"
{"/Users/headius/projects/jruby/test_cov.rb"=>[nil, nil, 1, 1, 1, nil, nil, nil, nil, nil, 1, nil, 0, nil, nil, nil, 1, nil, nil, nil, nil, nil, nil, nil, nil, nil, 1, 1, nil, nil]}

The changes I made:

• In ParserSupport.newline_node, call ParserConfiguration.coverLine with the appropriate line number
• In ParserConfiguration.coverLine, maintain a growing Integer[] as lines are encountered. Zero the line being covered.
• In Parser, at the end of parse associate the filename with that Integer[] only if coverage is enabled.
• In CoverageData's LINE event hook, only update lines for files that already have an associated Integer[].

This should basically mimic MRI's behavior, modulo any newline nodes we're missing.

There are a few places outside newline_node where we create newlines. I did not add coverage calls there, because I'm unsure if they're intended to be visitable lines or if they're outside of normal executable code.

I'm going to push these changes to JRuby master. I have no idea what's available for testing coverage beyond the one-off cases folks like you have provided, so I'd appreciate help finding tests for this.

FWIW, I'm not sure if the missing lines at the end of the file are important or not. JRuby's parser does not get a count of lines at any point during parsing, and if a line is not parsed as coverable, we will never grow the array to that point. In this case, the last two lines of the file are blank and so the parser never has anything to do there. They won't execute as code, and so they're not in the array.

I would be surprised if this affected coverage tools, but that remains to be seen.

Thanks for doing this fix. Much appreciated. When do you think this will be available in a released version of JRuby?

Hi all!
At first big thanks to Charles for tackling all this and to the JRuby dev team in general for well.. JRuby <3<3<3 you guys!

For reference, when I'm talking about jruby-head I'm talking about jruby with the latest commit being this one:
"commit ff42564963ac0d5b8b6471f23fd3441fb2db6fba
Author: Charles Oliver Nutter <headius@headius.com>
Date: Wed Aug 22 18:42:34 2012 -0500

Fix JRUBY-6827 (...)"
All of this on my LinudMint Debian Edition (based on Debian Testing) - 64 bit.

Good news first: it seems to kind of work. I created a minimal example where it seems to work after an adjustment to rspec-expectations. You can check it out at github - the fix is described there and the error output is documented. The error is an index out of bounds exception when trying to load the 'pp' library - no idea how that is triggered. And yes it only occurs when using jruby-head and simplecov.

Which gets us to the bad part, namely that error. I created a branch of my shoes4 fork where I use jruby-head and simplecov again. There the error can be seen again in this gist - again rspec-expectations. I tried to fix it there as well but then the error goes on and occurs in pry (another require of 'pp'). Uncommenting that one however leads to everything not working anymore even without simplecov. However if one keeps going the next error come for pry when trying to require 'tmpdir'. The stacktrace can be seen in this gist.

On a related note, jruby-head (without simplecov) seems to make 68 tests fail (all for the same reason apparently) while everything is green on jruby-1.6.7.2 but that's rather the topic for a separate issue.

I hope that I provided enough information, if some kind of information or an example or something is missing please tell me

Cheers,
Tobi

edit: Oh and as far as I know rcov doesn't work with 1.9 at least it was like that like half a year ago