YAML "intermittent" error for large data structures

Is this something that has caused problems in an application?

There is bound to be differences, since Syck and JvYAML are completely different YAML engines; but this shouldn't have any effect on functionality.

Yes this has caused problems in production scripts I have written. That is the reason I started to investigate this. Today I can't use JRuby on these scripts.

I think I am aware of the differences between JvYAML and Syck. But I hope that JvYAML still tries to follow the YAML spec. Today JvYAML even fails to read it's own output correctly in some situations (as I tried to show with the "test/unit" based testcase attachment).

I also hope that JvYAML should be interoperable with other YAML implementations for those types that are shared between implementations. I would for example expect that a YAML file written by JRuby should be possible to read with Syck in CRuby. Isn't that the intention?

What do you mean when you say "There is bound to be differences"? I can understand if JvYAML for example chooses to write strings that look like a number in different ways than Syck:

% jruby -ryaml -e 'puts YAML.dump("123")'
— !str 123
% ruby -ryaml -e 'puts YAML.dump("123")'
— "123"

But both JvYAML and Syck can read any of these formats, so that is no problem. Beside such "trivial" differences, should there be other differences in behaviour?

I don't know if my rake-based testcase is the best way to demonstrate the problems I have encountered. I hope my use of a mixture of Cruby and JRuby there didn't confuse. Assuming that YAML in CRuby and JRuby should be interoperable, I just thought that it was a good way to exhibit the problem I'm having.

One of the things that has made me confused is the fact that these problems only occurs sometimes. In my rake-based testcase for example I only got errors for i=50, 74, 78. When I re-run exactly the same tests today on an Ubuntu Linux (still with -r3415 of JRuby) I get different errors: only for i=34. When I reported the problem I used Mac OS X.

Isn't these kind of differences very strange? Is there anything in JvYAML that can behave in an seemingly "indeterministic" way?

No, JvYAML should follow the spec. There are still areas where it fails, obviously, but that we're taking care of. (Actually, JvYAML follows the spec more closely than SYCK in several places).

But yes, YAML should be interoperable between different YAML implementations.

It's very interesting; I can't imagine what could cause these intermittent failures, though. As you say, there isn't any indeterminism built in. =)

I will try to investigate, but I can't promise to much.

Maybe it is related to garbage collection? The error occurs for large data structures, and occurs with some interval in time. Both these facts would be consistent with such an explanation. To test this hypothesis I made the following setting in my shell (used by the "jruby" script):

$ setenv JAVA_OPTS -Xms100m

After this setting the error disappeared. Does this help explain the mystery?

Hey. I don't see why the stack size should have ANY effect at all on it.

But I realized that I used identityHashCode as anchoring point in some places. That wasn't a good decision, so I've changed that. Could you try again with trunk?

I tried the testcase again after your change. No errors any more. I also increased the constant COUNT in the file "gen-yaml-samples.rb" from 80 to 200 (so the script creates even larger data structures for YAML to convert). Still no errors. Your change seem to have removed the problem.

Tomorrow at work I will re-run the tests on Ubuntu Linux too (right now I'm using Mac OS X).

Just out of curiosity, could you give a short explanation why "identityHashCode" caused problems? From reading the Java API documentation I can't see how stack size would affect the working of the code.

identityHashCode is not guaranteed to be unique; so if you have a lot of object churn there's a possibility it will hit the same identity twice. By increasing the heap size I would guess you prevented objects from being collected, which could have had an effect on the identityHashCode of future objects. Or it could just be a crazy coincidence...

I'm closing this for now, since it seems ok.

This issue was fixed with using IdentityHashMaps for all references to already handled objects and nodes, and also not using only System.identityHashCode to generate Anchor IDs.

I have now re-run the testcase on Ubuntu Linux too, using revision 3435. It worked there too (no errors).

BUT, first I just made an "svn update" and got revision 3445. There the error (or something with similar effect) had been re-introduced. I hope it is easy for you too Ola, to run the testcase I supplied. Just unpack the file "testcase-using-rake.tar.gz" in an empty directory and run it as described earlier in this bug-report.

Otherwise I will try again later, when you have made all your current YAML changes, and things have stabilized.

Ok, that was a bit fast of me. =)

So, this issue has been finally resolved. The reason it came back seems to be that I by mistake backed out two changes from HashMap to IdentityHashMap I made before. It also seems identityHashCode wasn't the culprit this time; instead it was the fact that I used regular HashMaps to keep track on objects represented. Your test case works fine for me now, and I'm sure you won't find this problem again. =)

This is embarressing. It still seems it's the identity hash code.

Once again, this time for real! I did away with generated alias keys alltogether, and instead just saved everything with the data as key.