JRuby

JRuby --1.9 cannot load YAML output from JRuby --1.8

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: JRuby 1.6.5
  • Fix Version/s: JRuby 1.7.0.pre1
  • Component/s: Standard Library
  • Labels:
    None
  • Number of attachments :
    1

Description

I open this issue mostly for discussion since I suspect that there is not bug, but it represents a real world problem.

If you use JRuby in 1.8 mode and dump a string containing a TAB character, you get something like this:

macbeth:~ uwe$ jruby --1.8 -ryaml -S jirb
>> YAML.dump("36L\tDIESEL")
=> "--- 36L\tDIESEL\n"
>> YAML.load "--- 36L\tDIESEL\n"
=> "36L\tDIESEL"
>>

Notice the TAB character is embedded verbatim in the YAML stream. No quoting, no escaping. YAML.load works fine with this stream.

Now the same for --1.9 mode:

macbeth:~ uwe$ jruby --1.9 -ryaml -S jirb
irb(main):001:0> YAML.dump("36L\tDIESEL")
=> "--- ! \"36L\\tDIESEL\"\n...\n"
irb(main):002:0> YAML.load "--- ! \"36L\\tDIESEL\"\n...\n"
=> "36L\tDIESEL"

Notice that now the TAB character is both quoted and escaped. YAML.load work fine on this stream as well.

But what happens if we try to parse the stream generated by JRuby 1.8 with JRuby 1.9?

macbeth:~ uwe$ jruby --1.9 -ryaml -S jirb
irb(main):001:0> YAML.load "--- 36L\tDIESEL\n"
ArgumentError: syntax error in "<reader>", line 1, column 8:
    --- 36L	DIESEL
           ^
	from org/jruby/ext/psych/PsychParser.java:273:in `parse'
	from /Library/Frameworks/JRuby.framework/Versions/1.6.5/lib/ruby/1.9/psych.rb:148:in `parse_stream'
	from /Library/Frameworks/JRuby.framework/Versions/1.6.5/lib/ruby/1.9/psych.rb:119:in `parse'
	from /Library/Frameworks/JRuby.framework/Versions/1.6.5/lib/ruby/1.9/psych.rb:106:in `load'
	from (irb):3:in `evaluate'
	from org/jruby/RubyKernel.java:1093:in `eval'
	from org/jruby/RubyKernel.java:1420:in `loop'
	from org/jruby/RubyKernel.java:1206:in `catch'
	from org/jruby/RubyKernel.java:1206:in `catch'
	from /Library/Frameworks/JRuby.framework/Versions/1.6.5/bin/jirb:13:in `(root)'

It blows up. We are seeing this in production, so we need a solution to this problem. On the receiver (consumer) side of the exchange, we use event based parsing with Psych::Parser and Psych::Handler so, as far as I know, reverting to syck is not an option.

Any help on this is greatly appreciated, and time is of the essence. We are currently just avoiding the problem by disallowing offending input, but we need a permanent solution for this:

How can we safely generate YAML with JRuby 1.8 and parse it with JRuby 1.9 using event based parsing?

P.S. On the consumer side, we are actually running JRuby in 1.8 mode, but we pull in Psych explicitly. I mention this only to open up more possibilities for a solution.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Uwe Kubosch added a comment - - edited

Using MRI 1.8 I get this:

macbeth:~ uwe$ ruby -ryaml -S irb
>> YAML.dump "36L\tDIESEL"
=> "--- 36L\tDIESEL\n"

MRI gives the exact same output as JRuby --1.8. So JRuby --1.9 has problems parsing YAML generated by MRI as well.

Show
Uwe Kubosch added a comment - - edited Using MRI 1.8 I get this: macbeth:~ uwe$ ruby -ryaml -S irb >> YAML.dump "36L\tDIESEL" => "--- 36L\tDIESEL\n" MRI gives the exact same output as JRuby --1.8. So JRuby --1.9 has problems parsing YAML generated by MRI as well.
Hide
Permalink
Uwe Kubosch added a comment -

Reading the spec at http://yaml.org/spec/1.2/spec.html it looks like the generated YAML stream with the verbatim TAB character is valid. That suggests that JRuby failing to parse it is a bug in Psych.

Show
Uwe Kubosch added a comment - Reading the spec at http://yaml.org/spec/1.2/spec.html it looks like the generated YAML stream with the verbatim TAB character is valid. That suggests that JRuby failing to parse it is a bug in Psych.
Hide
Permalink
Uwe Kubosch added a comment -

I have now traced this back to snakeyaml. It throws a ScannerException on the example string. I have registered issue http://code.google.com/p/snakeyaml/issues/detail?id=136 to see if we can get it fixed upstream.

Show
Uwe Kubosch added a comment - I have now traced this back to snakeyaml. It throws a ScannerException on the example string. I have registered issue http://code.google.com/p/snakeyaml/issues/detail?id=136 to see if we can get it fixed upstream.
Hide
Permalink
Uwe Kubosch added a comment -

A fix has been proposed at

http://code.google.com/r/py4fun-tabinscalar/

I will see if a new release is planned before JRuby 1.6.6/1.7.0 If so, I think we should update our version of SnakeYAML.

Show
Uwe Kubosch added a comment - A fix has been proposed at http://code.google.com/r/py4fun-tabinscalar/ I will see if a new release is planned before JRuby 1.6.6/1.7.0 If so, I think we should update our version of SnakeYAML.
Hide
Permalink
Uwe Kubosch added a comment -

A fixed JAR is available here:

http://code.google.com/p/snakeyaml/downloads/detail?name=snakeyaml-1.10-SNAPSHOT.jar

Show
Uwe Kubosch added a comment - A fixed JAR is available here: http://code.google.com/p/snakeyaml/downloads/detail?name=snakeyaml-1.10-SNAPSHOT.jar
Hide
Permalink
Uwe Kubosch added a comment -

I have added the snapshot jar to build_lib and run the tests. It fixes the problem in the example above.

Is it OK if I commit the jar? We can always roll it back if there is an issue.

Show
Uwe Kubosch added a comment - I have added the snapshot jar to build_lib and run the tests. It fixes the problem in the example above. Is it OK if I commit the jar? We can always roll it back if there is an issue.
Hide
Permalink
Thomas E Enebo added a comment -

Uwe,

Any way we can convince them to put out 1.10 for us soon? Tough to release with a snapshot...

Show
Thomas E Enebo added a comment - Uwe, Any way we can convince them to put out 1.10 for us soon? Tough to release with a snapshot...
Hide
Permalink
Uwe Kubosch added a comment -

I will ask. Probably nothing this year, and I am sure they want a definitive answer from Kirill first.

Show
Uwe Kubosch added a comment - I will ask. Probably nothing this year, and I am sure they want a definitive answer from Kirill first.
Hide
Permalink
Uwe Kubosch added a comment -

The next SnakeYAML release is scheduled for february. They will include the fix, pending an official verdict on wether the YAML stream is valid.

I hope we can release using the snapshot and upgrade to the release version when it comes. Can we do that?

Show
Uwe Kubosch added a comment - The next SnakeYAML release is scheduled for february. They will include the fix, pending an official verdict on wether the YAML stream is valid. I hope we can release using the snapshot and upgrade to the release version when it comes. Can we do that?
Hide
Permalink
Uwe Kubosch added a comment -

We've got confirmation that the YAML stream is correct and that the fix is included in the current SnakeYAML SNAPSHOT and that it will be released in february.

Is it OK that I change to the current snapshot and then update to the final version when it is released?

I am having trouble testing JRuby in 1.9 mode, but I have included a test for the change that fails with SnakeYAML 1.9 and succeeds with the new snapshot.

Show
Uwe Kubosch added a comment - We've got confirmation that the YAML stream is correct and that the fix is included in the current SnakeYAML SNAPSHOT and that it will be released in february. Is it OK that I change to the current snapshot and then update to the final version when it is released? I am having trouble testing JRuby in 1.9 mode, but I have included a test for the change that fails with SnakeYAML 1.9 and succeeds with the new snapshot.
Hide
Permalink
Uwe Kubosch added a comment -

Added a patch that updates SnakeYAML.

Could you review it for me?

Show
Uwe Kubosch added a comment - Added a patch that updates SnakeYAML. Could you review it for me?
Hide
Permalink
Hiro Asari added a comment -

Is this a critical issue to rely on a SNAPSHOT release? I'm not thrilled to have an unreleased jar. (I agree with Tom on this.)

Then I see an existing coro-mock-1.0-SNAPSHOT.jar. Hmm.

Show
Hiro Asari added a comment - Is this a critical issue to rely on a SNAPSHOT release? I'm not thrilled to have an unreleased jar. (I agree with Tom on this.) Then I see an existing coro-mock-1.0-SNAPSHOT.jar. Hmm.
Hide
Permalink
Uwe Kubosch added a comment -

The bug itself is major in that a YAML stream with a verbatim tab character in a string scalar causes a syntax error and a runtime exception.

For our project (fleet control for airport busses and fuel trucks) it is critical since we have no workaround for this.

The use of the snapshot is important for us, but we can build our own version of JRuby with the snapshot included. However, we would naturally strongly prefer to use the release branch and, in time, an official release.

The use of the snapshot would be temporary. SnakeYAML has announced that they plan to release version 1.10 in february. I can add the final release to JRuby then.

Show
Uwe Kubosch added a comment - The bug itself is major in that a YAML stream with a verbatim tab character in a string scalar causes a syntax error and a runtime exception. For our project (fleet control for airport busses and fuel trucks) it is critical since we have no workaround for this. The use of the snapshot is important for us, but we can build our own version of JRuby with the snapshot included. However, we would naturally strongly prefer to use the release branch and, in time, an official release. The use of the snapshot would be temporary. SnakeYAML has announced that they plan to release version 1.10 in february. I can add the final release to JRuby then.
Hide
Permalink
Thomas E Enebo added a comment -

I think the main concern is other companies (like RedHat) do maven builds to generate their products. Depending on snapshot builds is really icky for maven folks. I recommend rolling your own build for your case and we will get the real release of snakeYAML when they put it out for our next release.

Show
Thomas E Enebo added a comment - I think the main concern is other companies (like RedHat) do maven builds to generate their products. Depending on snapshot builds is really icky for maven folks. I recommend rolling your own build for your case and we will get the real release of snakeYAML when they put it out for our next release.
Hide
Permalink
Uwe Kubosch added a comment -

OK. I'll add the final SnakeYAML 1.10 jar when it is released.

When are we planning to release JRuby 1.7.0?

Show
Uwe Kubosch added a comment - OK. I'll add the final SnakeYAML 1.10 jar when it is released. When are we planning to release JRuby 1.7.0?
Hide
Permalink
Charles Oliver Nutter added a comment -

JRuby 1.7 is at least a month away, after we release 1.6.6. Maybe more, depending on how much we want to get in, but we don't want to go too long.

Show
Charles Oliver Nutter added a comment - JRuby 1.7 is at least a month away, after we release 1.6.6. Maybe more, depending on how much we want to get in, but we don't want to go too long.
Hide
Permalink
Uwe Kubosch added a comment -

I just pushed the final version of SnakeYAML 1.10.

Show
Uwe Kubosch added a comment - I just pushed the final version of SnakeYAML 1.10.

People

  • Assignee:
    Thomas E Enebo
    Reporter:
    Uwe Kubosch
Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved: