Jackson JSON Processor

Add support for non-blocking ("async") parsing

Details

  • Type: New Feature New Feature
  • Status: Closed Closed
  • Priority: Minor Minor
  • Resolution: Duplicate
  • Affects Version/s: None
  • Fix Version/s: None
  • Component/s: None
  • Labels:
    None
  • Number of attachments :
    0

Description

(suggested by Dimitri M on user list)

There are use cases where it'd be good to be able to feed input to parser, instead of trying to provide an input stream for parser to read from. This would cover use cases where input comes in chunks; for example, part of logical document in one chunk, then after a delay (perhaps in a separate request) another one and so forth. In these cases it may be difficult to implement InputStream (Reader etc) abstraction; instead, it would be better if application could feed (push) data to parser.
But if so, parser must be able to indicate cases where no data is YET available (but may become available).

This is similar to how Aalto Xml processor (http://www.cowtowncoder.com/hatchery/aalto/index.html) operatesd in its async mode. However, since Json is much simple format than xml, implementation might be simpler.

Based on my experiences with Aalto, implementation is a non-trivial thing however. One problem is that even UTF-8 decoding needs to be somewhat aware of chunk boundaries, so in the end a separate parser may be required: this because current parser uses blocking to handle these split cases. A smaller problem is that of indicating "not-yet-available" case – this can probably be handled by introducing a new member in JsonToken enumeration.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Jon Brisbin added a comment -

I would really, really like to see NIO-based parsing for JSON, as I'm working on a non-blocking project right now but I'm a bit stymied by the blocking nature of the library. This isn't just a "nice to have" but a showstopper if the parsing is done in the selector thread (which sometimes happens depending on how the application routes requests).

I think non-blocking apps on Java will be the big "thing" of this year, so it makes sense to start thinking about how non-blocking web apps will processing incoming JSON requests and send outgoing ones.

IMHO this should be a fairly high priority item if one wants to stay ahead of the curve.

Show
Jon Brisbin added a comment - I would really, really like to see NIO-based parsing for JSON, as I'm working on a non-blocking project right now but I'm a bit stymied by the blocking nature of the library. This isn't just a "nice to have" but a showstopper if the parsing is done in the selector thread (which sometimes happens depending on how the application routes requests). I think non-blocking apps on Java will be the big "thing" of this year, so it makes sense to start thinking about how non-blocking web apps will processing incoming JSON requests and send outgoing ones. IMHO this should be a fairly high priority item if one wants to stay ahead of the curve.
Hide
Permalink
Tatu Saloranta added a comment -

If there is actual demand (i.e. requests), I wouldn't mind tackling this: I wrote Aalto XML parser (http://wiki.fasterxml.com/AaltoHome) which is non-blocking/async parser for XML, so I know exactly how to do it. But it is not a trivial undertaking when done properly.

One work-around to alleviate need for non-blocking parser typically is to use external framing, and read contents in non-blocking fashion, then handing complete document to blocking parser. This works for many cases.

The main concern I have, still, is with respect to actual need (different from perceived need).
I am said to say that considering effort for Aalto, very VERY few developers are using it, or are actually activelly using it. And most who contacted me either did not have good understanding what non-blocking means, or were (IMO) confused with respect to needing it.
This is why I am cautious at jumping to do things that are cool – while I know node.js, Akka et al are gaining popularity, I am still waiting for flow of requests for this feature.

... still, I take your comment as a strong vote; and encourage everyone to voice their support. Like I said, this is only matter of priorization – if there is need, I know how to do it.
Or – anyone who is brave enough to tackle it, Aalto might have useful nuggets. The main challenge in understanding code there is that it is basically a state machine. But one reusable piece is name-parsing, which is a major PITA to get right (bytes-to-Symbol, with non-blocking handling); and Aalto does it, in a way that is very similar to how Jackson does it.

Show
Tatu Saloranta added a comment - If there is actual demand (i.e. requests), I wouldn't mind tackling this: I wrote Aalto XML parser ( http://wiki.fasterxml.com/AaltoHome ) which is non-blocking/async parser for XML, so I know exactly how to do it. But it is not a trivial undertaking when done properly. One work-around to alleviate need for non-blocking parser typically is to use external framing, and read contents in non-blocking fashion, then handing complete document to blocking parser. This works for many cases. The main concern I have, still, is with respect to actual need (different from perceived need). I am said to say that considering effort for Aalto, very VERY few developers are using it, or are actually activelly using it. And most who contacted me either did not have good understanding what non-blocking means, or were (IMO) confused with respect to needing it. This is why I am cautious at jumping to do things that are cool – while I know node.js, Akka et al are gaining popularity, I am still waiting for flow of requests for this feature. ... still, I take your comment as a strong vote; and encourage everyone to voice their support. Like I said, this is only matter of priorization – if there is need, I know how to do it. Or – anyone who is brave enough to tackle it, Aalto might have useful nuggets. The main challenge in understanding code there is that it is basically a state machine. But one reusable piece is name-parsing, which is a major PITA to get right (bytes-to-Symbol, with non-blocking handling); and Aalto does it, in a way that is very similar to how Jackson does it.
Hide
Permalink
Tatu Saloranta added a comment -

Despite being mentioned on and off over time, I really do not feel there is much actual concrete interest with this. I have played a bit with async/non-blocking Smile parser; and it may be something to write first as it's easier to do. And something I may have more use for myself.

Anyway: will move this to https://github.com/FasterXML/jackson-core/issues/57 to keep it open, while Codehaus Jackson Jira will be (eventually) closed.

Show
Tatu Saloranta added a comment - Despite being mentioned on and off over time, I really do not feel there is much actual concrete interest with this. I have played a bit with async/non-blocking Smile parser; and it may be something to write first as it's easier to do. And something I may have more use for myself. Anyway: will move this to https://github.com/FasterXML/jackson-core/issues/57 to keep it open, while Codehaus Jackson Jira will be (eventually) closed.

People

  • Assignee:
    Tatu Saloranta
    Reporter:
    Tatu Saloranta
Vote (5)
Watch (4)

Dates

  • Created:
    Updated:
    Resolved: