Streaming JSON String Values

[meshcalero]

Idea: Provide means to stream huge values (e.g JSON Strings containing a base64 encoding of some large binary file) when using a NonBlockingJsonParser, rather than requiring string values to fit into memory. It would be great if I would could specify a max value-textbuffer size. When a given value exceeds that size, the parser would make the value accessible in some streaming fashion, rather than trying to fully load it into memory.

Use Case: I'm implementing a proxy-like application that provides some filter mechanism on JSON payload. For this I need to parse the proxied responses, in order to detect the keys to filter, but I'm usually not interested in reading the values other than passing them along. In order to enable proxying arbitrary json-payloads (incl. those with huge string values) while keeping the memory-footprint under control, I'd love to have the above streaming access to individual CLOB values.

[pjfanning]

Have you looked at other JSON libraries? I'm not against Jackson supporting something like this but it is likely to be a lot of work - especially since the pressure will come on to have jackson-databind and other related libs also support this.

https://github.com/jettison-json/jettison is worth looking at since it is basically JSON support but using the StAX API. A large json value could be represented as many CharacterEvents.

You can override the StreamReadConstraints if you are just hitting issues with the 20m char default limit for String values.

Just to be contrary, I just don't think JSON was designed for placing gigabytes of data in string values. There are so many better ways to represent this kind of data.

[meshcalero]

Thanks for the swift response.

I unfortunately just realized that you already have a similar thread open: #1288. So sorry for the duplication.

Will look into jettison.

While I fully agree with your comment on "inappropriate use of JSON", as a proxy, agnostic to the environment where I'm used, I have litte say on what people put into their payload. :-)

[cowtowncoder]

@meshcalero Just a friendly note: do not waste your time with Jettison. I don't have time to go into too many details but it's.... not ... the thing you are looking for.

[cowtowncoder]

But to expand on that: Jettison does not really offer ability to split String values, even though XML parsers do (and Jettison uses Woodstox). This because actual JSON side of things uses ancient "json.org" parser/generator which uses simple Tree model for JSON content. So it's not even streaming across tokens/events, much less within String values.

[pjfanning]

The issue goes beyond JSON parsing and the JSON parsing issue is a big set of changes. We can't use arrays or Strings because they are limited to Integer.MAX_VALUE bytes or chars. We would need temp files or multidimensional arrays. You would probably also need a specialised Base64 decoder that avoids arrays and Strings too.

[cowtowncoder]

Issue #1288 shows one way to handle exposing chunked/incremental String values, fwtw - without need for temp files or multi-dimensional array on Jackson side of things (caller might need buffering, depending).
Assuming that stateful consumption (non-idempotent access) is acceptable.

Implementation could be relatively straight-forward for JSON parser, but there are 4 backends and of course ideally other formats' parser would also support such access.

[cowtowncoder]

@pjfanning I assume you do not have context on Jettison, but as I recall, it's more of a joke than serious thing to use. Exposing JSON via XML API is a bad idea to begin with (due to structural impedance), and implementation of JSON handling is not any better. I know Dan D and he considered the library to be one his less good ideas. :)

So while it might ostensibly look like a useful thing, it is not. Starting with a joke ("badgerFish convention") mixing up API and content... nope.

[meshcalero]

I will definitively park the problem for now (there are more important challenges to address first). But given that it was a potential problem that I had spottet, and given that the JSON parser that I will use for the project might not be so easy to change at a later point in time, I wanted to evaluate if you would be open to address that problem.

Sound like you're not. And since I could also not find any other parser that would support it, I will for now use Jackson and see if I find the energy and time to fork jackson-core once I have solved my other problems.

[cowtowncoder]

Also: +1 for "why don't people just use Arrays of String values for Chunking?!". I realize that as Proxy author you have no power over them but man, chunking is CS 101, very basic stuff.
This is also why I was very surprised to learn lots of usage for JSON Strings of 20 megs and more in size: surely devs are not short-sighted enough NOT to chunk things? (spoiler alert: they are :) ).

[cowtowncoder]

@meshcalero Sorry if I gave wrong impression: I am open to addressing this issue -- that's why there are earlier issues. I have solved this problem with XML (that is, Stax API exposes it and both Woodstox and Aalto support CDATA chunking). I think it is a valid, if advanced, problem.

But I also want so set expections such that there is probably not a quick and easy, immediate solution available.
However: not saying there won't be eventual solution.

[meshcalero]

I did for sure not expect you to jump on the idea immediately. I just wanted to see if there would be a chance that you support this in jackson at some point in time in the future.

So thanks for considering the problem.

Maybe I find time to draft a solution on a fork. Will let you know if I ever get this running.

[cowtowncoder]

Best of luck @meshcalero !

Will leave this discussion open.