Hi all,
I wanted to pick up a question from a thread that's since been closed: No_std streaming JSON parser.

Disclosure: I'm the author of the bufjson crate.

@Vorpal, I noticed your requirement:

I have need for a streaming JSON parser that preserves formatting so that a document can round trip through my program exactly. I.e. I would need spaces and newlines to roundtrip as well as how numbers are formatted (for example 2 vs 2.0).

These requirements are solved by the bufjson streaming JSON parser! In bufjson:

• All token text is available as the exact original text from the underlying JSON text being parsed.
• This includes numbers and whitespace (there's a dedicated pseudo-token for whitespace).

You didn't mention strings in your post, but bufjson also allows perfect round-tripping of strings. This is important because most JSON parsers immediately expand escape sequences, so that, for example, the input string "\u0066oo" will be quietly expanded to "foo", preventing you from roundtripping it back as "\u0066oo". The bufjson crate solves this by explicitly separating tokenization/parsing from escape expansion, so that escape expansion is a choice and you always have access to the underlying literal string token if you need it.

The string escape expansion feature is very powerful and there are lots more examples of tricky roundtripping edge cases. Here's just one more. The forward slash or solidus (U+002F) / can be optionally escaped in JSON, so there are at least four ways of writing the same thing: /, \/, \u002f, and \u002F. A typical JSON parser will normalize them all to \ and then whatever output layer you are using may then attempt to re-normalize it: some JSON serializers will write it as \ and others as \/. This creates a lot of possible ways the input can be permuted when you want it to be echoed exactly. Again, bufjson is your solve!

If this sounds useful and you want to find the right place in the crate, take a look at:

• The lexical::Content trait.
• The lexical::Content::unescaped method.
• The free functions lexical::unescape and lexical::unescaped_cmd.

2 posts - 1 participant

Read full topic