Harden builtin Json parsing and encoding with escape, round-trip, and golden tests

[lukewilliamboswell]

Summary

The builtin Json implementation has coverage for many structural shapes, number bounds, nullability, trailing commas, and parser generation, but string escaping and real-world payload coverage are still thin.

One concrete gap is Json.split_json_string_tail in src/build/roc/Builtin.roc. It currently finds the next " and rejects the string if any \ appears before that quote. That means valid JSON strings containing escapes like \n, \", \\, \t, \/, or \uXXXX are rejected instead of decoded.

That helper is used by string parsing, record/object keys, unknown-field skipping, and tag parsing, so this affects more than top-level Str.

The encoder has explicit string escaping logic in append_json_quoted_string / append_json_string_byte, but the current tests only lightly exercise exact escaped output and do not broadly prove that encoded output is accepted by Json.parse and round-trips through realistic shapes.

Relevant Code

• src/build/roc/Builtin.roc
  • Json.parse
  • Json.parse_trailing_commas
  • Json.to_str
  • Json.to_str_try
  • Json.split_json_string_tail
  • Json.append_json_quoted_string
  • Json.append_json_string_byte
• test/cli/JsonEncodeRoundTrip.roc
• test/cli/JsonEncodeEdgeCases.roc
• test/cli/JsonEncodeNumberEdgeCases.roc
• test/cli/JsonEncodeNullContainerEdgeCases.roc
• test/cli/JsonEncodeNestedContainerEdgeCases.roc
• test/json-decoder/app.roc
• src/cli/test/json_decoder_platform_test.zig
• test/snapshots/repl/

Background

Json support is implemented as structural parser_for / encoder_for methods over builtin encoding state. The public helpers are ordinary convenience APIs over those generated methods.

For someone new to this area: src/build/roc/Builtin.roc defines the builtin Json surface and implementation. The compiler derives structural parsers and encoders that call methods on JsonEncoding and thread JsonState / JsonEncodeState.

Existing tests already cover many structural cases, but we should add a dedicated layer of tests for JSON string correctness and realistic escaped payloads.

Proposed Tests

Add REPL snapshot golden tests under test/snapshots/repl/ for user-visible Json behavior.

Good snapshot candidates:

• Valid JSON string escapes: \", \\, \/, \b, \f, \n, \r, \t
• Unicode escapes: \u0041, non-ASCII escapes, and documented surrogate-pair behavior
• Invalid escapes: \x, incomplete \u, non-hex \u, trailing backslash, unterminated escaped strings
• Escaped strings in top-level Str, record fields, list elements, tuple elements, dict keys, skipped unknown fields, and tag names
• Encoder output for quotes, backslashes, control characters, Unicode text, and nested values
• Parse-after-encode round trips for string-heavy records/lists/tuples/dicts/tag payloads

Snapshots are useful here because Json behavior is user-visible and golden output diffs make parser/encoder changes easy to review.

Also add or extend roc test fixtures where semantic assertions are clearer than textual REPL output:

• test/cli/JsonEncodeRoundTrip.roc for parse/encode round trips
• A new focused fixture such as test/cli/JsonStringEscapeEdgeCases.roc if the existing file would become too large

Extend the native Json decoder platform path with at least one escaped-string case:

• Known fields containing escaped article/body/bio-style text
• Unknown skipped fields containing escaped strings, nested escaped objects, and escaped array values
• This keeps coverage for the compiled platform/no-runtime-allocation path in addition to interpreter/REPL snapshot coverage

Real-World Payload Shape

Include small representative payloads that look like API responses rather than minimal unit cases:

• Article/body text with escaped newlines
• Bios or descriptions with embedded quotes
• URLs with escaped slashes if supported
• Unicode names or text
• Nested arrays/objects
• Optional missing fields
• Unknown fields that must be skipped

These should stay small and readable; the goal is to catch common real JSON payload features, not to import a large external corpus.

Acceptance Criteria

• Valid escaped JSON strings parse successfully and decode to the expected Roc Str.
• Invalid JSON string forms return Err(Json.invalid_json).
• Json.to_str emits valid JSON for strings containing quotes, backslashes, control characters, and Unicode text.
• Json.parse(Json.to_str(value)) round-trips for supported string-heavy shapes.
• Escaped strings work in object keys, dict keys, skipped unknown fields, tag names, and nested values.
• Golden snapshot coverage exists under test/snapshots/repl/.
• Semantic roc test coverage exists for round-trip behavior.
• The test/json-decoder platform path includes at least one escaped real-world-style payload.