iceberg/conversion: refactor JSON schema to constraint-based deduction #29139
+468
−223
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nvartolomei
force-pushed
the
nv/iceberg-json-schema
branch
2 times, most recently
from
6c57ad1 to
708ad06
Compare
nvartolomei
force-pushed
the
nv/iceberg-json-schema
branch
from
708ad06 to
668387c
Compare
nvartolomei
changed the title
nvartolomei
force-pushed
the
nv/iceberg-json-schema
branch
2 times, most recently
from
c15f9f1 to
98ea1dc
Compare
nvartolomei
requested review from
andrwng,
Copilot,
oleiman,
rockwotj and
wdberkeley
and removed request for
oleiman
Contributor
There was a problem hiding this comment.
Pull request overview
This PR refactors the JSON schema to Iceberg type conversion system from a direct conversion approach to a constraint-based deduction model. The new approach collects constraints from JSON schema (allowed types, formats, properties, items) and then resolves them to Iceberg types, enabling better handling of type unions and nullable types.
Changes:
- Refactored the core conversion logic from direct type mapping to a two-phase collect-then-resolve model
- Added support for nullable type unions (e.g.,
["null", "integer"]) that previously failed - Updated error messages to be more descriptive when type constraints are ambiguous
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
tests/rptest/tests/datalake/datalake_e2e_test.py |
Added test case for array items with nullable types |
src/v/iceberg/conversion/tests/iceberg_json_tests.cc |
Updated error messages and added tests for type unions, nullable unions, and format handling |
src/v/iceberg/conversion/json_schema/ir.h |
Added explicit enum values and validation for json_value_type to support bitset operations |
src/v/iceberg/conversion/ir_json.cc |
Completely refactored from direct conversion to constraint-based deduction model |
src/v/iceberg/conversion/BUILD |
Added dependency on base module for formatting utilities |
Refactor the JSON Schema to Iceberg type conversion to use a two-phase constraint solver architecture instead of the previous mixed traversal/conversion approach. Phase 1 (collect): Traverses the JSON Schema and builds a constraint structure with a bitfield tracking possible types, format annotations, and nested constraints for objects/arrays. Phase 2 (resolve): Resolves constraints to Iceberg types, rejecting schemas where type cannot be unambiguously determined. This design provides a cleaner separation of concerns and is more extensible for future JSON Schema keywords (allOf, anyOf, oneOf, $ref). No functional changes intended.
nvartolomei
force-pushed
the
nv/iceberg-json-schema
branch
from
98ea1dc to
5d90791
Compare
Collaborator
CI test resultstest results on build#78967
|
rockwotj
approved these changes
Jan 14, 2026
Contributor
rockwotj
left a comment
rockwotj
left a comment
There was a problem hiding this comment.
This is really good and clean. I really like the constraint based approach!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backports Required
Release Notes