Skip to content

[CH]support complex type in parquetv3#12079

Open
zhanglistar wants to merge 3 commits into
apache:mainfrom
zhanglistar:feat/support-complex-type-in-parquetv3
Open

[CH]support complex type in parquetv3#12079
zhanglistar wants to merge 3 commits into
apache:mainfrom
zhanglistar:feat/support-complex-type-in-parquetv3

Conversation

@zhanglistar
Copy link
Copy Markdown
Contributor

@zhanglistar zhanglistar commented May 12, 2026

What changes are proposed in this pull request?

Depends on Kyligence/ClickHouse#523.

This PR fixes several ClickHouse backend failures around nullable complex types, especially arrays, structs, and nested null values.

The changes include:

  • Preserve nullable semantics when parsing nested Substrait LIST types, so nested arrays can correctly represent values such as Array(Nullable(Array(...))).
  • Avoid inserting Field::Null into non-nullable ClickHouse complex columns during Spark row conversion and constant column creation.
  • Preserve source nullability when casting nullable columns and when aligning the final output schema.
  • Fix Parquet reading for nullable complex columns such as Nullable(Tuple(...)), including reconstructing struct-level nullability from nullable child fields.

These fixes prevent errors such as Bad get: has Null, requested Array and Cannot convert NULL value to non-Nullable type, and also restore correct semantics for array lambda operations over nullable structs.

Related to #2340.

How was this patch tested?

Tested with affected ClickHouse backend function validation cases, including:

Also verified C++ compilation for the touched objects, including parser and Parquet reader components.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: GPT-5.5

@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant