Skip to content

[VL] Local files node variant fallback fix#12074

Draft
felipepessoto wants to merge 2 commits into
apache:mainfrom
felipepessoto:LocalFilesNode_variant_fallback_fix
Draft

[VL] Local files node variant fallback fix#12074
felipepessoto wants to merge 2 commits into
apache:mainfrom
felipepessoto:LocalFilesNode_variant_fallback_fix

Conversation

@felipepessoto
Copy link
Copy Markdown
Contributor

What changes are proposed in this pull request?

Reading the Change Data Feed (CDF) of a Delta table that contains a VARIANT column
fails under the Velox native engine when the variant column is not projected:

spark.sql("""CREATE TABLE tbl (id INT, data VARIANT) USING delta
             TBLPROPERTIES ('delta.enableChangeDataFeed' = 'true')""")
spark.sql("""INSERT INTO tbl
             SELECT 1, parse_json('{"status":"active"}') UNION ALL
             SELECT 2, parse_json('{"status":"active"}')""")

// Works (variant projected → existing variant validation falls back to vanilla Spark)
spark.read.format("delta").option("readChangeFeed", "true").option("startingVersion", 0)
  .table("tbl").select("id", "data", "_change_type").show()

// Fails with GlutenNotSupportException: Type VariantType not supported
spark.read.format("delta").option("readChangeFeed", "true").option("startingVersion", 0)
  .table("tbl").select("id", "_change_type").show()
org.apache.gluten.exception.GlutenNotSupportException: Type VariantType not supported.
   at org.apache.gluten.expression.ConverterUtils$.getTypeNode(ConverterUtils.scala:248)
   at org.apache.gluten.substrait.rel.LocalFilesNode.buildNamedStruct(LocalFilesNode.java:115)
   at org.apache.gluten.substrait.rel.LocalFilesNode.toProtobuf(LocalFilesNode.java:199)
   at org.apache.gluten.backendsapi.velox.VeloxIteratorApi
        .$anonfun$genFirstStageIterator$3(VeloxIteratorApi.scala:224)
   ...

How was this patch tested?

New unit tests. Also tested with Delta unit tests.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Github Copilot CLI v1.0

This suite tests the `LocalFilesNode.containsUnsupportedFileSchemaType` method for various data types, ensuring correct identification of unsupported types like 'variant'. It includes tests for primitive types, nested structures, arrays, and maps.
@github-actions github-actions Bot added the CORE works for Gluten Core label May 12, 2026
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@felipepessoto felipepessoto marked this pull request as draft May 12, 2026 06:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CORE works for Gluten Core

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant