Skip to content

Renamed columns can return swapped values when projection order differs from file order #3475

Open

Description

@lyne7-sc

Apache Iceberg Rust version

main(acd6109)

Describe the bug

Reading older Parquet files after a column rename can silently return values under the wrong columns when the projection order differs from the file order and the corresponding column types and nullability match.

For example, an orders table contains paid_amount and refund_amount, both nullable DECIMAL(18, 2). After renaming paid_amount to payment_amount, requesting [refund_amount, payment_amount] can return the payment amount as the refund amount and vice versa.

The Parquet reader returns selected columns in file order. RecordBatchTransformer can mistake the required reordering for a name-only change because its schema comparison does not check field IDs.

To Reproduce

  1. Write a Parquet file with these fields in physical order:
    • paid_amount: field ID 1, nullable DECIMAL(18, 2), value 100.00.
    • refund_amount: field ID 2, nullable DECIMAL(18, 2), value 20.00.
  2. Construct an Iceberg scan schema with payment_amount (ID 1) and refund_amount (ID 2), preserving their types and nullability. Leave the Parquet file unchanged.
  3. Pass this schema to FileScanTask via .with_schema(...), set .with_project_field_ids(vec![2, 1]), and read through ArrowReader.

The output columns are [refund_amount, payment_amount], but the values remain [100.00, 20.00].

Expected behavior

The output values should follow the requested field IDs regardless of column renames or physical column order.

Willingness to contribute

I can contribute a fix for this bug independently

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions