You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Renamed columns can return swapped values when projection order differs from file order #3475
Reading older Parquet files after a column rename can silently return values under the wrong columns when the projection order differs from the file order and the corresponding column types and nullability match.
For example, an orders table contains paid_amount and refund_amount, both nullable DECIMAL(18, 2). After renaming paid_amount to payment_amount, requesting [refund_amount, payment_amount] can return the payment amount as the refund amount and vice versa.
The Parquet reader returns selected columns in file order. RecordBatchTransformer can mistake the required reordering for a name-only change because its schema comparison does not check field IDs.
To Reproduce
Write a Parquet file with these fields in physical order:
paid_amount: field ID 1, nullable DECIMAL(18, 2), value 100.00.
refund_amount: field ID 2, nullable DECIMAL(18, 2), value 20.00.
Construct an Iceberg scan schema with payment_amount (ID 1) and refund_amount (ID 2), preserving their types and nullability. Leave the Parquet file unchanged.
Pass this schema to FileScanTask via .with_schema(...), set .with_project_field_ids(vec![2, 1]), and read through ArrowReader.
The output columns are [refund_amount, payment_amount], but the values remain [100.00, 20.00].
Expected behavior
The output values should follow the requested field IDs regardless of column renames or physical column order.
Activity