Date-to-timestamp promoted column reads as NULL when another column is projected #3404

Related to #2411 (v3 tracking). Related: #3387 (same date-to-timestamp promotion, manifest bounds only), #3378 (same projection path, different trigger), #3374 (same projection-mask mechanism, name-mapping trigger).

Apache Iceberg Rust version

main at 9092073 (crate iceberg 0.10.1; arrow/parquet 59.3.0 per Cargo.lock). The code links below are permalinks at a766783, where the same behaviour was first reproduced.

Describe the bug

Format v3 allows promoting date to timestamp and timestamp_ns (Iceberg spec, Schema Evolution, valid primitive type promotions: https://iceberg.apache.org/spec/). When a data file stores a column as date and the table schema declares it timestamp, the read returns NULL for every row if another column is projected in the same read. Projecting only the promoted column returns the correct values.

The cause is in get_arrow_projection_mask_with_field_ids. type_promotion_is_valid (projection.rs#L99-L121) has no arm for Date to Timestamp/TimestampNs, so the Parquet leaf is left out of the projection mask (projection.rs#L208). The reader then treats the column as absent from the file, and RecordBatchTransformer fills it with NULL, or with initial-default if one is set (record_batch_transformer.rs#L818-L870). For a required column the read fails with Missing required field. Projecting only the promoted column works because no leaf matches, so the mask falls back to ProjectionMask::all() (projection.rs#L231-L234) and the cast in Promote runs.

Verified locally on a766783: adding (Some(PrimitiveType::Date), Some(PrimitiveType::Timestamp | PrimitiveType::TimestampNs)) => true to type_promotion_is_valid makes the test below pass, and cargo test -p iceberg --lib passes with that change. Tests in other crates were not run.

To Reproduce

Add this test at the end of mod tests in crates/iceberg/src/arrow/reader/projection.rs, then run:

cargo test -p iceberg --lib test_date_column_promoted_to_timestamp_with_other_column_projected

    /// A Date32 column written to Parquet must read back as the promoted timestamp type
    /// when another column is projected alongside it.
    #[tokio::test]
    async fn test_date_column_promoted_to_timestamp_with_other_column_projected() {
        use arrow_array::types::TimestampMicrosecondType;
        use arrow_array::{Date32Array, Int32Array, TimestampMicrosecondArray};

        let table_schema = Arc::new(
            Schema::builder()
                .with_schema_id(2)
                .with_fields(vec![
                    NestedField::optional(1, "d", Type::Primitive(PrimitiveType::Timestamp)).into(),
                    NestedField::optional(2, "a", Type::Primitive(PrimitiveType::Int)).into(),
                ])
                .build()
                .unwrap(),
        );

        let file_schema = Arc::new(ArrowSchema::new(vec![
            Field::new("d", DataType::Date32, true).with_metadata(HashMap::from([(
                PARQUET_FIELD_ID_META_KEY.to_string(),
                "1".to_string(),
            )])),
            Field::new("a", DataType::Int32, true).with_metadata(HashMap::from([(
                PARQUET_FIELD_ID_META_KEY.to_string(),
                "2".to_string(),
            )])),
        ]));

        let days = vec![18000, 18001, 18002];
        let to_write = RecordBatch::try_new(file_schema.clone(), vec![
            Arc::new(Date32Array::from(days.clone())) as ArrayRef,
            Arc::new(Int32Array::from(vec![1, 2, 3])) as ArrayRef,
        ])
        .unwrap();

        let tmp_dir = TempDir::new().unwrap();
        let path = format!("{}/date_file.parquet", tmp_dir.path().to_str().unwrap());
        let file = File::create(&path).unwrap();
        let mut writer = ArrowWriter::try_new(file, file_schema, None).unwrap();
        writer.write(&to_write).unwrap();
        writer.close().unwrap();

        let reader = ArrowReaderBuilder::new(FileIO::new_with_fs(), Runtime::current()).build();
        let tasks = Box::pin(futures::stream::iter(
            vec![Ok(FileScanTask::builder()
                .with_file_size_in_bytes(std::fs::metadata(&path).unwrap().len())
                .with_start(0)
                .with_length(0)
                .with_data_file_path(path.clone())
                .with_data_file_format(DataFileFormat::Parquet)
                .with_schema(table_schema)
                .with_project_field_ids(vec![1, 2])
                .with_case_sensitive(false)
                .build()
                .unwrap())]
            .into_iter(),
        )) as FileScanTaskStream;

        let batches = reader
            .read(tasks)
            .unwrap()
            .stream()
            .try_collect::<Vec<RecordBatch>>()
            .await
            .unwrap();

        let expected = TimestampMicrosecondArray::from(
            days.iter()
                .map(|d| i64::from(*d) * 86_400_000_000)
                .collect::<Vec<_>>(),
        );
        assert_eq!(batches.len(), 1);
        assert_eq!(
            batches[0].column(0).as_primitive::<TimestampMicrosecondType>(),
            &expected
        );
    }

Observed output on main 9092073 (macOS arm64):

thread 'arrow::reader::projection::tests::test_date_column_promoted_to_timestamp_with_other_column_projected' panicked at crates/iceberg/src/arrow/reader/projection.rs:2246:9:
assertion `left == right` failed
  left: PrimitiveArray<Timestamp(µs)>
[
  null,
  null,
  null,
]
 right: PrimitiveArray<Timestamp(µs)>
[
  2019-04-14T00:00:00,
  2019-04-15T00:00:00,
  2019-04-16T00:00:00,
]

Probe results on a766783 (3-row file, d is date, a is int; these probes are not part of the repro):

Out-of-range values (separate defect)

The spec says values outside the promoted type's range "must result in a runtime failure". The read currently returns NULL instead. This is a separate defect in the Promote cast, and the mask fix above does not change it. With a 2-row file holding date values 18000 and 200000 days since epoch, d as optional timestamp_ns, and only d projected:

[
  2019-04-14T00:00:00,
  null,
]

With d and a projected, both rows are NULL. A required timestamp_ns column with the value 200000 days fails with Arrow Schema Error ... Column 'd' is declared as non-nullable but contains null values. The same value in an optional timestamp (µs) column reads correctly as 2517-08-01T00:00:00.

The NULL comes from the Promote cast at record_batch_transformer.rs#L936, cast(&*columns[*source_index], target_type). That call uses arrow's default cast options, which are safe (safe: true in arrow-cast 59.3.0), so an overflowing value becomes NULL. 200000 days in nanoseconds is about 1.7e19, which is larger than i64::MAX.

Expected behavior

• Promoted values read back as the target type (days × 86,400,000,000 µs for timestamp, days × 86,400,000,000,000 ns for timestamp_ns), whether or not other columns are projected, as the single-column case already does.
• A value outside the target type's range fails the read, as the spec requires.
• A required promoted column is read, not reported as missing.
• initial-default is not used when the file contains the column.

Willingness to contribute

I would be willing to contribute a fix with guidance from the Iceberg community (any contribution would be agent-assisted and I would disclose that).

Found during an agent-assisted audit of v3 support (see #2411). The reproduction above was run against main at 9092073.