feat: add framework for File Format Options #3794

corwinjoy · 2025-09-29T21:59:00Z

Description

This PR adds encryption support and other advanced file options to delta-rs by implementing a comprehensive framework for file format settings. The changes enable users to configure encryption settings, customize writer properties, and apply file-level formatting options when reading and writing Delta tables.

Introduces a FileFormatOptions trait and related infrastructure to handle file-specific configurations
Adds support for both simple property-based encryption and KMS-based encryption through new factory patterns
Updates all operation builders to accept and propagate file format options throughout the write/read pipeline

In general, we have added a new trait called FileFormatOptions at the root DeltaTable level to unify how files within a delta table are read and written with specific formatting. The idea is that you can apply these settings once, at the top level, and then seamlessly perform any operations with the necessary settings.

This PR leverages the DataFusion TableOptions structure to support format options for multiple underlying file formats. (The idea being that delta-rs may eventually want to support storage formats beyond Parquet, such as Vortex or Lance.) Additionally, it centralizes file format options in a single, consistent location. This avoids the current difficulties where one has to separately set WriterProperties; then reader properties as part of the SessionState. (This is in line with comments from @roeap about how file configuration might be improved: #3300 (comment)). We would also like to eventually extend this upgrade to add notations about these file configurations to the delta table properties. For example, if the files are encrypted, one could add a KMS configuration for where to retrieve encryption keys.

Review Suggestion

This PR turned out to be larger than we hoped, so apologies for that, but I don't know how to split it into smaller pieces.
When reviewing, we suggest starting with the file crates/core/src/table/file_format_options.rs to get an overview of the new file format trait that can be applied to delta tables.

Related Issue(s)

Support Parquet Modular Encryption:
#3300

Documentation

Parquet Modular Encryption: https://docs.google.com/document/d/1MUg1J7u5VdLkgejJ4ybzfZt1OmwhQkq2iGPxsn4gqLI/edit?tab=t.0#heading=h.34wvmhc1zdch

Attribution

This PR was created in collaboration with @adamreeve

Signed-off-by: Corwin Joy <[email protected]>

github-actions · 2025-09-29T21:59:24Z

ACTION NEEDED

delta-rs follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

corwinjoy · 2025-09-29T22:00:30Z

Note that fully supporting Parquet encryption requires being able to get write and read properties per-file, which is why the existing ability to set WriterProperties isn't sufficient, and why WriterPropertiesFactory::create_writer_properties is called per file and requires a file path. This allows generating new random data encryption keys per file and performing tasks such as specifying a per-file AAD prefix or supporting the external storage of encryption keys that can be looked up using the file path.

crates/core/src/operations/optimize.rs

corwinjoy · 2025-09-29T22:13:35Z

@rtyler @roeap @alamb Tagging you here per our previous discussion on adding encryption support to delta-rs.

Signed-off-by: Corwin Joy <[email protected]>

rtyler · 2025-09-30T13:19:23Z

I have marked this pull request as draft. This does not compile as is, I can come back to it once it is able to compile and pass unit tests

Signed-off-by: Corwin Joy <[email protected]>

corwinjoy · 2025-10-01T01:31:24Z

I have marked this pull request as draft. This does not compile as is, I can come back to it once it is able to compile and pass unit tests

@rtyler OK. It seems that when I auto-merged the main branch it introduced a build error. I have resolved this and the code is once again building and passing unit tests.

ion-elgreco

I can see the benefit but we really need to reduce the surface of change that are being introduced

crates/core/src/operations/add_feature.rs

crates/core/src/operations/delete.rs

roeap · 2025-10-01T15:26:23Z

@corwinjoy - awesome to see this come to fruition! Will find some time to give this a review hopefully tomorrow.

At first glance one quick question. Do we see a way to "bundle" the datafusion specific stuff a bit more? It's a bit hard to keep track of all the individual flags while reviewing :)

corwinjoy · 2025-10-01T23:48:14Z

@roeap

At first glance one quick question. Do we see a way to "bundle" the datafusion specific stuff a bit more? It's a bit hard to keep track of all the individual flags while reviewing :)

What we did to minimize this dependency is define an abstract FileFormatOptions trait. Everything just passes around a FileFormatRef defined as Arc<dyn FileFormatOptions>. Then, only when needed, do we grab final table options or writer properties. Furthermore, we've gated these instances of getting final details behind three function calls in file_format_options.rs:

pub fn build_writer_properties_factory_ffo(
    file_format_options: Option<FileFormatRef>,
) -> Option<Arc<dyn WriterPropertiesFactory>> {...}

pub fn to_table_parquet_options_from_ffo(
    file_format_options: Option<&FileFormatRef>,
) -> Option<TableParquetOptions> {...}

pub fn state_with_file_format_options(
    state: SessionState,
    file_format_options: Option<&FileFormatRef>,
) -> DeltaResult<SessionState> {...}

There might be some ways to refine this further, but in general we've tried to isolate and abstract these file properties where possible and not require datafusion.

corwinjoy · 2025-10-01T23:59:12Z

@roeap From a user point of view, we've tried hard to make the settings as easy as possible. This can be seen in crates/deltalake/examples/basic_operations_encryption.rs. Here, we demonstrate different kinds of operations on tables. (We have a more formal unit test at crates/core/tests/commands_with_encryption.rs). Thes code examples all look like ordinary operations; all we needed was a common function call when creating DeltaOps:

async fn ops_with_crypto(
    uri: &str,
    file_format_options: &FileFormatRef,
) -> Result<DeltaOps, DeltaTableError> {
    let prefix_uri = format!("file://{}", uri);
    let url = Url::parse(&*prefix_uri).unwrap();
    let ops = DeltaOps::try_from_uri(url).await?;
    Ok(ops.with_file_format_options(file_format_options.clone()))
}

Calling with_file_format_options is sufficient to apply the needed encryption settings for all operations.

# Conflicts: # crates/core/src/delta_datafusion/table_provider.rs # crates/core/src/operations/delete.rs # crates/core/src/operations/drop_constraints.rs # crates/core/src/operations/filesystem_check.rs # crates/core/src/operations/load.rs # crates/core/src/operations/merge/mod.rs # crates/core/src/operations/mod.rs # crates/core/src/operations/optimize.rs # crates/core/src/operations/restore.rs # crates/core/src/operations/update.rs # crates/core/src/operations/write/mod.rs # crates/core/tests/command_optimize.rs # crates/core/tests/integration_datafusion.rs

Signed-off-by: Corwin Joy <[email protected]>

# Conflicts: # crates/core/src/operations/optimize.rs

Signed-off-by: Corwin Joy <[email protected]>

ion-elgreco · 2025-10-15T08:40:53Z

@ion-elgreco OK. I have migrated these file options to the config property in DeltaTable. This definitely reduced the changes quite a bit so thanks for the suggestion! I think it looks pretty solid and look forward to your feedback when you are ready. @adamreeve

I'll do another review over the weekend

codecov · 2025-10-15T08:42:39Z

Codecov Report

❌ Patch coverage is 2.29358% with 426 lines in your changes missing coverage. Please review.
✅ Project coverage is 25.44%. Comparing base (5ac0629) to head (020d409).

Files with missing lines	Patch %	Lines
crates/core/src/table/file_format_options.rs	0.00%	118 Missing ⚠️
crates/core/src/operations/optimize.rs	0.00%	70 Missing ⚠️
crates/core/src/operations/encryption.rs	0.00%	44 Missing ⚠️
crates/core/src/operations/delete.rs	0.00%	30 Missing ⚠️
crates/core/src/operations/write/writer.rs	0.00%	30 Missing ⚠️
crates/core/src/writer/record_batch.rs	0.00%	26 Missing ⚠️
crates/core/src/delta_datafusion/table_provider.rs	0.00%	23 Missing ⚠️
crates/core/src/operations/mod.rs	15.78%	16 Missing ⚠️
crates/core/src/operations/update.rs	0.00%	14 Missing ⚠️
crates/core/src/operations/merge/mod.rs	0.00%	13 Missing ⚠️
... and 6 more

❗ There is a different number of reports uploaded between BASE (5ac0629) and HEAD (020d409). Click for more details.

HEAD has 3 uploads less than BASE

Flag BASE (5ac0629) HEAD (020d409)

8 5

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #3794       +/-   ##
===========================================
- Coverage   73.76%   25.44%   -48.33%     
===========================================
  Files         152      126       -26     
  Lines       39524    20084    -19440     
  Branches    39524    20084    -19440     
===========================================
- Hits        29156     5110    -24046     
- Misses       9044    14605     +5561     
+ Partials     1324      369      -955

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

crates/core/src/delta_datafusion/find_files.rs

crates/core/src/delta_datafusion/table_provider.rs

ion-elgreco · 2025-10-20T13:17:49Z

crates/core/src/operations/optimize.rs

    Ok((operation, metrics))
 }

+async fn get_file_decryption_properties(


Why can this return None as well?

This can return None because FileFormatOptions may be set in order to specify a particular parquet formatting. But, this may not include any encryption. FileFormatOptions is for more than just encryption.

crates/core/src/table/file_format_options.rs

ion-elgreco · 2025-10-20T13:25:57Z

crates/core/src/table/file_format_options.rs

+    file_format_options: Option<FileFormatRef>,
+) -> WriterPropertiesFactoryRef {
+    build_writer_properties_factory_ffo(file_format_options)
+        .unwrap_or_else(|| build_writer_properties_factory_default())


unwrap_or_default is more idiomatic

Not quite sure how to do that since WriterPropertiesFactoryRef is just a typedef and not a true type.

crates/core/src/table/file_format_options.rs

ion-elgreco · 2025-10-20T14:45:40Z

@corwinjoy looks already better but I think we can reduce the amount of line changes even more!

I still have to take a better look at why we need a WriterPropertiesFactory :s but maybe you can explain it shortly for me?

adamreeve · 2025-10-20T21:47:35Z

I still have to take a better look at why we need a WriterPropertiesFactory :s but maybe you can explain it shortly for me?

Corwin is busy travelling this week so might be slow to reply, but I can help answer this part. For some use cases it might be fine to have a single WriterProperties instance for all files, but there are a few reasons why you could want to generate new encryption properties per-file so need a factory:

To generate new random data encryption keys per file, as it's good security practice to limit how widely one key is used
To set a different AAD prefix per-file, which prevents attackers from being able to swap out encrypted modules between files and tamper with data
Be able to handle schema changes like adding columns while specifying per-column encryption keys
Enables use of external key material, which is when you write a JSON file alongside each Parquet file containing the key metadata. This allows rotation of master keys without having to re-write Parquet files, only the JSON files need to be rewritten. We don't implement this in the Rust parquet-key-management crate but it's supported by the Java and C++/Python Parquet implementations.

This also aligns with the DataFusion EncryptionFactory trait that takes the file path as a parameter when creating encryption and decryption properties.

Signed-off-by: Corwin Joy <[email protected]>

…s_scan functions. Signed-off-by: Corwin Joy <[email protected]>

Signed-off-by: Corwin Joy <[email protected]>

…Builder Signed-off-by: Corwin Joy <[email protected]>

Signed-off-by: Corwin Joy <[email protected]>

# Conflicts: # Cargo.toml # crates/core/src/operations/delete.rs # crates/core/src/operations/optimize.rs # crates/core/src/operations/update.rs # crates/core/src/operations/write/execution.rs # crates/core/src/operations/write/writer.rs # crates/core/src/table/builder.rs

Signed-off-by: Corwin Joy <[email protected]>

…ies. Signed-off-by: Corwin Joy <[email protected]>

corwinjoy · 2025-10-30T03:37:18Z

@ion-elgreco OK. I have applied the changes you suggested from two weeks ago and re-merged the latest from main. So, I am ready for another review when you get the chance. Thanks again for the suggestions!

ion-elgreco · 2025-11-01T09:29:19Z

crates/core/src/writer/record_batch.rs

+        let writer_properties_factory = build_writer_properties_factory_wp(writer_properties);
+        self.writer_properties_factory = writer_properties_factory;


I prefer the use of extension traits here, example:

pub trait IntoWriterPropertiesFactoryRef { fn into_factory_ref(self) -> WriterPropertiesFactoryRef; } impl IntoWriterPropertiesFactoryRef for WriterProperties { fn into_factory_ref(self) -> WriterPropertiesFactoryRef { Arc::new(SimpleWriterPropertiesFactory::new(self)) } }

And this can be applied across the board where we do func(a) -> b

ion-elgreco · 2025-11-01T09:33:55Z

crates/core/src/operations/write/writer.rs

 impl PartitionWriter {
    /// Create a new instance of [`PartitionWriter`] from [`PartitionWriterConfig`]
-    pub fn try_with_config(
+    pub async fn try_with_config(


Why has this been made async?

ion-elgreco · 2025-11-01T09:36:57Z

crates/core/src/table/file_format_options.rs

+// More advanced factory with KMS support
+#[cfg(feature = "datafusion")]
+#[derive(Clone, Debug)]
+pub struct KMSWriterPropertiesFactory {


I think this can be moved (all KMS stuff) to your own private crate Since we have the WriterPropetiesFactory trait

ion-elgreco · 2025-11-01T09:44:08Z

crates/core/src/table/file_format_options.rs

+
+#[cfg(feature = "datafusion")]
+#[derive(Clone, Debug, Default)]
+pub struct SimpleFileFormatOptions {


What's the purpose for this for other delta-rs users?

Update delta-rs to add FileFormatOptions

0aeb7ea

Signed-off-by: Corwin Joy <[email protected]>

corwinjoy requested review from hntd187, ion-elgreco, roeap and rtyler as code owners September 29, 2025 21:59

github-actions bot added binding/python Issues for the Python package binding/rust Issues for the Rust crate labels Sep 29, 2025

corwinjoy commented Sep 29, 2025

View reviewed changes

crates/core/src/operations/optimize.rs Show resolved Hide resolved

corwinjoy changed the title ~~feat: Add framework for File Format Options~~ feat: add framework for File Format Options Sep 29, 2025

corwinjoy added 2 commits September 29, 2025 16:21

Update optimize to use zstd compression by default. Fix clippy warnings.

c081278

Signed-off-by: Corwin Joy <[email protected]>

Merge branch 'main' into file_format_options_squashed

9059d8e

rtyler self-assigned this Sep 30, 2025

rtyler marked this pull request as draft September 30, 2025 13:18

corwinjoy added 2 commits September 30, 2025 18:05

Merge branch 'main' into file_format_options_squashed

a41d3d3

Fix build error created by merging main

cd88ef8

Signed-off-by: Corwin Joy <[email protected]>

corwinjoy marked this pull request as ready for review October 1, 2025 01:31

ion-elgreco requested changes Oct 1, 2025

View reviewed changes

crates/core/src/operations/add_feature.rs Outdated Show resolved Hide resolved

crates/core/src/operations/delete.rs Outdated Show resolved Hide resolved

corwinjoy added 5 commits October 2, 2025 13:44

Update optimize.rs to fix merge conflicts from main

56ec968

Signed-off-by: Corwin Joy <[email protected]>

cargo format

1b7d6c2

Signed-off-by: Corwin Joy <[email protected]>

Merge branch 'main' into file_format_options_squashed

4a89682

# Conflicts: # crates/core/src/operations/optimize.rs

Merge latest from main

56a0cb5

Signed-off-by: Corwin Joy <[email protected]>