Environment
deltalake 0.32.4 (latest release), deltalake-aws 0.15.0object_store 0.13.2 (pinned transitively)- AWS S3
What happens
VacuumBuilder fails partway through a large vacuum:
Delta table error: Failed to read delta log object: Generic S3 error:
Got invalid DeleteObjects response: unknown variant `Code`, expected `Deleted` or `Error`
(The Failed to read delta log object prefix is just the text on DeltaTableError::ObjectStore; nothing is being read — this is the delete path.)
The whole maintenance run aborts, so nothing is reclaimed.
Root cause
Two changes compose into this:
1. object_store 0.13 routes all S3 deletes through the bulk API.
0.13 added a delete_stream override to PrefixStore that forwards to the inner store (src/prefix.rs). In 0.12.x that override did not exist, so PrefixStore used the trait's default implementation — individual DELETEs at .buffered(10).
delta-rs vacuum deletes via the prefix-wrapped store:
// deltalake-core/src/operations/vacuum.rs:517
let files_deleted = store
.object_store(Some(operation_id))
.delete_stream(locations)That now reaches AmazonS3::delete_stream → .try_chunks(1_000) → bulk_delete_request → POST /?delete.
(0.13 also removed delete from the ObjectStore trait — ObjectStoreExt::delete funnels a single path through delete_stream — so single-object deletes take the same route.)
2. bulk_delete_request in 0.13.2 has no guard for S3's 200-with-error-body.
send_retry returns Ok only for 2xx, so this is not a 4xx/5xx: S3 answered HTTP 200 with a body whose first child element is <Code>, i.e. an <Error> document rather than a <DeleteResult>. object_store handles exactly this case via body_contains_error() (InternalError / SlowDown), but only when retry_error_body is set — and in 0.13.2 bulk_delete_request is the one write path that doesn't set it (copy_request and create_multipart both do).
Given this fired ~15 minutes into a delete-heavy run doing up to 20 concurrent × 1000-key batches, S3 throttling (SlowDown) is the likely body — though I can't confirm the exact code, since the parse error doesn't carry it.
This is already fixed upstream
Both in object_store 0.14.0 (2026-06-18):
The ask
Bump object_store to 0.14 in the workspace. main currently pins:
object_store = { version = "0.13.2" }There is no 0.13.x backport (0.13.2 → 0.14.0 directly), and ^0.13.2 excludes 0.14, so downstream users cannot pick the fix up by overriding the dependency. Nor can it be worked around downstream:
retry_error_body is pub(crate), so it isn't reachable from a custom store.- A custom
ObjectStoreFactory wrapper that overrides delete_stream has nothing per-object to delegate to, since 0.13 removed delete from the trait — ObjectStoreExt::delete just calls delete_stream again.
So today the only options are pinning back to a delta-rs release on object_store 0.12, or carrying a patched fork.
Bumping would additionally expose aws_disable_bulk_delete through storage_options, giving users on S3-compatible backends a supported way to avoid DeleteObjects entirely.
I realise 0.14.0 is not a drop-in — it carries at least refactor!: remove cloud feature alias (#751, and deltalake-core enables object_store/cloud), Add Extensionsin*Result objects (#743), Make reqwest optional (#724) and Pluggable Crypto / Update reqwest 0.13 (#707). So this is a migration rather than a version-string change, and the reqwest 0.13 move in particular is a call for maintainers rather than a drive-by.
Happy to do the work if the bump is wanted and you can say how you'd like reqwest handled. Failing that, would a 0.13.3 backport of #733 alone be considered? It is a three-line change to bulk_delete_request and applies cleanly to the v0.13.2 tag, which would unblock every delta-rs user on the current release line without any of the 0.14 churn.
Environment
deltalake0.32.4 (latest release),deltalake-aws0.15.0object_store0.13.2 (pinned transitively)What happens
VacuumBuilderfails partway through a large vacuum:(The
Failed to read delta log objectprefix is just the text onDeltaTableError::ObjectStore; nothing is being read — this is the delete path.)The whole maintenance run aborts, so nothing is reclaimed.
Root cause
Two changes compose into this:
1.
object_store0.13 routes all S3 deletes through the bulk API.0.13 added a
delete_streamoverride toPrefixStorethat forwards to the inner store (src/prefix.rs). In 0.12.x that override did not exist, soPrefixStoreused the trait's default implementation — individualDELETEs at.buffered(10).delta-rs vacuum deletes via the prefix-wrapped store:
That now reaches
AmazonS3::delete_stream→.try_chunks(1_000)→bulk_delete_request→POST /?delete.(0.13 also removed
deletefrom theObjectStoretrait —ObjectStoreExt::deletefunnels a single path throughdelete_stream— so single-object deletes take the same route.)2.
bulk_delete_requestin 0.13.2 has no guard for S3's 200-with-error-body.send_retryreturnsOkonly for 2xx, so this is not a 4xx/5xx: S3 answered HTTP 200 with a body whose first child element is<Code>, i.e. an<Error>document rather than a<DeleteResult>.object_storehandles exactly this case viabody_contains_error()(InternalError/SlowDown), but only whenretry_error_bodyis set — and in 0.13.2bulk_delete_requestis the one write path that doesn't set it (copy_requestandcreate_multipartboth do).Given this fired ~15 minutes into a delete-heavy run doing up to 20 concurrent × 1000-key batches, S3 throttling (
SlowDown) is the likely body — though I can't confirm the exact code, since the parse error doesn't carry it.This is already fixed upstream
Both in
object_store0.14.0 (2026-06-18):fix(aws): set retry_error_body on bulk_delete_request— the fix for this exact failure.feat: add option to disable bulk delete for aws— addsaws_disable_bulk_delete/AmazonS3ConfigKey::DisableBulkDelete, issuing single-objectDELETE /keyinstead. Added for S3-compatible providers that don't implementDeleteObjects; also a useful escape hatch here.The ask
Bump
object_storeto 0.14 in the workspace.maincurrently pins:There is no 0.13.x backport (0.13.2 → 0.14.0 directly), and
^0.13.2excludes 0.14, so downstream users cannot pick the fix up by overriding the dependency. Nor can it be worked around downstream:retry_error_bodyispub(crate), so it isn't reachable from a custom store.ObjectStoreFactorywrapper that overridesdelete_streamhas nothing per-object to delegate to, since 0.13 removeddeletefrom the trait —ObjectStoreExt::deletejust callsdelete_streamagain.So today the only options are pinning back to a delta-rs release on
object_store0.12, or carrying a patched fork.Bumping would additionally expose
aws_disable_bulk_deletethroughstorage_options, giving users on S3-compatible backends a supported way to avoidDeleteObjectsentirely.I realise 0.14.0 is not a drop-in — it carries at least
refactor!: removecloudfeature alias(#751, anddeltalake-coreenablesobject_store/cloud),AddExtensionsin*Resultobjects(#743),Makereqwestoptional(#724) andPluggable Crypto / Update reqwest 0.13(#707). So this is a migration rather than a version-string change, and the reqwest 0.13 move in particular is a call for maintainers rather than a drive-by.Happy to do the work if the bump is wanted and you can say how you'd like
reqwesthandled. Failing that, would a 0.13.3 backport of #733 alone be considered? It is a three-line change tobulk_delete_requestand applies cleanly to thev0.13.2tag, which would unblock every delta-rs user on the current release line without any of the 0.14 churn.