Skip to content

[CORE-15023]: Cloud Storage Clients: Implement batch delete for GCS #29144

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
oleiman wants to merge 13 commits into from
Closed

[CORE-15023]: Cloud Storage Clients: Implement batch delete for GCS #29144

oleiman wants to merge 13 commits into from

Conversation

@oleiman
Copy link
Member

@oleiman oleiman commented Jan 5, 2026 •
edited
Loading

Builds on #29143

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v25.3.x
  • v25.2.x
  • v25.1.x

Release Notes

Improvements

  • Add batch delete support for GCS cloud storage clients

@oleiman oleiman self-assigned this Jan 5, 2026
@oleiman oleiman mentioned this pull request Jan 5, 2026
Closed
7 tasks
@oleiman oleiman force-pushed the ct/core-15023/multi-part-delete-gcs branch 5 times, most recently from 9140138 to 00a459a Compare January 7, 2026 21:40
@oleiman
Copy link
Member Author

oleiman commented Jan 7, 2026

/ci-repeat 1
release
tests/rptest/tests/cloud_topics/l0_gc_test.py

@oleiman
Copy link
Member Author

oleiman commented Jan 8, 2026

/cdt
provider=gcp
region=us-west2
cdt_instance_type=n2d-standard-4
tests/rptest/tests/cloud_topics/l0_gc_test.py

@oleiman
Copy link
Member Author

oleiman commented Jan 8, 2026

/cdt
rp_version=build
provider=gcp
region=us-west2
cdt_instance_type=n2d-standard-4
tests/rptest/tests/cloud_topics/l0_gc_test.py

@oleiman
Copy link
Member Author

oleiman commented Jan 8, 2026

/cdt
rp_version=pr
provider=gcp
region=us-west2
cdt_instance_type=n2d-standard-4
tests/rptest/tests/cloud_topics/l0_gc_test.py

@oleiman oleiman force-pushed the ct/core-15023/multi-part-delete-gcs branch from a244702 to 146cf30 Compare January 8, 2026 07:17
@oleiman
Copy link
Member Author

oleiman commented Jan 8, 2026

/cdt
rp_version=build
provider=gcp
region=us-west2
cdt_instance_type=n2d-standard-4
tests/rptest/tests/cloud_topics/l0_gc_test.py

@oleiman oleiman force-pushed the ct/core-15023/multi-part-delete-gcs branch from 146cf30 to 813a915 Compare January 8, 2026 18:48
@oleiman
Copy link
Member Author

oleiman commented Jan 8, 2026

/cdt
rp_version=build
provider=gcp
region=us-west2
cdt_instance_type=n2d-standard-4
tests/rptest/tests/cloud_topics/l0_gc_test.py

@oleiman oleiman force-pushed the ct/core-15023/multi-part-delete-gcs branch from 813a915 to 0ca9a5a Compare January 8, 2026 21:10
@oleiman
Copy link
Member Author

oleiman commented Jan 8, 2026

/ci-repeat 1
release
skip-rebase
skip-units
arch=amd64
tests/rptest/tests/cloud_topics/l0_gc_test.py

@oleiman
Copy link
Member Author

oleiman commented Jan 8, 2026

/cdt
rp_version=build
provider=gcp
region=us-west2
cdt_instance_type=n2d-standard-4
tests/rptest/tests/cloud_topics/l0_gc_test.py

@oleiman oleiman force-pushed the ct/core-15023/multi-part-delete-gcs branch from 0ca9a5a to a5cf5a4 Compare January 11, 2026 10:35
@oleiman
Copy link
Member Author

oleiman commented Jan 11, 2026

/cdt
rp_version=build
provider=gcp
region=us-west2
cdt_instance_type=n2d-standard-4
tests/rptest/tests/cloud_topics/l0_gc_test.py

1 similar comment
@oleiman
Copy link
Member Author

oleiman commented Jan 11, 2026

/cdt
rp_version=build
provider=gcp
region=us-west2
cdt_instance_type=n2d-standard-4
tests/rptest/tests/cloud_topics/l0_gc_test.py

@oleiman oleiman force-pushed the ct/core-15023/multi-part-delete-gcs branch from 600e6ee to aab0bcd Compare January 12, 2026 02:22
@oleiman
Copy link
Member Author

oleiman commented Jan 12, 2026

/cdt
rp_version=build
provider=gcp
region=us-west2
cdt_instance_type=n2d-standard-4
tests/rptest/tests/cloud_topics/l0_gc_test.py

2 similar comments
@oleiman
Copy link
Member Author

oleiman commented Jan 12, 2026

/cdt
rp_version=build
provider=gcp
region=us-west2
cdt_instance_type=n2d-standard-4
tests/rptest/tests/cloud_topics/l0_gc_test.py

@oleiman
Copy link
Member Author

oleiman commented Jan 12, 2026

/cdt
rp_version=build
provider=gcp
region=us-west2
cdt_instance_type=n2d-standard-4
tests/rptest/tests/cloud_topics/l0_gc_test.py

@oleiman oleiman force-pushed the ct/core-15023/multi-part-delete-gcs branch from 99758a3 to 8453fb8 Compare January 12, 2026 18:54
@oleiman
Copy link
Member Author

oleiman commented Jan 12, 2026

/cdt
rp_version=build
provider=gcp
region=us-west2
cdt_instance_type=n2d-standard-4
tests/rptest/tests/cloud_topics/l0_gc_test.py

@oleiman
Copy link
Member Author

oleiman commented Jan 12, 2026

/cdt
provider=gcp
region=us-west2
cdt_instance_type=n2d-standard-4
tests/rptest/tests/cloud_topics/l0_gc_test.py

@oleiman
http/tests: Selectively override content_type in flexible function ha…
4fd35f3 
…ndler

Signed-off-by: Oren Leiman <[email protected]>
@oleiman
s3_imposter: Wire content_type overrides through s3_imposter
69f560b 
Signed-off-by: Oren Leiman <[email protected]>
@oleiman
s3_imposter: Add support for multipart batch delete
70398ed 
Very crude request parsing, but it works well enough for existing tests.

Signed-off-by: Oren Leiman <[email protected]>
@oleiman
cs_clients/s3: Batch delete (multipart)
1e98ffd 
This is only needed for GCS, so various identifiers will reflect that.

Includes json response body parsing to get GCS-native error reasons.

Signed-off-by: Oren Leiman <[email protected]>
@oleiman
cloud_io: Enable batch deletes for GCS
7263045 
- Wire the batch request limit (100) into remote.cc
- Add switching logic in s3_client
- Update cloud_io/remote_test

Signed-off-by: Oren Leiman <[email protected]>
@oleiman
cs_clients/tests: Add gcs_client_test
ed98627 
Bit simpler than retrofitting s3_client_test.

Just flexing the batch delete code.

Signed-off-by: Oren Leiman <[email protected]>
@oleiman
[DNM] some logging
5f01bd1
@oleiman
try out GCP native style urls
3ce89e0
@oleiman oleiman force-pushed the ct/core-15023/multi-part-delete-gcs branch from 8453fb8 to 93cf120 Compare January 12, 2026 23:28
@oleiman
Copy link
Member Author

oleiman commented Jan 12, 2026

/cdt
rp_version=build
provider=gcp
region=us-west2
cdt_instance_type=n2d-standard-4
tests/rptest/tests/cloud_topics/l0_gc_test.py

1 similar comment
@oleiman
Copy link
Member Author

oleiman commented Jan 13, 2026

/cdt
rp_version=build
provider=gcp
region=us-west2
cdt_instance_type=n2d-standard-4
tests/rptest/tests/cloud_topics/l0_gc_test.py

@oleiman
strip useless headers
f686669 
stating content-type: json might cause the batch backend to expect a
generation or version specified, resulting in 404.
@oleiman
fancy extra client
0fa4bf8
@oleiman
logging
afc0c2b
@oleiman oleiman force-pushed the ct/core-15023/multi-part-delete-gcs branch from 61a56ed to 9bc5489 Compare January 13, 2026 18:58
@oleiman
Copy link
Member Author

oleiman commented Jan 13, 2026

/cdt
rp_version=build
provider=gcp
region=us-west2
cdt_instance_type=n2d-standard-4
tests/rptest/tests/cloud_topics/l0_gc_test.py

@oleiman oleiman force-pushed the ct/core-15023/multi-part-delete-gcs branch from 9bc5489 to aa577c1 Compare January 13, 2026 21:11
@oleiman
Copy link
Member Author

oleiman commented Jan 13, 2026

/cdt
rp_version=build
provider=gcp
region=us-west2
cdt_instance_type=n2d-standard-4
tests/rptest/tests/cloud_topics/l0_gc_test.py

@oleiman
Copy link
Member Author

oleiman commented Jan 13, 2026

/cdt
provider=gcp
region=us-west2
cdt_instance_type=n2d-standard-4
tests/rptest/tests/cloud_topics/l0_gc_test.py

@oleiman
Copy link
Member Author

oleiman commented Jan 14, 2026

superseded by #29246

@oleiman oleiman closed this Jan 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Awaiting requested review from Copilot Copilot will automatically review once the pull request is marked ready for review

Assignees

@oleiman oleiman

Labels

area/build area/redpanda

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@oleiman