dl/coordinator: periodically remove snapshots #24813
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
CI test resultstest results on build#60759
test results on build#60885
test results on build#60999
|
dotnwat
reviewed
Jan 17, 2025
andrwng
force-pushed
the
dl-remove-snapshots
branch
from
f2a4c4b to
e1c059c
Compare
dotnwat
reviewed
Jan 18, 2025
Adds a method that takes as inputs a list of files to delete, and then calls into the cloud_io::remote to delete them. This serves primarily to translate std::filesystem::paths (which we may translate from Iceberg URIs) into cloud_storage_clients::object_keys (used by cloud_io). This will be used by the datalake coordinator to remove expired snapshots.
Introduces a new abstraction to remove old snapshots from a given table. This effectively reduces the window with which users can perform point-in-time queries. It is intended that this operation is standalone and doesn't require additional changes to the coordinator STM to run. This commit only introduces the remover and an no-op implementation for ease of testing, similar to the file_committer abstraction. A subsequent change will add this to the coordinator.
Adds snapshot removal to the coordinator loop. It is currently run before any appends.
andrwng
force-pushed
the
dl-remove-snapshots
branch
from
e1c059c to
1bba0de
Compare
dotnwat
approved these changes
Jan 22, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds periodic removal of Iceberg snapshots to the coordinator. As is, the removal runs at the same cadence as table commits, though that may change in the future.
Once snapshots are removed from the table metadata, the coordinator examines the resulting table and extracts any snapshots that no longer exist. These snapshots are deleted from object storage synchronously (it's left as future work to do this in the background, or to delay removal if desired).
The code uses a similar structure to the file committer, and as such will need to be updated later to support dead letter tables.
Backports Required
Release Notes
Improvements