Scope trailing delete policy to blobs prefix only
#190
Conversation
| filter { | ||
| # We only want to expire objects with the `blobs/` prefix, i.e. Asset Blobs. | ||
| # Other objects in this bucket are not subject to this lifecycle policy. | ||
| prefix = "blobs/" |
There was a problem hiding this comment.
could it be multiple? IIRC there was some bug and we were breeding versions for dandiset manifest exports etc. And also then we keep updating manifests for draft version, and all those could be GCed as well. Eg. for 000003 alone for draft version we have over 200 in the history of those keys
(3) dandi@drogon:~/proj/datalad/datalad-deprecated$ datalad ls -aL s3://dandiarchive/dandisets/000003/draft/assets.yaml | grep DeleteMarker | wc -l
[INFO] S3 session: Connecting to the bucket dandiarchive with authentication
263
so what about trailing delete also for dandisets/?
There was a problem hiding this comment.
The asset trailing delete mechanism requires the dandi-archive application itself to put a delete marker on objects that are ready to be garbage collected. For manifest files, garbage collection is presumably more straightforward and can likely be done with just a single s3 bucket lifecycle rule - to keep this PR scoped to its original purpose, I've opened an issue to discuss that further #192.
Addresses #186 (comment).
Currently, the trailing delete lifecycle policy applies to the entire bucket. This PR updates it to only apply to the
blobs/prefix, i.e. the prefix where asset blobs are stored. With this change, other objects in the bucket such as Zarr data and manifest files will not be impacted by the lifecycle policy.