Drop transient object tags when publishing files to S3#7344
Open
pditommaso wants to merge 1 commit into
Open
Conversation
When copying an S3 object bucket-to-bucket (e.g. publishDir), the copy request only set the REPLACE tagging directive when the target had explicit tags. With no tags the directive defaulted to the S3 COPY behaviour, so the destination inherited the source object's tags. This propagated transient tags such as Fusion's nextflow.io/temporary=true to published outputs, causing retention policies on the output directory to delete permanent results. Always use the REPLACE tagging directive so the destination gets exactly the requested tag set (empty = no tags) instead of inheriting source tags. Closes #7339 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
✅ Deploy Preview for nextflow-docs canceled.
|
jorgee
reviewed
Jul 16, 2026
Contributor
There was a problem hiding this comment.
I think it is the simplest case to solve the issue, and it is the correct behaviour for PublishDir. What I don't like about the solution is that it will apply to all s3-to-s3 copies done by Nextflow. Basically, it will only affect nextflow fs cp/mv commands because the file porter will not perform any s3-to-s3 copy because they are not considered foreign files.
So I see two options: we can keep the current code that removes tags infscommand and document this change; or add a new field in TagAwareFile to set the tagging directive (replace or not) and set it just in the PublishDir case.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When Nextflow copies an S3 object from one bucket to another (e.g.
publishDir), it was propagating the source object's tags to the destination.By default Fusion tags all task outputs with
nextflow.io/temporary=true(viafusion.tags) so intermediates can be reaped by a retention policy. Because these tags were carried over on publish, a retention policy on the output directory would also delete "permanent" published outputs.Root cause
S3Client.copyFile()only set theREPLACEtagging directive when the publish target had explicit tags:With no tags the directive was left unset, so the S3
CopyObjectAPI defaulted toCOPY— inheriting the source object's tags on the destination.Fix
Always set the
REPLACEtagging directive so the destination gets exactly the requested tag set (empty = no tags) rather than inheriting source tags. Extracted into a testableapplyTagginghelper.Test
Added a unit test in
S3ClientTestverifyingREPLACEis always used and tags are set only when provided.Closes #7339
🤖 Generated with Claude Code