Skip to content

Drop transient object tags when publishing files to S3#7344

Open
pditommaso wants to merge 1 commit into
masterfrom
fix-7339-drop-temporary-tag-on-publish
Open

Drop transient object tags when publishing files to S3#7344
pditommaso wants to merge 1 commit into
masterfrom
fix-7339-drop-temporary-tag-on-publish

Conversation

@pditommaso

Copy link
Copy Markdown
Member

Problem

When Nextflow copies an S3 object from one bucket to another (e.g. publishDir), it was propagating the source object's tags to the destination.

By default Fusion tags all task outputs with nextflow.io/temporary=true (via fusion.tags) so intermediates can be reaped by a retention policy. Because these tags were carried over on publish, a retention policy on the output directory would also delete "permanent" published outputs.

Root cause

S3Client.copyFile() only set the REPLACE tagging directive when the publish target had explicit tags:

if( tags != null && !tags.isEmpty()) {
    reqBuilder.taggingDirective(TaggingDirective.REPLACE);
    reqBuilder.tagging(...);
}

With no tags the directive was left unset, so the S3 CopyObject API defaulted to COPY — inheriting the source object's tags on the destination.

Fix

Always set the REPLACE tagging directive so the destination gets exactly the requested tag set (empty = no tags) rather than inheriting source tags. Extracted into a testable applyTagging helper.

Test

Added a unit test in S3ClientTest verifying REPLACE is always used and tags are set only when provided.

Closes #7339

🤖 Generated with Claude Code

When copying an S3 object bucket-to-bucket (e.g. publishDir), the copy
request only set the REPLACE tagging directive when the target had explicit
tags. With no tags the directive defaulted to the S3 COPY behaviour, so the
destination inherited the source object's tags.

This propagated transient tags such as Fusion's nextflow.io/temporary=true
to published outputs, causing retention policies on the output directory to
delete permanent results.

Always use the REPLACE tagging directive so the destination gets exactly the
requested tag set (empty = no tags) instead of inheriting source tags.

Closes #7339

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
@netlify

netlify Bot commented Jul 16, 2026

Copy link
Copy Markdown

Deploy Preview for nextflow-docs canceled.

Name Link
🔨 Latest commit cf1fdc8
🔍 Latest deploy log https://app.netlify.com/projects/nextflow-docs/deploys/6a58878208720f0008f2b875

@pditommaso
pditommaso requested review from bentsherman and jorgee July 16, 2026 07:35

@jorgee jorgee left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is the simplest case to solve the issue, and it is the correct behaviour for PublishDir. What I don't like about the solution is that it will apply to all s3-to-s3 copies done by Nextflow. Basically, it will only affect nextflow fs cp/mv commands because the file porter will not perform any s3-to-s3 copy because they are not considered foreign files.

So I see two options: we can keep the current code that removes tags infscommand and document this change; or add a new field in TagAwareFile to set the tagging directive (replace or not) and set it just in the PublishDir case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nextflow preserves the temporary tag when publishing files

2 participants