Skip to content

Fix race condition calling workflow onComplete twice#7349

Merged
bentsherman merged 2 commits into
masterfrom
6692-fix-shutdown-called-twice
Jul 17, 2026
Merged

Fix race condition calling workflow onComplete twice#7349
bentsherman merged 2 commits into
masterfrom
6692-fix-shutdown-called-twice

Conversation

@bentsherman

Copy link
Copy Markdown
Member

Fix #6692

PR #4299 created a race condition where certain errors (e.g. a failed retry) can cause onComplete handlers (from a trace observer) to be called twice

This is because certain error conditions abort() the session directly from a separate thread instead of throwing an exception. The main thread still tears down the session via destroy(). Both methods call shutdown0() -> notfiyFlowComplete(), in this case concurrently.

The shutdown0() already has a flag to prevent adding new hooks after shutdown, so this PR simply expands the flag to also make the method idempotent. Doesn't matter for the shutdown hooks since they are drained, but it does matter for the trace observers.

Signed-off-by: Ben Sherman <bentshermann@gmail.com>
@bentsherman
bentsherman requested a review from jorgee July 16, 2026 17:20
@netlify

netlify Bot commented Jul 16, 2026

Copy link
Copy Markdown

Deploy Preview for nextflow-docs canceled.

Name Link
🔨 Latest commit de2227e
🔍 Latest deploy log https://app.netlify.com/projects/nextflow-docs/deploys/6a5a2427a3baee00089473b3

@bentsherman

Copy link
Copy Markdown
Member Author

Ideally I'd prefer the runtime to always throw an exception instead of calling session.abort(), or interacting with the session at all. But the session-runtime coupling is a harder problem to tackle

I suspect the runtime is doing this when it's on a separate thread, to ensure the exception isn't lost. If we can establish a clean pattern for propagating exceptions from threads and apply it everywhere, that should greatly reduce the dependence on the session object

For now, I think this fix is sensible given the current constraints

@bentsherman
bentsherman merged commit 14d5f26 into master Jul 17, 2026
23 checks passed
@bentsherman
bentsherman deleted the 6692-fix-shutdown-called-twice branch July 17, 2026 12:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TraceObserverV2 - onFlowComplete firing twice when a process was retried

2 participants