Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nextflow eagerly terminates Azure Batch jobs during execution #5839

Open
adamrtalbot opened this issue Mar 3, 2025 · 0 comments
Open

Nextflow eagerly terminates Azure Batch jobs during execution #5839

adamrtalbot opened this issue Mar 3, 2025 · 0 comments

Comments

@adamrtalbot
Copy link
Collaborator

New feature

Based on comment: #3927 (comment)

Azure Batch jobs are terminated during pipeline closing down, see #3927.

During that PR, we discussed eagerly changing the setting of each job to onAllTasksComplete = 'terminate' but never looked further into it.

Use case

The main benefit of doing it this way is that if Nextflow fails to close properly (i.e. is killed), most active jobs will be marked as to be terminated and be removed from the active job quota.

Suggested implementation

We just need an "onProcessClose" event to notify when all tasks are created. I've already added it in a number of PRs, like this one: #3818

The suggestion from the first PR which could be implemented ☝

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants