Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Job prioritisation mode #254

Open
bosterholz opened this issue Nov 15, 2022 · 3 comments
Open

Job prioritisation mode #254

bosterholz opened this issue Nov 15, 2022 · 3 comments
Labels
enhancement New feature or request

Comments

@bosterholz
Copy link
Collaborator

While running multiple metagenomes it happens easily, that lots of tasks of a module flood the slurm queue and therefore "block" other modules, as they don't get process time. To give a better example, I want to run hundreds of metagenomes and I am mainly looking for MAGs, their quality and annotation. But I would also like to get everything else to look at at a later point in time. If I start everything together (for example) the plasmid module jobs could get a head start, as they just need bins to run. If so the plasmid jobs will flood the queue and block everything else until this module completes.

It would be nice if we could use slurms nice flag:

--nice[=adjustment]
    Run the job with an adjusted scheduling priority within Slurm. With no adjustment value the scheduling priority is decreased by 100. A negative nice value increases the priority, otherwise decreases it. The adjustment range is +/- 2147483645. Only privileged users can specify a negative adjustment. 

to give prioritization for jobs in the queue and Nextflows maxForks, to make sure that not only one type of job fills the whole queue.
Using these two we could affect the order of execution, so that modules that are particularly interesting for a user at a given time are executed faster than others.

@bosterholz bosterholz added the enhancement New feature or request label Nov 15, 2022
@pbelmann
Copy link
Member

pbelmann commented Nov 15, 2022

Wouldn't be the --nice setting enough for this use case? Why do you also want to modify maxForks?

@bosterholz
Copy link
Collaborator Author

It looks like there is a finite queue size of jobs that Nextflow sends at a given time or Slurm allows to send.
If that is the case only using the --nice flag would be not enough, as for example 400 low priority jobs could just fill the whole queue and leave no room for high priority ones. Therefore the idea to use maxForks to give every job the chance to make it into the queue.

@pbelmann
Copy link
Member

I understand, but the issue with maxForks is that once the high priority jobs are done, all remaining ones will still be restricted by maxForks and thereby not efficiently using the underlying infrastructure.
Why not instead increasing the Nextflow queue size? We could add it as a user parameter.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants