Do not submit queued tasks after the session has aborted

[Mohit-Ak]

Bug report

Once a session aborts, submissions already sitting in ParallelPollingMonitor's
throttled executor queue still go through to the backend. The resulting jobs are
never killed and never polled — the shutdown kill sweep has already run and the
task monitor has already exited — so they run to completion in the backend with
nothing watching them. On a cloud batch service that is paid compute belonging to
a run that is already dead.

TaskPollingMonitor.submitPendingTasks() does guard the loop:

while( itr.hasNext() && session.canSubmitTasks() ) {

but ParallelPollingMonitor overrides submit() so that it only enqueues a
ThrottlingExecutor.Recoverable wrapper and returns immediately. By the time a
pool thread dequeues that wrapper and calls submit0(handler) — which runs
prepareLauncher(), handler.submit(), runningQueue.add(handler) and
notifyTaskSubmit(handler) — the session state is never re-read. The guard is
therefore an enqueue-time check on work that executes later, and everything
already queued when the abort lands is submitted regardless.

What this changes

One check inside the wrapper, so a queued submission becomes a no-op if the
session can no longer accept submissions by the time it actually runs:

@Override protected Object invoke() {
    if( !session.canSubmitTasks() ) {
        log.debug "Skipping task submission -- session no longer accepting submissions > $handler"
        return null
    }
    return submit0(handler)
}

This closes the queued-wrapper window, which is the bulk of the exposure. A
submission that is already inside submit0() when the abort lands can still
race through; closing that residue needs the kill sweep to cover late
runningQueue additions, which is a larger change and felt out of scope here —
happy to look at it separately if you want it folded in.

The debug line is deliberate: the silent bypass is a good part of what makes
orphaned jobs hard to track down after the fact.

How this was tested

Two tests added to ParallelPollingMonitorTest, one per side of the branch:

• should not submit a queued task once the session can no longer submit —
  stubs canSubmitTasks() >> false and asserts prepareLauncher(),
  submit() and notifyTaskSubmit() are never invoked.
• should submit a queued task while the session can still submit — the
  control; stubs canSubmitTasks() >> true and asserts all three are
  invoked exactly once, so the first test can't pass by simply breaking
  submission.

Against unpatched master the first test fails with
TooManyInvocationsError: 0 * handler.prepareLauncher() (1 invocation), which is
the bug stated as an assertion, while the control passes. With the change both
pass.

One existing test needed a stub added. should retry task builds its session
with Mock(Session) and never stubs canSubmitTasks(); Spock returns false
for unstubbed booleans, so under the new check that scenario would skip
submission and the retry counters would stay at zero. That test is about the
submit operation failing and being retried, not about an aborted session, so
_ * session.canSubmitTasks() >> true makes the intended precondition explicit.
No assertion was weakened.

./gradlew :nextflow:test --tests "nextflow.processor.ParallelPollingMonitorTest" \
                         --tests "nextflow.processor.TaskPollingMonitorTest"
  ParallelPollingMonitorTest   9 tests, 0 failures
  TaskPollingMonitorTest      26 tests, 0 failures

./gradlew :nextflow:test
  304 test classes, 3415 tests, 0 failures, 0 errors, 89 skipped

AwsBatchExecutor is the only current consumer of ParallelPollingMonitor, so
AWS Batch is where this is observable today, but the fix sits in core and covers
any future executor that adopts async submission.

Fixes #7445

[netlify]

✅ Deploy Preview for nextflow-docs canceled.

Name	Link
🔨 Latest commit	adfd2ac
🔍 Latest deploy log	https://app.netlify.com/projects/nextflow-docs/deploys/6a800539b7cc0b0008741062