Describe the bug
When a non-ephemeral self-hosted runner is dispatched a second job while it is already running one, JobDispatcher cancels the in-flight job to take the new one. The cancelled job is reported to the service as cancelled — its work is simply lost, the workflow run is marked cancelled, and any required status check that depends on it fails.
The mechanism is the same one described in #4357, but the damage here is different. #4357 covers the aftermath — the cancelled worker's TempDirectoryManager cleanup wiping the shared _temp directory and breaking the new job. We never reach that; our problem is that the old job is destroyed. Filing separately because a fix scoped to the temp-directory cleanup would leave this failure mode in place, and because a job being silently killed mid-execution is the more damaging half.
The runner logs the decision explicitly:
[ERR JobDispatcher] We are not yet checking the state of jobrequest <A> status. Cancel running worker right away.
[INFO JobDispatcher] Send job cancellation message to worker for job <A>.
To Reproduce
Requires two jobs to be dispatched to the same non-ephemeral self-hosted runner in quick succession. Both of our occurrences happened moments after the runner completed a previous job, so the window appears to be around the busy/idle transition.
- Register several non-ephemeral self-hosted runners sharing one label set
- Trigger a workflow with enough concurrent jobs to saturate them
- Observe a runner receiving a second job assignment while a job is already running
Observed — two occurrences, same runner, same day
A. The victim job never executes at all. The runner finished one matrix leg, was assigned the second, and was assigned a third job two seconds later:
[2026-09-03 18:34:29Z INFO Terminal] Job build-and-test (sqlite) completed with result: Succeeded
[2026-09-03 18:34:32Z INFO JobDispatcher] Job request 0 for plan 79b62e68 job fd42305a received.
[2026-09-03 18:34:32Z INFO JobDispatcher] Pull OrchestrationId 79b62e68.build-and-test-matrix.postgresql from JWT claims
[2026-09-03 18:34:32Z INFO Terminal] Running job: build-and-test (postgresql)
[2026-09-03 18:34:32Z INFO JobDispatcher] Successfully renew job fd42305a, job is valid till 9/3/2026 6:44:32 PM
[2026-09-03 18:34:34Z INFO JobDispatcher] Job request 0 for plan 79b62e68 job 029e843d received.
[2026-09-03 18:34:34Z INFO JobDispatcher] Pull OrchestrationId 79b62e68.build-and-test-matrix.sqlite from JWT claims
[2026-09-03 18:34:34Z ERR JobDispatcher] We are not yet checking the state of jobrequest fd42305a status. Cancel running worker right away.
[2026-09-03 18:34:34Z INFO JobDispatcher] Send job cancellation message to worker for job fd42305a.
Note the lease was renewed successfully two seconds before the cancel — the job was healthy. In the API, every step of that job reports skipped while the job reports cancelled, which makes it look like a workflow-level skip rather than a runner action:
1. Set up job: success
2. actions/checkout: skipped
...
9. Unit + integration tests: skipped
11. Upload test results: success (always())
13. Complete job: success
B. Two minutes of real work discarded. A container build was killed mid .NET AOT Generating native code:
[2026-09-03 19:01:18Z INFO Terminal] Job aot-publish completed with result: Succeeded
[2026-09-03 19:01:20Z INFO JobDispatcher] Job request 0 for plan 97ccc75b job 6c2d17a5 received.
[2026-09-03 19:01:20Z INFO JobDispatcher] Pull OrchestrationId 97ccc75b.container-build.__default from JWT claims
[2026-09-03 19:01:20Z INFO Terminal] Running job: Container build
[2026-09-03 19:03:23Z INFO JobDispatcher] Job request 0 for plan 97ccc75b job fa23b5e0 received.
[2026-09-03 19:03:23Z INFO JobDispatcher] Pull OrchestrationId 97ccc75b.build-and-test-matrix.postgresql from JWT claims
[2026-09-03 19:03:23Z ERR JobDispatcher] We are not yet checking the state of jobrequest 6c2d17a5 status. Cancel running worker right away.
[2026-09-03 19:03:23Z INFO JobDispatcher] Send job cancellation message to worker for job 6c2d17a5.
[2026-09-03 19:03:41Z INFO Terminal] Job Container build completed with result: Canceled
[2026-09-03 19:03:41Z INFO Terminal] Running job: build-and-test (postgresql)
The job step log ends with ##[error]The operation was canceled. and no error of its own.
Both runs re-ran green with no code change, on different runners.
Expected behavior
A runner that is already executing a job should not destroy it to accept another. Either the second assignment is rejected/returned to the queue so the service can place it elsewhere, or it is queued locally until the running job finishes. A job that has been successfully leased and renewed should not be cancellable by the arrival of an unrelated assignment.
Failing that, the outcome should at least be distinguishable from a user cancellation — as it stands, cancelled with all steps skipped gives no signal that the runner discarded the job, and the only way we found the cause was by reading _diag on the host.
Frequency
Not a one-off. Counting the error line across the _diag logs on one host with three non-ephemeral runners:
Seven events in two clusters — five on 2026-08-19 (21:23, 21:39, 21:48 on r02; 21:39, 21:48 on r03) and two on 2026-09-03 (18:34, 19:03, both r02). It affects more than one runner, so it does not look host- or runner-instance-specific.
Runner Version and Platform
Runner v2.337.0 (current latest at time of writing), Linux x64, Fedora Linux 44, kernel 7.1.4. Three non-ephemeral repo-level runners on one 12-core / 22 GB host, sharing the labels self-hosted, Linux, X64.
What's not the cause
Ruled out while diagnosing, in case it saves someone the same detour: not memory pressure (22 GB with 8 GB zram, 3 GB in use, no OOM kills), not disk, not network (the lease renewed seconds before each cancel), not two runners racing for one job (only the one runner's log ever references the cancelled job id), and not a duplicate runner registration (all four listeners on the host have distinct agentIds).
Public run links
Describe the bug
When a non-ephemeral self-hosted runner is dispatched a second job while it is already running one,
JobDispatchercancels the in-flight job to take the new one. The cancelled job is reported to the service ascancelled— its work is simply lost, the workflow run is markedcancelled, and any required status check that depends on it fails.The mechanism is the same one described in #4357, but the damage here is different. #4357 covers the aftermath — the cancelled worker's
TempDirectoryManagercleanup wiping the shared_tempdirectory and breaking the new job. We never reach that; our problem is that the old job is destroyed. Filing separately because a fix scoped to the temp-directory cleanup would leave this failure mode in place, and because a job being silently killed mid-execution is the more damaging half.The runner logs the decision explicitly:
To Reproduce
Requires two jobs to be dispatched to the same non-ephemeral self-hosted runner in quick succession. Both of our occurrences happened moments after the runner completed a previous job, so the window appears to be around the busy/idle transition.
Observed — two occurrences, same runner, same day
A. The victim job never executes at all. The runner finished one matrix leg, was assigned the second, and was assigned a third job two seconds later:
Note the lease was renewed successfully two seconds before the cancel — the job was healthy. In the API, every step of that job reports
skippedwhile the job reportscancelled, which makes it look like a workflow-level skip rather than a runner action:B. Two minutes of real work discarded. A container build was killed mid .NET AOT
Generating native code:The job step log ends with
##[error]The operation was canceled.and no error of its own.Both runs re-ran green with no code change, on different runners.
Expected behavior
A runner that is already executing a job should not destroy it to accept another. Either the second assignment is rejected/returned to the queue so the service can place it elsewhere, or it is queued locally until the running job finishes. A job that has been successfully leased and renewed should not be cancellable by the arrival of an unrelated assignment.
Failing that, the outcome should at least be distinguishable from a user cancellation — as it stands,
cancelledwith all stepsskippedgives no signal that the runner discarded the job, and the only way we found the cause was by reading_diagon the host.Frequency
Not a one-off. Counting the error line across the
_diaglogs on one host with three non-ephemeral runners:Seven events in two clusters — five on 2026-08-19 (21:23, 21:39, 21:48 on r02; 21:39, 21:48 on r03) and two on 2026-09-03 (18:34, 19:03, both r02). It affects more than one runner, so it does not look host- or runner-instance-specific.
Runner Version and Platform
Runner v2.337.0 (current latest at time of writing), Linux x64, Fedora Linux 44, kernel 7.1.4. Three non-ephemeral repo-level runners on one 12-core / 22 GB host, sharing the labels
self-hosted, Linux, X64.What's not the cause
Ruled out while diagnosing, in case it saves someone the same detour: not memory pressure (22 GB with 8 GB zram, 3 GB in use, no OOM kills), not disk, not network (the lease renewed seconds before each cancel), not two runners racing for one job (only the one runner's log ever references the cancelled job id), and not a duplicate runner registration (all four listeners on the host have distinct
agentIds).Public run links