Describe the bug
After Runner.Listener acknowledges a runner request, the request it issues next can hang forever. The process stays alive, its TCP connection to the broker stays ESTABLISHED with both queues empty, and no traffic passes over it again. There is no client-side timeout, no retry, and no exit — the listener simply stops making progress and never notices.
The job it claimed is never started. GitHub fails it after exactly ten minutes with "The self-hosted runner lost communication with the server", and the runner is dropped from the org's runner list.
The part that makes this expensive to operate around is that the process remains healthy from the outside. A supervisor watching the process, the VM, or the exit code sees nothing wrong, so the runner is never recycled and the slot is lost until someone intervenes by hand. For ephemeral runners this is permanent: the worker never comes back on its own.
Over 24 hours on two Linux x64 hosts, 10 of 14 workers ended up in this state and stayed there. Capacity only ever went down.
To Reproduce
We have not found a way to trigger the hang itself deterministically. What reproduces reliably is the connection fault that precedes it:
- Configure an ephemeral, org-level runner (2.336.0) on Linux x64.
- While the runner is starting up and creating its broker session, drop egress traffic to the broker for about three minutes — e.g.
iptables -I FORWARD -i <bridge> -d <broker ip> -j DROP.
- The listener logs
POST https://broker.actions.githubusercontent.com/session timed out after 100 seconds, then Runner connect error: The HTTP request timed out after 00:01:40.. Retrying until reconnected.
- Restore traffic. The listener reconnects and reaches
Listening for Jobs.
Most of the time that recovers cleanly — in a controlled run of this we induced the timeout twice and both runners recovered and went on to run jobs normally. Occasionally the runner instead hangs permanently on the first job it claims after reconnecting, which is the bug above. Note that the session-creation path does have a 100 second timeout and recovers correctly because of it; the path after the job is claimed appears not to.
Expected behavior
Either of the following would be enough:
GetJobMessageAsync on the broker path is bounded by a deadline of its own, the way AcknowledgeMessageAsync bounds itself at 5 seconds and session creation bounds itself at 100. A timeout would surface as an exception, RetryRequest would retry, and the runner would either recover or fail loudly.- Or, if it cannot make progress, the runner exits non-zero so that a supervisor can recycle it and the job is requeued, instead of staying alive holding a job it will never run.
What should not happen is the current behaviour: the process stays up indefinitely, the job is silently lost, and nothing observable from outside the process indicates a problem.
Runner Version and Platform
Version of your runner? 2.336.0
OS of the machine running the runner? Linux x64 — Ubuntu 24.04.4 LTS, kernel 6.8.0. Ephemeral runners (--ephemeral), one per KVM guest, registered at org level. .runner shows useV2Flow: true and serverUrlV2: https://broker.actions.githubusercontent.com/.
What's not working?
The job annotation GitHub produces:
The self-hosted runner lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.
None of the causes it suggests apply. At the time of inspection, four hours after the runner went silent:
$ ps aux | grep Runner.Listener
ubuntu 1929 0.0 1.5 274250012 113576 ? Sl 12:30 0:02 /home/ubuntu/actions-runner/bin/Runner.Listener run
$ ss -tanp | grep Runner.Listener
ESTAB 0 0 10.x.x.x:42888 20.85.130.105:443 users:(("Runner.Listener",pid=1929,fd=150))
The process had consumed 2 seconds of CPU across four hours, and its connection to the broker was established with both send and receive queues empty. The machine itself was idle: load 0.19 on 32 cores, 47 GB of 124 GB free, no OOM kills, no hung-task warnings, and 0.011% TCP retransmissions host-wide. Nothing terminated the process, starved it, or blocked its network.
Job Log Output
There is no step output to include, and that is itself the symptom: the API reports an empty steps array for every affected job. From one run (kumahq/kuma, run 33640499530):
| Job |
Runner |
Duration |
Steps |
| test / e2e (default, v1.35.3-k3s1, amd64) |
runner-A-worker-1 |
15:08:57 → 15:18:57 |
0 |
| test / e2e (kubernetes, v1.35.3-k3s1, amd64) |
runner-A-worker-7 |
15:08:59 → 15:19:00 |
0 |
| test / e2e (universal, kindIpv6, amd64) |
runner-A-worker-4 |
15:15:23 → 15:25:24 |
0 |
| test / e2e (universal, kind, amd64) |
runner-A-worker-2 |
15:18:18 → 15:28:18 |
0 |
Four different runners, no steps recorded on any of them, and each job lasting ten minutes to the second. The same signature has since appeared in a second, unrelated organisation on the same runner version.
Runner and Worker's Diagnostic Logs
_diag/Runner_20260902-123047-utc.log. This is the complete tail of the file — nothing was written after 12:33:07, and the file was still the newest one in _diag when inspected at 16:32:46Z.
[2026-09-02 12:32:27Z WARN GitHubActionsService] POST request to https://broker.actions.githubusercontent.com/session timed out after 100 seconds.
[2026-09-02 12:32:27Z ERR BrokerMessageListener] Catch exception during create session.
[2026-09-02 12:32:27Z ERR BrokerMessageListener] System.TimeoutException: The HTTP request timed out after 00:01:40.
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at GitHub.Services.Common.RawHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at GitHub.Actions.RunService.WebApi.BrokerHttpClient.CreateSessionAsync(TaskAgentSession session, CancellationToken cancellationToken)
at GitHub.Runner.Common.BrokerServer.CreateSessionAsync(TaskAgentSession session, CancellationToken cancellationToken)
at GitHub.Runner.Listener.BrokerMessageListener.CreateSessionAsync(CancellationToken token)
[2026-09-02 12:32:27Z INFO BrokerMessageListener] Retriable exception: The HTTP request timed out after 00:01:40.
[2026-09-02 12:32:27Z ERR Terminal] WRITE ERROR: 2026-09-02 12:32:27Z: Runner connect error: The HTTP request timed out after 00:01:40.. Retrying until reconnected.
[2026-09-02 12:32:27Z INFO BrokerMessageListener] Sleeping for 30 seconds before retrying.
[2026-09-02 12:32:57Z INFO BrokerMessageListener] Attempt to create session.
[2026-09-02 12:32:57Z INFO BrokerMessageListener] Connecting to the Broker Server...
[2026-09-02 12:32:57Z INFO RunnerServer] Establish connection with 100 seconds timeout.
[2026-09-02 12:32:58Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[2026-09-02 12:32:58Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[2026-09-02 12:32:58Z INFO BrokerMessageListener] VssConnection created
[2026-09-02 12:32:59Z INFO GitHubActionsService] AAD Correlation ID for this token request: Unknown
[2026-09-02 12:33:05Z INFO BrokerMessageListener] Session created.
[2026-09-02 12:33:05Z INFO Terminal] WRITE LINE: 2026-09-02 12:33:05Z: Runner reconnected.
[2026-09-02 12:33:05Z INFO Terminal] WRITE LINE: Current runner version: '2.336.0'
[2026-09-02 12:33:05Z INFO Terminal] WRITE LINE: 2026-09-02 12:33:05Z: Listening for Jobs
[2026-09-02 12:33:05Z INFO JobDispatcher] Set runner/worker IPC timeout to 30 seconds.
[2026-09-02 12:33:05Z INFO BrokerMessageListener] Acknowledging runner request '96a86da3-af55-55ca-9217-7454efd68633'.
[2026-09-02 12:33:06Z INFO ConfigurationStore] HasCredentials()
[2026-09-02 12:33:06Z INFO ConfigurationStore] stored True
[2026-09-02 12:33:06Z INFO CredentialManager] GetCredentialProvider
[2026-09-02 12:33:06Z INFO CredentialManager] Creating type OAuth
[2026-09-02 12:33:06Z INFO CredentialManager] Creating credential type: OAuth
[2026-09-02 12:33:06Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/ubuntu/actions-runner/.credentials_rsaparams
[2026-09-02 12:33:07Z INFO GitHubActionsService] AAD Correlation ID for this token request: Unknown
The last line is 12:33:07. Runner.Worker was never spawned, so there is no worker log for this job — _diag contains only the two Runner_ files from configuration and startup.
Note the contrast within this one log: the session-creation timeout at 12:32:27 is caught, reported, and retried successfully. The token request at 12:33:07, immediately after the job was acknowledged, produces nothing further at all.
Where the unbounded call appears to be
Reading v2.336.0, the calls either side of the hang are both bounded, and the one in between is not.
src/Runner.Listener/BrokerMessageListener.cs, AcknowledgeMessageAsync (~line 353) — bounded at 5 seconds:
using var timeoutCts = new CancellationTokenSource(TimeSpan.FromSeconds(5));
using var linkedCts = CancellationTokenSource.CreateLinkedTokenSource(
cancellationToken, timeoutCts.Token);
Trace.Info($"Acknowledging runner request '{runnerRequestId}'.");
await _brokerServer.AcknowledgeRunnerRequestAsync(..., linkedCts.Token);src/Runner.Listener/Runner.cs (~line 367) — the token used for the rest of the loop is linked only to shutdown, with no deadline of its own:
CancellationTokenSource messageQueueLoopTokenSource =
CancellationTokenSource.CreateLinkedTokenSource(HostContext.RunnerShutdownToken);src/Runner.Listener/Runner.cs (~line 557), the broker path — that token is passed straight through, with no timeout wrapper:
jobRequestMessage = await runServer.GetJobMessageAsync(
messageRef.RunnerRequestId,
messageRef.BillingOwnerId,
messageQueueLoopTokenSource.Token);src/Runner.Common/RunServer.cs (~line 54) forwards it unchanged into the retry helper:
public Task<AgentJobRequestMessage> GetJobMessageAsync(string id, string billingOwnerId, CancellationToken cancellationToken)
{
CheckConnection();
return RetryRequest<AgentJobRequestMessage>(
async () => await _runServiceHttpClient.GetJobMessageAsync(requestUri, id, VarUtil.OS, billingOwnerId, cancellationToken), cancellationToken,
shouldRetry: ex => ...);
}So messageQueueLoopTokenSource.Token only fires when the runner is shutting down. If the underlying request never completes and never faults, nothing else cancels it, RetryRequest never gets an exception to retry on, and the loop has nowhere to go — which matches what we see: no further log line, no retry, no exit, four hours of an idle ESTABLISHED socket.
I have not read RetryRequest itself or the HttpClient configuration behind _runServiceHttpClient, so it is possible a timeout is applied at a level I did not check. If so, it did not take effect here.
Anything else
Describe the bug
After
Runner.Listeneracknowledges a runner request, the request it issues next can hang forever. The process stays alive, its TCP connection to the broker staysESTABLISHEDwith both queues empty, and no traffic passes over it again. There is no client-side timeout, no retry, and no exit — the listener simply stops making progress and never notices.The job it claimed is never started. GitHub fails it after exactly ten minutes with "The self-hosted runner lost communication with the server", and the runner is dropped from the org's runner list.
The part that makes this expensive to operate around is that the process remains healthy from the outside. A supervisor watching the process, the VM, or the exit code sees nothing wrong, so the runner is never recycled and the slot is lost until someone intervenes by hand. For ephemeral runners this is permanent: the worker never comes back on its own.
Over 24 hours on two Linux x64 hosts, 10 of 14 workers ended up in this state and stayed there. Capacity only ever went down.
To Reproduce
We have not found a way to trigger the hang itself deterministically. What reproduces reliably is the connection fault that precedes it:
iptables -I FORWARD -i <bridge> -d <broker ip> -j DROP.POST https://broker.actions.githubusercontent.com/session timed out after 100 seconds, thenRunner connect error: The HTTP request timed out after 00:01:40.. Retrying until reconnected.Listening for Jobs.Most of the time that recovers cleanly — in a controlled run of this we induced the timeout twice and both runners recovered and went on to run jobs normally. Occasionally the runner instead hangs permanently on the first job it claims after reconnecting, which is the bug above. Note that the session-creation path does have a 100 second timeout and recovers correctly because of it; the path after the job is claimed appears not to.
Expected behavior
Either of the following would be enough:
GetJobMessageAsyncon the broker path is bounded by a deadline of its own, the wayAcknowledgeMessageAsyncbounds itself at 5 seconds and session creation bounds itself at 100. A timeout would surface as an exception,RetryRequestwould retry, and the runner would either recover or fail loudly.What should not happen is the current behaviour: the process stays up indefinitely, the job is silently lost, and nothing observable from outside the process indicates a problem.
Runner Version and Platform
Version of your runner? 2.336.0
OS of the machine running the runner? Linux x64 — Ubuntu 24.04.4 LTS, kernel 6.8.0. Ephemeral runners (
--ephemeral), one per KVM guest, registered at org level..runnershowsuseV2Flow: trueandserverUrlV2: https://broker.actions.githubusercontent.com/.What's not working?
The job annotation GitHub produces:
None of the causes it suggests apply. At the time of inspection, four hours after the runner went silent:
The process had consumed 2 seconds of CPU across four hours, and its connection to the broker was established with both send and receive queues empty. The machine itself was idle: load 0.19 on 32 cores, 47 GB of 124 GB free, no OOM kills, no hung-task warnings, and 0.011% TCP retransmissions host-wide. Nothing terminated the process, starved it, or blocked its network.
Job Log Output
There is no step output to include, and that is itself the symptom: the API reports an empty
stepsarray for every affected job. From one run (kumahq/kuma, run 33640499530):Four different runners, no steps recorded on any of them, and each job lasting ten minutes to the second. The same signature has since appeared in a second, unrelated organisation on the same runner version.
Runner and Worker's Diagnostic Logs
_diag/Runner_20260902-123047-utc.log. This is the complete tail of the file — nothing was written after 12:33:07, and the file was still the newest one in_diagwhen inspected at 16:32:46Z.The last line is 12:33:07.
Runner.Workerwas never spawned, so there is no worker log for this job —_diagcontains only the twoRunner_files from configuration and startup.Note the contrast within this one log: the session-creation timeout at 12:32:27 is caught, reported, and retried successfully. The token request at 12:33:07, immediately after the job was acknowledged, produces nothing further at all.
Where the unbounded call appears to be
Reading v2.336.0, the calls either side of the hang are both bounded, and the one in between is not.
src/Runner.Listener/BrokerMessageListener.cs,AcknowledgeMessageAsync(~line 353) — bounded at 5 seconds:src/Runner.Listener/Runner.cs(~line 367) — the token used for the rest of the loop is linked only to shutdown, with no deadline of its own:src/Runner.Listener/Runner.cs(~line 557), the broker path — that token is passed straight through, with no timeout wrapper:src/Runner.Common/RunServer.cs(~line 54) forwards it unchanged into the retry helper:So
messageQueueLoopTokenSource.Tokenonly fires when the runner is shutting down. If the underlying request never completes and never faults, nothing else cancels it,RetryRequestnever gets an exception to retry on, and the loop has nowhere to go — which matches what we see: no further log line, no retry, no exit, four hours of an idleESTABLISHEDsocket.I have not read
RetryRequestitself or theHttpClientconfiguration behind_runServiceHttpClient, so it is possible a timeout is applied at a level I did not check. If so, it did not take effect here.Anything else
Recreate session on RunnerSessionInvalid from broker,Cleanup session files on get message or session deleted error,Exit ephemeral runners on broker acknowledge job-not-found), so this area has clearly had attention — but the hang happens on 2.336.0 with those in place.Retry-Afteron 429 responses, which is a different path: here the server returns nothing at all rather than a status code.Runner.Workerbefore it reads the job message, also leaving the job orphaned. It may share a cause.