Skip to content

fix: local gateway connections recover after listener or token drift #1053

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
bkudiess wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

fix: local gateway connections recover after listener or token drift #1053

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3091303
fix: recover managed local gateway connections
bkudiess Jul 24, 2026
ff2ad8b
fix: verify WSL relay ownership during setup
bkudiess Jul 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions docs/CONNECTION_ARCHITECTURE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,24 @@ Node credential precedence follows the same invariant with a distinct stored tok

**`InteractiveGatewayCredentialResolver`** resolves credentials for HTTP surfaces (chat URL `?token=` auth). It **prefers SharedGatewayToken** over DeviceToken because HTTP endpoints expect the shared token, not the per-device WebSocket token. Browser proxy diagnostics should treat the missing shared token as a browser-control caveat, not as proof that the operator or node gateway connection is disconnected.

## Self-recovery and automatic local-gateway repair

Two orthogonal self-healing behaviors keep the connection reliable without dead-ending the user:

### Stale device-token self-recovery (operator + node)

The gateway may reject a stored device token with the structured code `AUTH_DEVICE_TOKEN_MISMATCH` (a rotated/revoked/replaced device token) — distinct from a wrong *shared* token. `GatewayErrorClassifier` is the single classifier for this: `ClassifyWithCode(message, ...codes)` inspects the structured `error.code`/`error.details.code` **before** the textual heuristic and returns the exact `GatewayErrorKind.DeviceTokenMismatch`, keeping a stale *device* token (auto-recoverable) separate from a wrong *shared* token (`Auth`, not device-recoverable). Broad `GatewayErrorKind.TokenDrift` remains a manual re-pair signal for UI copy.

On a device-token mismatch, the manager clears **only the rejected role's** device token and reconnects, letting `CredentialResolver` fall back to the same record's `SharedGatewayToken` (preferred) or `BootstrapToken`. This kills the post-setup "need a new token" dead end (setup clears the bootstrap token once pairing is durable, but the shared token remains). Operator recovery runs in `TryScheduleOperatorTokenRecovery`; node recovery is driven off the node client's classified `INodeConnectorTelemetryEvents.ConnectionFailure(GatewayErrorKind)` — the manager's `OnNodeConnectionFailure` queues `HandleNodeDeviceTokenMismatchAsync` off the connector's dispatch lock (capturing lifecycle+node generations at fire time and re-checking `IsCurrentNodeAttempt` before/after the transition semaphore). A per-gateway, per-role attempt guard (reset on handshake success / node pairing) prevents clear→reconnect→mismatch loops.

**Security — trust gate and endpoint provenance.** Clearing a device token downgrades to the more powerful shared/bootstrap credential, so `IsRecoverySafeEndpoint` restricts recovery to trusted endpoints: an owned SSH tunnel, a validated TLS (`wss`/`https`) endpoint, or — for a setup-managed WSL loopback gateway — a listener proven by `ManagedLocalGatewayPortProvenanceService` to be the Windows WSL relay. Loopback is not treated as identity by itself: an unknown listener or a proven obsolete native OpenClaw gateway blocks fallback, so a wrong local process cannot return a device-token mismatch to induce disclosure of the shared credential. A plain `ws://` remote endpoint is never eligible.

### Automatic managed-local WSL gateway repair (tray)

For an app-owned setup-managed local WSL gateway (`WslKeepAlivePolicy.IsSetupManagedLocalRecord` — never SSH/remote/ambiguous-localhost), the tray owns process supervision, keeping it out of the connection layer. `ManagedLocalGatewayAutoRepairMonitor` watches the operator connection and, when it is positively transport-unreachable (`GatewayErrorKind.Network`/`Server`, plus a cold-start `Connecting` state with no failure yet; never unknown/auth/pairing/rate-limit/scope/TLS/tunnel/token-drift) for a sustained window, invokes `ManagedLocalGatewayRepairCoordinator`. A typed `LocalPortConflict` is also repairable because its remediation is provenance-gated rather than a blind process restart. The monitor honors a **startup grace** (so a slow WSL cold start is not interrupted), a per-gateway unhealthy threshold and cooldown, a manager-owned explicit disconnect/stop intent, and a settings **kill switch** (`SettingsData.EnableManagedLocalGatewayAutoRepair`, default on).

`ManagedLocalGatewayRepairCoordinator` **probes before it restarts**: if the gateway is already reachable it just reconnects (the macOS "attach" path); only a genuinely-down gateway triggers a WSL distro restart (via `WslGatewayController`), a keepalive re-arm (`WslGatewayKeepAliveService.TryEnsureAsync`), and a reconnect. For the native-vs-WSL collision case, `ManagedLocalGatewayPortProvenanceService` classifies listeners by address and proves process command line plus scheduled-task/profile lineage. It automatically disables/stops only a fully proven obsolete native OpenClaw gateway; an unknown listener is never killed and produces precise `LocalPortConflict` diagnostics. The shared lifecycle lease serializes that destructive work with manual WSL actions. Reconnect is **gateway-pinned, intent-aware, and cancellable** (`GatewayConnectionManager.ReconnectIfCurrentAsync(gatewayId, ct)`), so gateway switches, explicit Disconnect/Stop, and shutdown always win. Repair is single-flight, verifies success by a real operator connection to the same gateway, is per-gateway restart-budget-bounded, and never reads or logs credentials.

## Client instance lifecycle

**Operator client** (`OpenClawGatewayClient`): Single instance at a time, owned by `GatewayConnectionManager`. Created via `GatewayClientFactory.Create()`. Old instance disposed before creating new one. `OperatorClientChanged` event notifies consumers of swaps.
Expand Down
19 changes: 19 additions & 0 deletions src/OpenClaw.Connection/ConnectionStateMachine.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ internal sealed class ConnectionStateMachine
private RoleConnectionState _operatorState = RoleConnectionState.Idle;
private RoleConnectionState _nodeState = RoleConnectionState.Idle;
private string? _operatorError;
private OpenClaw.Shared.GatewayErrorKind? _operatorErrorKind;
private string? _nodeError;
private string? _operatorCredentialSource;
private string? _nodeCredentialSource;
Expand Down Expand Up @@ -154,6 +155,7 @@ public void Reset()
_operatorState = RoleConnectionState.Idle;
_nodeState = _nodeEnabled ? RoleConnectionState.Idle : RoleConnectionState.Disabled;
_operatorError = null;
_operatorErrorKind = null;
_nodeError = null;
_operatorCredentialSource = null;
_nodeCredentialSource = null;
Expand Down Expand Up @@ -207,6 +209,12 @@ internal void SetOperatorCredentialResolution(GatewayCredentialResolution resolu
RebuildSnapshot();
}

internal void SetOperatorErrorKind(OpenClaw.Shared.GatewayErrorKind? kind)
{
_operatorErrorKind = kind;
RebuildSnapshot();
}

/// <summary>Update node info (device ID, pairing status, optional request ID) in the snapshot.</summary>
internal void SetNodeInfo(
string? deviceId,
Expand Down Expand Up @@ -293,6 +301,7 @@ private void ApplyTransition(ConnectionTrigger trigger, string? detail)
case ConnectionTrigger.ConnectRequested:
_operatorState = RoleConnectionState.Connecting;
_operatorError = null;
_operatorErrorKind = null;
break;

case ConnectionTrigger.ConnectRequestSent:
Expand All @@ -304,6 +313,7 @@ private void ApplyTransition(ConnectionTrigger trigger, string? detail)
case ConnectionTrigger.HandshakeSucceeded:
_operatorState = RoleConnectionState.Connected;
_operatorError = null;
_operatorErrorKind = null;
break;

case ConnectionTrigger.PairingPending:
Expand All @@ -317,16 +327,19 @@ private void ApplyTransition(ConnectionTrigger trigger, string? detail)
case ConnectionTrigger.PairingRejected:
_operatorState = RoleConnectionState.Error;
_operatorError = detail ?? "Pairing rejected";
_operatorErrorKind = OpenClaw.Shared.GatewayErrorKind.PairingRejected;
break;

case ConnectionTrigger.AuthenticationFailed:
_operatorState = RoleConnectionState.Error;
_operatorError = detail ?? "Authentication failed";
_operatorErrorKind ??= OpenClaw.Shared.GatewayErrorClassifier.ClassifyWithCode(_operatorError);
break;

case ConnectionTrigger.RateLimited:
_operatorState = RoleConnectionState.Error;
_operatorError = detail ?? "Rate limited";
_operatorErrorKind = OpenClaw.Shared.GatewayErrorKind.RateLimited;
break;

case ConnectionTrigger.WebSocketDisconnected:
Expand All @@ -339,19 +352,22 @@ private void ApplyTransition(ConnectionTrigger trigger, string? detail)
{
_operatorState = RoleConnectionState.Connecting;
_operatorError = null;
_operatorErrorKind = null;
}
break;

case ConnectionTrigger.WebSocketError:
_operatorState = RoleConnectionState.Error;
_operatorError = detail ?? "WebSocket error";
_operatorErrorKind ??= OpenClaw.Shared.GatewayErrorClassifier.Classify(_operatorError);
break;

case ConnectionTrigger.DisconnectRequested:
case ConnectionTrigger.Disposed:
_operatorState = RoleConnectionState.Idle;
_nodeState = _nodeEnabled ? RoleConnectionState.Idle : RoleConnectionState.Disabled;
_operatorError = null;
_operatorErrorKind = null;
_nodeError = null;
_operatorCredentialSource = null;
_nodeCredentialSource = null;
Expand All @@ -368,6 +384,7 @@ private void ApplyTransition(ConnectionTrigger trigger, string? detail)
case ConnectionTrigger.ReconnectScheduled:
_operatorState = RoleConnectionState.Connecting;
_operatorError = null;
_operatorErrorKind = null;
break;

case ConnectionTrigger.ReconnectSuppressed:
Expand All @@ -377,6 +394,7 @@ private void ApplyTransition(ConnectionTrigger trigger, string? detail)
case ConnectionTrigger.Cancelled:
_operatorState = RoleConnectionState.Idle;
_operatorError = null;
_operatorErrorKind = null;
break;

// ─── Node transitions ───
Expand Down Expand Up @@ -424,6 +442,7 @@ private void RebuildSnapshot()
OverallState = GatewayConnectionSnapshot.DeriveOverall(_operatorState, _nodeState, _nodeEnabled),
OperatorState = _operatorState,
OperatorError = _operatorError,
OperatorErrorKind = _operatorErrorKind,
OperatorCredentialSource = _operatorCredentialSource,
OperatorCredentialStatus = _operatorCredentialStatus,
OperatorCredentialFallbackUsed = _operatorCredentialFallbackUsed,
Expand Down
Loading
Loading