fix: reconnect after websocket heartbeat timeout

[aujxn]

Summary

Fixes #60.

This updates the websocket connection handler so heartbeat failures cause the active connection to terminate and reconnect.

Previously, the heartbeat task could time out waiting for PONG and exit without propagating that failure back to handle_connection. In that state, the connection manager did not necessarily observe a connection error, so automatic reconnect/resubscribe could be missed.

This patch makes handle_connection return a websocket timeout when:

• a received PONG cannot be delivered to the heartbeat task
• the heartbeat task exits and drops the ping channel

A regression test covers the missing-PONG case by using a mock websocket server that ignores PING, then asserting the client reconnects, resubscribes, and receives messages after reconnect.

Verification

• uv tool run pre-commit run --all-files (note: this introduced the whitespace change to .gitignore)
• cargo test --features clob,ws,tracing reconnects_when_heartbeat_pong_is_missing
• cargo test --workspace --all-features

---

Note

Medium Risk
Changes live WebSocket connection teardown and reconnection behavior when heartbeats fail; impact is limited to the ws connection loop and is covered by a new integration test.

Overview
Fixes stale WebSocket sessions where the heartbeat task could die on a missing PONG without the connection manager treating it as a failure, so reconnect and resubscribe might never run.

handle_connection now surfaces heartbeat failures as WsError::Timeout: when a PONG cannot be forwarded to the heartbeat task, and when the heartbeat task exits and the ping channel closes. Failed PING sends return ConnectionClosed instead of only breaking the loop. PONG handling is refactored alongside normal text parsing.

Adds reconnects_when_heartbeat_pong_is_missing (mock server ignores PING) to assert reconnect, resubscribe, and post-reconnect messages. .gitignore gets a trivial specs/* line-ending fix.

^{Reviewed by Cursor Bugbot for commit 2085ef9. Bugbot is set up for automated code reviews on this repo. Configure here.}

[qiuqiuaiweb3]

I reproduced #60 and ran a generation-aware follow-up against this PR. The production change looks correct, and the full three-file patch replays byte-for-byte on current main (222143d321eba97d5711a848265eb9aab3bc7ff4).

Source	Missing PONG	Healthy PONG control
crates.io 0.7.0	FAIL 3/3: timeout observed, but generation 1 remained open	PASS 3/3
PR base 8ba5008	same FAIL 3/3	PASS 3/3
current 222143d	same FAIL 3/3	PASS 3/3
PR head 2085ef9	PASS 3/3	PASS 3/3
current + PR patch	PASS 3/3	PASS 3/3

The loopback test gates the first handshake until the subscription reconnection handler is ready, then waits for a real text PING and releases the real heartbeat immediately before a zero-duration timeout. The missing-PONG path requires, in order: the observed timeout, generation-1 Close/error/EOF, a distinct generation-2 connection, a JSON-semantically-identical reconstruction request, a generation-2 PONG, an Ok(BookUpdate) from generation 2, and no third connection or extra reconstruction. The healthy control sends PONG plus a marker before release and proves the original Connected { since } remains, delivery continues on generation 1, and there is no reconnect or reconstruction. There is no sleep, yield, or scheduler-order oracle.

This adds a boundary that the current regression test does not distinguish: its server shares one subscription receiver across sockets, broadcasts messages to every live socket, and ignores PING on every generation. A later subscription plus Some(stream item) therefore does not identify the generation, prove the old socket closed, exclude repeated reconnects, or require the stream item to be Ok.

The full library suite (147 passed, 1 ignored), this PR's original missing-PONG integration test, strict clippy, and stable/pinned-nightly formatting all pass on the fixed variants. I suggest keeping the production change and strengthening the regression test with generation identity/old-socket closure plus the healthy-PONG control. I can adapt the deterministic test harness for this PR if that would help.

I used an AI coding assistant to help prepare and execute this synthetic loopback matrix; I reviewed the evidence and exact comment before posting.

[guoran8]

We independently root-caused this defect in production before finding this PR (analysis in #99; it is the same bug as #60), so reviewing it from that angle:

The fix is correct, and it targets the exact mechanism that makes the stall permanent.

The failure chain on main is: heartbeat_loop detects the dead connection and exits → ping_tx is dropped → the Some(()) = ping_rx.recv() arm's pattern match fails, which merely disables that branch. Meanwhile Some(msg) = read.next() on a half-open TCP connection is enabled-but-pending forever, so the else arm (which requires all branches disabled) is unreachable, and handle_connection never returns. The SDK knows the connection is dead but no layer that could act on it is ever told.

This PR changes the arm to bind the Option (ping = ping_rx.recv()), so channel closure completes the branch instead of disabling it, and the None case returns Err — turning the heartbeat task's own exit into the death notification, with no new channel or state. connection_loop then reconnects through its existing path. Elegant and minimal.

It also fixes the secondary leak we documented in #99: because handle_connection now returns, connection_loop's top-of-loop sender_rx.is_closed() check becomes reachable again, so dropping the Client reaps the background task and socket within one heartbeat cycle instead of leaking the half-open connection until OS keepalive (~2h).

Two additional observations, neither blocking:

• The pong_tx.send(...).is_err() → Err(Timeout) path (PONG arrives but the heartbeat task is gone) and the write-failure paths returning Err instead of break are consistent with the above: every "the peer or the heartbeat is gone" condition now funnels into the reconnect path.
• The regression test (server withholds PONG → assert re-subscription) covers the primary path. The Client-drop reap behavior is untested but follows structurally.

Production context for maintainers weighing priority: this defect cost us a 27-hour silent outage (process healthy, zero errors, zero data) before we added external inbound-silence watchdogs downstream. Those watchdogs work but leak one socket per redial cycle — this PR is what lets consumers delete that scaffolding. Would be glad to see it land; happy to run our reproduction matrix against the branch if useful.

[guoran8]

Production validation report, following up on the earlier review:

We cherry-picked this PR's commit onto the v0.7.0 release (it applies cleanly — src/ws/connection.rs was untouched between this PR's base and 0.7.0) and have been running it in production since 2026-08-10 across five services that use the WS layer — both the CLOB market channel and RTDS, which share the fixed ConnectionManager.

Results from the first ~24h:

• Before: our downstream workaround (inbound-silence watchdog → hard teardown → bounded redials → process exit + orchestrator restart) fired routinely; elevated pod restart counts were our accepted steady state.
• After: zero watchdog escalations and zero process restarts. Connection drops did still occur — aggregate feed blindness of ~13 min/day tells us the network was not quiet — but every one of them healed in-process through this PR's restored reconnect path, in seconds, with resubscription repopulating state as expected.

Test side, on the cherry-picked tree: the full websocket suite passes 42/42 including this PR's reconnects_when_heartbeat_pong_is_missing regression test, and 148 lib tests pass with clob,gamma,heartbeats,rtds,ws enabled.

For anyone needing the same backport while this is unmerged: foli0zone/rs-clob-client-v2, branch mercury/0.7.0-ws-heartbeat-reconnect (= upstream v0.7.0 + this PR's commit via cherry-pick -x, nothing else), usable via [patch.crates-io].

Maintainers: with an independent reproduction earlier in #60, the mechanism analysis in #99, and now a production soak, this fix looks as de-risked as an unmerged PR gets. Merging would resolve #60 and #99 together.