-
-
Notifications
You must be signed in to change notification settings - Fork 123
fix(client): release transport and feed guards before awaiting #1221
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -937,7 +937,11 @@ impl Client { | |
| } | ||
|
|
||
| // Close after flush; cleanup may also win this race on the run loop. | ||
| if let Some(transport) = self.transport.lock().await.as_ref() { | ||
| // Guard dropped first: edition 2024 keeps an `if let` scrutinee temporary alive | ||
| // for the whole matching arm, so holding it across `disconnect()` (an untimed | ||
| // socket write) would park `connect_internal`, which installs through this mutex. | ||
| let transport = self.transport.lock().await.clone(); | ||
| if let Some(transport) = transport { | ||
| transport.disconnect().await; | ||
| } | ||
| self.cleanup_connection_state().await; | ||
|
|
@@ -1003,7 +1007,8 @@ impl Client { | |
| .await; | ||
| self.notify_connection_shutdown(); | ||
|
|
||
| if let Some(transport) = self.transport.lock().await.as_ref() { | ||
| let transport = self.transport.lock().await.clone(); | ||
| if let Some(transport) = transport { | ||
| transport.disconnect().await; | ||
| } | ||
| } | ||
|
|
@@ -1038,7 +1043,8 @@ impl Client { | |
| .await; | ||
| self.notify_connection_shutdown(); | ||
|
|
||
| if let Some(transport) = self.transport.lock().await.as_ref() { | ||
| let transport = self.transport.lock().await.clone(); | ||
| if let Some(transport) = transport { | ||
| transport.disconnect().await; | ||
| } | ||
| } | ||
|
|
@@ -1165,11 +1171,16 @@ impl Client { | |
| // `Client::disconnect()`). Transport impls make `disconnect()` | ||
| // idempotent, so the redundant call from `Client::disconnect()` is | ||
| // safe. | ||
| if let Some(transport) = self.transport.lock().await.take() { | ||
| transport.disconnect().await; | ||
| } | ||
| // All three slots are cleared before the close is awaited, not after. The guard on | ||
| // `transport` no longer spans that await, so a `connect()` racing this teardown can | ||
| // publish its own transport, events and socket while the close is in flight; clearing | ||
| // afterwards would strip the replacement connection instead of the one being torn down. | ||
| let transport = self.transport.lock().await.take(); | ||
|
cubic-dev-ai[bot] marked this conversation as resolved.
|
||
| *self.transport_events.lock().await = None; | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P2: Release of the transport guard is the right fix and closes the long mid-close publish window, but the three slots are still cleared in three separate non-atomic lock acquisitions. If a racing Prompt for AI agents
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Answered in the parallel thread on line 1180. Short version: that narrow window predates this PR (it sat between the guard drop and the clears), a generation check cannot work until Generated by Claude Code |
||
| *self.noise_socket.lock().await = None; | ||
|
Comment on lines
+1178
to
1180
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Fresh evidence after the prior comment: moving these clears before Useful? React with 👍 / 👎.
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The narrow window between the three acquisitions is real, but it is not something this PR created or widened: before it, the same window sat between the guard drop and the two clears. What the PR did widen was the whole-close window, and that one is gone. Neither remedy is a small change here. The generation check does not work as written: It would also not be sufficient. Cleanup goes on after the close to reset Generated by Claude Code |
||
| if let Some(transport) = transport { | ||
| transport.disconnect().await; | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
| } | ||
| // Authoritative point for the gauge: every disconnect (intentional or a | ||
| // run-loop drop/reconnect) funnels through here, so disconnect()'s early | ||
| // set is just a prompt redundant signal. (`is_connected` was already cleared above, before | ||
|
|
@@ -1286,8 +1297,9 @@ impl Client { | |
| *self.media_conn.write().await = None; | ||
|
|
||
| // Clear app state key cache — keys will be re-fetched from DB on demand | ||
| if let Some(proc) = self.app_state_processor.lock().await.as_ref() { | ||
| proc.clear_key_cache().await; | ||
| let processor = self.app_state_processor.lock().await.clone(); | ||
| if let Some(processor) = processor { | ||
| processor.clear_key_cache().await; | ||
| } | ||
| #[cfg(feature = "client-lifecycle")] | ||
| drop(scope_close); | ||
|
|
@@ -1586,4 +1598,157 @@ mod tests { | |
| .expect("run() must still return on a terminal shutdown") | ||
| .expect("the run task must not panic"); | ||
| } | ||
|
|
||
| /// A transport whose `disconnect()` parks until released, standing in for the real | ||
| /// close-frame write: a TLS write with no timeout, behind the sender's own mutex. | ||
| struct ParkedDisconnect { | ||
| entered: async_channel::Sender<()>, | ||
| release: async_channel::Receiver<()>, | ||
| closes: Arc<AtomicUsize>, | ||
| } | ||
|
|
||
| #[async_trait::async_trait] | ||
| impl crate::transport::Transport for ParkedDisconnect { | ||
| async fn send(&self, _data: bytes::Bytes) -> Result<()> { | ||
| Ok(()) | ||
| } | ||
|
|
||
| async fn disconnect(&self) { | ||
| self.closes.fetch_add(1, Ordering::SeqCst); | ||
| let _ = self.entered.send(()).await; | ||
| let _ = self.release.recv().await; | ||
| } | ||
| } | ||
|
|
||
| fn parked_transport() -> ( | ||
| Arc<ParkedDisconnect>, | ||
| async_channel::Receiver<()>, | ||
| async_channel::Sender<()>, | ||
| Arc<AtomicUsize>, | ||
| ) { | ||
| let (entered_tx, entered_rx) = async_channel::bounded(4); | ||
| let (release_tx, release_rx) = async_channel::bounded(1); | ||
| let closes = Arc::new(AtomicUsize::new(0)); | ||
| let transport = Arc::new(ParkedDisconnect { | ||
| entered: entered_tx, | ||
| release: release_rx, | ||
| closes: closes.clone(), | ||
| }); | ||
| (transport, entered_rx, release_tx, closes) | ||
| } | ||
|
|
||
| /// The teardown paths must not hold the `transport` mutex across the socket close: | ||
| /// `connect_internal` installs the next connection's transport through that same mutex, | ||
| /// so a close that never returns would take the whole reconnect down with it. | ||
| #[tokio::test] | ||
| async fn an_in_flight_socket_close_does_not_park_the_transport_slot() { | ||
| let client = crate::test_utils::create_test_client().await; | ||
| let (transport, entered_rx, release_tx, _closes) = parked_transport(); | ||
| *client.transport.lock().await = Some(transport); | ||
|
|
||
| let cleanup = tokio::spawn({ | ||
| let client = Arc::clone(&client); | ||
| async move { client.cleanup_connection_state().await } | ||
| }); | ||
| tokio::time::timeout(Duration::from_secs(5), entered_rx.recv()) | ||
| .await | ||
| .expect("cleanup must reach the socket close") | ||
| .expect("the observer channel must stay open"); | ||
|
|
||
| // Exactly what `connect_internal` does once the next socket is up. | ||
| tokio::time::timeout(Duration::from_secs(5), async { | ||
| *client.transport.lock().await = Some(Arc::new(crate::transport::mock::MockTransport)); | ||
| }) | ||
| .await | ||
| .expect("installing the next transport must not wait on the in-flight close"); | ||
|
|
||
| drop(release_tx); | ||
| tokio::time::timeout(Duration::from_secs(5), cleanup) | ||
| .await | ||
| .expect("cleanup must finish once the close returns") | ||
| .expect("cleanup must not panic"); | ||
| } | ||
|
|
||
| /// The other half of releasing the guard: now that a `connect()` can publish its own | ||
| /// connection while the old close is still in flight, cleanup must not come back and | ||
| /// strip the replacement's state. Everything cleanup clears, it clears before the close. | ||
| #[tokio::test] | ||
| async fn a_connection_published_during_cleanup_is_not_stripped_by_it() { | ||
| let client = crate::test_utils::create_test_client().await; | ||
| let (transport, entered_rx, release_tx, _closes) = parked_transport(); | ||
| *client.transport.lock().await = Some(transport); | ||
| *client.transport_events.lock().await = Some(async_channel::bounded(1).1); | ||
|
|
||
| let cleanup = tokio::spawn({ | ||
| let client = Arc::clone(&client); | ||
| async move { client.cleanup_connection_state().await } | ||
| }); | ||
| tokio::time::timeout(Duration::from_secs(5), entered_rx.recv()) | ||
| .await | ||
| .expect("cleanup must reach the socket close") | ||
| .expect("the observer channel must stay open"); | ||
|
|
||
| // The publish half of `connect_internal`, minus the handshake. | ||
| *client.transport.lock().await = Some(Arc::new(crate::transport::mock::MockTransport)); | ||
| *client.transport_events.lock().await = Some(async_channel::bounded(1).1); | ||
|
|
||
| drop(release_tx); | ||
| tokio::time::timeout(Duration::from_secs(5), cleanup) | ||
| .await | ||
| .expect("cleanup must finish once the close returns") | ||
| .expect("cleanup must not panic"); | ||
|
|
||
| assert!( | ||
| client.transport.lock().await.is_some(), | ||
| "the replacement transport must survive the teardown it did not belong to" | ||
| ); | ||
| assert!( | ||
| client.transport_events.lock().await.is_some(), | ||
| "the replacement's event receiver must survive too, or its read loop never starts" | ||
| ); | ||
| } | ||
|
|
||
| /// The happy path the fix must preserve: cleanup still closes the socket it owns and | ||
| /// still leaves the slot empty for the next connection. | ||
| #[tokio::test] | ||
| async fn cleanup_closes_the_transport_and_clears_the_slot() { | ||
| let client = crate::test_utils::create_test_client().await; | ||
| let (transport, _entered_rx, release_tx, closes) = parked_transport(); | ||
| drop(release_tx); // never park: this is the ordinary, prompt close | ||
| *client.transport.lock().await = Some(transport); | ||
|
|
||
| tokio::time::timeout(Duration::from_secs(5), client.cleanup_connection_state()) | ||
| .await | ||
| .expect("cleanup must not block"); | ||
|
|
||
| assert_eq!( | ||
| closes.load(Ordering::SeqCst), | ||
| 1, | ||
| "the socket must be closed" | ||
| ); | ||
| assert!( | ||
| client.transport.lock().await.is_none(), | ||
| "cleanup owns the teardown and must leave the slot free" | ||
| ); | ||
| } | ||
|
|
||
| /// Same for the user-facing path: `disconnect()` closes the socket and hands a cleared | ||
| /// slot back, so nothing from the dead connection survives into the next one. | ||
| #[tokio::test] | ||
| async fn disconnect_closes_the_transport_and_clears_the_slot() { | ||
| let client = crate::test_utils::create_test_client().await; | ||
| let (transport, _entered_rx, release_tx, closes) = parked_transport(); | ||
| drop(release_tx); | ||
| *client.transport.lock().await = Some(transport); | ||
|
|
||
| tokio::time::timeout(Duration::from_secs(10), client.disconnect()) | ||
| .await | ||
| .expect("disconnect must not block"); | ||
|
|
||
| assert!( | ||
| closes.load(Ordering::SeqCst) >= 1, | ||
| "the socket must be closed" | ||
| ); | ||
| assert!(client.transport.lock().await.is_none()); | ||
| } | ||
| } | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When a new
connect()progresses while this old transport'sdisconnect()is parked—the exact concurrency the new regression test permits—it can install its transport, event receiver, and Noise socket at lines 845–847. Once the old close resumes, this cleanup leaves the replacement transport in place but unconditionally clears the replacement'stransport_eventsandnoise_socketat lines 1178–1179, after whichread_messages_loop()fails withNotStarted; the remaining cleanup also resets state belonging to that new connection. The teardown must either prevent publication until all old-generation cleanup is complete or conditionally clear only state associated with the transport/generation it took.Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Valid, and introduced by this change: the held guard used to block
connect_internal's first install, so it could not publish a whole connection inside the close window. Fixed in 13ddd6f by clearing all three slots before the close is awaited, with a test that fails the other way round.Generated by Claude Code