feat(realtime): lean livekit_join with passthrough flag - #37
Conversation
Reverse the bundle-initial-state-in-join design so the initial-state
upload and its ack no longer gate the local track.
- Send a lean `livekit_join` carrying only `{ type, passthrough }`.
- Fire the initial-state frame (set_image / prompt / null-bootstrap) as a
separate message immediately after the join, without awaiting its ack.
- Connect the room and publish the local track as soon as room_info
arrives; observe the ack out-of-band and surface an error only on
rejection. Its timeout is armed after room_info so a long queue wait
can't trip it.
- Derive `passthrough` from the initial prompt/image (override via
ConnectionConfig.passthrough).
BREAKING: ConnectionConfig.bundleInitialStateInJoin is replaced by
ConnectionConfig.passthrough (Bool?).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit e232327. Configure here.
| clearPendingInitialState() | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
Runtime prompts lose acks after connect
High Severity
connect can finish while isWaitingForInitialStateAck is still true. Any setPrompt/setImage in that window routes acks into the initial-state buffer; when the observer completes, clearPendingInitialState() drops unmatched acks, so runtime waiters time out or hang even though the server responded.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit e232327. Configure here.
| return | ||
| } catch { | ||
| DecartLogger.log("Initial-state acknowledgment failed: \(error.localizedDescription)", level: .error) | ||
| } |
There was a problem hiding this comment.
Initial ack timeout marks error
Medium Severity
The out-of-band initial-state observer reuses waitForPromptAck/waitForSetImageAck, which set connectionState to .error on timeout. After a successful connect, a late initial-state ack timeout can flip an already-connected session to error, though the PR describes surfacing errors only on server rejection.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit e232327. Configure here.
1 participant
What
Implements the realtime-connect spec: the signaling-WS handshake no longer bundles initial state into the join, and the local track is no longer gated on the initial-state ack.
Connect sequence (new)
livekit_join:{ type, passthrough }— no nestedinitial_state.set_image/prompt) as a separate message, without awaiting its ack.livekit_room_info.connectedas soon as the room connects.This takes the (often multi-MB) image upload and its ack off the critical path to first frame.
passthroughflagRequired boolean on the join. Derived:
falsewhen the user set a real image/prompt,trueotherwise.ConnectionConfig.passthrough(Bool?) overrides the derived value.The ack timeout is armed only after
room_infoarrives, so a longqueue_positionwait can't trip it; acks landing before the observer starts are buffered.Breaking change
ConnectionConfig.bundleInitialStateInJoinis removed and replaced byConnectionConfig.passthrough(Bool?, defaultnil= derive).Notes
sendInitial*AndWait,waitForBundledInitialStateAck,handleInitialStateAfterRoomInfo) and the unusedInitialStateMessageenum /sendMessagehelper. Net −82 lines.Test plan
swift buildclean.swift test— 96/96 pass, incl. new lean-join encoding tests andpassthroughderivation tests (prompt/image/none/override).🤖 Generated with Claude Code
Note
Medium Risk
Changes the realtime signaling contract and connect timing (breaking
ConnectionConfig), which can affect server compatibility and when clients seeconnectedvs initial-state errors.Overview
Realtime connect now follows a lean signaling handshake:
livekit_joinsends onlytypeandpassthrough(no nestedinitial_state), then the SDK immediately sends the initialset_image/promptas a separate WebSocket message.ConnectionConfig.bundleInitialStateInJoinis removed in favor ofpassthrough: Bool?(derive from prompt/image whennil).Connect no longer waits on the initial-state ack before LiveKit connect/publish;
observeInitialStateAckruns afterroom_infoso queue time does not consume the ack timeout, with early acks buffered viaisWaitingForInitialStateAck. Sessionconnectedis reached when the room is up; ack failures are handled out-of-band (logged, not blockingperformConnect). Empty sessions always bootstrap with a passthroughset_imageviabuildInitialStateRequest().Tests cover lean join JSON and
passthroughderivation/overrides.Reviewed by Cursor Bugbot for commit e232327. Bugbot is set up for automated code reviews on this repo. Configure here.