feat: PolyVoice — WebRTC voice calling - #28
Merged
Conversation
New PolyVoice product/pod — live two-way WebRTC voice calls to a PolyAI agent, the iOS counterpart of Android's ai.poly:voice. Ships SEPARATELY so chat-only apps never link the WebRTC binary (PolyMessaging stays source-only). - Public media seam on PolyMessaging (CallMediaEngine / CallMediaState / ICECandidate + PolyCall.wired) so PolyVoice injects a real engine into the existing, already-tested CallCoordinator signaling pipeline. - WebRTCCallMediaEngine: real RTCPeerConnection (audio-only Opus, offer/answer/ trickle ICE, mute) + AudioSessionController (accessory-aware AVAudioSession). - PolyVoice.call(config:options:) + VoiceOptions (required webrtcToken, distinct from the API key). - SwiftUI + UIKit Voice examples (tap-to-call). - docs/PolyVoice.md + README voice section; version -> 0.9.0; PolyVoice.podspec. WebRTC via stasel/WebRTC (pinned M149, iOS-only so the macOS test suite is unaffected). Builds + runs on the simulator, driving the full pipeline with a real WebRTC engine (auth -> session w/ device_type/platform -> link -> signaling -> real offer). Two-way audio needs a physical device — WebRTC media isn't reliable on the simulator.
Zakariya1057
force-pushed
the
feat/polyvoice-webrtc-calling
branch
from
July 1, 2026 16:13
4e23e01 to
88d3ae2
Compare
…sion, doc/version nits Pre-merge review fixes (docs + config + one real bug): - docs/README: SPM product snippet used `package: "PolyMessaging"` (the package *name*) — a consumer can't resolve that. Corrected to `package: "ios-sdk"` (the dependency identity = repo name). - AudioSessionController.deactivate() now guards on an `activated` flag so a call that failed before activating never clobbers the host app's global audio session (mirrors Android's guard). - PolyCall KDoc no longer claims voice is "not yet available" (PolyVoice ships the engine). - README install pins bumped 0.8.0 → 0.9.0; PolyVoice.podspec pins WebRTC-SDK ~> 149.0 to match the SPM `exact: 149.0.0`. - Dropped the `-autostart` test hook from the SwiftUI example; gitignore internal QA scratch. Note: known follow-ups tracked in the PR — gateway ICE/TURN fetch, signaling reconnect, AVAudioSession interruption handling, and Voice error-case parity.
Zakariya1057
force-pushed
the
feat/polyvoice-webrtc-calling
branch
from
July 1, 2026 16:32
9e2ef8f to
6ec060a
Compare
…tions, error parity Closes the Android-parity gaps found in review so PolyVoice is production-ready. - Gateway ICE/TURN fetch (GET /api/v1/ice-servers?token=…, best-effort → public STUN fallback), threaded through a new CallMediaEngine.createOffer(iceServers:) seam so calls connect behind symmetric NAT / CGNAT. New IceServer type + IceServersProviding. - Signaling auto-reconnect: an unexpected socket drop reconnects with backoff (1s/2s/4s, same session) and re-flushes buffered ICE before failing. GatewaySignalingChannel.open() is now re-openable. - AVAudioSession interruption handling: began → mute, endedResume → unmute (preserving the user's mute intent), endedStop → end the call. New CallInterruption + setInterruptionHandler. - PolyError.Voice.disconnected + .interrupted (both isRetryable); a post-connect media drop now maps to the retryable .disconnected, not .mediaFailed. Tests: +7 coordinator tests (reconnect survive/exhaust, ICE-to-engine, disconnected mapping, interruption mute/restore/preserve-user-mute/end) + ICE-parsing tests. Full suite 229 green. macOS build + iOS PolyVoice build clean.
Zakariya1057
changed the title
…ity pass Doc-parity review against the Android voice guide: - docs/PolyVoice.md: add a "Backgrounding" section (UIBackgroundModes: [audio]) — the one real gap vs Android; add a region/environment note to Credentials. - Examples: enable the `audio` background mode in both Voice examples so a call survives backgrounding (UIKit committed here; SwiftUI folded in at cleanup). - PolyVoice.podspec: pin WebRTC-SDK to exact 149.0.0 (matches the SPM pin; M141 broken). - Drop incidental Android cross-references from the iOS guide/podspec for a clean public repo.
Examples/README.md covered the chat ladder (01-07) but not the Voice examples — add a Voice callout linking both toolkits + the guide.
…reconnect re-flush, error cases Closes coverage gaps in the new code: - GatewayIceServersFetcher.fetch() HTTP path via a URLProtocol mock: success→parse, non-2xx→STUN, network-error→STUN, empty-list→STUN, nil-url→STUN. - VoiceEnvironment.iceServersURL(token:) endpoint + token encoding; per-region signaling URL. - PolyError.Voice.disconnected/.interrupted description + isRetryable classification. - Reconnect re-flushes ICE buffered during the gap — plus a real fix: local candidates now buffer while the socket is reconnecting instead of being sent into a dead socket and lost. Full suite 238 green (4 network-gated skipped). WebRTCCallMediaEngine + AudioSessionController remain integration-only (real WebRTC / AVAudioSession) — device-verified, like Android's adapters.
A four-agent file-by-file comparison against the Android polyvoice module surfaced these: HIGH (real-call correctness): - Gateway host: prod regions now use webrtc-gateway.<region>.platform.polyai.app (dev stays standalone) — iOS was omitting `.platform`, hitting the wrong domain for signaling + ICE on every non-dev region. - Inbound ICE: buffer remote candidates until the SDP answer is applied, then flush — adding them before setRemoteDescription silently dropped early trickle candidates. API / behavior parity: - VoiceOptions.signalingHost (custom/self-hosted gateway); PolyVoice.call now throws + validates blank apiKey/webrtcToken and .custom-without-host (mirrors Android). - AudioSessionController observes routeChangeNotification → mid-call re-routing (a headset connected during a call is no longer ignored by the sticky speaker override). - Signaling offer caller/callee aligned to the proven "Polyphone"; ICE sdpMid/sdpMLineIndex sent as explicit null — matching the web/Android wire shape. Tests (+7): connection-timeout, disconnect grace-recovery, inbound-ICE buffering, graceful close-frame-on-failure, pre-connect reconnect-exhaustion→signalingFailed, gateway host/.platform, signalingHost override, .custom-throws. Full suite 245 green; iOS + both examples build clean. Kept iOS's post-connect FAILED→.disconnected (retryable) as an intentional improvement.
Closes the last deferred Android-parity feature (AudioDevice/AudioState/setAudioDevice/audio/isMuted). - Public AudioDevice (type/name/id) + AudioState (availableDevices/selectedDevice) types. - PolyCall.audioState (AsyncStream), setAudioDevice(_:) (nil = automatic), isMuted getter. - CallMediaEngine seam gains setAudioStateHandler + selectAudioDevice; CallCoordinator relays AudioState snapshots + forwards selection + exposes isMuted. - AudioSessionController enumerates outputs (earpiece/speaker/wired/Bluetooth via availableInputs + currentRoute), selects via setPreferredInput/overrideOutputAudioPort, and emits AudioState on activate / route-change / selection / teardown. - Both examples ship a working output picker (SwiftUI buttons, UIKit segmented control). - Docs audio section + 3 coordinator tests (isMuted, selection forwarding, state relay). 248 green.
… (iOS idiom) iOS keeps one active output and auto-routes accessories, so a force-select device list fights the system. Keep the audioState/setAudioDevice/isMuted API (Android parity + honest observation), but present the iOS-idiomatic control: - Both examples now show the current output + a Speaker on/off toggle (speaker ↔ earpiece is the one output an app reliably controls; headsets/Bluetooth are system-routed). Docs note AVRoutePickerView for picking among connected outputs the standard way. - Bump the two reconnect-exhaustion tests' deadline (real-timer reconnect can drift under full-suite concurrency; nominal ~1.3s). Full suite 248 green.
Mirrors the Android repo's example layout: each Voice example now lives under Voice/01-Hello/ so later rungs can slot in beside it. Expands both 01-Hello READMEs, updates the examples index and PolyVoice doc links, and gitignores .idea/.
Voice had solid state-machine unit tests but none of the categories the chat side leans on. Adds four suites (24 tests), all network-free: - VoiceSessionLinkerTests: the messaging-WS leg, previously live-probe only — URL shape, SESSION_START gating, link frame, timeout, and the tolerated link-send failure. - VoiceE2EScenarioTests: scenario coverage at the public PolyCall surface (the voice twin of E2EScenarioTests) — the exact states progression the example apps render, failure surfacing, late- subscriber replay, mute/audio-device round-trips, end-before-start. - StressCallLifecycleTests: races and storms — end() mid-start() aborts cleanly, double start() arms once, 4x drop→reconnect cycles deliver every gap ICE candidate, 50-candidate pre-answer bursts flush in order, terminal states are sticky, teardown runs once. - StressMalformedSignalingTests: adversarial wire input (the voice twin of StressMalformedWire) — garbage/type-confused/oversized/ deeply-nested frames parse to nil, and a live call connects through and survives garbage bursts.
…product split
Mirrors the Voice reorg (and the Android repo's layout): the seven chat
rungs now live in Examples/{SwiftUI,UIKit}/Chat/, beside
Examples/{SwiftUI,UIKit}/Voice/.
- xcodegen package path is one level deeper (../../../..); all 14
projects regenerated, SwiftUI Chat/01-Hello and UIKit
Chat/06-FullReference verified building for the simulator
- every reference to the old paths updated: root README, examples
index, in-example README cross-links, the Voice READMEs' chat-ladder
links, CONTRIBUTING, agent briefs (CLAUDE.md/copilot-instructions),
the bug-report template, build-all.sh, and e2e-validation.sh
Voice had a section but no discoverability hooks. Mirrors the Android root README: intro says chat **and** voice with a TOC bullet, a voice row in the features table, runnable-demo links under the Voice calling section, and the example-apps section split into Chat/Voice product subsections with the Voice ladder table.
Mirrors the Android README's sentence: two distinct credentials, both on the same connector in Agent Studio › Connector Settings — the API key (connector auth) and the WebRTC token (media-gateway auth).
Zakariya1057
changed the title
Keeps each comment's actual content (host rules, guards, wire shapes) and removes the sibling-SDK attributions.
…-session ownership Addresses the confirmed findings from the external review: - PolyVoice.podspec: depend on WebRTC-lib (stasel's pod — WebRTC-SDK has no 149.0.0, so CocoaPods installs could never resolve); lint verified. - CI: new ios-build-test job runs the whole package on an iOS simulator (the real WebRTC engine and the new PolyVoiceTests target are compiled out of the macOS leg), and podspec-lint now lints PolyVoice too. - GatewaySignalingChannel: per-connection generations — delayed open/close/error callbacks from a cancelled socket can no longer terminate or falsely open the connection that replaced it; send() reports failure instead of swallowing it. - CallCoordinator: the offer stays buffered until it actually reaches the wire and is re-sent after a reconnect; failed/interleaved local ICE sends are requeued (snapshot-and-clear flush) instead of dropped; close codes are classified (1000 ends cleanly, policy/protocol/4xxx fail immediately, transient codes keep the reconnect ladder). - Teardown is stored and awaited: end() returns only after the sockets, messaging link, media engine, and audio session are released, and dropping a live PolyCall now triggers the same cleanup from deinit. - AudioSessionController: the activated flag is only set after the session configuration succeeds, so cleanup can't deactivate a session the call never owned; the device list now folds in current-route outputs (output-only A2DP routes were invisible via availableInputs). - VoiceSessionLinker preserves query params already on the ws base URL. - VoiceEnvironment: host:port signaling hosts now build a valid ICE URL instead of silently falling back to public STUN. - README voice snippet: PolyVoice.call throws — add the missing try. - build-all.sh / verify.sh now cover the two Voice examples (16 apps).
The ios-build-test job picked the first available iPhone by name and passed a bare name= destination, letting xcodebuild default to OS:latest. On the Xcode 16.2 runner the newest runtime is iOS 18.x, which has no iPhone 15 Pro simulator (17.x-only) — so device resolution failed with exit 70. Select the simulator by UDID instead, which pins device and runtime together, and fail loudly if none is available.
Opt-in manual-audio mode: with callKit: true the SDK never activates or deactivates the audio session and defers the WebRTC audio unit to CallKit (RTCAudioSession.useManualAudio/isAudioEnabled). Three statics forward the CXProviderDelegate moments — configure in the start action (never self-activate: that's the didActivate-never-fires bug), enable audio in didActivate (activation notification first, it clears WebRTC's stale-interruption latch), disable in didDeactivate. A non-CallKit call resets the process-global manual flag so a prior CallKit call can't silence it. The SDK's own interruption handling stands down in this mode — cellular calls arrive as CallKit hold/deactivate instead. New Voice 02-CallKit example rung (SwiftUI + UIKit): full CXProvider/ CXCallController wiring with the request-vs-report split, system-UI mute sync, recents opt-out, and a simulator fallback (CallKit is broken on iOS 17+ simulators). Docs: voice guide CallKit section replaces the "no CallKit" note; ladder tables, 01-Hello cross-links, CHANGELOG, and build scripts (18 apps) updated. Seam covered by CallKitAudioSeamTests on the iOS-simulator CI leg, including real-engine offer creation under manual audio.
…were refused Code=1 (unentitled) from CXCallController on device: UIBackgroundModes must include voip for CallKit, not just audio. Also surface transaction errors in the example UI via CallKitController.onRequestError instead of silently swallowing them (a refused Start tap looked like a dead button), and document the requirement in the voice guide + example READMEs.
… gate Signaling takes seconds, so CallKit activates the session (didActivate → isAudioEnabled = true) before createOffer runs; the engine then re-set isAudioEnabled = false, stopping the audio unit for good — connected but silent in both directions. createOffer now only arms useManualAudio and never touches the enable gate (owned by configure + didActivate/ didDeactivate). Regression-tested: createOffer must leave an earlier activation intact.
…rung 02-CallKit READMEs list everything preconfigured (mic usage description, audio+voip background modes); the voice guide's mic and backgrounding sections note how CallKit changes activation ownership and adds voip.
…lly hit Token/environment/host mismatch 401s, silent CallKit calls (missing hook forwarding or voip mode), silent plain calls (mic permission), the 30s media timeout on relay-only networks, retryable disconnects, and simulator expectations.
Five defects in the voice pipeline, all reachable through normal use: - CallKit calls came out of the EARPIECE, not the speaker. activate() skipped applyRoute() in callKitMode, relying on CallKit's activation route-change to route instead — but handleRouteChange() guards on `activated`, which is still false when that notification arrives. The route was never applied. This is the other half of 86c4a75: that commit fixed the audio GATE under CallKit and left the ROUTE broken. - end() during call setup leaked the peer connection and left the mic hot. createOffer() only publishes `peer` after its awaits, so a close() landing in the ~1-3s auth/session/ICE window released nothing, then createOffer resumed and built a live mic track nobody owned — recording indicator stuck on for the rest of the app's lifetime. close() now latches, and createOffer releases its own peer when it sees the latch. - end() during a signaling reconnect leaked a WebSocket + URLSession. channel.open() has no cancellation points, so cancelling reconnectTask didn't stop it: it resumed a socket after teardown's close() had run, and nothing ever released it. The reconnect loop now closes what it opened when the call is gone. - GatewaySignalingChannel's lock was one-sided: send() took it, but open()/close() wrote task/urlSession/delegate/receiveTask without it. Since close() (teardown) races open() (reconnect), that was a genuine concurrent read/write on strong refs, not just a stale read — @unchecked Sendable was asserting a guarantee the code didn't provide. - Media states were delivered via a Task per event, which has no ordering guarantee. The routine ICE blip .disconnected -> .connected could apply in reverse, stranding lastMediaState on .disconnected so the grace timer failed a perfectly healthy call. Now a single ordered AsyncStream. Also: ICE candidates gathered between a reconnect's .opened and the loop noticing it (a 100ms poll tick) were buffered with nothing left to flush them — lost for the rest of the call. And the audio controller's two sink closures were written from the actor and read from the AVAudioSession notification thread with no synchronization. Regression tests cover the ordering and reconnect-window cases; the reconnect-window test was verified to fail without its fix. 283 green, iOS build clean. The CallKit route fix needs on-device confirmation.
Zakariya1057
added a commit
that referenced
this pull request
Jul 20, 2026
* refactor(voice)!: settle the public API before 0.9.0 is permanent Once 0.9.0 is on CocoaPods trunk none of this can change without a major bump, and a published pod version can never be withdrawn. So fix the naming, placement and over-exposure now while it's still free. Naming: - ICECandidate -> IceCandidate. It sat next to IceServer in the same seam with a different spelling of the same acronym. - IceServer.default -> IceServer.defaultServers. It returns an ARRAY, so `IceServer.default` read wrong at every call site. - AudioDevice.DeviceType -> AudioDevice.Kind (and `.type` -> `.kind`). DeviceType collided with the pre-existing top-level DeviceType in the same module, forcing `AudioDevice.DeviceType` in full everywhere. Placement: CallMediaEngine, CallMediaState, CallInterruption, IceServer and IceCandidate were `public` but living under Internal/, against the repo's own Public/ vs Internal/ split. Moved to Public/Voice/. Exposure: CallMediaEngine and PolyCall.wired(...) exist ONLY so PolyVoice can inject its engine across the module boundary, and AudioDevice.init only so it can build the values it reports. As public API they'd be semver-locked forever — wired() in particular hard-codes the SDK's whole internal composition in its signature. They're now @_spi(PolyVoice), so they stay ours to refactor. CallMediaEngine also gains protocol-extension defaults for its optional capabilities, so adding one later is additive. Deliberately no defaults for the core requirements: a no-op default there would turn a missing implementation into a silently broken call. Observability: PolyCall is now @mainactor + ObservableObject with @published state/audioState, matching ChatSession instead of diverging from it. Every SwiftUI example was hand-rolling `Task { for await ... }` plus a Task? to cancel; that's gone. The states/audioStates AsyncStreams remain for UIKit and for side-effect consumers that must see every transition (the CallKit example reports each one to its provider). The old `audioState` stream is renamed `audioStates` so the singular name can be the published value — singular is the current value, plural is the stream. Docs: the "chat-only apps never pull the WebRTC binary" claim was overstated — SPM resolves package-level dependencies regardless of target conditions, so a chat-only consumer still downloads the xcframework, it just never links it. CocoaPods genuinely pulls nothing extra. Also pinned the install snippets to upToNextMinor, since pre-1.0 this project bumps MINOR for breaking changes, and fixed the wired() symbol drift. CHANGELOG: [0.9.0] was dated and written as though released, with an [Unreleased] section describing a CocoaPods failure in "0.9.0's spec" — but 0.9.0 was never tagged and PolyVoice was never registered on trunk. Folded [Unreleased] in, redated, and dropped the phantom regression. 283 tests green, iOS build clean, all 18 examples build, podspec lints. * test(chat): deflake startNewSession by ordering the id change before sessionStart test_startNewSession_clearsTranscriptAndGreetsAgain failed ~1-in-5 (it went red on CI for PR #28, and reproduces on the pre-existing base commit too, so it is not new). The test emitted the new session's `sessionStart` immediately after `startNewSession()`, racing the session-id change. Those arrive on separate streams from different actors, and `applySessionIdChange` resets both `messages` and `hasStarted`. When it lands AFTER `sessionStart`: - the fresh greeting is wiped, and - `hasStarted` goes back to false — permanently, because `hasStarted` is only ever set by `sessionStart`, which has already been consumed. Now the test waits for the id change to be applied (observable as the cleared transcript) before replaying the new session's events, which is the order a real app sees anyway: create-session resolves before the socket opens. Also made the trailing `hasStarted` check an assertEventually — it was a synchronous assertion on asynchronously-delivered state. 20/20 in isolation, 3 clean full-suite runs. NOTE: the underlying product behaviour is still order-dependent — an id change delivered after a sessionStart leaves ChatSession.hasStarted stuck false. Reachable only if those two streams invert, which the real create-then-connect sequence makes unlikely, so this is a test fix, not a papering-over. Worth a follow-up on ChatSession's side.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Adds PolyVoice — live, two-way WebRTC voice calls to a PolyAI agent — as a separate product/pod (the iOS counterpart of Android's
ai.poly:voice), so chat-only apps never link the WebRTC binary. It reuses the messagingConfigurationand the sameCallState/PolyErrortypes.What's new
PolyVoice product/pod:
PolyVoice.call(config:options:)+VoiceOptions(webrtcToken:)(required, distinct from the API key). A realRTCPeerConnectionaudio engine (WebRTCCallMediaEngine) + accessory-awareAVAudioSession(AudioSessionController). WebRTC viastasel/WebRTCpinned M149, iOS-only (macOS test suite unaffected).Public media seam on
PolyMessaging(stays source-only):CallMediaEngine/CallMediaState/ICECandidate/IceServer/CallInterruptionmade public +PolyCall.wired(config:webrtcToken:mediaEngine:).Production hardening (full Android parity — added after review):
PolyError.Voice.disconnected/.interrupted(bothisRetryable); a post-connect drop maps to the retryable.disconnected.Examples / docs / release: SwiftUI + UIKit Voice examples,
docs/PolyVoice.mdguide + README section, version 0.9.0, newPolyVoice.podspec.Testing
swift test— 229 green (4 network-gated probes skipped), including new coordinator tests for reconnect (survive / exhaust), ICE-servers-reach-the-engine, the.disconnectedmapping, and interruption mute / restore / preserve-user-mute / end — plus ICE-response parsing tests.swift build+ iOSPolyVoicebuild clean. CI green (build-test,podspec-lint,pr-title).device_type/platform→ link → signaling → real offer).Notes
YOUR_API_KEY/YOUR_WEBRTC_TOKENplaceholders — no committed credentials.PolyMessaging.podspecto CocoaPods trunk beforePolyVoice.podspec(Voice has an exact-version dependency on Messaging 0.9.0).