diff --git a/guides/voip-calls.mdx b/guides/voip-calls.mdx index ff264efa..a3320b5e 100644 --- a/guides/voip-calls.mdx +++ b/guides/voip-calls.mdx @@ -1,26 +1,28 @@ --- title: VoIP Calls -description: Place and answer end-to-end encrypted 1:1 voice calls with whatsapp-rust +description: Place and answer end-to-end encrypted 1:1 voice and video calls with whatsapp-rust --- ## Overview -whatsapp-rust supports end-to-end encrypted 1:1 voice calls that interoperate with the official WhatsApp app. The full media path is implemented in pure Rust: mic capture, encoding, E2E-SRTP encryption, relay transport, decryption, decoding, and playout. +whatsapp-rust supports end-to-end encrypted 1:1 voice and video calls that interoperate with the official WhatsApp app. The full audio media path is implemented in pure Rust — encoding, E2E-SRTP encryption, relay transport, decryption, and decoding — while you supply mic capture and speaker playout (see Audio I/O below). + +Video is **codec-neutral**: you hand the library complete H.264 Annex-B access units and it owns signaling, RTP packetization/reassembly, E2E-SRTP encryption, relay transport, and PLI/FIR-driven keyframe recovery. Encoding, decoding, capture, and display stay outside the library — the bundled CLI example drives them through `ffmpeg`/`ffplay`. - Voice calling is behind the optional `voip` feature flag. The default build is entirely unaffected — none of the codec or relay dependencies are compiled unless you opt in. + Voice and video calling are behind the optional `voip` feature flag. The default build is entirely unaffected — none of the codec or relay dependencies are compiled unless you opt in. -## Enabling the Feature +## Enabling the feature - The `voip` feature landed on `main` with [PR #918](https://github.com/oxidezap/whatsapp-rust/pull/918) and will be included in the next published release. Until then, depend on the git source: + The `voip` feature landed on `main` with [PR #918](https://github.com/oxidezap/whatsapp-rust/pull/918) (audio) and [PR #1024](https://github.com/oxidezap/whatsapp-rust/pull/1024) (video) and will be included in the next published release. Until then, depend on the git source: ```toml [dependencies] whatsapp-rust = { git = "https://github.com/oxidezap/whatsapp-rust", features = ["voip"] } -async-channel = "2" # needed to implement AudioSource / AudioSink +async-channel = "2" # needed to implement AudioSource / AudioSink / VideoSource / VideoSink tokio = { version = "1.48", features = ["macros", "rt-multi-thread"] } ``` @@ -46,12 +48,17 @@ async fn on_event(client: &Client, event: Event) -> anyhow::Result<()> { match event { Event::IncomingCall(incoming) => { // Borrow the action so `incoming` is not partially moved before `.accept(&incoming)`. - if matches!(&incoming.action, CallAction::Offer { .. }) { - let handle = client.voip() + if let CallAction::Offer { is_video, .. } = &incoming.action { + let mut call = client.voip() .accept(&incoming) - .audio(mic_source, speaker_sink) - .start() - .await?; + .audio(mic_source, speaker_sink); + if *is_video { + // Required to answer a video-from-the-start offer. + call = call.video(camera_source, video_sink); + } + // If `is_video` was false, `start()` accepts audio-only — video can still be + // added later via `CallHandle::start_video`. + let handle = call.start().await?; // Resolves when either side hangs up handle.wait_ended().await; @@ -77,6 +84,7 @@ let peer: Jid = "15551234567@s.whatsapp.net".parse()?; let handle = client.voip() .call(&peer) .audio(mic_source, speaker_sink) + .video(camera_source, video_sink) // omit for an audio-only call .start() .await?; @@ -98,13 +106,15 @@ handle.wait_ended().await; You supply the audio I/O by implementing the `AudioSource` and `AudioSink` traits. Both traits are channel-based — the library reads from a `Receiver` and writes decoded PCM to a `Sender`. The bundled `examples/voip-cli/src/main.rs` wires up [cpal](https://crates.io/crates/cpal)/PipeWire as a reference. -The CLI exposes three subcommands: +The CLI exposes three subcommands, each accepting a trailing `--video`: | Subcommand | Description | |------------|-------------| -| `loopback` | Mic → Opus → E2E-SRTP protect/unprotect → Opus → speaker. No WhatsApp connection — hear yourself processed by the full VoIP stack. | -| `listen [accept]` | Connect to WhatsApp and print incoming calls. Rejects by default; pass `accept` to auto-answer. | -| `call ` | Connect to WhatsApp and place an outgoing call to the given JID. | +| `loopback [--video]` | Mic → Opus → E2E-SRTP protect/unprotect → Opus → speaker. No WhatsApp connection — hear yourself processed by the full VoIP stack. `--video` replaces this with a separate, video-only loopback: an ffmpeg source is looped straight into an ffplay window with no E2E-SRTP step, unlike the audio path — it's a pipeline check for the capture/encode/render plumbing, not the video RTP/SRTP plane. | +| `listen [accept] [--video]` | Connect to WhatsApp and print incoming calls. Rejects by default; pass `accept` to auto-answer. `--video` implies `accept` too (there's no point answering with video and then rejecting), and the accepted call answers with video media. | +| `call [--video]` | Connect to WhatsApp and place an outgoing call to the given JID. With `--video` it's a video call from the start. | + +During a live call, single-key stdin commands (terminal only) work regardless of how the call started: `v` toggles video — upgrades to video, accepts a pending peer upgrade request, or downgrades back to audio — and `q` performs a signaled hangup. To test the audio stack locally without a WhatsApp session: @@ -143,6 +153,80 @@ impl AudioSink for MySpeaker { The library owns the call key, relay handshake, codec, and crypto. You only provide mic input and speaker output. +## Video I/O + +Video works the same way, one layer up: you supply `VideoSource` and `VideoSink` implementations that hand the library complete **H.264 Annex-B access units** (start codes included). The library never touches pixels — encoding and decoding are entirely your consumer's responsibility (the CLI example shells out to `ffmpeg`/`ffplay`). + +```rust +use whatsapp_rust::voip::{VideoSource, VideoSink, VideoFrame}; +use async_channel; + +struct MyCamera { receiver: async_channel::Receiver> } +struct MyDisplay { sender: async_channel::Sender } + +impl VideoSource for MyCamera { + // Each item is one complete H.264 Annex-B access unit. + fn frames(&self) -> async_channel::Receiver> { + self.receiver.clone() + } + + // RTP clock increment between access units (90_000 / fps). Must be non-zero; + // defaults to the 15 fps cadence if not overridden. + fn rtp_timestamp_stride(&self) -> u32 { + 90_000 / 20 // 20 fps + } +} + +impl VideoSink for MyDisplay { + fn playout(&self) -> async_channel::Sender { + self.sender.clone() + } +} +``` + +`VideoFrame` carries the reassembled peer access unit plus `keyframe` (safe point to (re)start a decoder) and `orientation` (from `