Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 16 additions & 3 deletions advanced/signal-protocol.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -1137,14 +1137,27 @@ impl SessionStore for SessionAdapter {

*When* the dirty Signal cache reaches the backend differs by direction, because the two directions have different recovery properties:

- **Send (DM/1:1 sessions)** persists through a batched **counter lease**. `SessionRecord` reserves its outbound sender-chain counter `SENDER_CHAIN_RESERVATION_BATCH` (64) values at a time, via `SessionRecord::reserve_sender_chain_counters`. A send covered by an unexhausted lease is already durable — it only schedules the same coalesced write-behind as the receive path below. The send that exhausts the lease, roughly 1 in 64, raises the ceiling and flushes **synchronously, before the stanza reaches the wire**. If that flush fails, the send aborts instead of transmitting an advance it couldn't save. Reusing an outbound counter reuses its message key and IV, so no counter can ever be used before its lease is durable. The lease field is local-only: it's field 100 in the encoded `SessionRecord`, outside the vendored `whatsapp.proto`. On every load, `SessionRecord::deserialize` fast-forwards the sender chain to the lease ceiling, so a crash or reconnect mid-lease can never re-derive a possibly-spent counter.
- **Send (group and status sends)** ignores the lease. It still flushes **synchronously, before the stanza reaches the wire**, on every send. Sender-key leasing is a potential follow-up, not implemented yet.
- **Send (DM/1:1 sessions)** persists through a batched **counter lease**. `SessionRecord` reserves its outbound sender-chain counter `SENDER_CHAIN_RESERVATION_BATCH` (64) values at a time, via `SessionRecord::reserve_sender_chain_counters`. A send covered by an unexhausted lease is already durable — it only schedules the same coalesced write-behind as the receive path below. The send that exhausts the lease, roughly 1 in 64, raises the ceiling and flushes **synchronously, before the stanza reaches the wire**. If that flush fails, the send aborts instead of transmitting an advance it couldn't save. Reusing an outbound counter reuses its message key and IV, so no counter can ever be used before its lease is durable. The lease field is local-only: it's field 100 in the encoded `SessionRecord`, outside the vendored `whatsapp.proto`. By default (`SessionRecord::deserialize`), every load fast-forwards the sender chain to the lease ceiling, so a crash mid-lease can never re-derive a possibly-spent counter — the store-backed load path relaxes this only for a *trusted* reload, via the incarnation marker described below.
- **Send (group and status sends)** follows the same lease pattern as DMs. `SenderKeyRecord` reserves its outbound chain iteration `SENDER_CHAIN_RESERVATION_BATCH` (64) at a time via `SenderKeyRecord::reserve_iterations`, using the same field-100 local-only encoding and the same fast-forward-on-load recovery as `SessionRecord`. A send within an unexhausted lease rides the coalesced write-behind; only the send that raises the ceiling, roughly 1 in 64, flushes synchronously before the stanza reaches the wire. Status posts go through the same group-encrypt path and inherit this behavior; status *reactions* are a DM-branch send and always followed the DM lease instead. Production encryption (`wacore::send::encrypt_group_message`) delegates to the same `group_encrypt` primitive these guarantees live in, so there is exactly one sender-key encrypt/advance/store implementation — earlier, a second unguarded copy on the production path meant most warm group sends skipped the pre-wire flush entirely.
- **Receive** (live traffic, outside the offline-drain batcher) routes through a single-flight coalescing scheduler (`src/signal_flush.rs`) instead of flushing per stanza: a burst of receives folds into one flush per ~25ms window, retried with exponential backoff (up to a 5s cap) on backend failure. This is safe because a lost receive-side advance simply re-derives forward on the next message (the receiving chain derives `CK_n → CK_n+1`), and a consumed one-time prekey stays buffered until its session is durable — a crash inside the window is recoverable.

<Warning>
Downgrading to a version that predates the counter lease after running a leased version: the older version ignores the lease field and could reuse counters that were only reserved (not yet actually sent) by the lease. Avoid downgrading a device's local state across this boundary.
Downgrading to a version that predates the counter lease after running a leased version: the older version ignores the lease field(s) and could reuse counters/iterations that were only reserved (not yet actually sent) by the lease. This applies to `SenderKeyRecord` (group/status sends) as well as `SessionRecord` (DM sends). Avoid downgrading a device's local state across this boundary.
</Warning>

### Clean reload vs. crash recovery

Fast-forwarding past a lease's reserved ceiling on every reload is the safe default, but it's also overly conservative for the common case: a clean reconnect or a same-process store re-creation never actually risked losing an in-flight send, yet unconditionally fast-forwarding still burned a full unused batch every time. 32 clean reconnects could push a sender-key chain 2,048 iterations ahead and get rejected once a peer who missed the intervening messages hit `MAX_FORWARD_JUMPS` (2,000).

`SignalStoreCache`, and the direct, non-cached `Device` store, now tag a durably-reserved record with a random 128-bit **store incarnation**, carried in a second local-only field (101, alongside the lease's field 100) on both `SessionRecord` and `SenderKeyRecord`:

- A live `SignalStoreCache` generates one incarnation marker when it's constructed. A reload that observes a matching marker proves the record came from *this same live cache* and was never lost to a crash, so it's exact: the chain resumes at the next iteration instead of fast-forwarding.
- A reload with no marker, a different marker, or a malformed/duplicated marker is untrusted and keeps the conservative fast-forward — this covers process restarts, a freshly constructed cache, and any genuinely lossy discard.
- [`SignalStoreCache::clear_after_flush()`](/concepts/architecture#disconnect-cleanup), used by connection teardown, only evicts a store once it's fully settled — no dirty, deleted, checked-out, or pending-wire-gate entries. Anything a concurrent write installs after the flush stays resident (and the incarnation stays put) until a later flush settles it; only an actual lossy discard rotates the marker.
- Direct `Device` stores hold one process-level incarnation in an `OnceLock` instead of a per-cache one. Since these stores synchronously await the backend write before returning ciphertext, a new `Device` wrapping the same backend in the same process is a clean, trusted reload; a process restart gets a fresh marker and stays conservative.

This adds no field to the public `Device` struct and no synchronous I/O — a marker is generated only when a cache or process starts, or when a lossy boundary invalidates trust.

The scheduler is generation-scoped (embeds the connection generation in its atomic state), so a reconnect during an in-flight flush needs no explicit reset: a stale worker from the previous connection cannot mutate the new generation's state, and stands down when it observes a foreign generation.

The offline drain, retry-receipt recovery, identity-change recovery, and teardown all keep their own **synchronous** flushes — they gate acks, receipts, or follow-up reads on durability and are not routed through the receive coalescer. See [Inbound Durability Hook](/advanced/inbound-durability) for the drain-batch commit ordering, which this coalescing does not change.
Expand Down
2 changes: 1 addition & 1 deletion api/client.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -1463,7 +1463,7 @@ pub async fn flush_pending_signal_state(&self) -> Result<(), anyhow::Error>

Forces any pending write-behind Signal cache state to the backend, returning once the flush completes (or fails).

Ordinarily — without calling this method — the backend trails the in-memory cache. Most DM sends are already covered by a durable sender-chain counter lease, so they only schedule the coalesced write-behind; only the roughly-1-in-64 send that exhausts the current lease flushes synchronously — and because the pre-wire flush check is global, a pending flush on an unrelated session can force a synchronous flush too. Group sends and status posts flush synchronously on every send; status reactions are the DM-branch exception and follow the lease behavior instead. The live receive path always schedules a coalesced flush, on a ~25ms window (see [flush scheduling](/advanced/signal-protocol#flush-scheduling-send-vs-receive)).
Ordinarily — without calling this method — the backend trails the in-memory cache. Most sends — DM, group, and status alike — are already covered by a durable lease (`SessionRecord`'s sender-chain counter lease for DMs, `SenderKeyRecord`'s chain iteration lease for group/status), so they only schedule the coalesced write-behind; only the roughly-1-in-64 send that exhausts the current lease flushes synchronously — and because the pre-wire flush check is global, a pending flush on an unrelated session or sender key can force a synchronous flush too. Status reactions are the DM-branch exception and follow the DM lease behavior instead of the group/status one. The live receive path always schedules a coalesced flush, on a ~25ms window (see [flush scheduling](/advanced/signal-protocol#flush-scheduling-send-vs-receive)).

A successful call to `flush_pending_signal_state()` closes that gap deterministically: everything dirty as of the call is persisted by the time it returns `Ok`. The call has **no hard wall-clock bound** — it can wait on locks, or on slow or failing storage, and a backend outage extends it until the retry loop succeeds. Check the returned `Result`: a failure means the flush did not complete, and state is still pending, not persisted.

Expand Down
2 changes: 1 addition & 1 deletion api/send.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ pub async fn send_message(
</ResponseField>

<Note>
For DMs, the outbound Signal ratchet advance is persisted through a batched counter lease. The sender-chain counter is reserved 64 at a time, so most sends are already covered by a durable lease and only schedule a coalesced write-behind — though the pre-wire flush check is global, so a pending flush on an unrelated session can still force this send to flush synchronously. The send that exhausts the current lease, roughly 1 in 64, persists to the backend **synchronously, before the stanza is transmitted**. Group sends, and status posts sent via `client.status()`, always persist their sender-key ratchet advance synchronously, before the stanza is transmitted. Status *reactions* (`send_reaction` targeting `status@broadcast`) are the exception: they route through the same DM branch as an ordinary 1:1 message, addressed to the status author's device, so they follow the DM counter-lease behavior instead. Reusing an outbound counter would reuse its message key and IV, so the advance is always durable before it can be reused. If a required persistence write fails, `send_message` returns `Err` instead of transmitting an advance that couldn't be saved. See [Signal Protocol — flush scheduling](/advanced/signal-protocol#flush-scheduling-send-vs-receive) for the full durability model.
The outbound Signal ratchet advance is persisted through a batched counter lease, for DMs and for group/status sends alike. For DMs, `SessionRecord` reserves its sender-chain counter 64 at a time; for group sends and status posts sent via `client.status()`, `SenderKeyRecord` reserves its chain iteration 64 at a time the same way. Most sends are already covered by a durable lease and only schedule a coalesced write-behind — though the pre-wire flush check is global, so a pending flush on an unrelated session or sender key can still force this send to flush synchronously. The send that exhausts the current lease, roughly 1 in 64, persists to the backend **synchronously, before the stanza is transmitted**. Status *reactions* (`send_reaction` targeting `status@broadcast`) are the exception: they route through the same DM branch as an ordinary 1:1 message, addressed to the status author's device, so they follow the DM counter-lease behavior instead of the group/status one. Reusing an outbound counter or iteration would reuse its message key and IV, so the advance is always durable before it can be reused. If a required persistence write fails, `send_message` returns `Err` instead of transmitting an advance that couldn't be saved. See [Signal Protocol — flush scheduling](/advanced/signal-protocol#flush-scheduling-send-vs-receive) for the full durability model.
</Note>

### SendResult
Expand Down
4 changes: 3 additions & 1 deletion concepts/architecture.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -347,7 +347,7 @@ When a connection is lost or `disconnect()` is called, `cleanup_connection_state
| `is_connected` | Set to `false` (Release ordering) | After socket is `None` so no task sees connected with a cleared socket |
| `chat_lanes` | Invalidated | Drop per-chat queue senders so workers exit via channel close — prevents stale workers from the old connection surviving reconnects with outdated signal/crypto state |
| `pending_retries` | Cleared | Stale keys from detached scope guard cleanup would otherwise suppress the first retry after reconnect |
| `signal_cache` | **Flushed, then cleared** | Pending identity / sender-key writes are persisted to the backend *before* the cache is dropped. If the flush fails, the cache is kept (not cleared) so dirty state survives. |
| `signal_cache` | **Flushed, then settled entries cleared** | Pending identity / session / sender-key writes are persisted to the backend before eviction. If the flush fails, the cache is kept (not cleared) so dirty state survives. Even after a successful flush, an entry a concurrent write touched in the gap stays resident — only fully-settled entries are evicted. |
| `response_waiters` | Drained | Pending IQ waiters fail fast with `InternalChannelClosed` instead of hanging until the 75s timeout |
| Offline sync state | Reset | Counters, timing, and semaphore replaced with fresh single-permit instance |
| Dead-socket timestamps | Reset to 0 | Prevents stale values from triggering an immediate reconnect on the next connection |
Expand All @@ -359,6 +359,8 @@ Chat lane invalidation is critical for correctness. Without it, stale message pr

<Note>
**Flush-before-clear (v0.6).** The signal cache is now flushed to the backend before being cleared on disconnect. Previously the cache was dropped immediately, so a just-advanced sender-key chain that hadn't been persisted yet was lost — on the next send the client would treat the chain as fresh and re-distribute the SKDM to every group device unnecessarily. Disconnect is therefore no longer a "forget all Signal state" operation; it's a "persist, then forget" one. This is a behavior change for anyone relying on the old drop-everything semantics.

**Settled-only eviction.** Teardown now calls `SignalStoreCache::clear_after_flush()` instead of an unconditional clear. A write that lands between the flush releasing its lock and teardown running — e.g. a concurrently raised sender-key reservation — used to be silently discarded, which could let ciphertext reach the wire before its new durability ceiling was ever persisted. Teardown now only evicts a store (sessions, identities, sender keys) once it has no dirty, deleted, checked-out, or pending-wire-gate entries; anything installed after the flush stays resident for the next successful flush to settle. See [Signal Protocol — clean reload vs. crash recovery](/advanced/signal-protocol#clean-reload-vs-crash-recovery) for how this interacts with the counter/iteration lease.
</Note>

**Stream error behavior:**
Expand Down