Recurring own-device session corruption ("implausibly far ahead") and retry receipts failing with "backend store error" on fresh messages

[JeanCapixaba]

Setup: fleet of 17 independent accounts running headless group-reply bots on this crate. Git pin via [patch.crates-io] (all workspace crates on the same rev), SqliteStore (one whatsapp.db per account), tokio transport, Debian slim containers, large groups (several hundred devices each). Observed across revs d37b1756 (post #1043), ffed9e8d (post #1059, includes #1048) and 99d9aa89 (post #1137).

Symptom 1: session row toward the account's own primary device rots, bot goes deaf to its owner

Every few days one of the accounts starts failing like this:

• inbound messages from the owner's primary phone stop decrypting, with the counter error saying the message is "implausibly far ahead";
• every group send logs an encrypt failure toward one single address, always the account's own lid with device 0 (<owner-lid>@lid, device .0), while the other several hundred devices in the group encrypt fine.

From the owner's point of view the bot simply ignores their commands. We had five separate accounts hit within one week in late July. The most recent case (Jul 24) happened on ffed9e8d, which already includes #1048, so whatever that closed, this vector is still alive.

Workaround that cured every case with no re-pairing and no user action:

1. stop the container;
2. back up the db, then delete exactly one row: DELETE FROM sessions WHERE address = '<owner-lid>.0';
3. start again. The session renegotiates by itself within about 3 minutes and stays clean afterwards.

One detail that may help: the rotten session blob is consistently ~1096 bytes, while the freshly renegotiated one is ~414 bytes. We kept backups of the rotten rows.

Symptom 2: retry receipt for a message sent half a second earlier fails with "backend store error"

On the busiest accounts (notably the same ones that suffered symptom 1) many group replies produce this sequence:

15:30:44.559 app: reply sent, id=3EB01B6254FFBC8627472B
15:30:45.032 WARN whatsapp_rust::receipt: Failed to handle retry receipt for 3EB01B6254FFBC8627472B: backend store error in backend
15:30:47.142 app: same id, server ack 267ms, delivery receipts arriving normally

Some device in the group asks for a retransmission of a message we sent 0.5s earlier and serving that retry dies in the store layer. The message itself is fine for the rest of the group (server ack, delivery receipts and group confirmations all normal), so the device behind the failed retry presumably stays on the permanent "waiting for this message" placeholder.

Rates are identical on ffed9e8d and 99d9aa89 (we ran both on the same day on the same accounts), so this is not a regression from the recent perf work; it looks older.

We also see bursts of the same warning for pre-restart message ids in the first minutes after a container restart (80 to 110 within 20 minutes on busy accounts). If retransmission is served from an in-memory cache, that part is probably expected after a restart; the fresh-id case above is the one that looks wrong.

Hypothesis

The overlap between the two symptoms makes us suspect a single cause: one peer device with a corrupted session row. That device cannot decrypt anything we send, requests a retransmission every time, and the retry handler fails on the same rotten row with the store error. Symptom 1 would be the special case where the rotten row is the one toward our own account's primary device, which is also the loudest one because it makes the bot deaf to its owner.

Questions

• Is this corruption vector known? Was perf(signal): reuse leases across ciphertext APIs #1048 expected to close it? We got a fresh case on a rev that includes it.
• For the failed retry receipts, would a debug log of the requesting participant help pin the device down? We can build with extra logging if you point us at the right spot.
• We can share redacted rotten-row blobs, full timelines and db dumps privately if useful.

[JeanCapixaba]

Field report after deploying 8bf3466f (includes #1149 and #1151) to the same fleet, about 30 minutes in:

• The retry receipt failures on freshly sent messages are gone. Baseline measured this afternoon on ffed9e8d: 4 cases in a 20 minute window on the two busiest accounts, roughly one per group reply. Same filter after the deploy: zero, with replies flowing in the same groups.
• The post-restart bursts of Failed to handle retry receipt ... backend store error are also gone. Every restart today on ffed9e8d produced 80 to 110 of them within minutes on busy accounts; the rolling restart onto 8bf3466f produced zero across all accounts.
• For the own-device corruption we will keep watching over the next days, since that one needs a restart plus accumulated traffic to show. Nothing so far.

Thanks for the fast turnaround and the clear diagnosis. The monologue-peer explanation matches exactly what we saw: it was always our own primary device, the one address that receives everything and almost never replies.