fix(recv): serialize the group inbound sender-key chain with a lock - #992
Conversation
process_group_enc_batch called group_decrypt (load -> advance chain -> store) with no lock, while the 1:1 receive path holds session_lock_for around its decrypt. So two workers for the same (group, sender) could advance the receiving sender-key ratchet at once and the last store wins, losing a chain advance and its persisted skipped-message keys. A duplicate worker is reachable when a live ChatLane is capacity-evicted and a later stanza recreates a second worker at the same generation; the global processing semaphore is widened online, so the per-chat worker is the only serializer. Result: later legitimate skmsg fail (bad-mac / NoSenderKeyState / spurious DuplicatedMessage) and retry-storm until the sender rotates an SKDM. Acquire sender_key_lock(sender_key_name) around group_decrypt, mirroring the 1:1 path. The lock is the shared per-(group, sender) mutex from the signal cache, so two inbound workers serialize; it wraps only the ratchet mutation, released before plaintext handling.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Warning Review limit reached
Next review available in: 30 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Repository UI (base), Organization UI (inherited) Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThis PR adds explicit sender-key chain locking around group message decryption in ChangesSender-key lock for group decryption
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
Suggested reviewers: Look, this is exactly the kind of fix we need — concurrency bugs in messaging are not something you move fast and break things on. Locking the sender-key chain before decrypt is the right call, non-negotiable for correctness at scale. The tests cover both the happy path and the undecryptable path, which is good, because at our scale, edge cases aren't edge cases — they're Tuesday. Ship it, but I want the lock contention monitored in production. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/message/receive.rs`:
- Around line 1228-1235: Hoist the sender-key lock lookup out of the payload
loop in receive.rs: `sender_key_name` is already ثابت for the batch, so avoid
calling `adapter.sender_key_store.sender_key_lock(&sender_key_name).await` on
every iteration. Fetch the `Arc<Mutex<()>>` once before the loop in the same
`receive`/payload-processing flow, then only call `.lock().await` inside the
loop before `group_decrypt`, keeping the existing `group_decrypt` and
`sender_key_store` usage unchanged.
In `@src/message/tests.rs`:
- Around line 6673-6678: The test in message tests uses a fixed
tokio::time::sleep before asserting decrypted group content, which can be flaky
under load. Update this happy-path case to use the same polling/retry approach
as the undecryptable-event test nearby: repeatedly check message_texts_for_id
for the expected "hello group" result until it appears or a timeout is reached.
Keep the assertion anchored around the existing message_texts_for_id helper and
the chain-lock decrypt behavior so the test waits for dispatch completion
instead of assuming a single 50ms delay is enough.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 9cd7844f-b2e6-4a27-80e4-252484232c5b
📒 Files selected for processing (3)
src/message.rssrc/message/receive.rssrc/message/tests.rs
There was a problem hiding this comment.
2 issues found across 3 files
Confidence score: 4/5
- In
src/message/tests.rs, the fixed 50ms sleep can make the async event assertion flaky, so merges could intermittently fail CI even when runtime behavior is correct — switch to polling for the expected text with a bounded timeout before merging to de-risk test stability. - In
src/message/receive.rs, repeatedly callingsender_key_lock(&sender_key_name).awaitinside the payload loop adds avoidable async lookup overhead, which could degrade batch processing efficiency as payload counts grow — hoist the lock lookup outside the loop (or track as a near-term follow-up).
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
📦 Binary size report
.text per crate
Baseline: |
…poll in test sender_key_name is loop-invariant, so fetch the sender_key_lock Arc once before the payload loop and just .lock() it per iteration around group_decrypt, instead of re-looking it up each payload. Tighten the comment to the why. Replace the happy-path test's fixed 50ms sleep with the same bounded poll the bad-path test uses, so a loaded CI runner can't flake it.
There was a problem hiding this comment.
0 issues found across 2 files (changes from recent commits).
Auto-approved: Adds missing sender-key lock to prevent race in group decryption, with tests. Low-risk synchronization fix.
Re-trigger cubic
What
Acquire the per-
(group, sender)sender-key chain lock aroundgroup_decryptin the inboundprocess_group_enc_batch, mirroring what the 1:1 receive path already does withsession_lock_for.Why (bug)
process_group_enc_batchcalledgroup_decrypt(load → advance chain → store, non-atomic) with no lock, while the sibling 1:1 path holdssession_lock_foraround its decrypt. So two workers for the same(group, sender)could advance the receiving sender-key ratchet concurrently, and the last store wins — losing a chain advance and its persisted skipped-message keys.A duplicate worker is reachable: a live
ChatLanegets capacity-evicted (its worker keeps drainingrx, only exiting on aconnection_generationmismatch), and a later stanza for that chat misses the cache and spawns a second worker at the same generation. The global processing semaphore doesn't help — it's widened online, so the per-chat worker is the only serializer.Result: later legitimate skmsg fail (bad-mac /
NoSenderKeyState/ spuriousDuplicatedMessage) and retry-storm until the sender rotates an SKDM.How
group_decryptinsender_key_lock(&sender_key_name).lock(). The lock is the shared per-(group, sender)mutex from the signal cache (same instance the store hands out everywhere), so two inbound workers serialize on it.handle_decrypted_plaintext).Note: this closes the ratchet-corruption (P1) regardless of the duplicate-worker window. The eviction-spawns-a-duplicate-worker root cause (and the 1:1 FIFO-ordering symptom) is a separate concern handled elsewhere;
sender_key_lockhere makes the group path robust to it either way.Tests
group_skmsg_decrypts_under_sender_key_lock(happy) — an skmsg with an established sender key decrypts through the batch under the new lock and surfaces its content.group_skmsg_without_sender_key_takes_retry_path(bad) — an skmsg whose sender key was never distributed hitsNoSenderKeyStateunder the lock and takes the retry path (one undecryptable event, no user content).cargo fmt/clippyclean; the 48 group/skmsg receive tests pass.