fix(retry): bound outbound resend rate per group to prevent AccountLocked by jlucaso1 · Pull Request #871 · oxidezap/whatsapp-rust

jlucaso1 · 2026-06-15T12:44:27Z

Problem

A bot running this lib hit <failure reason="403" location="vll"/> (AccountLocked). Root cause from production logs: a sustained high rate of outbound retry resends to a single group. During a mass PN->LID migration, hundreds of distinct devices each fail to decrypt the same messages and each send a retry receipt, and we pairwise-resend to every one. The signal WhatsApp anti-abuse penalizes is the aggregate per-chat resend rate (many distinct devices), not any single device's depth. The existing MAX_RETRY_COUNT=5 trusts the peer-echoed count, which stays at 1, so it never fires.

This builds on the investigation in #870 by @Salientekill (thanks for the logs and the diagnosis). Measured against the real logs, a per-(chat,msg,requester) cap cuts under 0.5% of the storm, because the storm is a cross-device fan-out rather than a single device looping, so a per-device key never accumulates. The fix has to aggregate across devices.

Fix

A per-group token-bucket rate limiter on the resends we actually perform.

Keyed by chat, so it bounds the aggregate resend rate regardless of how many distinct devices drive it.
Drops (does not queue) when over budget: the requester was already marked for fresh SKDM earlier in the retry path, so a throttled device still recovers on the next normal group send and re-requests this message on its own timer. No timers, no queue, hot path stays allocation-free.
Lazy refill off the monotonic clock: correct over long sessions, immune to wall-clock jumps, no background task. Capacity-only cache bounds memory; the same-chat read-modify-write is race-free via single-flight get_with plus a per-bucket mutex.
Group-only: DMs have no SKDM fallback, so they keep the unconditional resend (bounded by MAX_RETRY_COUNT) rather than risk a dropped delivery.
Conservative defaults (burst 20, refill 10/min per chat), tunable live via Client::set_resend_rate_limit and BotBuilder::with_resend_rate_limit. resends_throttled_total() surfaces storm chats.

Not WA Web behavior (WA Web caps only on the peer-echoed count plus delivery eligibility); added as a deliberate safety valve for the production 403, since the fan-out is something WA Web itself does not defend against.

Tests

Unit (resend_rate_limiter): under/over burst, refill over time, burst cap, disabled, per-chat isolation, live retune clamp, and a multi-threaded no-bypass-under-contention test.
Integration (retry): the limiter is reachable and tunable through the public Client API, and handle_retry_receipt drops a throttled group resend (returns Ok, sends nothing) while keeping the message cached and clearing the in-progress marker.

Verify

cargo fmt --all, cargo clippy --all-targets -- -D warnings clean.
cargo test -p whatsapp-rust --lib (826 tests) pass.

…cked A bot running this lib hit a 403 AccountLocked. Root cause from production logs: a sustained high rate of outbound retry resends to a single group. During a mass PN->LID migration, hundreds of distinct devices each fail to decrypt the same messages and each send a retry, and we pairwise-resend to every one. The penalized signal is the aggregate per-chat resend rate, not any single device's depth, so MAX_RETRY_COUNT (which trusts the peer-echoed count, stuck at 1) never fires and a per-(chat,msg,requester) cap cuts under 0.5% of the storm. Add a per-group token-bucket rate limiter on the resends we perform, keyed by chat so it bounds the aggregate rate regardless of how many devices drive it. It drops rather than queues when over budget: the requester was already marked for fresh SKDM, so it recovers on the next send and re-requests on its own timer. Refill is lazy off the monotonic clock (no timers, correct over long sessions), the bucket cache is capacity-only (bounded memory), and the same-chat read-modify-write is race-free via single-flight get_with plus a per-bucket mutex. Group-only because DMs have no SKDM fallback. Defaults are conservative (burst 20, refill 10/min) and tunable live via Client::set_resend_rate_limit and BotBuilder::with_resend_rate_limit; resends_throttled_total surfaces storm chats. Builds on the investigation in #870.

coderabbitai · 2026-06-15T12:44:40Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: c1c431ca-54e4-46ee-b54f-1aa6838403da

📥 Commits

Reviewing files that changed from the base of the PR and between a725074 and aff6b2a.

📒 Files selected for processing (8)

src/bot.rs
src/cache_config.rs
src/client.rs
src/client/accessors.rs
src/client/lifecycle.rs
src/lib.rs
src/resend_rate_limiter.rs
src/retry.rs

📝 Walkthrough

Summary by CodeRabbit

New Features
- Added per-chat outbound resend rate limiting to prevent excessive message resends
- Rate limit parameters are configurable at startup and dynamically adjustable
- Added monitoring metrics to track throttled resends
Tests
- Added unit and integration tests for rate limiter configuration and behavior

Walkthrough

Adds a per-chat outbound resend rate limiter using a token-bucket algorithm. A new ResendRateLimiter module stores one Mutex<TokenBucket> per Jid in a Cache, supports live retuning via atomics, and gates group retries in handle_retry_receipt. The limiter is initialized in Client, configured via CacheConfig, exposed through BotBuilder, and surfaced in MemoryDiagnostics.

Changes

Per-chat outbound resend rate limiter

Layer / File(s)	Summary
ResendRateLimiter core: TokenBucket and limiter `src/lib.rs`, `src/resend_rate_limiter.rs`	New `pub(crate)` module adds `TokenBucket` (fractional token refill, monotonic `try_take`) and `ResendRateLimiter` (per-`Jid` `Cache<Mutex<TokenBucket>>`, atomic live rate retuning via `set_rate`, async `try_acquire`, `throttled_total`, `entry_count`), plus `DEFAULT_RESEND_BURST = 20` and `DEFAULT_RESEND_REFILL_PER_MIN = 10` constants. Unit and concurrency tests are included.
Client struct, construction, and diagnostics `src/cache_config.rs`, `src/client.rs`, `src/client/lifecycle.rs`, `src/client/accessors.rs`	`CacheConfig` gains `resend_rate_limiter_capacity` (default 4096). `Client` gains a `resend_rate_limiter` field initialized in `new_with_cache_config`. `MemoryDiagnostics` gains `resend_rate_limiter_chats` and `resends_throttled_total` fields populated in `memory_diagnostics`. New public accessors `set_resend_rate_limit` and `resends_throttled_total` are added to `Client`.
Resend gate in handle_retry_receipt `src/retry.rs`	`handle_retry_receipt` inserts a group-only `resend_rate_limiter.try_acquire` check and returns `Ok(())` without resending when throttled. Two new tests verify API wiring and end-to-end throttle behavior (dropped resend, preserved cached message, cleared pending-retry marker).
BotBuilder configuration `src/bot.rs`	Adds `resend_rate_limit: Option<(u32, u32)>` to `BotBuilder` state, initializes to `None`, carries through typestate `cast`, adds `with_resend_rate_limit(burst, refill_per_min)` builder method, and applies it via `client.set_resend_rate_limit` in `build_graph`.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

oxidezap/whatsapp-rust#550: Modifies handle_retry_receipt resend-path logic in the same function where this PR inserts the rate-limiting gate, making them directly overlapping in src/retry.rs.

Suggested labels

api-design

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately captures the main change: implementing a per-group resend rate limiter to prevent AccountLocked errors, which is the core objective of this PR.
Description check	✅ Passed	The description is comprehensive and directly related to the changeset, explaining the problem, solution, implementation details, configuration options, and test coverage.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch fix/per-chat-resend-rate-limit

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

github-actions · 2026-06-15T12:48:40Z

📦 Binary size report

Metric	main	PR	Δ
bin size (stripped)	10.58 MiB	10.59 MiB	+11.28 KiB (+0.10%) 🔺
bin .text	8.67 MiB	8.68 MiB	+10.81 KiB (+0.12%) 🔺
bin allocated (text+data+bss)	10.58 MiB	10.59 MiB	+12.12 KiB (+0.11%) 🔺
llvm-lines wacore	644,384	644,384	0
llvm-lines wacore copies	17,673	17,673	0
llvm-lines whatsapp-rust lib	652,742	657,128	+4,386 (+0.67%) 🔺
llvm-lines whatsapp-rust lib copies	19,857	20,013	+156 (+0.79%) 🔺
deps crates (Cargo.lock)	354	354	0

.text per crate

Crate	main	PR	Δ
.text whatsapp_rust	1.47 MiB	1.48 MiB	+8.51 KiB (+0.57%) 🔺
.text wacore	543.77 KiB	543.77 KiB	0
.text wacore_binary	103.64 KiB	103.64 KiB	0
.text wacore_libsignal	168.32 KiB	168.32 KiB	0
.text wacore_appstate	35.26 KiB	35.26 KiB	0
.text wacore_noise	30.68 KiB	30.68 KiB	0
.text waproto	895.34 KiB	895.34 KiB	0
.text whatsapp_rust_sqlite_storage	206.21 KiB	206.21 KiB	0
.text whatsapp_rust_tokio_transport	33.09 KiB	33.09 KiB	0
.text whatsapp_rust_ureq_http_client	6.19 KiB	6.19 KiB	0
.text std	1.13 MiB	1.13 MiB	+2.34 KiB (+0.20%) 🔺
.text other deps	4.02 MiB	4.02 MiB	-57 B (-0.00%) 🔽

Top movers (cargo-bloat attribution)

Crate	main	PR	Δ
whatsapp_rust	1.47 MiB	1.48 MiB	+8.51 KiB (+0.57%)
std	1.13 MiB	1.13 MiB	+2.34 KiB (+0.20%)
prost	466.39 KiB	464.26 KiB	-2.13 KiB (-0.46%)
regex_automata	1.61 KiB	2.88 KiB	+1.27 KiB (+79.10%)

Baseline: a725074cb (latest main run) · Head: d43d4b569 · Graphs

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: aff6b2ad14

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

codspeed-hq · 2026-06-15T12:52:00Z

Merging this PR will not alter performance

✅ 172 untouched benchmarks

_{Comparing fix/per-chat-resend-rate-limit (aff6b2a) with main (a725074)}

cubic-dev-ai

1 issue found across 8 files

_{Reply with feedback, questions, or to request a fix.

Re-trigger cubic}

A chronically undecryptable sender (a broken Signal session the peer never re-establishes, or a draining offline queue of messages that can no longer decrypt) fails every message, and we send a retry receipt for each one. The per-message cap (MAX_DECRYPT_RETRIES) bounds receipts per message id, not per sender, so a sender flooding distinct ids is never gated and the aggregate climbs into AccountLocked -- the same anti-abuse class the per-chat resend limiter (oxidezap#871) bounds for outbound resends. Observed in the wild: ~25k retry receipts from ~15 senders over two days. Add a per-sender token-bucket limiter mirroring resend_rate_limiter, keyed by sender. When a sender exhausts its budget, ack-and-drop the stanza (drain the offline queue so the server stops redelivering) instead of asking for a resend forever. The session still self-heals: the bucket refills lazily and any fresh handshake recovers it, so a transient desync (a handful of retries under the burst) is never penalized. The UndecryptableMessage event still fires, so app-side recovery (local session reset) is unaffected. Wired through the public Client API (set_retry_receipt_rate_limit / retry_receipts_throttled_total) and memory diagnostics, mirroring the resend limiter. TokenBucket is shared with resend_rate_limiter. Defaults: burst 10, refill 2/min, capacity 4096 senders.

coderabbitai Bot added the api-design label Jun 15, 2026

coderabbitai Bot approved these changes Jun 15, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Jun 15, 2026

View reviewed changes

Comment thread src/retry.rs

cubic-dev-ai Bot reviewed Jun 15, 2026

View reviewed changes

Comment thread src/retry.rs

jlucaso1 merged commit b23a9f7 into main Jun 15, 2026
19 checks passed

jlucaso1 deleted the fix/per-chat-resend-rate-limit branch June 15, 2026 13:02

jlucaso1 mentioned this pull request Jun 15, 2026

docs: per-chat resend rate limiter (whatsapp-rust PR #871) oxidezap/whatsapp-rust-docs#334

Closed

Salientekill mentioned this pull request Jun 28, 2026

feat(retry): bound inbound retry-receipt rate per sender #924

Closed

Salientekill mentioned this pull request Jul 4, 2026

feat(retry): quarantine per-(chat, requester) inbound retry receipts #982

Closed

coderabbitai Bot mentioned this pull request Jul 5, 2026

feat(retry): opt-in RetryAdmission hook (compliant supersede of #982) #985

Merged

marchugon mentioned this pull request Jul 5, 2026

Review of upstream PR #982 (oxidezap/whatsapp-rust) marchugon/whatsapp-rust#2

Open

Navigation Menu

Uh oh!

Uh oh!

fix(retry): bound outbound resend rate per group to prevent AccountLocked - #871

fix(retry): bound outbound resend rate per group to prevent AccountLocked#871
jlucaso1 merged 1 commit into
mainfrom
fix/per-chat-resend-rate-limit

jlucaso1 commented Jun 15, 2026 •

edited by cubic-dev-ai Bot

Loading

Uh oh!

coderabbitai Bot commented Jun 15, 2026 •

edited

Loading

Summary by CodeRabbit

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Suggested labels

Uh oh!

github-actions Bot commented Jun 15, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

codspeed-hq Bot commented Jun 15, 2026

Uh oh!

cubic-dev-ai Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Uh oh!

Conversation

jlucaso1 commented Jun 15, 2026 • edited by cubic-dev-ai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Fix

Tests

Verify

Uh oh!

coderabbitai Bot commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Suggested labels

Uh oh!

github-actions Bot commented Jun 15, 2026

📦 Binary size report

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

codspeed-hq Bot commented Jun 15, 2026

Merging this PR will not alter performance

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

jlucaso1 commented Jun 15, 2026 •

edited by cubic-dev-ai Bot

Loading

coderabbitai Bot commented Jun 15, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading