perf(appstate): index-sort dedup for large patches, O(n²) scan stays for small

[jlucaso1]

What

collect_unique_index_macs (in wacore/src/appstate_sync.rs) deduplicates a patch's index MACs in first-seen order, feeding the batched previous-value-MAC backend lookup. It used an O(n²) linear scan (out.iter().any(|v| v == index_mac)) per element.

Now:

• Small patches (≤ 64 mutations): unchanged linear scan — cache-friendly, zero scratch allocation (a HashSet measured 6–120% slower at small N here).
• Large patches (> 64): dedup via an O(n log n) sort of position indices. Sort positions by MAC content (ties broken by position so each run's first occurrence leads), drop the non-leaders, then re-sort by position to restore first-seen order. Only scratch is a single pre-sized Vec<u32>.

Why

A fresh CodSpeed sweep of main flagged this as the one heavy benchmark dominated by avoidable algorithmic work (everything else heavy is genuine SHA/curve crypto or zlib). At N=1000 (the resume-sync patch ceiling), ~87% of the 5.7 ms was byte-slice equality from the O(n²) scan (~500K 32-byte compares).

An earlier revision used a HashSet<&[u8]>, which was ×17 faster but added ~34 KB of transient allocation per patch and tripped CodSpeed's memory gate. The index sort keeps the bulk of the win with only a ~4 KB Vec<u32>.

Results (CodSpeed, fd6c7ff vs main)

• bench_collect_unique_index_macs[1000]: 5,678 µs → 550.8 µs (×10).
• 1 improved, 0 regressed, 171 untouched — no memory regression.
• Binary size +13 KiB stripped (std sort/dedup monomorphization), within the 64 KiB budget.

Correctness

• First-seen order preserved on both paths.
• dedup_tests assert the scan and sort paths agree across n = 8, 64, 65, 1000 (with duplicates), that mutations without an index MAC are skipped, and — exercising the sort path specifically — that first-seen order is restored when it runs opposite to byte-sort order (so an order-restoration bug can't pass by coincidence).

No wire/behavior change; pure internal dedup. Ran cargo fmt/clippy and the new tests locally; full suite + CodSpeed on CI.

[coderabbitai]

📝 Walkthrough

Walkthrough

collect_unique_index_macs in appstate_sync.rs is refactored to use two deduplication paths controlled by a MAC_DEDUP_SCAN_LIMIT constant: a linear scan for small inputs and a sort-based order-preserving approach for larger ones. A private mutation_index_mac helper is added. Unit tests in dedup_tests verify both paths produce identical first-seen-order results and skip mutations without an index MAC.

Changes

Index MAC Deduplication Strategy

Layer / File(s)	Summary
Dedup strategy, helper, and tests wacore/src/appstate_sync.rs	MAC_DEDUP_SCAN_LIMIT and mutation_index_mac are introduced; collect_unique_index_macs conditionally uses a linear scan (≤ limit) or a sort-and-dedup path (> limit), both preserving first-seen order; dedup_tests assert path equivalence at small, boundary, and large sizes and confirm MAC-absent mutations are skipped.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

• oxidezap/whatsapp-rust#821: Modifies collect_unique_index_macs as a dependency for batching previous-MAC prefetches in build_patch, directly coupling to the dedup step this PR changes.
• oxidezap/whatsapp-rust#856: Adds a benchmark specifically targeting collect_unique_index_macs' index-MAC dedup behavior, which is the exact algorithm being replaced here.

Suggested labels

performance

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title accurately describes the main optimization: a hybrid deduplication strategy that uses index-sort for large patches while preserving the O(n²) scan for small ones.
Description check	✅ Passed	The description is comprehensive and directly related to the changeset, clearly explaining the what, why, results, and correctness validation of the optimization.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch claude/whatsapp-rust-pr-review-mz0gyy

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[cubic-dev-ai]

No issues found across 1 file

_{Re-trigger cubic}

[github-actions]

📦 Binary size report

Metric	main	PR	Δ
bin size (stripped)	10.66 MiB	10.67 MiB	+13.22 KiB (+0.12%) 🔺
bin .text	8.75 MiB	8.76 MiB	+12.38 KiB (+0.14%) 🔺
bin allocated (text+data+bss)	10.66 MiB	10.67 MiB	+12.01 KiB (+0.11%) 🔺
llvm-lines wacore	661,539	666,701	+5,162 (+0.78%) 🔺
llvm-lines wacore copies	17,918	18,047	+129 (+0.72%) 🔺
llvm-lines whatsapp-rust lib	652,742	652,742	0
llvm-lines whatsapp-rust lib copies	19,857	19,857	0
deps crates (Cargo.lock)	354	354	0

.text per crate

Crate	main	PR	Δ
.text whatsapp_rust	1.47 MiB	1.47 MiB	0
.text wacore	552.60 KiB	556.04 KiB	+3.44 KiB (+0.62%) 🔺
.text wacore_binary	103.64 KiB	103.64 KiB	0
.text wacore_libsignal	168.32 KiB	168.32 KiB	0
.text wacore_appstate	35.26 KiB	35.26 KiB	0
.text wacore_noise	30.68 KiB	30.68 KiB	0
.text waproto	960.72 KiB	960.72 KiB	0
.text whatsapp_rust_sqlite_storage	206.21 KiB	206.21 KiB	0
.text whatsapp_rust_tokio_transport	33.09 KiB	33.09 KiB	0
.text whatsapp_rust_ureq_http_client	6.19 KiB	6.19 KiB	0
.text std	1.12 MiB	1.13 MiB	+8.88 KiB (+0.77%) 🔺
.text other deps	4.04 MiB	4.04 MiB	0

Top movers (cargo-bloat attribution)

Crate	main	PR	Δ
std	1.12 MiB	1.13 MiB	+8.88 KiB (+0.77%)
wacore	552.60 KiB	556.04 KiB	+3.44 KiB (+0.62%)

Baseline: 23130ca26 (latest main run) · Head: cf0c3a951 · Graphs

[codspeed-hq]

Merging this PR will improve performance by ×10

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
✅ 171 untouched benchmarks

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	Simulation	bench_collect_unique_index_macs[1000]	5,678.6 µs	550.8 µs	×10

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing claude/whatsapp-rust-pr-review-mz0gyy (fd6c7ff) with main (23130ca)}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@wacore/src/appstate_sync.rs`:
- Around line 786-795: The test scan_and_sort_paths_agree currently only tests
cases with lexicographically ordered first appearances (using the generated
build function), which doesn't properly exercise the "restore first-seen order"
step. Add a test case with a non-lexicographic fixture where first-seen order
differs from byte-sort order, such as manually constructing a sequence with
distinct values that appear in a different order than their sorted order. This
ensures that any regression in the order restoration logic will be caught
immediately instead of passing silently when first-seen order happens to match
lexicographic order.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: d6c33a4a-1283-4f5d-8a01-f5bc4dd1b98b

📥 Commits

Reviewing files that changed from the base of the PR and between 23130ca and f04b922.

📒 Files selected for processing (1)

• wacore/src/appstate_sync.rs