Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
62ca987
feat: add integration benchmarks with mock server
jlucaso1 Apr 15, 2026
8efa7a8
perf: reduce transport event channel capacity from 10K to 1K
jlucaso1 Apr 15, 2026
f46417e
perf: eliminate intermediate allocations in prekey upload and send paths
jlucaso1 Apr 15, 2026
da9bf0c
perf: pool zlib decompressor to avoid repeated 48KB allocations
jlucaso1 Apr 15, 2026
a1f68b9
perf: use Bytes for prekey store to eliminate O(n) clones
jlucaso1 Apr 15, 2026
d692743
perf: CompactString for props cache + Bytes transport + sender buffer…
jlucaso1 Apr 15, 2026
5e20548
perf: shared prekey encode buffer + reusable sender encryption buffer
jlucaso1 Apr 15, 2026
140a487
perf: zero-copy digest validation, smarter marshal sizing, smaller ha…
jlucaso1 Apr 15, 2026
e801322
refactor: split handle_incoming_message into classify + process phases
jlucaso1 Apr 15, 2026
79a282c
perf: zero-copy EncPayload extraction via OwnedNodeRef::slice_bytes
jlucaso1 Apr 15, 2026
9e1adbc
fix: clippy --all --tests errors and warnings
jlucaso1 Apr 15, 2026
f47371a
fix: zlib pool BufError spin + scratch buffer retention
jlucaso1 Apr 15, 2026
1563f49
fix: address review findings - race condition, buffer reuse, EncType …
jlucaso1 Apr 15, 2026
2ba5f6e
fix: address all review findings
jlucaso1 Apr 15, 2026
8c599aa
fix: normalize retry stanza to JID to bare (no device suffix)
jlucaso1 Apr 15, 2026
c1abd3d
fix: relax retry e2e test to tolerate device-qualified JIDs
jlucaso1 Apr 15, 2026
36224a8
fix: restore WA Web-compliant original_from (raw receipt JID with dev…
jlucaso1 Apr 15, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
186 changes: 186 additions & 0 deletions .github/workflows/bench-integration.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,186 @@
name: Integration Benchmark

on:
pull_request:
branches: [main]
Comment on lines +4 to +5

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Use a secret-capable trigger for benchmark PR job

This workflow runs on pull_request, but the PR benchmark job authenticates the service container with secrets.BARTENDER_GHCR_TOKEN. For PRs from forks, repository secrets are not provided, so the mock-server service cannot authenticate/pull and the benchmark job fails before tests run. That makes the new benchmark CI path unreliable for external contributions.

Useful? React with 👍 / 👎.

push:
branches: [main]

concurrency:
group: bench-integration-${{ github.head_ref || github.run_id }}
cancel-in-progress: true

env:
CARGO_TERM_COLOR: always
PROTOC_VERSION: "3.25.3"
SCCACHE_GHA_ENABLED: "true"
RUSTC_WRAPPER: "sccache"

jobs:
bench-integration-pr:
if: github.event_name == 'pull_request'
name: Integration Benchmark (PR)
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write
services:
mock-server:
image: ghcr.io/whiskeysockets-devtools/bartender:latest
credentials:
username: ${{ github.actor }}
password: ${{ secrets.BARTENDER_GHCR_TOKEN }}
ports:
- 8080:8080
env:
CHATSTATE_TTL_SECS: "3"
options: --log-driver none
steps:
- uses: actions/checkout@v6

- uses: dtolnay/rust-toolchain@master
with:
toolchain: nightly-2026-04-05

Comment on lines +28 to +44

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

cat -n .github/workflows/bench-integration.yml | sed -n '1,60p'

Repository: jlucaso1/whatsapp-rust

Length of output: 1873


🏁 Script executed:

# Search for any version tags or references to bartender in the repo
rg -i "bartender" --type yaml --type md

Repository: jlucaso1/whatsapp-rust

Length of output: 48


🌐 Web query:

dtolnay rust-toolchain action pinning best practices recommended version

💡 Result:

Best practices for pinning the dtolnay/rust-toolchain GitHub Action: Use specific tags or commit SHAs for reproducibility and security. The action's @rev determines the default toolchain (e.g., dtolnay/rust-toolchain@stable installs the current stable Rust, dtolnay/rust-toolchain@1.94.0 installs Rust 1.94.0). Recommended approaches: - For latest stable: - uses: dtolnay/rust-toolchain@stable - For pinned version: - uses: dtolnay/rust-toolchain@1.94.0 (matches current stable as of 2026-04-15) - For explicit toolchain with master: - uses: dtolnay/rust-toolchain@master with: toolchain: '1.94.0' For maximum security, pin to a full commit SHA from master branch history (e.g., dtolnay/rust-toolchain@315e265), as tags like @stable are lightweight and follow latest in channel. Combine with rust-toolchain.toml in repo root for project-specific pinning (e.g., channel = "1.94.0"), which rustup respects automatically. This ensures consistent builds, avoids surprise lints from un-pinned updates, and follows GitHub security guidelines for third-party actions. Current stable Rust version (2026-04-15): 1.94.1 (released 2026-03-26), recommended for most projects.

Citations:


🌐 Web query:

whiskeysockets bartender docker image releases versions

💡 Result:

No official whiskeysockets/bartender Docker image or releases found. WhiskeySockets maintains Baileys (WhatsApp Web API library) with latest release v7.0.0-rc.9 on 2025-11-21. "Bartender" appears in Baileys PR #2191 context as an internal testing/benchmark tool for socket changes (e.g., "destructive benchmark test on bartender"), but no public repository, Docker image, or version releases for it exist in search results. Related Baileys Docker images like devjuanda/baileys-server-pro exist but are third-party. Unrelated projects like barassistant/server (cocktail app) have Docker tags including 'dev'.

Citations:


Pin the benchmark environment to immutable refs.

bartender:latest and dtolnay/rust-toolchain@master both use mutable references that drift over time, making benchmark baselines non-reproducible and risking unexpected workflow breakage. Pin the image to a digest and the action to a commit SHA or tag.

Proposed pinning pattern
-        image: ghcr.io/whiskeysockets-devtools/bartender:latest
+        image: ghcr.io/whiskeysockets-devtools/bartender@sha256:<verified-digest>
...
-      - uses: dtolnay/rust-toolchain@master
+      - uses: dtolnay/rust-toolchain@<verified-commit-sha>
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/bench-integration.yml around lines 29 - 45, The workflow
uses mutable refs: the mock-server image
"ghcr.io/whiskeysockets-devtools/bartender:latest" and the action
"dtolnay/rust-toolchain@master"; replace them with pinned immutable refs by
updating the mock-server image to the exact image digest (sha256:...) instead of
:latest and change dtolnay/rust-toolchain@master to a specific commit SHA or
released tag; ensure the new image digest and action SHA/tag are recorded in the
workflow so benchmarks remain reproducible and the steps referencing
"mock-server" and the "dtolnay/rust-toolchain" action use those pinned values.

- name: Install protoc
uses: taiki-e/install-action@v2
with:
tool: protoc@${{ env.PROTOC_VERSION }}

- name: Setup sccache
uses: mozilla-actions/sccache-action@v0.0.9

- name: Cache Rust registry
uses: Swatinem/rust-cache@v2
with:
prefix-key: ${{ runner.os }}-cargo-bench-int
cache-targets: "false"

- name: Wait for mock server
run: |
for i in $(seq 1 30); do
if curl -sk https://localhost:8080/ > /dev/null 2>&1; then
echo "Mock server is ready"
exit 0
fi
sleep 1
done
echo "Mock server failed to become ready"
exit 1

- name: Run integration benchmarks
env:
MOCK_SERVER_URL: "wss://127.0.0.1:8080/ws/chat"
RUST_LOG: warn
run: |
cargo run -p bench-integration --release \
> bench_integration_results.json \
2> bench_integration.log
cat bench_integration.log >&2
echo "Results:"
cat bench_integration_results.json

- name: Upload benchmark artifacts
uses: actions/upload-artifact@v4
with:
name: bench-integration-results
path: |
bench_integration_results.json
bench_integration.log
retention-days: 30

- name: Compare against baseline
uses: benchmark-action/github-action-benchmark@v1
with:
name: "whatsapp-rust integration benchmarks"
tool: "customSmallerIsBetter"
output-file-path: bench_integration_results.json
github-token: ${{ secrets.GITHUB_TOKEN }}
auto-push: false
save-data-file: false
benchmark-data-dir-path: dev/bench-integration
summary-always: true

bench-integration-push:
if: github.event_name == 'push'
name: Integration Benchmark (push)
runs-on: ubuntu-latest
permissions:
contents: write
deployments: write
services:
mock-server:
image: ghcr.io/whiskeysockets-devtools/bartender:latest
credentials:
username: ${{ github.actor }}
password: ${{ secrets.BARTENDER_GHCR_TOKEN }}
ports:
- 8080:8080
env:
CHATSTATE_TTL_SECS: "3"
options: --log-driver none
steps:
- uses: actions/checkout@v6

- uses: dtolnay/rust-toolchain@master
with:
toolchain: nightly-2026-04-05

- name: Install protoc
uses: taiki-e/install-action@v2
with:
tool: protoc@${{ env.PROTOC_VERSION }}

- name: Setup sccache
uses: mozilla-actions/sccache-action@v0.0.9

- name: Cache Rust registry
uses: Swatinem/rust-cache@v2
with:
prefix-key: ${{ runner.os }}-cargo-bench-int
cache-targets: "false"

- name: Wait for mock server
run: |
for i in $(seq 1 30); do
if curl -sk https://localhost:8080/ > /dev/null 2>&1; then
echo "Mock server is ready"
exit 0
fi
sleep 1
done
echo "Mock server failed to become ready"
exit 1

- name: Run integration benchmarks
env:
MOCK_SERVER_URL: "wss://127.0.0.1:8080/ws/chat"
RUST_LOG: warn
run: |
cargo run -p bench-integration --release \
> bench_integration_results.json \
2> bench_integration.log
cat bench_integration.log >&2
echo "Results:"
cat bench_integration_results.json

- name: Upload benchmark artifacts
uses: actions/upload-artifact@v4
with:
name: bench-integration-results
path: |
bench_integration_results.json
bench_integration.log
retention-days: 30

- name: Store baseline
uses: benchmark-action/github-action-benchmark@v1
with:
name: "whatsapp-rust integration benchmarks"
tool: "customSmallerIsBetter"
output-file-path: bench_integration_results.json
github-token: ${{ secrets.GITHUB_TOKEN }}
auto-push: true
benchmark-data-dir-path: dev/bench-integration
max-items-in-chart: 100
summary-always: true
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,4 @@ docs
.claude
__pycache__
.codex
dhat-heap.json
18 changes: 17 additions & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 7 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ members = [
".",
"http_clients/ureq-client",
"storages/sqlite-storage",
"tests/bench-integration",
"tests/e2e",
"transports/tokio-transport",
"wacore",
Expand Down Expand Up @@ -161,3 +162,9 @@ inherits = "release"
lto = "thin"
debug = 1
strip = false

# Profiling: optimized with debug symbols for DHAT / heaptrack / perf
[profile.profiling]
inherits = "release"
debug = 1
strip = false
2 changes: 1 addition & 1 deletion src/appstate_sync.rs
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ mod tests {
async fn store_prekey(&self, _: u32, _: &[u8], _: bool) -> StoreResult<()> {
Ok(())
}
async fn load_prekey(&self, _: u32) -> StoreResult<Option<Vec<u8>>> {
async fn load_prekey(&self, _: u32) -> StoreResult<Option<bytes::Bytes>> {
Ok(None)
}
async fn remove_prekey(&self, _: u32) -> StoreResult<()> {
Expand Down
3 changes: 1 addition & 2 deletions src/client.rs
Original file line number Diff line number Diff line change
Expand Up @@ -893,9 +893,8 @@ impl Client {
/// [`send_node`](Client::send_node) for normal stanza sending.
pub async fn send_raw_bytes(&self, plaintext: Vec<u8>) -> Result<(), ClientError> {
let noise_socket = self.get_noise_socket().await?;
let encrypted_buf = Vec::with_capacity(plaintext.len() + 32);
noise_socket
.encrypt_and_send(plaintext, encrypted_buf)
.encrypt_and_send(bytes::Bytes::from(plaintext))
.await?;
self.last_data_sent_ms
.store(wacore::time::now_millis().max(0) as u64, Ordering::Relaxed);
Expand Down
4 changes: 2 additions & 2 deletions src/features/signal.rs
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ impl<'a> Signal<'a> {
wacore::send::create_sender_key_distribution_message_for_group(
&mut adapter.sender_key_store,
group_jid,
&own_jid,
&sender_addr,
)
.await?,
)
Expand All @@ -162,7 +162,7 @@ impl<'a> Signal<'a> {
let ciphertext = wacore::send::encrypt_group_message(
&mut adapter.sender_key_store,
group_jid,
&own_jid,
&sender_addr,
plaintext,
&mut rng,
)
Expand Down
90 changes: 47 additions & 43 deletions src/handlers/message.rs
Original file line number Diff line number Diff line change
Expand Up @@ -38,51 +38,11 @@ impl StanzaHandler for MessageHandler {
}
};

// Single cache lookup: get or create the lane (lock + queue + worker).
// Single-flight: get_with_by_ref guarantees exactly one init runs per key,
// preventing duplicate workers for the same chat (TOCTOU race).
let lane = client
.chat_lanes
.get_with_by_ref(&chat_jid, async {
let (tx, rx) = async_channel::unbounded::<Arc<wacore_binary::OwnedNodeRef>>();

let client_for_worker = client.clone();
let spawn_generation = client
.connection_generation
.load(std::sync::atomic::Ordering::Acquire);

client
.runtime
.spawn(Box::pin(async move {
while let Ok(msg_node) = rx.recv().await {
if client_for_worker
.connection_generation
.load(std::sync::atomic::Ordering::Acquire)
!= spawn_generation
{
log::debug!(target: "MessageQueue", "Stale worker exiting; remaining messages will be redelivered by server");
break;
}
let start = wacore::time::now_millis() as u64;
let client = client_for_worker.clone();
Box::pin(client.handle_incoming_message(msg_node)).await;
let elapsed =
(wacore::time::now_millis() as u64).saturating_sub(start);
if elapsed > MAX_MESSAGE_DELAY_MS {
warn!(
target: "MessageQueue",
"Message processing took {:.1}s (MAX_MESSAGE_DELAY is {}s)",
elapsed as f64 / 1000.0,
MAX_MESSAGE_DELAY_MS / 1000
);
}
}
}))
.detach();

ChatLane {
enqueue_lock: Arc::new(async_lock::Mutex::new(())),
queue_tx: tx,
}
})
.get_with_by_ref(&chat_jid, async { create_chat_lane(&client) })
.await;

// Lock serializes enqueue order for this chat
Expand All @@ -97,3 +57,47 @@ impl StanzaHandler for MessageHandler {
true
}
}

/// Construct a ChatLane with a spawned worker task. Extracted to keep the
/// init closure passed to `get_with_by_ref` small.
fn create_chat_lane(client: &Arc<Client>) -> ChatLane {
let (tx, rx) = async_channel::unbounded::<Arc<wacore_binary::OwnedNodeRef>>();

let client_for_worker = client.clone();
let spawn_generation = client
.connection_generation
.load(std::sync::atomic::Ordering::Acquire);

client
.runtime
.spawn(Box::pin(async move {
while let Ok(msg_node) = rx.recv().await {
if client_for_worker
.connection_generation
.load(std::sync::atomic::Ordering::Acquire)
!= spawn_generation
{
log::debug!(target: "MessageQueue", "Stale worker exiting; remaining messages will be redelivered by server");
break;
}
let start = wacore::time::now_millis() as u64;
let client = client_for_worker.clone();
Box::pin(client.handle_incoming_message(msg_node)).await;

@coderabbitai coderabbitai Bot Apr 15, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Verify handle_incoming_message signature and current call sites.
# Expectation: if it returns an awaitable future directly, Box::pin is unnecessary.

set -euo pipefail

rg -n -C3 '\bhandle_incoming_message\s*\(' --type rust

Repository: jlucaso1/whatsapp-rust

Length of output: 3552


Remove unnecessary Box::pin() wrapping in the worker loop.

handle_incoming_message is an async fn that returns a future directly; it can be awaited without boxing. All other call sites in the codebase use direct .await, and the heap allocation from Box::pin() is unnecessary in this hot path.

♻️ Suggested change
-                let client = client_for_worker.clone();
-                Box::pin(client.handle_incoming_message(msg_node)).await;
+                let client = client_for_worker.clone();
+                client.handle_incoming_message(msg_node).await;
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/handlers/message.rs` at line 85, The worker loop is unnecessarily boxing
the future returned by the async function client.handle_incoming_message; remove
the Box::pin(...) wrapper and await the future directly (i.e., replace
Box::pin(client.handle_incoming_message(msg_node)).await with
client.handle_incoming_message(msg_node).await) to avoid the heap allocation in
this hot path, and remove any now-unused imports or allocations introduced
solely for boxing.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Box::pin here is intentional -- handle_incoming_message takes self: Arc and returns a large async future. Without Box::pin, that future is inlined into the worker loop's state machine, making the outer spawned future ~29KB (the exact problem we were trying to fix). With Box::pin, the outer future stays small and the large message-handling future is heap-allocated only when actually processing a message.

This is the opposite of unnecessary -- it's a deliberate optimization. Removing it would undo the future-size reduction we achieved.

The Box::pin on line 73 (runtime.spawn(Box::pin(async move { ... }))) is required by the spawn API which takes Pin<Box>.

No change needed here. The review suggestion is incorrect for this case.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jlucaso1, you're absolutely right — thank you for the detailed explanation. The Box::pin is a well-known technique for controlling async state machine sizes: without it, the compiler inlines the entire handle_incoming_message future into the worker loop's state machine, bloating every spawned task instance regardless of whether a message is currently being processed. With Box::pin, the outer task remains compact and the large future is only heap-allocated on demand. My suggestion was incorrect. No change needed.


✏️ Learnings added
Learnt from: jlucaso1
Repo: jlucaso1/whatsapp-rust PR: 551
File: src/handlers/message.rs:85-85
Timestamp: 2026-04-15T16:06:29.137Z
Learning: In `src/handlers/message.rs` (whatsapp-rust), the `Box::pin(client.handle_incoming_message(msg_node)).await` pattern inside the per-chat worker loop is intentional and must NOT be removed. `handle_incoming_message` returns a large async future; without `Box::pin`, the Rust compiler inlines that future into the outer worker loop's state machine, inflating every spawned task to ~29KB. `Box::pin` keeps the outer future small and heap-allocates the large inner future only when a message is actually being processed. Removing this would undo a deliberate future-size optimization. Do not flag this as unnecessary boxing.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.


🧠 Learnings used
Learnt from: jlucaso1
Repo: jlucaso1/whatsapp-rust PR: 506
File: src/message.rs:309-329
Timestamp: 2026-04-08T14:24:14.247Z
Learning: In whatsapp-rust (src/message.rs), when an `<unavailable>` child node is present in an incoming message, retry receipts (RetryReason::NoSession or any other reason) must NOT be sent. The correct recovery is PDO-only: call spawn_pdo_request_with_options and dispatch an UndecryptableMessage event. Sending a retry receipt here is protocol-incorrect because <unavailable> is a server-side routing failure (bare-JID fanout to companion devices), not a Signal session failure. This matches WA Web and whatsmeow behavior. Do not suggest routing the <unavailable> path through handle_decrypt_failure.

Learnt from: jlucaso1
Repo: jlucaso1/whatsapp-rust PR: 511
File: src/send.rs:658-661
Timestamp: 2026-04-11T04:49:14.681Z
Learning: In whatsapp-rust (src/send.rs), response_waiters uses a simple String key (message_id) for phash ACK waiters. This is intentional and safe: every send path produces a structurally unique ID via generate_message_id(), so the same ID can never be registered twice within the 10-second waiter window. The SendOptions.message_id override (for explicit caller-controlled resends) does not create a collision risk because any prior waiter for that ID is always removed before a retry. Adding compound registration tokens would be unnecessary complexity. This matches WhatsApp Web's own ackHandlers (ackHandlers array in Comms.js). Do not flag response_waiters cleanup as a race condition.

Learnt from: jlucaso1
Repo: jlucaso1/whatsapp-rust PR: 425
File: src/send.rs:111-119
Timestamp: 2026-03-22T23:20:20.069Z
Learning: In `src/send.rs` (whatsapp-rust), `SendOptions.extra_stanza_nodes` is intentionally a transparent pass-through. The `infer_stanza_metadata` helper always prepends its inferred `<meta>` node without checking for existing nodes in `extra_stanza_nodes`. Deduplication/idempotency is deliberately NOT performed — if a caller provides their own `<meta>` node alongside a poll/event message, it is treated as intentional and both nodes are sent. Do not flag this as a bug.

Learnt from: jlucaso1
Repo: jlucaso1/whatsapp-rust PR: 506
File: src/message.rs:310-321
Timestamp: 2026-04-08T14:22:21.072Z
Learning: In whatsapp-rust (src/message.rs), all `<unavailable>` child node types — including `type="view_once"` (UnavailableType::ViewOnce) and the default Unknown — are handled identically: they all unconditionally trigger spawn_pdo_request_with_options and an UndecryptableMessage event. This matches WA Web and whatsmeow behavior. Do not suggest gating the PDO request path based on unavailable_type.

Learnt from: jlucaso1
Repo: jlucaso1/whatsapp-rust PR: 316
File: src/handlers/notification.rs:620-628
Timestamp: 2026-03-11T16:19:23.298Z
Learning: In Rust code path src/handlers/, for hash-based picture notifications, avoid relying on upgrading to a JID via from.clone() as a general fallback. This should be treated as a temporary approximation only in rare edge cases where a contact hash has no JID. A proper fix would involve a contact hash registry (e.g., getContactRecordByHash()) backed by address book sync. Until such infrastructure exists, document this fallback's as-is behavior and consider revisiting for maintainability and correctness in future refactors. This guidance applies broadly to similar files handling hash-based notifications in src/handlers, not only to this exact function.

let elapsed = (wacore::time::now_millis() as u64).saturating_sub(start);
if elapsed > MAX_MESSAGE_DELAY_MS {
warn!(
target: "MessageQueue",
"Message processing took {:.1}s (MAX_MESSAGE_DELAY is {}s)",
elapsed as f64 / 1000.0,
MAX_MESSAGE_DELAY_MS / 1000
);
}
}
}))
.detach();

ChatLane {
enqueue_lock: Arc::new(async_lock::Mutex::new(())),
queue_tx: tx,
}
}
Loading
Loading