Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion api/client.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -1964,10 +1964,13 @@ Entry counts plus estimated retained heap bytes for the client's internal collec
| `signal_sessions` | `CollectionStats` | Cached Signal sessions |
| `signal_identities` | `CollectionStats` | Cached Signal identities |
| `signal_sender_keys` | `CollectionStats` | Cached sender keys |
| `history_sync_tasks` | `CollectionStats` | Queued/running history-sync tasks and their logical compressed-payload byte sum. A shared `Bytes` slice may retain a larger backing allocation, whose capacity isn't exposed by the type |
| `history_sync_tasks_peak` | `u64` | Lifetime high-water mark of queued/running history-sync tasks |
| `history_sync_payload_bytes_peak` | `u64` | Lifetime high-water mark of logical compressed-payload bytes |
| `chatstate_handlers` | `usize` | Registered chat state handlers |
| `custom_enc_handlers` | `usize` | Registered custom encryption handlers |

`CollectionStats` carries both `entries: u64` and `bytes: u64`. `MemoryReport::total_estimated_bytes(&self) -> u64` sums `.bytes` across every byte-carrying field, and `MemoryReport` implements `Display` for a pretty-printed, human-readable breakdown.
`CollectionStats` carries both `entries: u64` and `bytes: u64`. `MemoryReport::total_estimated_bytes(&self) -> u64` sums `.bytes` across every byte-carrying field, and `MemoryReport` implements `Display` for a pretty-printed, human-readable breakdown, including an `--- In-flight history sync ---` section with the two peak fields above.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated

**Example:**

Expand Down
17 changes: 15 additions & 2 deletions api/store.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -360,7 +360,7 @@ async fn put_msg_secret(
chat: &str,
sender: &str,
msg_id: &str,
secret: &[u8],
secret: &[u8; 32], // MessageSecret, the protocol-fixed message-secret size
) -> Result<()>;

/// Batch upsert. Each MsgSecretEntry carries its own absolute expires_at
Expand Down Expand Up @@ -390,7 +390,20 @@ async fn get_msg_secret_with_ts(
async fn delete_expired_msg_secrets(&self, cutoff_timestamp: i64) -> Result<u32>;
```

`MsgSecretEntry` is `{ chat, sender, msg_id, secret, expires_at, message_ts }`. The SQLite table is:
```rust
pub type MessageSecret = [u8; 32]; // wacore::reporting_token::MESSAGE_SECRET_SIZE

pub struct MsgSecretEntry {
pub chat: Arc<str>, // shared per conversation instead of one String per row
pub sender: Arc<str>,
pub msg_id: Arc<str>,
pub secret: MessageSecret,
pub expires_at: i64,
pub message_ts: i64,
}
```

`chat`, `sender`, and `msg_id` are `Arc<str>` rather than `String` so buffered batch inserts can clone entries cheaply, and `secret` is the fixed-size `MessageSecret` array rather than `Vec<u8>` — an invalid-length secret is unrepresentable instead of needing a runtime check. This is a breaking change for custom backends constructing `MsgSecretEntry` directly: build the JID/ID fields with `Arc::from(...)`/`.into()` and pass a `[u8; 32]` for `secret`. The SQLite table itself is unchanged:
Comment thread
greptile-apps[bot] marked this conversation as resolved.
Outdated

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Split the migration guidance into concise sentences

This paragraph combines both field-type changes, two separate rationales, compatibility impact, and migration instructions in long compound sentences. This conflicts with the applicable AGENTS.md requirements to “Keep sentences concise” and use “one idea per sentence,” making the breaking-change guidance harder for custom-backend authors to scan. Split each type change, rationale, and required migration step into its own sentence.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Split sentences and use the second person ("you") for instructions.

The explanation combines multiple concepts (string types, cloning, array size, runtime checks) into a long run-on sentence and does not use the second-person perspective for the breaking change instruction. As per coding guidelines, documentation must keep sentences concise (one idea per sentence) and use the second person ("you") and active voice.

♻️ Proposed rewrite for clarity and compliance
-`chat`, `sender`, and `msg_id` are `Arc<str>` rather than `String` so buffered batch inserts can clone entries cheaply, and `secret` is the fixed-size `MessageSecret` array rather than `Vec<u8>` — an invalid-length secret is unrepresentable instead of needing a runtime check. This is a breaking change for custom backends constructing `MsgSecretEntry` directly: build the JID/ID fields with `Arc::from(...)`/`.into()` and pass a `[u8; 32]` for `secret`. The SQLite table itself is unchanged:
+The `chat`, `sender`, and `msg_id` fields are `Arc<str>` rather than `String`. This allows buffered batch inserts to clone entries cheaply. The `secret` field uses the fixed-size `MessageSecret` array rather than `Vec<u8>`. This makes an invalid-length secret unrepresentable. You no longer need to perform a runtime length check. This is a breaking change if you construct `MsgSecretEntry` directly in a custom backend. You must build the JID/ID fields with `Arc::from(...)` or `.into()`. You must also pass a `[u8; 32]` for `secret`. The SQLite table itself is unchanged:
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
`chat`, `sender`, and `msg_id` are `Arc<str>` rather than `String` so buffered batch inserts can clone entries cheaply, and `secret` is the fixed-size `MessageSecret` array rather than `Vec<u8>`an invalid-length secret is unrepresentable instead of needing a runtime check. This is a breaking change for custom backends constructing `MsgSecretEntry` directly: build the JID/ID fields with `Arc::from(...)`/`.into()` and pass a `[u8; 32]` for `secret`. The SQLite table itself is unchanged:
The `chat`, `sender`, and `msg_id` fields are `Arc<str>` rather than `String`. This allows buffered batch inserts to clone entries cheaply. The `secret` field uses the fixed-size `MessageSecret` array rather than `Vec<u8>`. This makes an invalid-length secret unrepresentable. You no longer need to perform a runtime length check. This is a breaking change if you construct `MsgSecretEntry` directly in a custom backend. You must build the JID/ID fields with `Arc::from(...)` or `.into()`. You must also pass a `[u8; 32]` for `secret`. The SQLite table itself is unchanged:
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@api/store.mdx` at line 406, Rewrite the documentation paragraph around
MsgSecretEntry using short, single-idea sentences. Explain the Arc<str> fields,
fixed-size MessageSecret secret, and unchanged SQLite table separately, then
address the breaking change directly in second person by instructing users to
construct JID/ID fields with Arc::from(...) or .into() and provide a [u8; 32]
secret.

Source: Coding guidelines


```sql
CREATE TABLE msg_secrets (
Expand Down
31 changes: 31 additions & 0 deletions api/wacore.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -335,6 +335,36 @@ pub struct HistoryMsgSecretRecordRef<'a> {

`conversation_index` is a zero-based counter shared across records from the same conversation, letting a filter cache per-conversation classification instead of recomputing it per record. See [Architecture — RAM optimization layers](/concepts/architecture#ram-optimization-layers) for the measured allocation win.

For callers that want to skip the intermediate owned `HistoryMsgSecretRecord` entirely, `wacore::history_sync` also exports a streaming visitor path:
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated

```rust
pub trait HistoryMsgSecretRecordVisitor {
fn visit(&mut self, record: HistoryMsgSecretRecordRef<'_>) -> usize;
fn reserve(&mut self, _additional: usize) {}
fn retained_item_size(&self) -> Option<std::num::NonZeroUsize> { None }
}

pub fn process_history_sync_bytes_with_record_visitor<F>(
compressed_data: Bytes,
own_user: Option<&str>,
retain_blob: bool,
visitor: F,
) -> Result<HistorySyncResult, HistorySyncError>
where
F: for<'a> FnMut(HistoryMsgSecretRecordRef<'a>)

pub fn process_history_sync_bytes_with_record_sink<V>(
compressed_data: Bytes,
own_user: Option<&str>,
retain_blob: bool,
visitor: V,
) -> Result<HistorySyncResult, HistorySyncError>
where
V: HistoryMsgSecretRecordVisitor
```

`process_history_sync_bytes_with_record_visitor` takes a plain `FnMut` closure for the common case — the closure builds the caller's own row directly from the borrowed `HistoryMsgSecretRecordRef`, so the owned `HistoryMsgSecretRecord` is never allocated. `process_history_sync_bytes_with_record_sink` takes a full `HistoryMsgSecretRecordVisitor` implementation instead of a closure: `visit`'s return value reports the byte size the caller retained for that record (for accounting), and the optional `reserve`/`retained_item_size` hooks let a capacity-aware caller (e.g. a batched SQL insert buffer) size its own collection up front rather than growing it one record at a time. Reach for `process_history_sync_bytes_filtered` (above) when a simple accept/reject predicate is enough; reach for the visitor/sink pair when you also want to avoid materializing the owned record.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated

### Time

```rust
Expand Down Expand Up @@ -787,6 +817,7 @@ The library includes several allocation-reduction strategies that the integratio
- **Secret-presence pre-scan in history sync** — before buffa decodes a `HistorySyncMsg` (30+ fields, String allocations), a shallow varint walk checks whether the message carries `message_secret` at any level. Messages without a secret are skipped entirely. Bench (20k messages, secret-dense fixture): −29.5% allocations (56,020 → 39,520), −4.3% allocated bytes. Production blobs are secret-sparse so the saving is larger in practice
- **Inline storage in `HistoryMsgSecretRecord`** — `chat_id` is `Arc<str>` (allocated once per conversation, shared across all records in that conversation), `msg_id` is `CompactString` (inline for typical 20–22 char WA IDs on 64-bit targets; smaller inline limit on 32-bit/wasm32), and `secret` is `SecretBytes` (inline for secrets ≤32 bytes, heap for larger)
- **Filter-before-materialize in history sync** — `process_history_sync_bytes_filtered` runs a caller-supplied retention predicate against a borrowed [`HistoryMsgSecretRecordRef`](#history_sync-types) before the owned `HistoryMsgSecretRecord` is built; a rejected record is never materialized. Bench (500-conversation blob, upstream PR's rejection-heavy fixture): allocation churn 21.61 MiB → 14.00 MiB, allocation count ~84k → ~26k. The reduction scales with how much of the record set the predicate rejects — the default `process_history_sync_bytes` (accept-all) sees none of it
- **Streaming visitor/sink for history-sync records** — `process_history_sync_bytes_with_record_visitor` and `process_history_sync_bytes_with_record_sink` (via the [`HistoryMsgSecretRecordVisitor`](#history_sync-types) trait) go a step further than the filter predicate above: the caller builds its own storage row directly from the borrowed `HistoryMsgSecretRecordRef`, so the intermediate owned `HistoryMsgSecretRecord` is never allocated at all. Bench (allocator-instrumented synthetic history extraction): 20.20 MiB → 14.43 MiB allocated (-28.6%); CodSpeed history stream-drain memory: 243.8 KB → 115.2 KB (2.1× less)
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated
- **Borrowed JID parsing** — [`jid::parse_jid_ref`](#wacore-binary) parses common protocol JIDs into a `JidRef` with zero allocation, falling back to `Jid`'s owned parser only for edge cases. Cut allocations 4,093 → 97 in the history-sync task's JID classification path
- **In-place buffered media decrypt** — non-streaming `HttpClient` downloads now authenticate the already-buffered response body and decrypt AES-256-CBC in place (`DownloadUtils::verify_and_decrypt_in_place`, see [download](/api/download#downloadutils)), truncating the MAC/padding tail instead of allocating a second file-sized output buffer. Streaming clients are unaffected. Buffered download/decrypt span: 4.355 MiB → 2.146 MiB

Expand Down
1 change: 1 addition & 0 deletions concepts/architecture.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -525,6 +525,7 @@ graph LR
8. **Secret-presence pre-scan** — before running buffa decode on each `HistorySyncMsg`, a shallow varint walk checks whether the message carries `message_secret` at any level (`WebMessageInfo.message_secret` or `Message.message_context_info.message_secret`). Messages without a secret (the majority in production blobs) are discarded immediately — no struct decode, no allocation. The scan mirrors protobuf merge semantics so repeated field occurrences and malformed bytes are handled identically to a full decode
9. **Shared conversation id** — `HistoryMsgSecretRecord.chat_id` is `Arc<str>`, allocated once per conversation and reference-counted into every record within that conversation (10k clones → 500 on the bench fixture). `msg_id` uses `CompactString` (inline for typical 20–22 char WA IDs) and `secret` uses `SecretBytes` (inline for secrets ≤32 bytes)
10. **Filter-before-materialize retention hook** — `process_history_sync_bytes_filtered` runs a caller-supplied retention predicate against a borrowed `HistoryMsgSecretRecordRef` (a zero-copy view into the not-yet-built record) before allocating the owned, heap-backed `HistoryMsgSecretRecord`. A record the predicate rejects is discarded without ever being materialized. `process_history_sync_bytes` — and `process_history_sync` itself — keep their existing accept-all behavior by wrapping the filtered entry point with an always-`true` predicate, so callers that don't own a retention policy are unaffected. Bench (synthetic 500-conversation blob, upstream PR's rejection-heavy fixture): allocation churn 21.61 MiB → 14.00 MiB, allocation count ~84k → ~26k. The reduction scales with how much of the record set the predicate rejects — the default accept-all behavior sees none of it. See [`wacore` — history_sync types](/api/wacore#history_sync-types) for the exported signatures
11. **Streaming record-visitor path** — `process_history_sync_bytes_with_record_visitor` (closure-based) and `process_history_sync_bytes_with_record_sink` (trait-based, via `HistoryMsgSecretRecordVisitor`) go further than the filter hook above: the caller builds its own storage row directly from the borrowed `HistoryMsgSecretRecordRef`, so the owned `HistoryMsgSecretRecord` is never allocated at all, even for accepted records. The visitor trait's optional `reserve`/`retained_item_size` hooks let a capacity-aware caller (e.g. a batched SQL insert buffer) size its own collection up front. Bench (allocator-instrumented synthetic history extraction): 20.20 MiB → 14.43 MiB allocated (-28.6%); CodSpeed history stream-drain memory: 243.8 KB → 115.2 KB (2.1× less). See [`wacore` — history_sync types](/api/wacore#history_sync-types)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Point the visitor link at the generated anchor

The destination heading is history_sync types, so the generated slug preserves the underscore and replaces the space, producing #history_sync-types; the repository follows the same behavior for anchors such as #passkey-linking-shortcake_passkey. Consequently, clicking this newly added link does not jump to the documented signatures. Change this link and the new same-page link in api/wacore.mdx to use #history_sync-types.

Useful? React with 👍 / 👎.

Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated

### Skip mode

Expand Down