diff --git a/api/client.mdx b/api/client.mdx index d10de123..fcf5a187 100644 --- a/api/client.mdx +++ b/api/client.mdx @@ -1964,10 +1964,13 @@ Entry counts plus estimated retained heap bytes for the client's internal collec | `signal_sessions` | `CollectionStats` | Cached Signal sessions | | `signal_identities` | `CollectionStats` | Cached Signal identities | | `signal_sender_keys` | `CollectionStats` | Cached sender keys | +| `history_sync_tasks` | `CollectionStats` | Queued/running history-sync tasks and their logical compressed-payload byte sum. A shared `Bytes` slice may retain a larger backing allocation, whose capacity isn't exposed by the type | +| `history_sync_tasks_peak` | `u64` | Lifetime high-water mark of queued/running history-sync tasks | +| `history_sync_payload_bytes_peak` | `u64` | Lifetime high-water mark of logical compressed-payload bytes | | `chatstate_handlers` | `usize` | Registered chat state handlers | | `custom_enc_handlers` | `usize` | Registered custom encryption handlers | -`CollectionStats` carries both `entries: u64` and `bytes: u64`. `MemoryReport::total_estimated_bytes(&self) -> u64` sums `.bytes` across every byte-carrying field, and `MemoryReport` implements `Display` for a pretty-printed, human-readable breakdown. +`CollectionStats` carries both `entries: u64` and `bytes: u64`. `MemoryReport::total_estimated_bytes(&self) -> u64` sums `.bytes` across every byte-carrying field. `MemoryReport` implements `Display` for a pretty-printed, human-readable breakdown. This output includes an `--- In-flight history sync ---` section with the two peak fields above. **Example:** diff --git a/api/store.mdx b/api/store.mdx index cf41ab45..925c0525 100644 --- a/api/store.mdx +++ b/api/store.mdx @@ -360,7 +360,7 @@ async fn put_msg_secret( chat: &str, sender: &str, msg_id: &str, - secret: &[u8], + secret: &[u8; 32], // MessageSecret, the protocol-fixed message-secret size ) -> Result<()>; /// Batch upsert. Each MsgSecretEntry carries its own absolute expires_at @@ -390,7 +390,24 @@ async fn get_msg_secret_with_ts( async fn delete_expired_msg_secrets(&self, cutoff_timestamp: i64) -> Result; ``` -`MsgSecretEntry` is `{ chat, sender, msg_id, secret, expires_at, message_ts }`. The SQLite table is: +`get_msg_secret` and `get_msg_secret_with_ts` still return `Vec` rather than `MessageSecret`. Reads don't carry the same fixed-length guarantee as writes, since the persisted `secret BLOB` column has no length constraint at the SQL level. + +```rust +pub type MessageSecret = [u8; 32]; // wacore::reporting_token::MESSAGE_SECRET_SIZE + +pub struct MsgSecretEntry { + pub chat: Arc, // shared per conversation instead of one String per row + pub sender: Arc, + pub msg_id: Arc, + pub secret: MessageSecret, + pub expires_at: i64, + pub message_ts: i64, +} +``` + +The `chat`, `sender`, and `msg_id` fields are `Arc` rather than `String`. This allows buffered batch inserts to clone entries cheaply. The `secret` field uses the fixed-size `MessageSecret` array rather than `Vec`. This makes an invalid-length secret unrepresentable, so you no longer need a runtime length check. + +This is a breaking change if you build `MsgSecretEntry` directly in a custom backend, or if you override the defaulted `put_msg_secret` method. If you construct entries directly, build the JID/ID fields with `Arc::from(...)` or `.into()`, and pass a `[u8; 32]` for `secret`. If you override `put_msg_secret` (most backends don't — see the Note below), update its signature to take `secret: &[u8; 32]` instead of `secret: &[u8]`. The SQLite table itself is unchanged: ```sql CREATE TABLE msg_secrets ( diff --git a/api/wacore.mdx b/api/wacore.mdx index f90587c7..1d06a075 100644 --- a/api/wacore.mdx +++ b/api/wacore.mdx @@ -335,6 +335,36 @@ pub struct HistoryMsgSecretRecordRef<'a> { `conversation_index` is a zero-based counter shared across records from the same conversation, letting a filter cache per-conversation classification instead of recomputing it per record. See [Architecture — RAM optimization layers](/concepts/architecture#ram-optimization-layers) for the measured allocation win. +If you want to skip the intermediate owned `HistoryMsgSecretRecord` entirely, `wacore::history_sync` also exports a streaming visitor path: + +```rust +pub trait HistoryMsgSecretRecordVisitor { + fn visit(&mut self, record: HistoryMsgSecretRecordRef<'_>) -> usize; + fn reserve(&mut self, _additional: usize) {} + fn retained_item_size(&self) -> Option { None } +} + +pub fn process_history_sync_bytes_with_record_visitor( + compressed_data: Bytes, + own_user: Option<&str>, + retain_blob: bool, + visitor: F, +) -> Result +where + F: for<'a> FnMut(HistoryMsgSecretRecordRef<'a>) + +pub fn process_history_sync_bytes_with_record_sink( + compressed_data: Bytes, + own_user: Option<&str>, + retain_blob: bool, + visitor: V, +) -> Result +where + V: HistoryMsgSecretRecordVisitor +``` + +`process_history_sync_bytes_with_record_visitor` takes a plain `FnMut` closure for the common case. The closure builds your own row directly from the borrowed `HistoryMsgSecretRecordRef`. This ensures the owned `HistoryMsgSecretRecord` is never allocated. `process_history_sync_bytes_with_record_sink` takes a full `HistoryMsgSecretRecordVisitor` implementation instead of a closure. The `visit` return value reports the byte size you retained for that record for accounting. The optional `reserve` and `retained_item_size` hooks let you size your own collection (e.g., a batched SQL insert buffer) up front rather than growing it one record at a time. Reach for `process_history_sync_bytes_filtered` (above) when a simple accept/reject predicate is enough. Reach for the visitor/sink pair when you also want to avoid materializing the owned record. + ### Time ```rust @@ -787,6 +817,7 @@ The library includes several allocation-reduction strategies that the integratio - **Secret-presence pre-scan in history sync** — before buffa decodes a `HistorySyncMsg` (30+ fields, String allocations), a shallow varint walk checks whether the message carries `message_secret` at any level. Messages without a secret are skipped entirely. Bench (20k messages, secret-dense fixture): −29.5% allocations (56,020 → 39,520), −4.3% allocated bytes. Production blobs are secret-sparse so the saving is larger in practice - **Inline storage in `HistoryMsgSecretRecord`** — `chat_id` is `Arc` (allocated once per conversation, shared across all records in that conversation), `msg_id` is `CompactString` (inline for typical 20–22 char WA IDs on 64-bit targets; smaller inline limit on 32-bit/wasm32), and `secret` is `SecretBytes` (inline for secrets ≤32 bytes, heap for larger) - **Filter-before-materialize in history sync** — `process_history_sync_bytes_filtered` runs a caller-supplied retention predicate against a borrowed [`HistoryMsgSecretRecordRef`](#history_sync-types) before the owned `HistoryMsgSecretRecord` is built; a rejected record is never materialized. Bench (500-conversation blob, upstream PR's rejection-heavy fixture): allocation churn 21.61 MiB → 14.00 MiB, allocation count ~84k → ~26k. The reduction scales with how much of the record set the predicate rejects — the default `process_history_sync_bytes` (accept-all) sees none of it +- **Streaming visitor/sink for history-sync records** — `process_history_sync_bytes_with_record_visitor` and `process_history_sync_bytes_with_record_sink` (via the [`HistoryMsgSecretRecordVisitor`](#history_sync-types) trait) go a step further than the filter predicate above. You can build your own storage row directly from the borrowed `HistoryMsgSecretRecordRef`. This ensures the intermediate owned `HistoryMsgSecretRecord` is never allocated at all. Bench (allocator-instrumented synthetic history extraction): 20.20 MiB → 14.43 MiB allocated (-28.6%); CodSpeed history stream-drain memory: 243.8 KB → 115.2 KB (2.1× less) - **Borrowed JID parsing** — [`jid::parse_jid_ref`](#wacore-binary) parses common protocol JIDs into a `JidRef` with zero allocation, falling back to `Jid`'s owned parser only for edge cases. Cut allocations 4,093 → 97 in the history-sync task's JID classification path - **In-place buffered media decrypt** — non-streaming `HttpClient` downloads now authenticate the already-buffered response body and decrypt AES-256-CBC in place (`DownloadUtils::verify_and_decrypt_in_place`, see [download](/api/download#downloadutils)), truncating the MAC/padding tail instead of allocating a second file-sized output buffer. Streaming clients are unaffected. Buffered download/decrypt span: 4.355 MiB → 2.146 MiB diff --git a/concepts/architecture.mdx b/concepts/architecture.mdx index ba44220c..833616e4 100644 --- a/concepts/architecture.mdx +++ b/concepts/architecture.mdx @@ -525,6 +525,7 @@ graph LR 8. **Secret-presence pre-scan** — before running buffa decode on each `HistorySyncMsg`, a shallow varint walk checks whether the message carries `message_secret` at any level (`WebMessageInfo.message_secret` or `Message.message_context_info.message_secret`). Messages without a secret (the majority in production blobs) are discarded immediately — no struct decode, no allocation. The scan mirrors protobuf merge semantics so repeated field occurrences and malformed bytes are handled identically to a full decode 9. **Shared conversation id** — `HistoryMsgSecretRecord.chat_id` is `Arc`, allocated once per conversation and reference-counted into every record within that conversation (10k clones → 500 on the bench fixture). `msg_id` uses `CompactString` (inline for typical 20–22 char WA IDs) and `secret` uses `SecretBytes` (inline for secrets ≤32 bytes) 10. **Filter-before-materialize retention hook** — `process_history_sync_bytes_filtered` runs a caller-supplied retention predicate against a borrowed `HistoryMsgSecretRecordRef` (a zero-copy view into the not-yet-built record) before allocating the owned, heap-backed `HistoryMsgSecretRecord`. A record the predicate rejects is discarded without ever being materialized. `process_history_sync_bytes` — and `process_history_sync` itself — keep their existing accept-all behavior by wrapping the filtered entry point with an always-`true` predicate, so callers that don't own a retention policy are unaffected. Bench (synthetic 500-conversation blob, upstream PR's rejection-heavy fixture): allocation churn 21.61 MiB → 14.00 MiB, allocation count ~84k → ~26k. The reduction scales with how much of the record set the predicate rejects — the default accept-all behavior sees none of it. See [`wacore` — history_sync types](/api/wacore#history_sync-types) for the exported signatures +11. **Streaming record-visitor path** — `process_history_sync_bytes_with_record_visitor` (closure-based) and `process_history_sync_bytes_with_record_sink` (trait-based, via `HistoryMsgSecretRecordVisitor`) go further than the filter hook above. You can build your own storage row directly from the borrowed `HistoryMsgSecretRecordRef`. This ensures the owned `HistoryMsgSecretRecord` is never allocated, even for accepted records. The visitor trait's optional `reserve` and `retained_item_size` hooks let you size your own collection (e.g., a batched SQL insert buffer) up front. Bench (allocator-instrumented synthetic history extraction): 20.20 MiB → 14.43 MiB allocated (-28.6%); CodSpeed history stream-drain memory: 243.8 KB → 115.2 KB (2.1× less). See [`wacore` — history_sync types](/api/wacore#history_sync-types) ### Skip mode