Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion api/client.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -1964,10 +1964,13 @@ Entry counts plus estimated retained heap bytes for the client's internal collec
| `signal_sessions` | `CollectionStats` | Cached Signal sessions |
| `signal_identities` | `CollectionStats` | Cached Signal identities |
| `signal_sender_keys` | `CollectionStats` | Cached sender keys |
| `history_sync_tasks` | `CollectionStats` | Queued/running history-sync tasks and their logical compressed-payload byte sum. A shared `Bytes` slice may retain a larger backing allocation, whose capacity isn't exposed by the type |
| `history_sync_tasks_peak` | `u64` | Lifetime high-water mark of queued/running history-sync tasks |
| `history_sync_payload_bytes_peak` | `u64` | Lifetime high-water mark of logical compressed-payload bytes |
| `chatstate_handlers` | `usize` | Registered chat state handlers |
| `custom_enc_handlers` | `usize` | Registered custom encryption handlers |

`CollectionStats` carries both `entries: u64` and `bytes: u64`. `MemoryReport::total_estimated_bytes(&self) -> u64` sums `.bytes` across every byte-carrying field, and `MemoryReport` implements `Display` for a pretty-printed, human-readable breakdown.
`CollectionStats` carries both `entries: u64` and `bytes: u64`. `MemoryReport::total_estimated_bytes(&self) -> u64` sums `.bytes` across every byte-carrying field. `MemoryReport` implements `Display` for a pretty-printed, human-readable breakdown. This output includes an `--- In-flight history sync ---` section with the two peak fields above.

**Example:**

Expand Down
21 changes: 19 additions & 2 deletions api/store.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -360,7 +360,7 @@ async fn put_msg_secret(
chat: &str,
sender: &str,
msg_id: &str,
secret: &[u8],
secret: &[u8; 32], // MessageSecret, the protocol-fixed message-secret size
) -> Result<()>;

/// Batch upsert. Each MsgSecretEntry carries its own absolute expires_at
Expand Down Expand Up @@ -390,7 +390,24 @@ async fn get_msg_secret_with_ts(
async fn delete_expired_msg_secrets(&self, cutoff_timestamp: i64) -> Result<u32>;
```

`MsgSecretEntry` is `{ chat, sender, msg_id, secret, expires_at, message_ts }`. The SQLite table is:
`get_msg_secret` and `get_msg_secret_with_ts` still return `Vec<u8>` rather than `MessageSecret`. Reads don't carry the same fixed-length guarantee as writes, since the persisted `secret BLOB` column has no length constraint at the SQL level.

```rust
pub type MessageSecret = [u8; 32]; // wacore::reporting_token::MESSAGE_SECRET_SIZE

pub struct MsgSecretEntry {
pub chat: Arc<str>, // shared per conversation instead of one String per row
pub sender: Arc<str>,
pub msg_id: Arc<str>,
pub secret: MessageSecret,
pub expires_at: i64,
pub message_ts: i64,
}
```

The `chat`, `sender`, and `msg_id` fields are `Arc<str>` rather than `String`. This allows buffered batch inserts to clone entries cheaply. The `secret` field uses the fixed-size `MessageSecret` array rather than `Vec<u8>`. This makes an invalid-length secret unrepresentable, so you no longer need a runtime length check.

This is a breaking change if you build `MsgSecretEntry` directly in a custom backend, or if you override the defaulted `put_msg_secret` method. If you construct entries directly, build the JID/ID fields with `Arc::from(...)` or `.into()`, and pass a `[u8; 32]` for `secret`. If you override `put_msg_secret` (most backends don't — see the Note below), update its signature to take `secret: &[u8; 32]` instead of `secret: &[u8]`. The SQLite table itself is unchanged:

```sql
CREATE TABLE msg_secrets (
Expand Down
31 changes: 31 additions & 0 deletions api/wacore.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -335,6 +335,36 @@ pub struct HistoryMsgSecretRecordRef<'a> {

`conversation_index` is a zero-based counter shared across records from the same conversation, letting a filter cache per-conversation classification instead of recomputing it per record. See [Architecture — RAM optimization layers](/concepts/architecture#ram-optimization-layers) for the measured allocation win.

If you want to skip the intermediate owned `HistoryMsgSecretRecord` entirely, `wacore::history_sync` also exports a streaming visitor path:

```rust
pub trait HistoryMsgSecretRecordVisitor {
fn visit(&mut self, record: HistoryMsgSecretRecordRef<'_>) -> usize;
fn reserve(&mut self, _additional: usize) {}
fn retained_item_size(&self) -> Option<std::num::NonZeroUsize> { None }
}

pub fn process_history_sync_bytes_with_record_visitor<F>(
compressed_data: Bytes,
own_user: Option<&str>,
retain_blob: bool,
visitor: F,
) -> Result<HistorySyncResult, HistorySyncError>
where
F: for<'a> FnMut(HistoryMsgSecretRecordRef<'a>)

pub fn process_history_sync_bytes_with_record_sink<V>(
compressed_data: Bytes,
own_user: Option<&str>,
retain_blob: bool,
visitor: V,
) -> Result<HistorySyncResult, HistorySyncError>
where
V: HistoryMsgSecretRecordVisitor
```

`process_history_sync_bytes_with_record_visitor` takes a plain `FnMut` closure for the common case. The closure builds your own row directly from the borrowed `HistoryMsgSecretRecordRef`. This ensures the owned `HistoryMsgSecretRecord` is never allocated. `process_history_sync_bytes_with_record_sink` takes a full `HistoryMsgSecretRecordVisitor` implementation instead of a closure. The `visit` return value reports the byte size you retained for that record for accounting. The optional `reserve` and `retained_item_size` hooks let you size your own collection (e.g., a batched SQL insert buffer) up front rather than growing it one record at a time. Reach for `process_history_sync_bytes_filtered` (above) when a simple accept/reject predicate is enough. Reach for the visitor/sink pair when you also want to avoid materializing the owned record.

### Time

```rust
Expand Down Expand Up @@ -787,6 +817,7 @@ The library includes several allocation-reduction strategies that the integratio
- **Secret-presence pre-scan in history sync** — before buffa decodes a `HistorySyncMsg` (30+ fields, String allocations), a shallow varint walk checks whether the message carries `message_secret` at any level. Messages without a secret are skipped entirely. Bench (20k messages, secret-dense fixture): −29.5% allocations (56,020 → 39,520), −4.3% allocated bytes. Production blobs are secret-sparse so the saving is larger in practice
- **Inline storage in `HistoryMsgSecretRecord`** — `chat_id` is `Arc<str>` (allocated once per conversation, shared across all records in that conversation), `msg_id` is `CompactString` (inline for typical 20–22 char WA IDs on 64-bit targets; smaller inline limit on 32-bit/wasm32), and `secret` is `SecretBytes` (inline for secrets ≤32 bytes, heap for larger)
- **Filter-before-materialize in history sync** — `process_history_sync_bytes_filtered` runs a caller-supplied retention predicate against a borrowed [`HistoryMsgSecretRecordRef`](#history_sync-types) before the owned `HistoryMsgSecretRecord` is built; a rejected record is never materialized. Bench (500-conversation blob, upstream PR's rejection-heavy fixture): allocation churn 21.61 MiB → 14.00 MiB, allocation count ~84k → ~26k. The reduction scales with how much of the record set the predicate rejects — the default `process_history_sync_bytes` (accept-all) sees none of it
- **Streaming visitor/sink for history-sync records** — `process_history_sync_bytes_with_record_visitor` and `process_history_sync_bytes_with_record_sink` (via the [`HistoryMsgSecretRecordVisitor`](#history_sync-types) trait) go a step further than the filter predicate above. You can build your own storage row directly from the borrowed `HistoryMsgSecretRecordRef`. This ensures the intermediate owned `HistoryMsgSecretRecord` is never allocated at all. Bench (allocator-instrumented synthetic history extraction): 20.20 MiB → 14.43 MiB allocated (-28.6%); CodSpeed history stream-drain memory: 243.8 KB → 115.2 KB (2.1× less)
- **Borrowed JID parsing** — [`jid::parse_jid_ref`](#wacore-binary) parses common protocol JIDs into a `JidRef` with zero allocation, falling back to `Jid`'s owned parser only for edge cases. Cut allocations 4,093 → 97 in the history-sync task's JID classification path
- **In-place buffered media decrypt** — non-streaming `HttpClient` downloads now authenticate the already-buffered response body and decrypt AES-256-CBC in place (`DownloadUtils::verify_and_decrypt_in_place`, see [download](/api/download#downloadutils)), truncating the MAC/padding tail instead of allocating a second file-sized output buffer. Streaming clients are unaffected. Buffered download/decrypt span: 4.355 MiB → 2.146 MiB

Expand Down
1 change: 1 addition & 0 deletions concepts/architecture.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -525,6 +525,7 @@ graph LR
8. **Secret-presence pre-scan** — before running buffa decode on each `HistorySyncMsg`, a shallow varint walk checks whether the message carries `message_secret` at any level (`WebMessageInfo.message_secret` or `Message.message_context_info.message_secret`). Messages without a secret (the majority in production blobs) are discarded immediately — no struct decode, no allocation. The scan mirrors protobuf merge semantics so repeated field occurrences and malformed bytes are handled identically to a full decode
9. **Shared conversation id** — `HistoryMsgSecretRecord.chat_id` is `Arc<str>`, allocated once per conversation and reference-counted into every record within that conversation (10k clones → 500 on the bench fixture). `msg_id` uses `CompactString` (inline for typical 20–22 char WA IDs) and `secret` uses `SecretBytes` (inline for secrets ≤32 bytes)
10. **Filter-before-materialize retention hook** — `process_history_sync_bytes_filtered` runs a caller-supplied retention predicate against a borrowed `HistoryMsgSecretRecordRef` (a zero-copy view into the not-yet-built record) before allocating the owned, heap-backed `HistoryMsgSecretRecord`. A record the predicate rejects is discarded without ever being materialized. `process_history_sync_bytes` — and `process_history_sync` itself — keep their existing accept-all behavior by wrapping the filtered entry point with an always-`true` predicate, so callers that don't own a retention policy are unaffected. Bench (synthetic 500-conversation blob, upstream PR's rejection-heavy fixture): allocation churn 21.61 MiB → 14.00 MiB, allocation count ~84k → ~26k. The reduction scales with how much of the record set the predicate rejects — the default accept-all behavior sees none of it. See [`wacore` — history_sync types](/api/wacore#history_sync-types) for the exported signatures
11. **Streaming record-visitor path** — `process_history_sync_bytes_with_record_visitor` (closure-based) and `process_history_sync_bytes_with_record_sink` (trait-based, via `HistoryMsgSecretRecordVisitor`) go further than the filter hook above. You can build your own storage row directly from the borrowed `HistoryMsgSecretRecordRef`. This ensures the owned `HistoryMsgSecretRecord` is never allocated, even for accepted records. The visitor trait's optional `reserve` and `retained_item_size` hooks let you size your own collection (e.g., a batched SQL insert buffer) up front. Bench (allocator-instrumented synthetic history extraction): 20.20 MiB → 14.43 MiB allocated (-28.6%); CodSpeed history stream-drain memory: 243.8 KB → 115.2 KB (2.1× less). See [`wacore` — history_sync types](/api/wacore#history_sync-types)

### Skip mode

Expand Down