feat(desktop): NIP-AM agent-usage backend — P2 emission/transport/archive + P4a aggregation/D6

[wpfleger96]

What

Implements Phases 2 and 4a of the Usage v2 plan (plan events d0268cd0/0e95b035), extending the archive backend to emit, transport, archive, and aggregate both cache categories and billing identity fail-closed.

P2 — emission, transport, archive

Tri-state accumulators (Unseen/Exact/Unknown) for cache-read and cache-write in buzz-agent turn and session state. Absent field = Unknown (never zero) through the full pipeline. No unwrap_or(0) on the cache path. Both cache folds are gated on usage-bearing responses (same gate as the total-state and identity folds) — a response with no usage at all must not poison either accumulator.

Overflow-aware input token parsing and accumulation — closed end-to-end from parse through wire to ACP:

• sum_usage() returns SumUsageResult (Exact(u64) | Overflow) — checked arithmetic, never clamps. anthropic_input_tokens() returns Option<SumUsageResult> since it sums three fields (input_tokens + cache_read_input_tokens + cache_creation_input_tokens) that can collectively overflow. Single-field callers (prompt_tokens, completion_tokens, etc.) convert via .into_exact() — their single-field sums cannot overflow.
• LlmResponse.input_tokens_overflowed: bool propagates the parse-layer signal into the run loop. When set, input_tokens is None (clamped value discarded), the context-gate baseline (last_request_input_tokens) is frozen at its prior reading, and turn_input_tokens is poisoned to TurnIOState::Poisoned before any emission — including mid-turn emit_usage_update calls. A dedicated enum on LlmResponse.input_tokens would ripple into ~20 existing test assertions on r.input_tokens == Some(...); the bool flag confines the change to the two call sites that check it.
• TurnIOState (Unseen/Exact/Poisoned) for input and output: per-round fold uses checked_add; overflow poisons permanently at turn and session level, no healing. Absence does not poison (pass-2-cleared contract unchanged). Wire emission omits accumulatedInputTokens/accumulatedOutputTokens when poisoned — never null, never u64::MAX. ACP treats absent = publisher-poisoned: delta_reliable: false, null turn fields, null cumulative for that category; session cumulative stays unknown for all subsequent turns once poisoned.

Conditional wire emission for accumulatedCachedInputTokens and new accumulatedCacheWriteTokens: fields are omitted when the cumulative is Unseen or Unknown. ACP _goose/unstable/session/update contract documented next to the payload with tests for all absence/zero variants.

PricingIdentity stamping (publisher-side):

• pricing_authority(): canonical parsed-URL endpoint comparison against the official allowlist — HTTPS only, exact allowlisted host (lookalike-safe), default port (omitted or explicit :443), required API base path, rejects userinfo/query/fragment/path-prefix lookalikes.
• Model: the actually-requested request_model after mesh/auto resolution (not effective_model_str).
• Turn discipline: identity retained only while ALL usage in the current turn carries one identical proven identity; any mismatch, unproven-usage-bearing response, or unpaired cumulative snapshot poisons to absent; a later matching notification does not heal a mixed turn.

ACP UsageTracker identity fold: per-in-flight-turn tri-state identity accumulator replacing last-update-wins. Any absent identity on a token-advancing notification or exact mismatch poisons to absent; poison survives later updates; reset in begin_turn()/take(); reset also when a request fails (baseline cleared so preflight gate cannot stay frozen sub-threshold on retries).

M3 migration: adds turn_cache_write_tokens, cumulative_cache_write_tokens, pricing_authority, pricing_model, pricing_cache_class to agent_metric_index. Additive, idempotent, guarded per-column by marker. M2 migration also guarded per-column (turn and cumulative cache-read columns checked and added independently; marker commits only after both are present). Fresh-DB schema includes all columns.

First-turn baselines: seed_zero_baseline seeds last_input: Some(0), last_output: Some(0), last_cached_input: Some(0), last_cache_write: Some(0), and last_total: Some(0) — all have the known-zero-at-spawn argument. Absent fields from incoming snapshots still produce unknown (tri-state unchanged). Sessions buzz-acp did not spawn (no seed) remain fail-closed on turn one.

ReportedUsage TS mirror: cacheReadTokens, cacheWriteTokens, freshInputTokens added to tauriArchive.ts as UsageField members, field-for-field with the Rust struct.

P4a — aggregation layer

Extended S-1 ladder to cache-read and cache-write via the same ladder_token path as the existing token fields.

freshInputTokens derivation: checked arithmetic, fail-closed — absent cache fields produce Unknown (not zero), overflow and cacheRead+cacheWrite > input both produce incomplete: true. Aggregated as a UsageField.

D6 comparator: sort_value() = provider total when known, else input+output when both known, else None (unknown-last). Replaces the prior total-only comparator for both agent-level and model-level sort. Ships a pinned test vector that the TS render layer (P5) must match.

Test coverage

• buzz-agent: 440 lib + 15 integration (golden_transcripts) — includes 13 new cache_total_state_tests; 14 new turn_io_state_tests; 3 new sum_usage_* tests (exact single-field, exact two-field, overflow signals correctly); 3 new parse_anthropic_* tests (overflow flag set + value cleared, normal sum no flag, absent usage no flag); end-to-end golden transcript drives real subprocess with Anthropic-shaped input_tokens: u64::MAX, cache_read: 1 response and asserts accumulatedInputTokens absent from the emitted usage_update — no logic duplication; 3 wire pin tests; 4 fold_pricing_identity_* tests; pricing_authority() explicit-:443 acceptance
• buzz-acp: 700 tests (691 lib + 9 integration) — 4 new usage tests (absent input → unreliable+null; absent output → unreliable+null; goose-shaped both present unchanged; poison mid-session); 3 ACP behavior tests; 7 pool lifecycle tests
• Desktop (Rust): 2259+ tests — 14 new P4a pinned tests; 2 M3 round-trip tests; 1 serde key-shape test; 2 M2 partial-schema migration tests; first-turn cache round-trip test

Related PRs

• P1 NIP-AM spec: #4632
• P3 pricing table: #4629
• UI (P5): #4001

[wesbillman]

Commenting on Wes’s behalf. [P2] Input/output overflow is silently published as an exact saturated count.

Per-round accumulation and turn-to-session emission use saturating_add (crates/buzz-agent/src/agent.rs:239-243, 375-382). Two valid provider reports of u64::MAX and 1 therefore produce u64::MAX; ACP treats that as a reliable delta and the archive reports it as exact. This conflicts with the checked/unknown behavior implemented for total and cache counters in this PR.

Please use checked tri-state accumulation (or otherwise poison reliability) for input/output, with both within-turn and turn-to-session overflow tests.

[wesbillman]

Commenting on Wes’s behalf.

Requesting changes because the remaining overflow blocker is still present at head b564ecd99eee8d3700c422d87c43787447add96a.

The updated commit fixes the two inline findings (independent M2 column repair and known-zero first-turn cache baselines), but input/output accumulation still uses saturating_add both within a turn and when merging a completed turn into the session (crates/buzz-agent/src/agent.rs:239-242, 375-381; crates/buzz-agent/src/lib.rs:762-767). Overflow therefore remains silently serialized as an exact u64::MAX cumulative count, unlike the checked/unknown behavior now used for total and cache counters.

Please use overflow-aware tri-state/reliability handling for input and output, with tests for both within-turn and turn-to-session overflow. This is the unresolved P2 reported in #4000 (comment).

[wesbillman]

Commenting on Wes’s behalf.

Requesting changes at head d64e4d35c2fdec6b2a58c0da51ddcbe32b4e8c82 for one accounting-summary inconsistency: hasUnknownUsage ignores the newly exposed cache-read/cache-write fields and derived freshInputTokens. A response can therefore contain { incomplete: true } while its bucket, agent, model, and top-level coverage summary claims there is no unknown usage. I left the exact reproductions and requested coverage inline.

The previously requested overflow blocker is resolved at this head: input/output accumulation now uses checked, permanently poisoned optional state through per-round, turn-to-session, ACP, and wire boundaries rather than publishing saturated values as exact counts.

[wesbillman]

Two fail-closed accounting defects remain at head d64e4d35c2fdec6b2a58c0da51ddcbe32b4e8c82. The overflow representation is now correct publisher-side, but ACP can heal a poisoned cumulative input/output stream if a later producer snapshot reintroduces the field, and P4a's aggregate completeness flag ignores the two new cache categories. Both can make downstream accounting claim reliability/completeness after observing unknown data. Details inline.

[wesbillman]

Details for my requested-changes review (GitHub could not anchor these lines in the very large diff):

1. [P1] Sticky ACP poison — crates/buzz-acp/src/usage.rs:549-560. take() replaces SessionState.last_input/last_output with the latest optional snapshot. Reproducer: seeded baseline → absent input (publisher poison) → later Some(100) → later Some(150). The final turn becomes delta_reliable=true with delta 50, healing an unknown prefix. Preserve a sticky poisoned state for both fields and test absent→present→present.

2. [P1] Cache completeness omitted — desktop/src-tauri/src/archive/agent_usage.rs:472-477. UsageAccumulator::has_unknown() checks input/output/total/cost but not cache-read/cache-write. Exact primary counters plus omitted cache fields therefore yield cache/fresh-input fields marked incomplete while bucket/model/agent/coverage can say hasUnknownUsage=false. Include both cache accumulators and assert scope/coverage flags in the existing omitted-cache regression.

[wesbillman]

The two prior findings are directionally addressed, and the aggregate completeness fix is correct. One P1 escape remains in the sticky-poison repair at head 7ae974d8be1188434bf0decba804a252754db2b0:

[P1] An absent intermediate notification can still heal within the same turn. record() computes input_poisoned / output_poisoned from the committed SessionState plus only the current notification (crates/buzz-acp/src/usage.rs:349-364). During an in-flight turn, it does not persist those flags; take() derives the session flags only from the final pending record (:579-587). Because multiple usage notifications per turn are explicitly supported, this sequence still escapes:

1. seeded session, begin_turn();
2. notification A omits input → pending is unreliable, but committed input_ever_poisoned remains false;
3. notification B in the same turn sends input again → record() sees committed false + current present and replaces pending with a reliable delta;
4. take() sees the final cumulative input as present and commits input_ever_poisoned=false.

The omission has vanished completely. This is the same fail-closed violation as the cross-turn reproducer, just inside the existing multi-notification path. Track per-in-flight input/output poison (like pending_identity), reset it in begin_turn()/take(), fold every matching notification into it, and commit that sticky state. Please add symmetric same-turn absent→present tests for input and output.

The UsageAccumulator::has_unknown() update and its bucket/model/agent/coverage tests resolve the cache/fresh-input finding cleanly.

[wesbillman]

Commenting on Wes’s behalf.

Requesting changes at head 7ae974d8be1188434bf0decba804a252754db2b0. The follow-up fixes the aggregate hasUnknownUsage defect, and the new cross-turn input/output tests cover absent→present snapshots across published turns. However, the ACP sticky-poison fix still heals an absent snapshot when another usage notification restores the field within the same in-flight turn.

record() computes poison from committed SessionState plus only the current notification, then replaces pending (crates/buzz-acp/src/usage.rs:341-402, 440-490). It does not persist poison observed by earlier notifications in that turn. take() likewise derives committed poison only from the final pending cumulative fields (:572-600). Therefore seeded baseline → begin turn → input absent → input present produces a reliable final record and commits an unpoisoned session. Output is symmetric.

Please add per-in-flight input/output poison accumulators, fold every matching notification into them, clear them at the turn boundary, and add symmetric same-turn absent→present regression tests. This is a remaining fail-closed accounting blocker.

[wesbillman]

Commenting on Wes's behalf.

Requesting changes at head 080545a2a7e507557ee90a0926b35f85be4b91f8 for one remaining P1 fail-closed escape found in the final delta review.

UsageTracker::record() folds absent input/output only when the notification matches the current in-flight session (crates/buzz-acp/src/usage.rs:413, 426-433). A notification for session A while session B is in flight is discarded entirely (:625-627). Discarding A's cumulative counters is correct because advancing its baseline would undercount A's next turn, but discarding the observed absence is not: a later present A snapshot can then publish delta_reliable: true, healing an unknown prefix despite the sticky-poison contract.

Reproducer at this head:

1. Seed A and B; publish a reliable A snapshot at input/output 50/10.
2. Begin B; receive a late A snapshot with input absent and output 10; publish B normally.
3. Begin A; receive A at 100/20.
4. A is currently reported reliable; it must remain permanently unreliable because ACP observed the absent A input snapshot.

Please preserve the existing rule that cross-session notifications do not advance counters, while monotonically OR-ing any observed input/output absence into that notification's session poison flags. Add symmetric input and output regressions that prove both baseline preservation and permanent poison.