Skip to content

feat(acp): report standard adapter usage - #4950

Open
atishpatel wants to merge 2 commits into
mainfrom
workerbee/acp-prompt-usage
Open

feat(acp): report standard adapter usage#4950
atishpatel wants to merge 2 commits into
mainfrom
workerbee/acp-prompt-usage

Conversation

@atishpatel

Copy link
Copy Markdown
Contributor

Why

Claude Code and Codex expose standard ACP prompt-response usage, but Buzz only consumed Goose’s private cumulative usage notification. Their token use and Claude’s cumulative cost were therefore absent from NIP-AM metrics.

What

  • Read per-turn session/prompt response usage for known Claude and Codex adapters
  • Publish Claude’s raw cumulative cost separately from per-turn tokens without changing the NIP-AM schema
  • Keep Goose usage exclusive and cover Claude/Codex wire serialization

Risk Assessment

Low-to-medium: changes best-effort observability only and does not affect prompt execution. The adapter-specific mappings preserve source semantics and omit unavailable fields.

References

  • Validated with cargo fmt --check, cargo test -p buzz-acp --no-run, and full cargo test -p buzz-acp (678 passed at 652e373a before merge-trailer amendment).

Generated with Codex

@atishpatel
atishpatel force-pushed the workerbee/acp-prompt-usage branch 2 times, most recently from e20cb57 to 0540e3d Compare August 5, 2026 21:08
@atishpatel
atishpatel marked this pull request as ready for review August 5, 2026 21:12
@atishpatel
atishpatel requested a review from a team as a code owner August 5, 2026 21:12
@custard-pirate

Copy link
Copy Markdown

I validated this problem independently while building an owner-side NIP-AM usage projection for a mixed Codex/Claude/Goose fleet. This PR covers the important upstream seam, and its separate standard-adapter tracker is a better fit than duplicating the local integration.

Three attribution/coverage details from that deployment may be useful here or in a focused follow-up:

  1. Codex exposes the effective model at /_meta/quota/model_usage/0/model in the prompt result. Retaining that value prevents otherwise valid kind 44200 records from landing under an unknown model. When the adapter omits it, the operator-selected model from PromptContext is a useful fallback.
  2. Normalize the launched adapter command into a provider-neutral harness label (codex, claude, or goose) before publishing. Absolute paths and wrapper names otherwise fragment the dataset.
  3. Consider emitting a coverage-only kind 44200 record with null token fields when a completed turn supplies no usage. That distinguishes unsupported/missing telemetry from agent inactivity without treating unknown values as zero.

Cache-write preservation and the prompt-response token path are already covered here (and overlap #4000), so I do not plan to open a competing ACP usage PR. I can prepare a small dependent follow-up for these three points once the desired behavior and landing order are clear.

@atishpatel
atishpatel force-pushed the workerbee/acp-prompt-usage branch from 0540e3d to b542ac8 Compare August 10, 2026 12:27

@wesbillman wesbillman left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewing on Wes's behalf.

The new standard-adapter mapping publishes usage.inputTokens directly while also publishing cachedReadTokens/cachedWriteTokens. Both supported adapters define the ACP prompt-response inputTokens as non-cached input: Codex explicitly subtracts cached input in toTokenCount(), and Claude's sessionUsage() keeps fresh input, cache reads, and cache writes as separate accumulators. NIP-AM requires the opposite representation: inputTokens is the inclusive input-side total, and cache fields are informational subsets (docs/nips/NIP-AM.md, Numeric validity and token semantics; also TokenCounts' contract in crates/buzz-core/src/agent_turn_metric.rs).

As written, a Claude turn with inputTokens=100, cachedReadTokens=30, and cachedWriteTokens=25 publishes input=100 rather than 155; the added test currently pins that undercount. Codex has the same issue for cache reads. This corrupts NIP-AM input totals and any aggregation/cost logic built on them.

Please normalize standard usage before constructing TurnUsage: checked-add cache read/write into turn_input_tokens, fail closed on overflow, and update the wire tests to assert the inclusive input value while preserving each cache subset separately.

@atishpatel
atishpatel force-pushed the workerbee/acp-prompt-usage branch from b542ac8 to b259a8b Compare August 10, 2026 15:14
@atishpatel

Copy link
Copy Markdown
Contributor Author

Rebased onto latest main (5e4c05f9) and incorporated Wes's review feedback.

PR #4000 does not make this PR redundant. #4000 adds the cumulative _goose/unstable/session/update path used by Goose/buzz-agent plus the archive/accounting infrastructure around it. It does not ingest the standard ACP session/prompt response usage emitted by Claude/Codex, and it does not ingest Claude's standard session/update cumulative cost notification. Those are the remaining gaps covered here.

Per Wes's review, this revision now normalizes standard-adapter input to NIP-AM's inclusive contract with checked arithmetic:

  • Claude: inputTokens + cachedReadTokens + cachedWriteTokens (the test now proves 100 + 30 + 25 = 155 while preserving both cache subsets)
  • Codex: inputTokens + cachedReadTokens + cachedWriteTokens (the test now proves 90 + 40 = 130 while preserving Codex's provider total)
  • overflow fails closed instead of saturating or publishing an undercount

It also preserves #4000/current-main's fail-closed optional cumulative counters, cache-write support, and pricing identity, while keeping Goose/buzz-agent usage exclusive if both producer paths emit data.

Wes's broader attribution/coverage suggestions—effective model fallback, normalized provider-neutral harness labels, and optional coverage-only records for completed turns with no usage—remain useful follow-up scope; they are not supplied by #4000 and are not folded into this narrowly rebased ingestion change.

Validation: cargo test -p buzz-acp (722 unit + 9 lifecycle tests passed), cargo fmt --all -- --check, and the push hooks' Rust/Desktop Tauri checks.

— Brainy Bumble

@wpfleger96 wpfleger96 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Combined + deduped feedback from the dual review I ran on this (two independent passes; full writeups on our side). The current head already fixes the inclusive-input issue from the earlier review round — checked-add of cache read/write with fail-closed overflow looks right. What's left: two accounting gaps both passes converged on (the cost baseline and the initial_message lifecycle), plus a few nits. Overall the shape is exactly what we hoped for next to #4000: goose stays exclusive, standard adapters fill the gap, and omitting pricingIdentity/model for these records is the correct fail-closed call vs #4629's exact-lookup contract.

Comment thread crates/buzz-acp/src/usage.rs Outdated
Comment thread crates/buzz-acp/src/acp.rs
Comment thread crates/buzz-acp/src/usage.rs
Comment thread crates/buzz-acp/src/usage.rs Outdated
Comment thread crates/buzz-acp/src/acp.rs
WorkerBeeGPT and others added 2 commits August 10, 2026 15:01
Co-authored-by: Atish Patel <atish@squareup.com>
Signed-off-by: Atish Patel <atish@squareup.com>
Signed-off-by: Brainy Bumble <0ed7657b57c0e8a9f5288390dd6c8d5d0a3a06abe9b01b9006814f52077d6cdf@buzz.block.builderlab.xyz>
Co-authored-by: Atish Patel <atish@squareup.com>
Signed-off-by: Atish Patel <atish@squareup.com>
@atishpatel
atishpatel force-pushed the workerbee/acp-prompt-usage branch from b259a8b to 001af63 Compare August 10, 2026 20:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants