Skip to content

Cost Grok agents from the usage file their CLI already writes - #536

Open
gpinkham wants to merge 1 commit into
chaitanyagiri:mainfrom
gpinkham:fix/grok-usage-ledger
Open

gpinkham wants to merge 1 commit into
chaitanyagiri:mainfrom
gpinkham:fix/grok-usage-ledger

Conversation

@gpinkham

@gpinkham gpinkham commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

What & why

A Grok-backed agent ran normally and cost $0.00 / 0 tok forever. The telemetry env that makes an agent push OTel is injected for Claude Code alone (hive.ts ensureAgent), and a Grok agent writes no Claude transcript — so both of TelemetryCollector's usage sources came back empty. The result was no fleet numbers and no row in cost-ledger.jsonl at all: on my machine, 21,542 ledger rows, every one a Claude model, and none for the Grok agent that had been working alongside them.

Its real numbers were on disk the whole time. The Grok CLI keeps a cumulative per-session snapshot at ~/.grok/sessions/<encodeURIComponent(cwd)>/<sessionId>/usage.json, and both halves of that path are already resolved — the Grok hook bridge normalizes its camelCase payload to session_id (GROK_HOOK_SHIM), recordSession stores it, so resolveCwd/resolveSessionId answer for a Grok agent exactly as they do for a Claude one. This adds a reader, not new plumbing. Cost comes from the same file (costUsdTicks, 10^10 ticks to the dollar per Grok's docs), so there are no Grok prices to maintain.

The dedup is the part that needed care. appendCostLedger is fed a running total, so re-appending an unchanged sample writes the same row forever — that is #56, which left 2,417 identical rows. The transcript fallback dodges it by reporting an empty sessionId to disqualify itself from the ledger; this provider cannot, because it needs a real session id to be accounted at all. Hence CumulativeSampleGate: admit a row only when the numbers move.

The Claude path is untouched. grokFallback returns null for a Claude agent (its session id is not a directory under ~/.grok/sessions), and the ledger gate short-circuits on provider before the gate is consulted.

Closes #535

Type of change

  • Bug fix
  • New feature
  • Refactor / cleanup
  • Docs
  • Build / CI

Evidence

No visible UI to capture here — this is a main-process ingest path — so per the template's "no visible UI" rule the evidence is a red→green test plus a ledger diff taken against a real Grok agent.

Before

test/grok-usage-ledger.test.cjs on main, with the fix reverted:

### RED — on upstream/main, with the fix reverted ###
not ok 1 - a Grok agent is costed from the CLI’s own usage.json
not ok 2 - the sample carries a real session id, so the ledger accepts it
ok 3 - no usage.json means no data, not a zeroed sample
ok 4 - an unknown cwd or session id is not guessed at
not ok 5 - an idle Grok agent does not re-append the same ledger row (#56)
not ok 6 - the gate keeps agents apart and forgets on request
# tests 6
# pass 2
# fail 4

(3 and 4 pass on main too — they assert that a non-Grok agent gets nothing from this source, which was already true and must stay true.)

And the real ledger, against the live Grok agent on this machine:

Grok agent in registry.json:
  ryan-mu1mcvvx  provider=grok  sessionId=01a0a15a-0176-7f22-9a00-c81bcd4e3940

BEFORE  cost-ledger.jsonl: 21542 rows total, 0 for ryan-mu1mcvvx
        distinct models in ledger: retention_carry, claude-opus-4-8, claude-opus-5,
                                   claude-sonnet-4-6, (empty), claude-haiku-4-5-20251001

After

Same test file, same command, with the change applied:

### GREEN — with the fix ###
ok 1 - a Grok agent is costed from the CLI’s own usage.json
ok 2 - the sample carries a real session id, so the ledger accepts it
ok 3 - no usage.json means no data, not a zeroed sample
ok 4 - an unknown cwd or session id is not guessed at
ok 5 - an idle Grok agent does not re-append the same ledger row (#56)
ok 6 - the gate keeps agents apart and forgets on request
# tests 6
# pass 6
# fail 0

Same agent, same cost-ledger.jsonl, with the fix — the row that now gets written, and the dedup holding across three idle beats:

AFTER   getAgentUsage('ryan-mu1mcvvx') ->
        {"agent_id":"ryan-mu1mcvvx","session_id":"01a0a15a-0176-7f22-9a00-c81bcd4e3940",
         "ts":1789417432899,"input":1782920,"output":14247,"cache_read":1524736,
         "cache_creation":0,"model":"grok-4.6-build","usd":0.53883412}

DEDUP   three consecutive 30s beats, agent idle: admitted=[true,false,false]
        -> 1 row written, not 3 (bug #56 not reproduced)

$0.5388 is costUsdTicks: 5388341200 / 1e10. Reading that number raw would have billed the session at $5.4 billion, which is why the tick scale has its own assertion.

How I tested it

  • OS: macOS (Darwin 21.6.0), Node 20.20.2
  • Steps:
    1. npm run typecheck — clean.
    2. npm run test:focused — 840/840 pass, including the 6 new ones.
    3. npm run build — succeeds.
    4. Wrote test/grok-usage-ledger.test.cjs first and confirmed it goes red on main (4 of 6 failing) before the fix existed, then green with it — output above. It sandboxes HOME to a temp dir, so it never reads a developer's real ~/.grok.
    5. Ran the provider against the real Grok agent in my hive (ryan-mu1mcvvx) and its real ~/.grok/sessions/.../usage.json, and confirmed the row it produces matches the file, and that three consecutive idle beats append once rather than three times.
    6. Confirmed the full suite failure set is byte-identical with and without the change, so nothing here regressed an existing test.
    7. Verified the file's write cadence on a live turn rather than assuming it: the agent took a turn on 2026-09-16, turns[] grew from three entries to four, and the new entry's endedAt (01:43:56.258856Z) equals the file's updatedAt and sits within a millisecond of its mtime — so the snapshot is rewritten at turn end, one entry per turn. The session totals moved with it ($0.5388 → $0.8907), which is exactly the change the dedup gate is there to admit.

Credit (optional)

Discord:

X:

Checklist

  • Before and after evidence is attached above, under both headings.
  • npm run typecheck passes.
  • npm run test:focused passes.
  • npm run build succeeds.
  • This PR is one change. Unrelated fixes belong in their own PR.
  • I read the diff myself before opening this, and there is no debug output,
    commented-out code, or unrelated formatting churn in it.
  • Any new UI derives from DESIGN.md / tokens.ts — no ad-hoc colors,
    spacing, or fonts. (No UI in this change.)
  • If I added art, it's my own or compatibly licensed, and listed in
    ATTRIBUTION.md. (No art.)

🤖 Generated with Claude Code

https://claude.ai/code/session_01YSioheUKVkWkZ3aRxjYnaJ

A Grok agent ran fine and cost $0.00 forever. The telemetry env that makes an
agent push OTel is injected for Claude Code alone (hive.ts `ensureAgent`), and
a Grok agent writes no Claude transcript, so both of the collector's sources
came back empty — no fleet numbers, and no row in cost-ledger.jsonl at all. On
this machine that is 21,542 ledger rows, every one of them a Claude model, and
none for the Grok agent that had been working alongside them.

Its real numbers were on disk the whole time. The Grok CLI keeps a cumulative
per-session snapshot at ~/.grok/sessions/<encodeURIComponent(cwd)>/<id>/
usage.json, and both halves of that path are already resolved: the Grok hook
bridge normalizes its camelCase payload to `session_id` (GROK_HOOK_SHIM), which
`recordSession` stores, so resolveCwd/resolveSessionId answer for a Grok agent
exactly as they do for a Claude one. So this adds a reader, not any plumbing.
Cost comes from the file too — `costUsdTicks`, 10^10 ticks to the dollar per
Grok's own docs — so there are no Grok prices to maintain here.

The dedup needs care. `appendCostLedger` is fed a running total, so re-appending
an unchanged sample writes the same row forever; that is chaitanyagiri#56, which left 2,417
identical rows. The transcript fallback dodges it by reporting an empty session
id to disqualify itself from the ledger, but this provider cannot — it needs a
real session id to be accounted at all. Hence CumulativeSampleGate: admit a row
only when the numbers move. An idle Grok agent re-reads the same file every beat
and is correctly silent. `ts` is not in the signature, because when only the
write time changed there is no new cost to record.

The Claude path is untouched. `grokFallback` returns null for a Claude agent —
its session id is not a directory under ~/.grok/sessions — and the ledger gate
short-circuits on provider before the gate is ever consulted.

Closes chaitanyagiri#535

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YSioheUKVkWkZ3aRxjYnaJ
@github-actions

Copy link
Copy Markdown
Contributor

🚫 This PR is missing its before/after evidence

Every pull request here has to show its work. Screenshots or a short screen recording, before the change and after it.

  • Before — no image or video under that heading
  • After — no image or video under that heading

How to fix it: edit the description, keep the ### Before and ### After headings from the template, and drag an image or video under each. GitHub uploads it inline. This check re-runs the moment you save.

A bug fix with no visible surface still needs it: show the failing behaviour, then the same steps passing. A terminal recording is fine.

Genuinely nothing to show — a CI tweak, a typo, a dependency bump? A maintainer can apply the no-visual-change label. Please don't ask unless it truly has no observable effect.

📖 CONTRIBUTING.md → Evidence is mandatory

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Non-Claude (Grok) agents record zero token/cost usage

1 participant