Skip to content

Make offline scorecard pricing provider-aware #4335

Description

@Hmbown

Problem

The offline scorecard currently accepts a model ID and usage but no effective provider. It calls model-only pricing helpers, so the same model name can be assigned API pricing even when the recorded turn came from a route where dollar pricing is unavailable or materially different (for example OpenAI API versus ChatGPT/Codex OAuth).

Draft PR #4332 fixes provider-aware pricing for active TUI turns, compaction/purge, runtime-thread records, tool children, and sub-agent mailbox usage. The standalone offline scorecard remains model-only and is intentionally called out as outside that claim.

The original scorecard work came from @findshan in #3388. Preserve that contributor credit in any redesign or harvested patch.

Expected behavior

  • Persist or ingest effective provider provenance alongside every scorecard turn.
  • Use provider-aware pricing; unknown/legacy provenance fails closed as unpriced rather than guessing.
  • Keep token/cache metrics available even when dollar cost is unavailable.
  • Preserve backward compatibility for old scorecard inputs and clearly mark their price as unknown.
  • Do not infer provider from model name.

Acceptance

  • Scorecard input/records carry optional effective provider identity.
  • Codex OAuth tokens aggregate with zero fabricated API dollar cost.
  • Same-model, different-provider fixtures produce different truthful pricing results.
  • Legacy model-only fixtures remain readable and explicitly unpriced.
  • CLI output and JSON distinguish zero cost from unavailable pricing.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingreliabilityReliability, flaky behavior, retries, fallbacks, and robustnesssubagentsSub-agent orchestration, lifecycle, and completion handlingtuiTerminal UI behavior, rendering, or interactionv0.8.69Targeting v0.8.69

    Projects

    Status
    Done

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions