-
Notifications
You must be signed in to change notification settings - Fork 3.4k
Make offline scorecard pricing provider-aware #4335
Copy link
Copy link
Closed
Labels
bugSomething isn't workingSomething isn't workingreliabilityReliability, flaky behavior, retries, fallbacks, and robustnessReliability, flaky behavior, retries, fallbacks, and robustnesssubagentsSub-agent orchestration, lifecycle, and completion handlingSub-agent orchestration, lifecycle, and completion handlingtuiTerminal UI behavior, rendering, or interactionTerminal UI behavior, rendering, or interactionv0.8.69Targeting v0.8.69Targeting v0.8.69
Milestone
Description
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't workingreliabilityReliability, flaky behavior, retries, fallbacks, and robustnessReliability, flaky behavior, retries, fallbacks, and robustnesssubagentsSub-agent orchestration, lifecycle, and completion handlingSub-agent orchestration, lifecycle, and completion handlingtuiTerminal UI behavior, rendering, or interactionTerminal UI behavior, rendering, or interactionv0.8.69Targeting v0.8.69Targeting v0.8.69
Projects
StatusShow more project fields
Done
Problem
The offline scorecard currently accepts a model ID and usage but no effective provider. It calls model-only pricing helpers, so the same model name can be assigned API pricing even when the recorded turn came from a route where dollar pricing is unavailable or materially different (for example OpenAI API versus ChatGPT/Codex OAuth).
Draft PR #4332 fixes provider-aware pricing for active TUI turns, compaction/purge, runtime-thread records, tool children, and sub-agent mailbox usage. The standalone offline scorecard remains model-only and is intentionally called out as outside that claim.
The original scorecard work came from @findshan in #3388. Preserve that contributor credit in any redesign or harvested patch.
Expected behavior
Acceptance