Skip to content

feat: Report token usage/cost per check run; add --dry-run #72

Description

@Tgenz1213

Problem

archguard check calls a paid LLM API (for any provider except ollama) once per (file, relevant-ADR) pair via llm.AnalyzeDrift (internal/llm/llm.go:117), plus one embedding call per analyzed file (fetchContext/CreateEmbedding path in internal/analysis/engine.go). None of this is measured or surfaced: a run against --all on a large repo with a cloud provider can rack up real cost with no visibility into how much, and no way to check before running whether it's about to be expensive.

check --suggest-fixes (added after this issue was filed, see docs/arch/0016-llm-suggested-remediation.md and docs/arch/0017-suggestion-cache-key-namespace.md) adds a second, distinct llm.Provider.Chat call type (llm.SuggestRemediation), made only after a violation is already confirmed by a judgment call. Any accounting/dry-run work here needs to account for it explicitly rather than silently lumping it in with judgment calls.

Acceptance Criteria

  • After a check run, print a summary of tokens consumed: input/output tokens for chat calls, and tokens for embedding calls, split out by provider role (chat vs. embedding, since vector_store.provider can differ from llm.provider per docs/arch/0004-decoupled-chat-and-embedding-providers.md). This can be computed without changing the llm.Provider interface at all: every provider already implements CountTokens(ctx, text) (internal/llm/llm.go:59) using its own real tokenizer, so the engine can call it on each prompt/response/embedded-text it already has in hand, purely as an accounting step alongside the existing calls.
    • When --suggest-fixes is set, SuggestRemediation calls get their own line in the summary (e.g. "judgment" vs. "suggestion" token/call counts), not merged into the judgment-call total -- they're gated by a different flag, only fire per confirmed violation rather than per (file, ADR) pair, and a suggestion cache hit (see docs/arch/0017-suggestion-cache-key-namespace.md) means zero incremental cost that a merged total would misrepresent.
  • Add an optional, user-supplied cost estimate: new config fields (e.g. llm.cost_per_1k_input_tokens, llm.cost_per_1k_output_tokens, vector_store.cost_per_1k_tokens in archguard.yaml) that, when set, turn the token summary into a dollar estimate. Do not hardcode a provider pricing table in the binary -- provider pricing changes over time and a stale hardcoded table would silently misreport cost, which defeats the point of a trust-building feature. Omit the fields and only token counts are shown.
  • Add archguard check --dry-run: runs the full pipeline including embedding calls (needed to determine which ADRs are actually relevant per file via vector search) but skips every judgment Chat call, reporting the tokens/estimated cost that would be spent on chat analysis for the (file, ADR) pairs that would have been checked, plus the real embedding cost already incurred. No violations are reported and the exit code is always ExitSuccess regardless of what a full run would have found.
    • --dry-run does not attempt to project suggestion-call cost, even with --suggest-fixes also passed: which files would produce a confirmed violation (the trigger for a suggestion call) is only known by actually running the judgment call, which --dry-run skips by design -- there's no sound way to estimate a count that depends on output --dry-run never produces. Document this explicitly as a known limitation (e.g. a one-line note in --dry-run's output when --suggest-fixes is also set) rather than leaving it an unstated gap.
  • README documents the new config fields and --dry-run, with a note that dollar figures are only as accurate as the rates the user configured, and the --dry-run + --suggest-fixes limitation above.
  • Tests cover token summary accuracy against llm.MockProvider (including the judgment/suggestion split), and --dry-run making zero judgment Chat calls.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions