Skip to content

Update Chapter 8 pricing artifacts, offline routing tests, and Prometheus validation - #8

Merged
brianray merged 5 commits into
mainfrom
copilot/update-pricing-verification
Sep 21, 2026
Merged

brianray merged 5 commits into
mainfrom
copilot/update-pricing-verification

Conversation

Copilot AI commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

This updates the Chapter 8 companion files so pricing stays JSON-driven, token routing/budget examples are covered by deterministic offline tests, and the Prometheus dashboard artifacts are validated in CI. It also reconciles Chapter 8 path references with the repository’s nested layout.

  • Pricing source of truth

    • Refreshes chapters/ch08/pricing.json metadata for the retrieval date and clarifies that live Anthropic pricing could not be re-verified from the sandbox due to unreachable public hosts.
    • Keeps Table 8.1 pricing flow JSON-driven via PRICING_PATH/table_8_1_rates() with no separate hardcoded rate table.
    • Fixes Chapter 8 companion-file path comments to match the actual repo layout.
  • Model router coverage

    • Expands Chapter 8 unit coverage for count_tokens, route_model, and cost_estimate_usd.
    • Asserts offline/local token fallback behavior, provider-failure fallback, small-vs-large routing thresholds, force_large=True, and that the chapter-opening cost figure matches both pricing.json and cost_estimate_usd().
  • Token budget management (Listing 8.2)

    • Refines chapters/ch08/token_budget.py around an explicit budgeted context shape that includes question.
    • Keeps retrieval trimming first, then history summarization via deterministic MockProvider behavior, and raises when the context still cannot fit.
    • Adds tests for trim-only, trim-then-summarize, and failure paths.
  • Prometheus dashboard artifacts (Listing 8.3)

    • Renames the rules file to chapters/ch08/dashboard_rules.yml.
    • Defines llm_token_price_usd_per_mtok as per-model, per-price-type constant recording rules sourced from the same values as pricing.json.
    • Updates dashboard_queries.promql to include commented Chapter 8 cost queries and restores the / 1e6 conversion so token-rate × USD-per-million-token math stays dimensionally correct.
  • CI / parser validation

    • Adds chapters/ch08/verify_promql.sh to validate both the rules file and each PromQL expression through promtool.
    • Wires validation into the existing CI workflow and adds a focused root-level Chapter 8 workflow with explicit contents: read permissions.
    • Unignores the verifier script so it can be tracked in the repo.

Example of the updated pricing join pattern:

sum by (model) (
  (label_replace(
    rate(gen_ai_client_token_usage_total{token_type=~"input|cached_input|output"}[5m]),
    "price_type", "$1", "token_type", "(.*)"
  ) / 1e6)
  * on (model, price_type) group_left
  llm_token_price_usd_per_mtok
)

Copilot AI and others added 4 commits September 21, 2026 17:42
Co-authored-by: brianray <676284+brianray@users.noreply.github.com>
Co-authored-by: brianray <676284+brianray@users.noreply.github.com>
Co-authored-by: brianray <676284+brianray@users.noreply.github.com>
Co-authored-by: brianray <676284+brianray@users.noreply.github.com>
Copilot AI changed the title [WIP] Verify and update pricing in pricing.json Update Chapter 8 pricing artifacts, offline routing tests, and Prometheus validation Sep 21, 2026
Copilot AI requested a review from brianray September 21, 2026 17:44
@brianray
brianray marked this pull request as ready for review September 21, 2026 17:47
@brianray
brianray merged commit 7f3293b into main Sep 21, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants