Agent provider model gap audit #134
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: Agent provider model gap audit | |
| on: | |
| schedule: | |
| - cron: "30 7 * * *" | |
| workflow_dispatch: | |
| concurrency: | |
| group: agent-provider-model-sync | |
| cancel-in-progress: false | |
| permissions: | |
| contents: read | |
| issues: write | |
| jobs: | |
| sync: | |
| runs-on: ubuntu-latest | |
| env: | |
| # Route Claude Code through the Braintrust gateway. The gateway | |
| # authenticates with a Braintrust API key (BRAINTRUST_AGENT_SPEND_API_KEY, | |
| # passed as anthropic_api_key below) and calls the provider on our behalf. | |
| # https://www.braintrust.dev/docs/deploy/gateway | |
| ANTHROPIC_BASE_URL: https://gateway.braintrust.dev | |
| # Attribute gateway spend to the automations-spend-control project. | |
| ANTHROPIC_CUSTOM_HEADERS: "x-bt-project-name: automations-spend-control" | |
| steps: | |
| - name: Checkout repository | |
| uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 | |
| with: | |
| fetch-depth: 0 | |
| - name: Attempt provider-direct sync with Claude Code | |
| # The audit covers ~15 providers (model list + pricing per provider) with | |
| # --max-turns 200. At the other Claude steps' rate (~80 turns / 15 min) | |
| # that needs ~35-40 min, so a 25-min cap killed the step mid-audit on | |
| # essentially every run. Give the turn budget room to finish. | |
| timeout-minutes: 45 | |
| uses: anthropics/claude-code-action@fbda2eb1bdc90d319b8d853f5deb53bca199a7c1 # v1.0.140 | |
| with: | |
| anthropic_api_key: ${{ secrets.BRAINTRUST_AGENT_SPEND_API_KEY }} | |
| github_token: ${{ github.token }} | |
| display_report: "true" | |
| claude_args: | | |
| --model claude-opus-4-6 | |
| --max-turns 200 | |
| --allowedTools "Read,Glob,Grep,LS,WebSearch,WebFetch,mcp__github__get_issue,mcp__github__get_issue_comments,mcp__github__search_issues,mcp__github__list_issues,mcp__github__create_issue" | |
| --disallowedTools "Bash,Edit,MultiEdit,Write,Replace,NotebookEditCell,mcp__github__create_issue_comment,mcp__github__update_issue,mcp__github__create_pr,mcp__github__create_or_update_file,mcp__github__delete_file,mcp__github_file_ops__commit_files,mcp__github_file_ops__delete_files" | |
| prompt: | | |
| # Goal | |
| Find gaps in the Braintrust proxy model catalog by comparing it to official provider sources directly. This is an issue-finding workflow — do not edit files, create commits, or open pull requests. | |
| # Files to inspect first | |
| - `packages/proxy/scripts/sync_models.ts` | |
| - `packages/proxy/schema/model_list.json` | |
| - `packages/proxy/schema/index.ts` | |
| - `packages/proxy/schema/models.ts` | |
| - `.github/workflows/sync-models.yaml` | |
| - `.github/workflows/agent-provider-model-sync.yaml` | |
| # Current provider inventory | |
| Treat these URLs as starting points — re-check the current official source if a page has redirected or been reorganized. | |
| - `anthropic` — models and pricing: `https://docs.anthropic.com/en/docs/about-claude/models/all-models` | |
| - `azure` — models: `https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models` ; pricing: `https://azure.microsoft.com/en-us/pricing/details/azure-openai/` | |
| - `baseten` — models: `https://www.baseten.co/library/` (Model APIs library; per-model pages live at `https://www.baseten.co/library/<id>/`) ; pricing: the per-model library page, else `https://www.baseten.co/pricing/` | |
| - `bedrock` — models: `https://docs.aws.amazon.com/en_us/bedrock/latest/userguide/foundation-models-reference.html` ; pricing: `https://aws.amazon.com/bedrock/pricing/` | |
| - `cerebras` — models: `https://inference-docs.cerebras.ai/models/overview` ; pricing: `https://inference-docs.cerebras.ai/support/pricing` | |
| - `databricks` — models: `https://docs.databricks.com/aws/en/machine-learning/model-serving/foundation-model-overview` | |
| - `fireworks` — models: `https://docs.fireworks.ai/models/overview` and `https://fireworks.ai/models` (catalog listing; per-model pages live at `https://fireworks.ai/models/<owner>/<id>`) ; pricing: `https://fireworks.ai/pricing` | |
| - `google` — models: `https://ai.google.dev/gemini-api/docs/models/gemini` ; pricing: `https://ai.google.dev/pricing` | |
| - `groq` — models and pricing: `https://console.groq.com/docs/models` | |
| - `mistral` — models: `https://docs.mistral.ai/getting-started/models` | |
| - `openai` — models: `https://platform.openai.com/docs/models/compare` ; pricing: `https://platform.openai.com/docs/pricing/` | |
| - `perplexity` — models and pricing: `https://docs.perplexity.ai/docs/agent-api/models` | |
| - `together` — models and pricing: `https://docs.together.ai/docs/serverless-models` | |
| - `vertex` — models: `https://cloud.google.com/vertex-ai/generative-ai/docs/model-garden/available-models` ; pricing: `https://cloud.google.com/vertex-ai/generative-ai/pricing` | |
| - `xAI` — models and pricing: `https://docs.x.ai/developers/models` | |
| # Process | |
| 0. Before any discovery, use `mcp__github__search_issues` to find the 5 most recently opened issues with the `[BOT ISSUE]` title prefix. Record their titles. Do not create an issue whose gap duplicates any of these titles. | |
| 1. Read the local schema and model list to understand current naming, ordering, and provider mapping conventions. | |
| 2. For each provider, fetch the current official model list and check for: models missing locally, local models that are stale or deprecated, and missing or drifted pricing or token limits. | |
| 3. Prefer official provider APIs for discovery. Fall back to official documentation. Never use third-party aggregators. | |
| 4. For pricing: use the model page if pricing is stated there; otherwise use the provider pricing page; otherwise skip pricing for that provider. | |
| 5. Do one bounded search for provider families not already in the current inventory. If inconclusive, do not create an issue. | |
| 6. Do not report models that are deprecated, retired, or have an official retirement date within 90 days of today. | |
| 7. Do not report embedding models. Embedding models are not supported in the playground and should not be recommended for `packages/proxy/schema/model_list.json`. | |
| 8. Do not report computer-vision-only models. The catalog only tracks text/chat LLMs (including multimodal LLMs that accept images as input alongside text). Skip any model whose sole purpose is computer vision — image generation (e.g. Imagen, DALL·E, Stable Diffusion, Flux), image editing/inpainting, image embeddings/classification, OCR, object detection, segmentation, video generation (e.g. Veo, Sora, Runway), or speech-only models. If unsure whether a model qualifies as a text/chat LLM, skip it rather than filing an issue. | |
| # Issue priority order | |
| When deciding which gaps to file first, use this order: | |
| 1. **New models** — models that exist on the provider's public inference surface but are absent from `model_list.json` | |
| 2. **Model cost updates** — stale or incorrect pricing or token limits for models already in the catalog | |
| 3. **Model deprecations** — local models that have been officially deprecated or retired by the provider | |
| 4. **New model providers** — provider families not yet represented in the catalog at all | |
| Work through providers in priority order. File higher-priority issues before lower-priority ones. If the run limit is reached, stop rather than filing lower-priority issues ahead of higher-priority ones. | |
| **Provider-specific constraints:** | |
| | Provider | Discovery | Pricing | | |
| |------------|--------------------------------------------------|----------------------------------------------------------------| | |
| | Mistral | Official models index → linked detail pages | Use detail pages; do not infer IDs from the generic pricing page | | |
| | xAI | Public docs only; no auth required | Public docs only | | |
| | Bedrock | Public AWS docs only; no credentials | `aws.amazon.com/bedrock/pricing` | | |
| | Databricks | Official model docs only | Skip if no stable public per-model pricing source found | | |
| # Discovery and pricing rules | |
| - Treat model discovery and pricing discovery as separate tasks. | |
| - Only report a missing model when you have an official model ID from official model docs or an official API surface. | |
| - Only report pricing gaps when you have an official pricing source that maps cleanly to that exact model. | |
| - Do not infer model existence from a pricing page alone, or pricing from a model docs page alone. | |
| - Only treat a model as available from a provider when it appears on that provider's public shared inference surface (serverless, general catalog). Ignore bring-your-own-model flows, custom deployments, template galleries, and self-hosted guides. | |
| - `azure`, `bedrock`, and `vertex` can serve models made by other companies. A model appearing there does not mean the local provider mapping is wrong — only report a mapping gap when official docs clearly show the current mapping is incomplete. | |
| - **Shared model ids span providers — union them.** Some ids are served by more than one provider under the exact same id (e.g. `zai-org/GLM-5.2` and `moonshotai/Kimi-K2.6` are on both `baseten` and `together`). When you find such an id, check EVERY provider in the inventory that serves that exact id and report the FULL set in `available_providers` — never attribute a shared id to only the first provider you happen to find it on. If the id is already in `model_list.json` but a provider that serves it (on that provider's official inference surface) is missing from its `available_providers`, that is itself a fileable gap (a provider-mapping update, priority alongside cost updates): report the id and the complete provider set, and set `kind: "stale_metadata"` in the metadata block (it is an update to an already-present id, not a new model — see the metadata rules below). This gap is distinct from an "add model" issue, so do not suppress it via the duplicate check just because the id already exists or a recent issue added the id for one provider. | |
| - Report the `model_list.json` `available_providers` set only. Do NOT assume `index.ts` (`AvailableEndpointTypes`) must equal it — index.ts is intentionally narrower for some models (e.g. an id pinned to a single provider for pricing), so a difference there is not necessarily a gap. | |
| - For pricing, prefer standard public on-demand pricing. Ignore free tiers, preview promos, enterprise, provisioned throughput, reserved capacity, and batch pricing. If pricing varies by region with no clear default, skip it. If pricing varies by prompt length or modality in a way that doesn't map cleanly to the existing schema, skip it. | |
| # Before filing an issue — verification checklist | |
| All three checks must pass before creating an issue: | |
| 1. **Cross-source**: Confirm the model ID appears in at least two independent official signals (e.g., model listing page + pricing page, or model listing page + changelog). A single docs page is not sufficient. | |
| 2. **Already fixed**: Read `packages/proxy/schema/model_list.json` (and `index.ts`) and confirm the model id / corrected value is not already present, so you do not re-file a gap that has already been addressed. (Do not use `git`/Bash — they are not available to this step.) | |
| 3. **ID format**: Confirm the proposed model ID matches the naming convention used by the same provider in `model_list.json` (e.g., `publishers/google/models/...` for vertex). Do not file an issue for a plausible-but-unverified ID. | |
| Then search for an equivalent open GitHub issue and do not create duplicates. | |
| # Constraints | |
| - Do not invent values, defaults, or placeholder models. | |
| - If a provider cannot be queried reliably in this run, skip it. | |
| - If a source is ambiguous about a rename versus a new model, do not create an issue unless the gap itself is still concrete. | |
| - Create at most 10 issues per run; at most one per distinct gap. | |
| - Each issue must target exactly one provider and fit exactly one shape: | |
| 1. a few missing models for one provider | |
| 2. adding one provider to a few existing local models | |
| 3. one new provider-family support gap | |
| - Do not mix missing-model additions with provider-mapping updates in the same issue. | |
| - Do not open umbrella issues. For model-focused issues, include at most 3 model IDs — if you find more, pick the highest-confidence subset. | |
| # Issue requirements | |
| If you create an issue: | |
| - Title: `[BOT ISSUE] ` prefix, naming the single provider and specific model(s). | |
| - Body must include: | |
| - The exact gap (missing model, missing pricing, stale metadata, missing provider-family support, or provider-mapping gap) and why it is high confidence | |
| - Official source URLs used and local files inspected | |
| - Every verified field that would help the downstream `fix-missing-model-bot-issues` job write the catalog entry directly — the issue should read like a proposed update to `packages/proxy/schema/model_list.json` | |
| - A short verification note linking each official source URL to the fields it verifies; call out fields that were not published rather than guessing | |
| - Include a machine-readable metadata block immediately before the closing summary: | |
| `<!-- fix-bot-issue-metadata -->` | |
| followed by: | |
| ```json | |
| { | |
| "kind": "missing_model", | |
| "provider": "vertex", | |
| "models": ["publishers/google/models/gemini-3.1-pro-preview"], | |
| "status": "active", | |
| "model_specs": { | |
| "publishers/google/models/gemini-3.1-pro-preview": { | |
| "format": "google", | |
| "flavor": "chat", | |
| "available_providers": ["vertex"], | |
| "locations": ["global"], | |
| "max_input_tokens": 1048576, | |
| "max_output_tokens": 65536 | |
| } | |
| }, | |
| "source_urls": ["https://example.com"] | |
| } | |
| ``` | |
| - Set `kind` correctly — this is required for the fix to apply. Use `"missing_model"` ONLY when the model id is not yet in `model_list.json`. For ANY change to an id that is already present — provider-mapping gaps (adding a provider to `available_providers`), pricing or token-limit corrections, deprecations — set `kind: "stale_metadata"`. The downstream fixer treats absent/`missing_model` issues as additions and CLOSES them without changes when the named models already exist (`fix_bot_issue.ts`), so a provider-mapping or metadata gap filed as `missing_model` is silently dropped. For `stale_metadata`, the `model_specs` you provide must be the full intended entry (e.g. the complete `available_providers` set), since the fixer updates the existing entry from it. | |
| - For a single-model issue you may use `model_spec` or `model_specs`. For multi-model issues, use `model_specs` with one fully populated spec per model ID. | |
| - Populate specs with exact token limits, pricing fields, verified `displayName`, provider list, deprecation metadata, and location info from official sources. Do not guess. | |
| - Include `parent` only when official sources explicitly document a relationship to a stable base alias and that parent model id exists in `model_list.json`. Recognized patterns: dated snapshot → stable alias (e.g. `claude-3-5-sonnet-20241022` → `claude-3-5-sonnet-latest`, `gpt-4o-2024-11-20` → `gpt-4o`), `@NNN`/`-vN` versioned snapshots (e.g. Vertex `@001`), location-scoped prefixes (`global.`, `us.`, `eu.`, `apac.`), and provider-documented tier variants (e.g. `:free`, `:nitro` on Together AI). Leave unset for base models or when the parent id is not in the local catalog. | |
| - Include replacement relationships only when officially published. | |
| # Output expectations | |
| - Do not edit files, create commits, or open pull requests. | |
| - If there are no high-confidence non-duplicate gaps, do nothing. |