Skip to content

Add Opus 5 and Kimi K3 benchmark scores - #26

Merged
qinxuye merged 2 commits into
mainfrom
agent/add-opus-5-benchmarks
Jul 26, 2026
Merged

Add Opus 5 and Kimi K3 benchmark scores#26
qinxuye merged 2 commits into
mainfrom
agent/add-opus-5-benchmarks

Conversation

@qinxuye

@qinxuye qinxuye commented Jul 26, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add a bundled Claude Opus 5 profile with official model ID, limits, pricing, and source links
  • record Opus 5 GPQA Diamond 93.2 from Artificial Analysis and LiveCodeBench 89.0 from Vals
  • fill Kimi K3's previously missing LiveCodeBench score with the new Vals result, 87.2
  • add registry regression coverage for both models and both capability benchmarks

Why

The production capability axis uses gpqa_diamond and livecodebench. Opus 5 is now available, and Vals has published LiveCodeBench results for both Opus 5 and Kimi K3, so both profiles can provide the complete benchmark pair.

Validation

  • XROUTER_LLM_ENTERPRISE_DISABLE_AUTOPATCH=1 PYTHONPATH=src python -m pytest -q — 81 passed, 21 skipped
  • PYTHONPATH=src python -m compileall -q src tests
  • git diff --check
  • repository secret scan

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds a new model configuration for claude-opus-5 and updates the existing kimi-k3 model configuration with new source URLs and benchmark scores. It also updates the corresponding unit tests to verify the loaded profiles. The review feedback points out an inconsistency in the source_quality field for claude-opus-5, which is marked as official despite relying on third-party benchmark evaluations, and suggests changing it to third_party to align with other models like kimi-k3.

Comment thread src/xrouter_llm/resources/config/models/claude-opus-5.yaml Outdated
@qinxuye
qinxuye merged commit 29b7761 into main Jul 26, 2026
4 checks passed
@qinxuye
qinxuye deleted the agent/add-opus-5-benchmarks branch July 26, 2026 04:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant