You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
test: add claude plugin eval suites for quality and dev-standards #25
quality and dev-standards have no claude plugin eval suites, so CI (.github/workflows/plugin-evals.yml, from #18) skips them and a PR changing their skills shows nothing. github-tools and cad-tools have suites.
Scope
Cases under plugins/quality-tools/evals/<skill>/<case>/ and plugins/dev-standards/evals/<skill>/<case>/, in the layout github-tools and cad-tools use (prompt.md + graders/).
Side-effect free: dry runs with read-only tools (allowed_tools: [Skill, Read]), no shell, so they run on a CI runner.
Each case has a free tool_used: Skill check, plus regex graders where the expected behaviour is a literal command or flag, and an llm grader only where the behaviour needs judging.
Prioritise skills whose failure costs the most: preflight, create-pr, full-pr-review, act-on-pr, make-ci-green-again, and dev-standards' git-workflow.
CI uses Sonnet 5.5 as both agent and judge; verify new cases locally with --model claude-sonnet-5-5 --judge-model claude-sonnet-5-5 at the default 3 runs.
Activity