[jimisola]

What & Why

Closes #18. Implements the workflow from that issue. On PRs that touch plugins/*/skills/** or plugins/*/evals/**, it runs claude plugin eval for each changed plugin that has evals; a change to the workflow itself runs every suite. workflow_dispatch runs chosen (or all) plugins, optionally with the no-plugin baseline.

• Auth: workload identity federation; no ANTHROPIC_API_KEY exists anywhere, and the step unsets it in case. Each GitHub OIDC token can be exchanged only once (check_jti), and one eval fires parallel judge calls, so the job exchanges a fresh token per eval case and runs that case on the minted access token (ANTHROPIC_AUTH_TOKEN, masked).
• Inert until set up: the job is skipped while the ANTHROPIC_FEDERATION_RULE_ID repository variable is empty, and for fork PRs (no OIDC token).
• Pinned: Claude Code 2.1.296, claude-sonnet-5-5 under test and as judge (Haiku failed correct long answers).
• Gates: --threshold 0.8, --max-cost-usd 1 per case, --ablation none unless the baseline is asked for.
• Cost tracking: the job summary shows total cost and duration, the Claude Code version, models and per-case scores; JSON results are uploaded as an artifact.

Setup (done)

• Console: issuer github-actions, rule claude-plugins-ci with subject prefix repo:jimisola@1577689/claude-plugins@1305641786:* (this repo uses GitHub's immutable subject), owner/repository claims, and a condition allowing pull_request, or workflow_dispatch on main. Workspace ci-evals-personal with a spend limit.
• Repository variables: ANTHROPIC_FEDERATION_RULE_ID, ANTHROPIC_ORGANIZATION_ID, ANTHROPIC_SERVICE_ACCOUNT_ID, ANTHROPIC_WORKSPACE_ID.

Test Plan

• actionlint clean (including shellcheck)
• Real runs against federation: no auth failures since the per-case exchange; a full github-tools run is ~12 min and ~$2.50
• Remaining eval failures on main's skills are fixed in fix(github-tools): merge-ready asks for scope in the reply when AskUserQuestion is unavailable #23; re-run here after it merges