Cross-family: cost-optimized cadence, judge experiment, and PR profiles - #1002
Conversation
Wire the IMPACT-ANALYSIS.md §10 cross-family (multi-model) executor/judge dimension into the existing Vally eval workflows as a strictly OPT-IN feature. Every current trigger keeps byte-for-byte behavior: the default `single` profile performs no expansion, so entries stay model-less and the runner falls back to the experiment file's existing executor/judge. evaluation.yml (discover job): - New `matrix_profile` workflow_dispatch choice input (single|default|full|newer). - Resolve a profile from the event: workflow_dispatch input, `/evaluate` --full-matrix|--newer|--cross-family flags, or a scheduled cadence day. Event bodies arrive via env and are only regex-matched, never executed. - Expand each plugin/shard entry across the profile's executor models, attaching a cross-family primary judge (judge is never the same model as the executor) and, on scheduled dual-judge days, an optional within-family second judge. - Extend the strict per-entry allowlist to model/judge/judge2. evaluation-run.yml (reusable runner): - Validate the optional ENTRY_MODEL/ENTRY_JUDGE/ENTRY_JUDGE2 against the same allowlist; prefer the injected executor/judge when present. - Pass --model/--judge-model to gen-experiment.mjs, with a sed fallback that patches the static experiment file only when a cross-family model is active. - After the primary judge, optionally re-score the SAME transcripts with the second judge (cheap: re-runs only `vally compare`, not the executor) into a separate `vally-crossjudge/` root, uploaded as a distinct `vally-crossjudge-*` artifact that downstream `vally-results-*` consumers do not glob. gen-experiment.mjs: - Add --model/--judge-model that rewrite the copied base experiment's overrides.model/judge_model lines, with model-id shape validation. Validated: actionlint clean on both workflows; PowerShell parse of the new discover block; gen-experiment functional injection; bash substitution + sed fallback; and an assertion that judge != executor holds for every route. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5fb19734-0288-416c-8c13-07b1a49dfde0
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5fb19734-0288-416c-8c13-07b1a49dfde0
Split the 18 runnable dotnet-test evals into three measured-duration shards so cross-family model legs stay below the three-hour GitHub Actions timeout. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5fb19734-0288-416c-8c13-07b1a49dfde0
…R mode
Repoint the default executor set to {sonnet-4.6, gpt-5.5} and add tiered
scheduled cadence so every model family runs at a regular-but-cost-optimized
interval:
- default (sonnet-4.6, gpt-5.5): Mon/Wed/Fri + every authorized PR
- mid (haiku-4.5, mai, gpt-5.3-codex): Tue/Sat
- opus (opus-4.8): Thu
- newer (gpt-5.6-sol, opus-5, sonnet-5): Sun
Heavy tiers replace the defaults on their day rather than co-running them.
PR gates run the default profile; an opt-in `full` profile (defaults + mid +
opus) is available via `/evaluate --full` with a warning that it is expensive.
Add an Opus-4.8 vs Sonnet-4.6 judge experiment: on scheduled runs the
GPT-executor legs carry a second Sonnet-4.6 judge, and a new step pairs the two
judges' verdicts into an isolated judge-comparison.json (kept out of the
dashboard components manifest).
Make the scheduled skip-guard per-profile by tagging scheduled runs
"schedule: <profile>" via run-name and matching on display_title, so one
profile no longer suppresses the others on an unchanged commit.
Also fix the token-usage and benchmark aggregation to group each plugin's
results by its own executor model before merging, so cross-family runs no
longer mislabel per-model rows.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e23e5efe-713f-4898-a158-8970f70fdcaf
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e23e5efe-713f-4898-a158-8970f70fdcaf
The single (one-model, self-judged claude-opus-4.6) profile was the only non-cross-family path left. Every trigger now resolves to a cross-family profile: manual dispatch and the PR gate default to the two DEFAULT models (sonnet-4.6 + gpt-5.5), heavier tiers stay opt-in. Drops the dropdown option, the --single comment flag, the single special-cases, and the now unused EVAL_PR_NUMBER env. /evaluate --single degrades gracefully to default. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e23e5efe-713f-4898-a158-8970f70fdcaf
judge-comparison.json now records the same stat set for both the primary (opus-4.8) and second (sonnet-4.6) judge per (model, skill): passed, regressed, conclusive, underpowered, meanScore, winRate, trialCount -- not just the pass/regressed booleans. Clarifies in the step comment that this data is experiment-only: scheduled dual-judge cadence only (no-op on PRs), excluded from components.json, and never posted to any PR comment. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e23e5efe-713f-4898-a158-8970f70fdcaf
The opus48 profile was reachable only via the Thursday schedule and the workflow_dispatch dropdown. Add a --opus48 comment flag so it matches the other opt-in tiers (--mid, --newer, --full) and can be requested from a PR. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e23e5efe-713f-4898-a158-8970f70fdcaf
…ugin Multi-model review (Opus-4.8, GPT-5.6, rubber-duck) surfaced two agreed, in-scope issues: - The /evaluate flag regexes matched anywhere as substrings, so a malformed near-miss (--fuller, --opus480, --midnight, --newer-model) silently selected an expensive profile. Anchor each flag on token boundaries ((?:^|\\s) before, (?=\\s|\$) after) so near-misses fall through to default; \\s boundaries still recognize a flag anywhere in a multi-line review body. - judge-comparison pairing keyed verdicts on model|skillName with no plugin qualifier, so two plugins sharing a skill name on the same executor collided and one was dropped from judge-comparison.json and the agreement denominator. Derive the plugin from the results.json path (<plugin>/<skill>/results.json) and key on plugin|model|skill; also emit plugin in each comparison record. actionlint clean; both fixes covered by parser/keying simulations. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e23e5efe-713f-4898-a158-8970f70fdcaf
There was a problem hiding this comment.
Pull request overview
Introduces a cross-family evaluation matrix for scheduled and PR-gated runs, adds an optional dual-judge (“crossjudge”) experiment path, and reworks aggregation so dashboard/token usage data is grouped by executor model rather than collapsed across models.
Changes:
- Adds
matrix_profileselection and scheduled per-day model cadence, including per-profile skip-guarding via scheduled run-name tags. - Fans evaluation matrices across executor models with explicit judge/judge2 routing; adds a scheduled dual-judge re-score path and emits isolated
judge-comparison.json. - Adds
executionShard:tags to severaldotnet-testevals to support sharded execution bucketing.
Show a summary per file
| File | Description |
|---|---|
| tests/dotnet-test/writing-mstest-tests/eval.yaml | Adds executionShard tag to influence workflow sharding/bucketing. |
| tests/dotnet-test/test-smell-detection/eval.yaml | Adds executionShard tag to influence workflow sharding/bucketing. |
| tests/dotnet-test/test-gap-analysis/eval.yaml | Adds executionShard tag to influence workflow sharding/bucketing. |
| tests/dotnet-test/test-anti-patterns/eval.yaml | Adds executionShard tag to influence workflow sharding/bucketing. |
| tests/dotnet-test/run-tests/eval.yaml | Adds executionShard tag to influence workflow sharding/bucketing. |
| tests/dotnet-test/platform-detection/eval.yaml | Adds executionShard tag to influence workflow sharding/bucketing. |
| tests/dotnet-test/mtp-hot-reload/eval.yaml | Adds executionShard tag to influence workflow sharding/bucketing. |
| tests/dotnet-test/migrate-static-to-wrapper/eval.yaml | Adds executionShard tag to influence workflow sharding/bucketing. |
| tests/dotnet-test/grade-tests/eval.yaml | Adds executionShard tag to influence workflow sharding/bucketing. |
| tests/dotnet-test/find-untested-sources/eval.yaml | Adds executionShard tag to influence workflow sharding/bucketing. |
| tests/dotnet-test/filter-syntax/eval.yaml | Adds executionShard tag to influence workflow sharding/bucketing. |
| tests/dotnet-test/detect-static-dependencies/eval.yaml | Adds executionShard tag to influence workflow sharding/bucketing. |
| tests/dotnet-test/code-testing-agent/eval.yaml | Adds executionShard tag to influence workflow sharding/bucketing. |
| eng/vally-adapter/gen-experiment.mjs | Adds CLI overrides for executor and judge model IDs when generating per-plugin experiment YAML. |
| .github/workflows/evaluation.yml | Adds per-day scheduled cadence + PR gate profile selection, cross-family matrix expansion, per-model aggregation, and judge-comparison output generation. |
| .github/workflows/evaluation-run.yml | Threads model/judge/judge2 through matrix entries, injects them into experiment generation, and adds optional second-judge re-score + artifact upload. |
Review details
Tip
Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- Files reviewed: 16/16 changed files
- Comments generated: 3
- Review effort level: Lite
|
👋 @AbhitejJohn — this PR has 3 unresolved review thread(s),merge conflict. When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the |
Cross-family evaluation interleaves several executor models in one plugin's benchmark history. The summary cards averaged the last 50 runs without grouping by model and the 'Model' card showed only the last-appended model, while each quality trend drew one line per variant across all models - blending families into a single average and a single zig-zag line. Group the recent window by model into a per-model quality table (Skilled / Plugin / Vanilla + deltas, one row per model) and redraw each trend with one line per model (colour = model, dash = variant, spanGaps:false) so a line never bridges two families. Per-point issue markers and tooltips are preserved. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e23e5efe-713f-4898-a158-8970f70fdcaf
There was a problem hiding this comment.
Review details
Suppressed comments (2)
.github/workflows/evaluation.yml:1131
- Minor typo in log output: “entr(ies)” reads like a placeholder and is hard to scan in logs. Prefer “entries”.
Write-Host "Cross-family profile '$matrixProfile' (dualJudge=$dualJudge): expanded to $($entries.Count) entr(ies) across $($models.Count) model(s)"
.github/workflows/evaluation-run.yml:164
- The comment says ENTRY_MODEL/ENTRY_JUDGE/ENTRY_JUDGE2 are “empty in the single-model default”, but the discover job now expands every profile into entries that always include
modelandjudge(andjudge2may be empty). This makes the comment misleading for future maintainers; update it to reflect that these fields are populated for cross-family matrix runs and only empty for any legacy one-model entries.
# Cross-family executor/judge fields (IMPACT-ANALYSIS.md §10) are
# OPTIONAL: they are empty in the single-model default and non-empty
# only when the discover job fans the matrix across models. When
# present they override the executor/judge below, so hold them to the
# same strict allowlist as names. Empty = "use the experiment-file
- Files reviewed: 17/17 changed files
- Comments generated: 0 new
- Review effort level: Lite
The summary container is a CSS grid, so the table card's flex:1 1 100% was ignored and the card was squeezed into one ~200px grid track, clipping the Delta Isolated / Delta Plugin columns. Add grid-column:1/-1 so the table spans the whole row. Caught by a Playwright render check of a 3-model fixture. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e23e5efe-713f-4898-a158-8970f70fdcaf
There was a problem hiding this comment.
Review details
Suppressed comments (1)
.github/workflows/evaluation-run.yml:532
RESULTS_DIR2uses backslash-escaped slashes in the replacement part of bash pattern substitution. In${var/pat/repl}, backslashes inreplare preserved (except for escaping&and\), so this will likely produce a path containing literal\characters (e.g..../TestResults\ /vally-crossjudge\ /...) and cause the secondary-judge re-score to write to a non-existent directory (and/or overwrite the primary output). Build the crossjudge output path without embedding backslashes in the replacement.
if [ -n "$JUDGE2_MODEL" ] && [ "$JUDGE2_MODEL" != "$JUDGE_MODEL" ]; then
RESULTS_DIR2="${RESULTS_DIR/\/vally\//\/vally-crossjudge\/}"
echo "Re-scoring $PLUGIN transcripts with secondary judge $JUDGE2_MODEL -> $RESULTS_DIR2"
- Files reviewed: 17/17 changed files
- Comments generated: 0 new
- Review effort level: Lite
The per-model view can get busy once several executor models share one plugin's history. Add a checkbox filter bar (all models enabled by default); toggling a model re-renders the summary table and every quality/efficiency chart for just the selected models. Model colours stay canonical (bound to full history) so hiding a model never recolours the others, and at least one model stays enabled so the view is never empty. Chart instances are now tracked and destroyed on each re-render to avoid canvas-reuse leaks. Verified with a Playwright render check: filter bar shows one checkbox per model all enabled; unchecking a model drops it from the summary and every chart while the rest keep their colours; re-checking restores it. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e23e5efe-713f-4898-a158-8970f70fdcaf
There was a problem hiding this comment.
Review details
Suppressed comments (2)
.github/workflows/evaluation.yml:1939
- This appends new judge-pair rows on every re-run (including re-running the same scheduled workflow for the same commit), which will double-count pairs and skew
agreementRate. Consider de-duplicating by a stable key (commit+plugin+model+skill+judge pair) before computing totals.
$all = @($existing) + @($new)
# Keep only entries inside the retention window.
$cutoff = (Get-Date).ToUniversalTime().AddDays(-1 * [int]$env:DASHBOARD_RETENTION_DAYS)
$all = @($all | Where-Object {
try { [datetime]::Parse($_.date).ToUniversalTime() -ge $cutoff } catch { $true }
eng/dashboard/dashboard.js:223
allModelsis derived only from Quality history. If a plugin ever has Efficiency entries without Quality (or if Quality is temporarily empty due to partial backfill), the model filter will default to an empty set anddraw()will filter out all entries. Build the canonical model list from the union of Quality + Efficiency histories so filtering and colours still work in that case.
const allModels = orderedModels(allQualityEntries);
activeModelColors = buildModelColorMap(allModels);
- Files reviewed: 17/17 changed files
- Comments generated: 0 new
- Review effort level: Lite
The model->colour map lived in a single module-level �ctiveModelColors that every enderPlugin overwrote. Because draw() now runs lazily on filter toggles (after a plugin was first rendered), switching plugin tabs and then toggling a filter on an earlier plugin recoloured its charts and summary dots using the later plugin's map. Capture the map in a plugin-scoped pluginModelColors const and restore the module global from it at the top of every draw(), so each plugin always renders with its own canonical colours. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e23e5efe-713f-4898-a158-8970f70fdcaf
📊 Skill Evaluation Results23 skill(s) evaluated — ✅ 7 improved, ❌ 6 no credible change, 🔻 0 regressed.
A skill passes only on a credible net win over baseline: more wins than losses, by an exact one-sided sign test at
ℹ️ Column legend
|
| Scenario | Net win | Δ Pref | Trials (W/T/L) |
|---|---|---|---|
| ▼ Diagnose build failures from binlog only (no source files) | -100.0% | -40.0% | 0/0/1 |
⚠️ build-parallelism — details
Reason: Net win +100.0% (1W/0T/0L over 1 trial(s), sign test p=0.500), mean preference +40.0% — underpowered (1 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs
| Scenario | Net win | Δ Pref | Trials (W/T/L) |
|---|---|---|---|
| ▲ Analyze build parallelism bottlenecks | +100.0% | +40.0% | 1/0/0 |
⚠️ build-perf-diagnostics — details
Reason: Net win +0.0% (0W/1T/0L over 1 trial(s), sign test p=1.000), mean preference +0.0% — underpowered (1 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs
| Scenario | Net win | Δ Pref | Trials (W/T/L) |
|---|---|---|---|
| = Diagnose slow build for a small project | +0.0% | +0.0% | 0/1/0 |
⚠️ check-bin-obj-clash — details
Reason: Net win +0.0% (0W/1T/0L over 1 trial(s), sign test p=1.000), mean preference +0.0% — underpowered (1 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs
| Scenario | Net win | Δ Pref | Trials (W/T/L) |
|---|---|---|---|
| = Diagnose bin/obj output path clashes | +0.0% | +0.0% | 0/1/0 |
❌ code-testing-agent — details
Reason: Net win -10.0% (1W/7T/2L over 10 trial(s), sign test p=0.500), mean preference -4.0% — no improvement
| Scenario | Net win | Δ Pref | Trials (W/T/L) |
|---|---|---|---|
| = Diagnose failing tests without generating a new suite | +0.0% | +0.0% | 0/2/0 |
| = Does not revert a gutted-looking workspace (workspace integrity) | +0.0% | +0.0% | 0/2/0 |
| = Extend an existing suite to the untested method only | +0.0% | +0.0% | 0/2/0 |
| ▼ Generate Vitest tests for the shopping-cart library (TypeScript polyglot) | -50.0% | -20.0% | 0/1/1 |
| = Keep a single-function request proportional | +0.0% | +0.0% | 1/0/1 |
⚠️ directory-build-organization — details
Reason: Net win +100.0% (1W/0T/0L over 1 trial(s), sign test p=0.500), mean preference +40.0% — underpowered (1 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs
| Scenario | Net win | Δ Pref | Trials (W/T/L) |
|---|---|---|---|
| ▲ Organize build infrastructure for a multi-project repo | +100.0% | +40.0% | 1/0/0 |
⚠️ extension-points — details
Reason: Net win +0.0% (1W/1T/1L over 3 trial(s), sign test p=0.750), mean preference +0.0% — underpowered (3 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs
| Scenario | Net win | Δ Pref | Trials (W/T/L) |
|---|---|---|---|
| = Diagnose NuGet package and repo extension conflicts | +0.0% | +0.0% | 0/1/0 |
| ▲ Diagnose build extension point failures | +100.0% | +40.0% | 1/0/0 |
| ▼ Fix extension point anti-patterns | -100.0% | -40.0% | 0/0/1 |
❌ filter-syntax — details
Reason: Net win +20.0% (2W/2T/1L over 5 trial(s), sign test p=0.500), mean preference +8.0% — not credible — 2 of 5 trial(s) tied, leaving only 3 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)
| Scenario | Net win | Δ Pref | Trials (W/T/L) |
|---|---|---|---|
| ▼ Filter a TUnit suite down to one class and one property value | -100.0% | -40.0% | 0/0/1 |
| ▲ Filter xUnit v3 tests that do not accept the generic filter expression | +100.0% | +40.0% | 1/0/0 |
| ▲ Pass a filter to a Microsoft.Testing.Platform project on the .NET 9 SDK | +100.0% | +40.0% | 1/0/0 |
| = Select one category and exclude another on a VSTest project | +0.0% | +0.0% | 0/1/0 |
| = Translate CI filter expressions after moving to xUnit v3 | +0.0% | +0.0% | 0/1/0 |
⚠️ item-management — details
Reason: Net win +33.3% (1W/2T/0L over 3 trial(s), sign test p=0.500), mean preference +13.3% — underpowered (3 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs
| Scenario | Net win | Δ Pref | Trials (W/T/L) |
|---|---|---|---|
| = Diagnose cascading item and batching bugs in code generation pipeline | +0.0% | +0.0% | 0/1/0 |
| ▲ Diagnose item group and batching issues | +100.0% | +40.0% | 1/0/0 |
| = Fix item management anti-patterns | +0.0% | +0.0% | 0/1/0 |
❌ migrate-static-to-wrapper — details
Reason: Net win +50.0% (3W/3T/0L over 6 trial(s), sign test p=0.125), mean preference +20.0% — not credible — 3 of 6 trial(s) tied, leaving only 3 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)
| Scenario | Net win | Δ Pref | Trials (W/T/L) |
|---|---|---|---|
| = Add the required using directive and update tests with a test double | +0.0% | +0.0% | 0/1/0 |
| ▲ Decline migration when wrapper does not exist yet | +100.0% | +40.0% | 1/0/0 |
| ▲ Migrate DateTime.UtcNow to TimeProvider in a service class | +100.0% | +40.0% | 1/0/0 |
| = Migrate a static helper class without breaking its callers | +0.0% | +0.0% | 0/1/0 |
| = Migrate only in scoped files, leaving others untouched | +0.0% | +0.0% | 0/1/0 |
| ▲ Preserve DateTimeKind when migrating to TimeProvider | +100.0% | +40.0% | 1/0/0 |
⚠️ msbuild-antipatterns — details
Reason: Net win -25.0% (0W/3T/1L over 4 trial(s), sign test p=0.500), mean preference -10.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs
| Scenario | Net win | Δ Pref | Trials (W/T/L) |
|---|---|---|---|
| = Add a module to an F# project | +0.0% | +0.0% | 0/1/0 |
| = Add a signature file to define public API | +0.0% | +0.0% | 0/1/0 |
| ▼ Fix broken file order causing FS0039 | -100.0% | -40.0% | 0/0/1 |
| = Review MSBuild files for anti-patterns and style issues | +0.0% | +0.0% | 0/1/0 |
❌ mtp-hot-reload — details
Reason: Net win +71.4% (6W/0T/1L over 7 trial(s), sign test p=0.063), mean preference +71.4% — not credible (sign test p=0.063 > 0.05)
| Scenario | Net win | Δ Pref | Trials (W/T/L) |
|---|---|---|---|
| ▲ Enable hot reload when package already installed | +100.0% | +100.0% | 1/0/0 |
| ▼ Negative: VSTest project cannot use MTP hot reload | -100.0% | -40.0% | 0/0/1 |
| ▲ Run specific failing test with hot reload filter | +100.0% | +100.0% | 1/0/0 |
| ▲ Suggest hot reload for failing test in MTP project (SDK 10) | +100.0% | +100.0% | 1/0/0 |
| ▲ Suggest hot reload for failing test in MTP project (SDK 9) | +100.0% | +100.0% | 1/0/0 |
| ▲ Suggest launchSettings.json configuration for hot reload | +100.0% | +100.0% | 1/0/0 |
| ▲ Use dotnet run not dotnet test for hot reload | +100.0% | +40.0% | 1/0/0 |
❌ platform-detection — details
Reason: Net win -20.0% (0W/4T/1L over 5 trial(s), sign test p=0.500), mean preference -20.0% — no improvement
| Scenario | Net win | Δ Pref | Trials (W/T/L) |
|---|---|---|---|
| ▼ MTP signal set in Directory.Build.props rather than the project file | -100.0% | -100.0% | 0/0/1 |
| = Microsoft.NET.Test.Sdk alongside an MTP runner property | +0.0% | +0.0% | 0/1/0 |
| = TUnit project is MTP-only | +0.0% | +0.0% | 0/1/0 |
| = global.json opts a plain xUnit v3 project into MTP on SDK 10 | +0.0% | +0.0% | 0/1/0 |
| = global.json runner outranks TestingPlatformDotnetTestSupport on SDK 10 | +0.0% | +0.0% | 0/1/0 |
⚠️ property-patterns — details
Reason: Net win +33.3% (1W/2T/0L over 3 trial(s), sign test p=0.500), mean preference +13.3% — underpowered (3 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs
| Scenario | Net win | Δ Pref | Trials (W/T/L) |
|---|---|---|---|
| ▲ Diagnose multi-level property hierarchy bugs | +100.0% | +40.0% | 1/0/0 |
| = Diagnose shared build property issues | +0.0% | +0.0% | 0/1/0 |
| = Fix shared property configuration | +0.0% | +0.0% | 0/1/0 |
⚠️ target-authoring — details
Reason: Net win +33.3% (2W/0T/1L over 3 trial(s), sign test p=0.500), mean preference +13.3% — underpowered (3 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs
| Scenario | Net win | Δ Pref | Trials (W/T/L) |
|---|---|---|---|
| ▼ Diagnose broken SDK target chain across files | -100.0% | -40.0% | 0/0/1 |
| ▲ Diagnose custom target build regression | +100.0% | +40.0% | 1/0/0 |
| ▲ Fix custom target anti-patterns | +100.0% | +40.0% | 1/0/0 |
❌ test-anti-patterns — details
Reason: Net win +12.5% (2W/5T/1L over 8 trial(s), sign test p=0.500), mean preference +5.0% — not credible — 5 of 8 trial(s) tied, leaving only 3 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)
| Scenario | Net win | Δ Pref | Trials (W/T/L) |
|---|---|---|---|
| = Audit a pytest suite using Python-specific anti-pattern markers | +0.0% | +0.0% | 0/1/0 |
| ▲ Detect coverage-touching pattern across a service facade | +100.0% | +40.0% | 1/0/0 |
| = Detect duplicated tests and magic values | +0.0% | +0.0% | 0/1/0 |
| ▲ Detect flakiness indicators and test coupling | +100.0% | +40.0% | 1/0/0 |
| ▼ Detect mixed severity anti-patterns in repository service tests | -100.0% | -40.0% | 0/0/1 |
| = Detect self-referential assertions in round-trip and identity tests | +0.0% | +0.0% | 0/1/0 |
| = Recognize well-written tests without inventing false positives | +0.0% | +0.0% | 0/1/0 |
| = Separate false-confidence assertions from cosmetic ones | +0.0% | +0.0% | 0/1/0 |
Per-scenario details for 7 skill(s) were omitted to keep this comment under GitHub's 65,536-character limit — open the job's step summary or Full Results for the complete breakdown.
🔍 Full Results - additional metrics and failure investigation steps
To investigate failures, paste this to your AI coding agent:
For PR 1002 in dotnet/skills, download eval artifacts with
gh run download 31542761520 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/1105254fa7c5d952e892dcf81a76a8c8ad5ff632/eng/vally-adapter/InvestigatingResults.md and follow it to analyze the results.json files. Diagnose each failure, suggest fixes to the eval.yaml and skill content, and tell me what to fix first.
▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions
Add the missing b/c executionShard markers from PR #1002 while preserving this PR's finer per-skill shards for its expanded evals. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a4d6df16-f4e4-4dd4-afa9-5623db19b3f4
|
Results are in. Full all-plugins run of both executors (each judged cross-family by
There's an ongoing investigation to understand what these underpowered evals are and how we'd go about fixing them. |
…or GPT arms Non-GPT executor legs (Claude/MAI) now judged by gpt-5.6-terra with no second judge (was gpt-5.6-sol). GPT executor legs keep claude-opus-4.8 as primary but use claude-haiku-4.5 as the dual-judge second judge (was claude-sonnet-4.6). Refreshes the stale judge-experiment comments to match. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e23e5efe-713f-4898-a158-8970f70fdcaf
There was a problem hiding this comment.
Review details
Suppressed comments (1)
.github/workflows/evaluation.yml:1956
- The judge-comparison output metadata labels the experiment as "opus-4.8-vs-sonnet-4.6", but this step (and the PR description) compares claude-opus-4.8 (primary judge) vs claude-haiku-4.5 (second judge). This incorrect label will misidentify the experiment in judge-comparison.json and any downstream analysis.
experiment = "opus-4.8-vs-sonnet-4.6"
- Files reviewed: 27/27 changed files
- Comments generated: 0 new
- Review effort level: Lite
There was a problem hiding this comment.
Review details
Suppressed comments (5)
tests/dotnet-test/writing-mstest-tests/eval.yaml:5
- This file now has two
executionShardkeys (band the existing shard). Duplicate keys are ambiguous in YAML and can break/override parsing depending on loader; this should be a single tag.
name: writing-mstest-tests
executionShard: b
description: Evaluates the dotnet-test/writing-mstest-tests skill
type: capability
executionShard: mstest
tests/dotnet-test/run-tests/eval.yaml:5
- This file now has two
executionShardkeys (cand the existing shard). Duplicate keys are ambiguous in YAML and can break/override parsing depending on loader; this should be a single tag.
name: run-tests
executionShard: c
description: Evaluates the dotnet-test/run-tests skill
type: capability
executionShard: execution
tests/dotnet-test/platform-detection/eval.yaml:5
- This file now has two
executionShardkeys (band the existing shard). Duplicate keys are ambiguous in YAML and can break/override parsing depending on loader; this should be a single tag.
name: platform-detection
executionShard: b
description: Evaluates the dotnet-test/platform-detection skill
type: capability
executionShard: execution
tests/dotnet-test/code-testing-agent/eval.yaml:5
- This file now has two
executionShardkeys (band the existing shard). Duplicate keys are ambiguous in YAML and can break/override parsing depending on loader; this should be a single tag.
name: code-testing-agent
executionShard: b
description: Evaluates the dotnet-test/code-testing-agent skill
type: capability
executionShard: generation
.github/workflows/evaluation.yml:1957
- The judge comparison file is meant to compare Opus-4.8 (primary) vs Haiku-4.5 (secondary), but the emitted
experimentlabel currently saysopus-4.8-vs-sonnet-4.6, which will mislabel/aggregate experiment data downstream.
[pscustomobject]@{
experiment = "opus-4.8-vs-sonnet-4.6"
lastUpdate = $date
- Files reviewed: 18/18 changed files
- Comments generated: 0 new
- Review effort level: Lite
The main merge (257df23) left both the branch's original letter shard (executionShard: b/c) and main's word shard (execution/generation/mstest) in these four eval.yaml files, tripping the eval-quality duplicate-key gate. Drop the stale letter key so each file keeps main's canonical word shard; the active value is unchanged (YAML already kept the last key). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e23e5efe-713f-4898-a158-8970f70fdcaf
There was a problem hiding this comment.
Review details
Suppressed comments (2)
.github/workflows/evaluation.yml:1957
- The judge-comparison output metadata labels the experiment as "opus-4.8-vs-sonnet-4.6", but this step is comparing primary judge claude-opus-4.8 vs secondary judge claude-haiku-4.5 (per the step header + PR description). This incorrect label will mislead downstream analysis of judge agreement.
[pscustomobject]@{
experiment = "opus-4.8-vs-sonnet-4.6"
lastUpdate = $date
eng/dashboard/dashboard.js:621
- Efficiency charts are still rendered as one continuous series per variant, which will connect points across different executor models when a plugin history contains multiple models. This contradicts the PR description (Quality/Efficiency trend charts split by executor model) and can make efficiency trends misleading on cross-family runs. Consider segmenting efficiency charts per model (null gaps / one dataset per model), similar to renderModelSegmentedChart used for quality.
const effChart = new Chart(canvas, {
type: 'line',
data: {
labels,
- Files reviewed: 14/14 changed files
- Comments generated: 0 new
- Review effort level: Lite
github-actions
Bot
added
pr-state/ready-for-eval
There was a problem hiding this comment.
Review details
Suppressed comments (2)
.github/workflows/evaluation.yml:1957
- The judge-comparison JSON metadata labels the experiment as "opus-4.8-vs-sonnet-4.6", but this step is comparing the primary judge (claude-opus-4.8) against the secondary judge (claude-haiku-4.5). This mismatch will make the dataset confusing to interpret downstream.
[pscustomobject]@{
experiment = "opus-4.8-vs-sonnet-4.6"
lastUpdate = $date
.github/workflows/evaluation.yml:1874
- The step header says pairing is done by "(executor model, skill)", but the implementation keys on "$plugin|$model|$skill" to avoid collisions when two plugins share a skill name. The comment should match the actual pairing key to prevent accidental refactors that reintroduce collisions.
# GPT-executor dual-judge comparison (§10.6). Pair each skill's PRIMARY
# verdict (claude-opus-4.8 judge) with its SECOND-judge verdict (claude-haiku-4.5) by
# (executor model, skill) and record the SAME stat set for both judges
- Files reviewed: 14/14 changed files
- Comments generated: 0 new
- Review effort level: Lite
📊 Skill Evaluation Results10 skill(s) evaluated — ✅ 0 improved, ❌ 0 no credible change, 🔻 0 regressed.
A skill passes only on a credible net win over baseline: more wins than losses, by an exact one-sided sign test at
ℹ️ Column legend
|
| Scenario | Net win | Δ Pref | Trials (W/T/L) |
|---|---|---|---|
| = Diagnose build failures from binlog only (no source files) | +0.0% | +0.0% | 0/1/0 |
⚠️ build-parallelism — details
Reason: Net win +100.0% (1W/0T/0L over 1 trial(s), sign test p=0.500), mean preference +40.0% — underpowered (1 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs
| Scenario | Net win | Δ Pref | Trials (W/T/L) |
|---|---|---|---|
| ▲ Analyze build parallelism bottlenecks | +100.0% | +40.0% | 1/0/0 |
⚠️ build-perf-diagnostics — details
Reason: Net win +100.0% (1W/0T/0L over 1 trial(s), sign test p=0.500), mean preference +40.0% — underpowered (1 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs
| Scenario | Net win | Δ Pref | Trials (W/T/L) |
|---|---|---|---|
| ▲ Diagnose slow build for a small project | +100.0% | +40.0% | 1/0/0 |
⚠️ check-bin-obj-clash — details
Reason: Net win +100.0% (1W/0T/0L over 1 trial(s), sign test p=0.500), mean preference +40.0% — underpowered (1 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs
| Scenario | Net win | Δ Pref | Trials (W/T/L) |
|---|---|---|---|
| ▲ Diagnose bin/obj output path clashes | +100.0% | +40.0% | 1/0/0 |
⚠️ directory-build-organization — details
Reason: Net win +0.0% (0W/1T/0L over 1 trial(s), sign test p=1.000), mean preference +0.0% — underpowered (1 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs
| Scenario | Net win | Δ Pref | Trials (W/T/L) |
|---|---|---|---|
| = Organize build infrastructure for a multi-project repo | +0.0% | +0.0% | 0/1/0 |
⚠️ extension-points — details
Reason: Net win +0.0% (1W/1T/1L over 3 trial(s), sign test p=0.750), mean preference +0.0% — underpowered (3 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs
| Scenario | Net win | Δ Pref | Trials (W/T/L) |
|---|---|---|---|
| = Diagnose NuGet package and repo extension conflicts | +0.0% | +0.0% | 0/1/0 |
| ▼ Diagnose build extension point failures | -100.0% | -40.0% | 0/0/1 |
| ▲ Fix extension point anti-patterns | +100.0% | +40.0% | 1/0/0 |
⚠️ item-management — details
Reason: Net win +66.7% (2W/1T/0L over 3 trial(s), sign test p=0.250), mean preference +26.7% — underpowered (3 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs
| Scenario | Net win | Δ Pref | Trials (W/T/L) |
|---|---|---|---|
| = Diagnose cascading item and batching bugs in code generation pipeline | +0.0% | +0.0% | 0/1/0 |
| ▲ Diagnose item group and batching issues | +100.0% | +40.0% | 1/0/0 |
| ▲ Fix item management anti-patterns | +100.0% | +40.0% | 1/0/0 |
⚠️ msbuild-antipatterns — details
Reason: Net win +0.0% (0W/4T/0L over 4 trial(s), sign test p=1.000), mean preference +0.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs
| Scenario | Net win | Δ Pref | Trials (W/T/L) |
|---|---|---|---|
| = Add a module to an F# project | +0.0% | +0.0% | 0/1/0 |
| = Add a signature file to define public API | +0.0% | +0.0% | 0/1/0 |
| = Fix broken file order causing FS0039 | +0.0% | +0.0% | 0/1/0 |
| = Review MSBuild files for anti-patterns and style issues | +0.0% | +0.0% | 0/1/0 |
⚠️ property-patterns — details
Reason: Net win +66.7% (2W/1T/0L over 3 trial(s), sign test p=0.250), mean preference +26.7% — underpowered (3 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs
| Scenario | Net win | Δ Pref | Trials (W/T/L) |
|---|---|---|---|
| ▲ Diagnose multi-level property hierarchy bugs | +100.0% | +40.0% | 1/0/0 |
| ▲ Diagnose shared build property issues | +100.0% | +40.0% | 1/0/0 |
| = Fix shared property configuration | +0.0% | +0.0% | 0/1/0 |
⚠️ target-authoring — details
Reason: Net win +33.3% (1W/2T/0L over 3 trial(s), sign test p=0.500), mean preference +13.3% — underpowered (3 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs
| Scenario | Net win | Δ Pref | Trials (W/T/L) |
|---|---|---|---|
| = Diagnose broken SDK target chain across files | +0.0% | +0.0% | 0/1/0 |
| ▲ Diagnose custom target build regression | +100.0% | +40.0% | 1/0/0 |
| = Fix custom target anti-patterns | +0.0% | +0.0% | 0/1/0 |
🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions
github-actions
Bot
added
waiting-on-review
JanKrivanek
left a comment
JanKrivanek
left a comment
There was a problem hiding this comment.
Reapproving - new changes look good
Summary
Adds a cost-optimized cross-family evaluation cadence, a judge comparison experiment, and PR gate profiles.
Model cadence (one profile per scheduled day, 07:00 UTC)
defaultmidopus48newerHeavy tiers (
opus48,newer) replace the defaults on their day rather than co-running them, to cap spend on the expensive models. Thedefaultprofile also carries every authorized PR.PR gates
/evaluateruns thedefaultprofile.fullprofile (defaults + mid + opus48) via/evaluate --full, which emits a warning that it is expensive and should be used sparingly.singleremains the legacy one-model path for infra dispatches.Sharding (runner-termination mitigation)
Plugins can bucket their skills with an optional
executionShard:tag ineval.yaml. Thediscoverjob groups skills by shard and fans each bucket out onto its own fresh runner, per model. Untagged skills fall into onedefaultbucket.dotnet-test, one Opus leg dropped from ~171 min unsharded to ~61 min sharded (~2.8x faster), with no change to token spend.Judge experiment (Opus-4.8 vs Haiku-4.5)
On scheduled runs, GPT-executor legs (judged primarily by claude-opus-4.8) also carry a second claude-haiku-4.5 judge. A new step pairs both judges' verdicts by (plugin, model, skill) and writes an isolated
judge-comparison.jsonwith an agreement rate. The file is deliberately excluded from the dashboardcomponents.jsonmanifest so it never renders as a plugin and is never posted on the PR.Dashboard (cross-family view)
The results dashboard (
eng/dashboard/dashboard.js) now understands multiple executor model families in one plugin's history:Fixes
run-name: "schedule: <profile>"and the skip guard matches ondisplay_titleper profile, so one profile no longer suppresses the others on an unchanged commit.Notes / open items
claude-sonnet-5was re-added tonewerper request.gpt-5.6-terrawith no second judge; GPT executor legs (gpt-5.6-luna,gpt-5.3-codex,gpt-5.6-sol) are judged primarily byclaude-opus-4.8withclaude-haiku-4.5as the dual-judge second judge.Executor comparison: gpt-5.6-luna vs gpt-5.3-codex
Full all-plugins evaluation of both executors, each judged cross-family by
claude-opus-4.8. Run from a disposable experiment branch — not part of this PR's diff.Coverage: all 16 plugins, no skill filter. Each executor produced a scored verdict for 91 skills — the symmetric set common to both models, out of ~95 scorable evals (99
eval.yamlminus 4agent.*evals excluded by design). Every skill is scored as skilled (skill loaded) vs baseline (no skill); a "pass" means loading the skill measurably improved the model's output.Of the 91 scored skills, 47 were underpowered (too few trials for a confident verdict) and fall into "no-change" because signal cannot be separated from noise — not because the skill had no effect. Read the pass/regress counts as coming from the meaningfully-powered subset; treat the rest as inconclusive.
Findings: skills help both executors (positive net-win, few regressions);
gpt-5.6-lunaclearly leadsgpt-5.3-codexon the identical skill set, supporting its place in thedefaultset;gpt-5.3-codexis the consistent laggard and stays confined to themid/fullprofiles; the only model-level conflict ismigrate-dotnet9-to-dotnet10(codex passes, luna regresses).