Add per-skill "Skill Value" view to the evaluation dashboard

[AbhitejJohn]

Adds a Skill Value tab to the eval dashboard answering, per skill: "Add this skill and model X uses ~M% fewer tokens and ~Y% less time for typical scenarios; without it ~p% of those scenarios do not pass their checks." Rows are grouped Plugin -> Skill -> Model and filterable by executor and judge model, with a trailing average over the last 20 CI runs keyed by (skill, executor, judge) so model versions never blend.

Data

Reuses the existing publish-eval-data job by emitting a new entries.SkillValue block from generate-benchmark-data.ps1 — no new job, no adapter change. Both arms (baseline = without skill, treatment = with skill) come straight from the verdict the adapter already produces. entries.SkillValue is scheduled-only.

Guardrails

• Min-trial gate (>=5 paired observations, else "insufficient signal").
• Not-passed rate comes from the aggregate per-arm pass booleans and is labelled as telemetry (may include judge-scored graders), N/A when a skill has no boolean graders — never presented as an objective gate.
• Activation% shown prominently; low-activation deltas dimmed/marked as diluted.
• Sample size n always visible; tokens headline = input+output (cache in drill-down).
• Deltas are never blended across models; a single-model group shows that model's real deltas.

Correctness & integrity (this PR)

• Quality/Efficiency integrity: the scheduled merge now groups by (model, judge) for Skill Value but passes -SkillValueOnly for the second+ judge of an executor, so dual-judge runs keep exactly one Quality/Efficiency point per model per run (no duplicated/​inflated trailing windows).
• Honest pass labelling: hasDeterministic -> hasPassData; the column reads "Not-passed"; the sentence reads "do not pass their checks" with a caveat tooltip — because adapt.mjs states these pass booleans may include LLM-grader results.
• Plugin-scoped keys: rows are keyed by (plugin, skill, executor, judge) so two plugins that share a skill name never blend histories.
• Rollup clarity: multi-model rollups read "N model/judge result(s) show measured value" instead of an ambiguous 0/m; Skill Value is the first tab.

Validation

node --check clean on both JS, PowerShell parse clean on the generator, actionlint 1.7.7 clean on the workflow. Rendered locally against a two-plugin fixture (gate, N/A, dilution, single-model rollup, and value cases).

Known limitations (not fixed here)

• Activation% is the adapter's OR-across-repetitions boolean, so it is "any-rep activation per scenario," labelled as such, not a true per-execution rate.
• No variance/confidence interval on the continuous deltas — the gate is an observations floor.

Preview

Fixture-rendered preview using the production dashboard code and styles (sample data):

cc @AbhitejJohn

[Copilot]

Pull request overview

Adds a new Skill Value view to the evaluation dashboard, backed by newly emitted per-run entries.SkillValue data, to summarize per-skill token/time deltas and not-passed telemetry per (executor model, judge model) without blending model versions.

Changes:

• Add eng/dashboard/skill-value.js and wire a new Skill Value dashboard tab with filtering and trailing-window aggregation.
• Extend eng/dashboard/generate-benchmark-data.ps1 to emit a new entries.SkillValue block (and add -SkillValueOnly to avoid duplicating Quality/Efficiency points in multi-judge scenarios).
• Update .github/workflows/evaluation.yml to group scheduled benchmark merges by (model, judgeModel) and prevent double-counting Quality/Efficiency while still emitting SkillValue per judge.

Show a summary per file

File	Description
eng/dashboard/skill-value.js	New client-side renderer/aggregator for Skill Value rows grouped Plugin → Skill → Model, with gating and drill-down.
eng/dashboard/generate-benchmark-data.ps1	Emits new entries.SkillValue per scheduled run; adds -SkillValueOnly to dedupe Q/E when needed.
eng/dashboard/dashboard.js	Adds a Skill Value tab as the default landing view and hooks tab switching to initSkillValue().
eng/dashboard/dashboard.html	Adds CSS for the Skill Value view and loads skill-value.js.
.github/workflows/evaluation.yml	Groups merged results by (model, judgeModel) and passes -SkillValueOnly for second+ judges per executor model.

Review details

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

• Files reviewed: 5/5 changed files
• Comments generated: 1
• Review effort level: Lite

[github-actions]

👋 @AbhitejJohn — this PR has 1 unresolved review thread(s),merge conflict. When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the no-stale label to silence further pings.)

[Copilot]

Review details

Suppressed comments (1)

eng/dashboard/skill-value.js:276

• pctSigned() also formats a zero reduction as "+0%". For consistency with the leaf rows and to avoid implying direction when there’s no change, format 0 as "0%".

  function pctSigned(r) { return r == null ? 'n/a' : `${r > 0 ? '−' : '+'}${Math.abs(r * 100).toFixed(0)}%`; }

• Files reviewed: 5/5 changed files
• Comments generated: 3
• Review effort level: Lite

[github-actions]

❌ Evaluation did not complete successfully (the evaluate job reported failure). Check the workflow run logs, then comment /evaluate 1daa97619f9d2f4e647300399ea0bacb8b46afc1 to retry this exact commit.

52 partial result file(s) were preserved for diagnosis but were not consolidated because the full matrix did not complete.

[Copilot]

Review details

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

eng/dashboard/skill-value.js:208

• The inline comment about when row.activation is null is backwards/misleading. row.activation is set to null when activationExpected === 0, which means no scenarios were expected to activate the skill (e.g., all stimuli were expectActivation:false), not that no scenario declared the annotation. This can confuse future readers interpreting the gating logic.

    // delta is diluted — so suppress the confident value claim. A null activation
    // means NO scenario declared expect_activation, so we cannot confirm the skill
    // fired at all; treat that as unverified rather than asserting value.

eng/dashboard/skill-value.js:258

• metricCell takes a third parameter (fmt) that is never used. This makes the call sites look like formatting affects the output when it doesn’t. If you want to keep the signature stable, prefix the unused parameter with _ to make the intent clear.

  function metricCell(base, treat, fmt) {
    // Delta cell already shows both arms; this is the paired-n column.
    const n = Math.min(base ? base.n : 0, treat ? treat.n : 0);
    const cls = gated(n) ? 'neutral' : 'sv-insufficient';
    return `<td class="num"><span class="${cls}">n=${n}</span></td>`;

• Files reviewed: 5/5 changed files
• Comments generated: 0 new
• Review effort level: Lite

[github-actions]

❌ Evaluation did not complete successfully (the evaluate job reported failure). Check the workflow run logs, then comment /evaluate 542d376c35990b350f4e994eacb65773dd453f73 to retry this exact commit.

22 partial result file(s) were preserved for diagnosis but were not consolidated because the full matrix did not complete.

[AbhitejJohn]

/evaluate f71cddd

[github-actions]

📊 Skill Evaluation Results

10 model/skill results across 5 skills and 2 models — ✅ 0 improved, ➖ 0 not proven improved, ⚠️ 10 invalid or underpowered, 📉 0 preference losses (report only).

Measurement identity: evaluated commit f71cddd4a77ee327b2d5ddb0411034107be8dbbc; 2 judge models.

Measurement health: 10 expected / 10 observed / 10 written; 0 missing, 0 unexpected, 10 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.

A result passes only when the aggregate net win across distinct-stimulus votes is at least 20% and an exact one-sided sign-test result of p ≤ 0.05. Repeated runs measure reliability only.

Skill	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
configuring-opentelemetry-dotnet	claude-sonnet-4.6	⚠️ Underpowered	n=3; 1W/1T/1L; d=2; p=0.750; net +0.0%	🔴 0.63	—	Predeclare more independent, discriminating stimuli; repeated runs do not add power.
configuring-opentelemetry-dotnet	gpt-5.6-luna	⚠️ Underpowered	n=3; 3W/0T/0L; d=3; p=0.125; net +100.0%	✅ 0.15	—	Predeclare more independent, discriminating stimuli; repeated runs do not add power.
convert-blazor-server-to-webapp	claude-sonnet-4.6	⚠️ Underpowered	n=1; 1W/0T/0L; d=1; p=0.500; net +100.0%	✅ 0.07	—	Predeclare more independent, discriminating stimuli; repeated runs do not add power.
convert-blazor-server-to-webapp	gpt-5.6-luna	⚠️ Underpowered	n=1; 1W/0T/0L; d=1; p=0.500; net +100.0%	✅ 0.06	—	Predeclare more independent, discriminating stimuli; repeated runs do not add power.
dotnet-webapi	claude-sonnet-4.6	⚠️ Underpowered	n=3; 2W/0T/1L; d=3; p=0.500; net +33.3%	🔴 0.66	—	Predeclare more independent, discriminating stimuli; repeated runs do not add power.
dotnet-webapi	gpt-5.6-luna	⚠️ Underpowered	n=3; 2W/1T/0L; d=2; p=0.250; net +66.7%	🟡 0.32	—	Predeclare more independent, discriminating stimuli; repeated runs do not add power.
minimal-api-file-upload	claude-sonnet-4.6	⚠️ Underpowered	n=3; 3W/0T/0L; d=3; p=0.125; net +100.0%	🟡 0.40	—	Predeclare more independent, discriminating stimuli; repeated runs do not add power.
minimal-api-file-upload	gpt-5.6-luna	⚠️ Underpowered	n=3; 2W/1T/0L; d=2; p=0.250; net +66.7%	✅ 0.10	—	Predeclare more independent, discriminating stimuli; repeated runs do not add power.
setup-local-sdk	claude-sonnet-4.6	⚠️ Underpowered	n=3; 3W/0T/0L; d=3; p=0.125; net +100.0%	🟡 0.26	—	Predeclare more independent, discriminating stimuli; repeated runs do not add power.
setup-local-sdk	gpt-5.6-luna	⚠️ Underpowered	n=3; 2W/1T/0L; d=2; p=0.250; net +66.7%	✅ 0.15	—	Predeclare more independent, discriminating stimuli; repeated runs do not add power.

ℹ️ How to read this report

• ✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
• ➖ Not proven improved — the result is valid but did not pass both gates. This is not automatically a regression.
• ⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the skill.
• 📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
• Gate evidence — n distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, and net win. The p value applies to one model/skill result; no matrix-wide multiple-comparison correction is applied.
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
• Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
• Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

⚠️ Underpowered — configuring-opentelemetry-dotnet (claude-sonnet-4.6)

Why: Net win +0.0% (1W/1T/1L over 3 stimulus vote(s), sign test p=0.750), mean preference +0.0% across 3 paired run(s) — underpowered (3 counted stimulus vote(s); a credible verdict needs at least 5) — add distinct, discriminating stimuli; repeated runs do not increase task breadth

Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.

State: INVALID_INCONCLUSIVE (underpowered)

Gate evidence: n=3; 1W/1T/1L; d=2; p=0.750; net +0.0%

Overfit: High (score 0.63)

Repeated-run reliability (not used by the gate): 3 paired runs (1W/1T/1L).

Weak or warning scenarios:

Scenario	Net win	Δ Pref	Runs (W/T/L)
= Propagate trace context across a message queue	+0.0%	+0.0%	0/1/0
▼ Set up OpenTelemetry tracing and metrics with custom spans in ASP.NET Core	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

• Propagate trace context across a message queue: Position-swap inconsistent (forward: A, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⚠️ Underpowered — configuring-opentelemetry-dotnet (gpt-5.6-luna)

Why: Net win +100.0% (3W/0T/0L over 3 stimulus vote(s), sign test p=0.125), mean preference +40.0% across 3 paired run(s) — underpowered (3 counted stimulus vote(s); a credible verdict needs at least 5, and this eval won every one of them) — add distinct, discriminating stimuli; repeated runs do not increase task breadth

Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.

State: INVALID_INCONCLUSIVE (underpowered)

Gate evidence: n=3; 3W/0T/0L; d=3; p=0.125; net +100.0%

Overfit: Low (score 0.15)

Repeated-run reliability (not used by the gate): 3 paired runs (3W/0T/0L).

⚠️ Underpowered — convert-blazor-server-to-webapp (claude-sonnet-4.6)

Why: Net win +100.0% (1W/0T/0L over 1 stimulus vote(s), sign test p=0.500), mean preference +100.0% across 1 paired run(s) — underpowered (1 counted stimulus vote(s); a credible verdict needs at least 5, and this eval won every one of them) — add distinct, discriminating stimuli; repeated runs do not increase task breadth

Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.

State: INVALID_INCONCLUSIVE (underpowered)

Gate evidence: n=1; 1W/0T/0L; d=1; p=0.500; net +100.0%

Overfit: Low (score 0.07)

Repeated-run reliability (not used by the gate): 1 paired run (1W/0T/0L).

⚠️ Underpowered — convert-blazor-server-to-webapp (gpt-5.6-luna)

Why: Net win +100.0% (1W/0T/0L over 1 stimulus vote(s), sign test p=0.500), mean preference +100.0% across 1 paired run(s) — underpowered (1 counted stimulus vote(s); a credible verdict needs at least 5, and this eval won every one of them) — add distinct, discriminating stimuli; repeated runs do not increase task breadth

Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.

State: INVALID_INCONCLUSIVE (underpowered)

Gate evidence: n=1; 1W/0T/0L; d=1; p=0.500; net +100.0%

Overfit: Low (score 0.06)

Repeated-run reliability (not used by the gate): 1 paired run (1W/0T/0L).

⚠️ Underpowered — dotnet-webapi (claude-sonnet-4.6)

Why: Net win +33.3% (2W/0T/1L over 3 stimulus vote(s), sign test p=0.500), mean preference +53.3% across 3 paired run(s) — underpowered (3 counted stimulus vote(s); a credible verdict needs at least 5) — add distinct, discriminating stimuli; repeated runs do not increase task breadth

Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.

State: INVALID_INCONCLUSIVE (underpowered)

Gate evidence: n=3; 2W/0T/1L; d=3; p=0.500; net +33.3%

Overfit: High (score 0.66)

Repeated-run reliability (not used by the gate): 3 paired runs (2W/0T/1L).

Weak or warning scenarios:

Scenario	Net win	Δ Pref	Runs (W/T/L)
▼ Add error handling with ProblemDetails and IExceptionHandler	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

• Add error handling with ProblemDetails and IExceptionHandler: A is a complete, directly usable answer with the full handler and wiring, exact mappings, logging, and explicit 500 detail suppression. B's final response is only a summary of files it says it created, so the user cannot actually implement the solution from the answer; its sta...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⚠️ Underpowered — dotnet-webapi (gpt-5.6-luna)

Why: Net win +66.7% (2W/1T/0L over 3 stimulus vote(s), sign test p=0.250), mean preference +46.7% across 3 paired run(s) — underpowered (3 counted stimulus vote(s); a credible verdict needs at least 5) — add distinct, discriminating stimuli; repeated runs do not increase task breadth

Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.

State: INVALID_INCONCLUSIVE (underpowered)

Gate evidence: n=3; 2W/1T/0L; d=2; p=0.250; net +66.7%

Overfit: Moderate (score 0.32)

Repeated-run reliability (not used by the gate): 3 paired runs (2W/1T/0L).

Weak or warning scenarios:

Scenario	Net win	Δ Pref	Runs (W/T/L)
= Add a new API endpoint to an existing controller-based project	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Add a new API endpoint to an existing controller-based project: Both agents investigated the workspace, found it effectively empty (only a skill file in B's case), and declined to produce any code rather than fabricating a solution. Neither meets any rubric criterion since no controller, DTOs, or endpoints were written. Their behavior and ...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⚠️ Underpowered — minimal-api-file-upload (claude-sonnet-4.6)

Why: Net win +100.0% (3W/0T/0L over 3 stimulus vote(s), sign test p=0.125), mean preference +60.0% across 3 paired run(s) — underpowered (3 counted stimulus vote(s); a credible verdict needs at least 5, and this eval won every one of them) — add distinct, discriminating stimuli; repeated runs do not increase task breadth

Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.

State: INVALID_INCONCLUSIVE (underpowered)

Gate evidence: n=3; 3W/0T/0L; d=3; p=0.125; net +100.0%

Overfit: Moderate (score 0.40)

Repeated-run reliability (not used by the gate): 3 paired runs (3W/0T/0L).

⚠️ Underpowered — minimal-api-file-upload (gpt-5.6-luna)

Why: Net win +66.7% (2W/1T/0L over 3 stimulus vote(s), sign test p=0.250), mean preference +26.7% across 3 paired run(s) — underpowered (3 counted stimulus vote(s); a credible verdict needs at least 5) — add distinct, discriminating stimuli; repeated runs do not increase task breadth

Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.

State: INVALID_INCONCLUSIVE (underpowered)

Gate evidence: n=3; 2W/1T/0L; d=2; p=0.250; net +66.7%

Overfit: Low (score 0.10)

Repeated-run reliability (not used by the gate): 3 paired runs (2W/1T/0L).

Weak or warning scenarios:

Scenario	Net win	Δ Pref	Runs (W/T/L)
= Stream very large file uploads without buffering	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Stream very large file uploads without buffering: Position-swap inconsistent (forward: A, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⚠️ Underpowered — setup-local-sdk (claude-sonnet-4.6)

Why: Net win +100.0% (3W/0T/0L over 3 stimulus vote(s), sign test p=0.125), mean preference +80.0% across 3 paired run(s) — underpowered (3 counted stimulus vote(s); a credible verdict needs at least 5, and this eval won every one of them) — add distinct, discriminating stimuli; repeated runs do not increase task breadth

Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.

State: INVALID_INCONCLUSIVE (underpowered)

Gate evidence: n=3; 3W/0T/0L; d=3; p=0.125; net +100.0%

Overfit: Moderate (score 0.26)

Repeated-run reliability (not used by the gate): 3 paired runs (3W/0T/0L).

⚠️ Underpowered — setup-local-sdk (gpt-5.6-luna)

Why: Net win +66.7% (2W/1T/0L over 3 stimulus vote(s), sign test p=0.250), mean preference +46.7% across 3 paired run(s) — underpowered (3 counted stimulus vote(s); a credible verdict needs at least 5) — add distinct, discriminating stimuli; repeated runs do not increase task breadth

Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.

State: INVALID_INCONCLUSIVE (underpowered)

Gate evidence: n=3; 2W/1T/0L; d=2; p=0.250; net +66.7%

Overfit: Low (score 0.15)

Repeated-run reliability (not used by the gate): 3 paired runs (2W/1T/0L).

Weak or warning scenarios:

Scenario	Net win	Δ Pref	Runs (W/T/L)
= Create team install scripts	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Create team install scripts: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔍 Full Results - all metrics and investigation details

To investigate non-passing or warning results, paste this to your AI coding agent:

For PR 1059 in dotnet/skills, download eval artifacts with gh run download 33053413438 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/f71cddd4a77ee327b2d5ddb0411034107be8dbbc/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[Evangelink]

Reviewed the full change set and found no actionable issues.

[github-actions]

❌ Evaluation did not complete successfully (the evaluate job reported failure). Check the workflow run logs, then comment /evaluate f71cddd4a77ee327b2d5ddb0411034107be8dbbc to retry this exact commit.

34 partial result file(s) were preserved for diagnosis but were not consolidated because the full matrix did not complete.

[github-actions]

Skill Coverage Report

	Plugin	Skill	Covered	Coverage
✅	dotnet-test	grade-tests	27/29	93.1%

Uncovered: dotnet-test/grade-tests

• [Pitfall] Inflating deductions to justify the grade (line 337)
• [Pitfall] Spilling a 500-row table into a PR comment (line 346)

[AbhitejJohn]

/evaluate 09e5bab

[Copilot]

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

eng/dashboard/skill-value.js:235

• The N/A tooltip is too definitive: hasPassData is false when there were no counted (baselinePassed/treatmentPassed) booleans, which can happen not only when a skill has no boolean graders, but also when all trials errored or pass booleans were missing. Reword the tooltip so it describes the observed condition rather than asserting the cause.

  function failureCell(row) {
    if (!row.hasPass || row.passTotal === 0) return '<td class="num" title="No pass/fail data — skill has no boolean graders">N/A</td>';
    if (!gated(row.passTotal)) return `<td class="num"><span class="sv-insufficient">n=${row.passTotal}</span></td>`;

• Files reviewed: 6/6 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

• Files reviewed: 6/6 changed files
• Comments generated: 0 new
• Review effort level: Lite

[AbhitejJohn]

/evaluate bce81f3

[github-actions]

📊 Skill Evaluation Results

2 model/skill results across 1 skill and 2 models — ✅ 2 improved, ➖ 0 not proven improved, ⚠️ 0 invalid or underpowered, 📉 0 preference losses (report only).

Measurement identity: evaluated commit 09e5babf493ebed39b1fa1d38899fef36789cf54; 2 judge models.

Measurement health: 2 expected / 2 observed / 2 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.

A result passes only when the aggregate net win across distinct-stimulus votes is at least 20% and an exact one-sided sign-test result of p ≤ 0.05. Repeated runs measure reliability only.

Skill	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
grade-tests	claude-sonnet-4.6	✅ Improved	n=8; 8W/0T/0L; d=8; p=0.004; net +100.0%	🔴 0.59	—	Review overfit evidence.
grade-tests	gpt-5.6-luna	✅ Improved	n=8; 8W/0T/0L; d=8; p=0.004; net +100.0%	🟡 0.39	—	Review overfit evidence.

ℹ️ How to read this report

• ✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
• ➖ Not proven improved — the result is valid but did not pass both gates. This is not automatically a regression.
• ⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the skill.
• 📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
• Gate evidence — n distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, and net win. The p value applies to one model/skill result; no matrix-wide multiple-comparison correction is applied.
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
• Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
• Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

✅ Improved — grade-tests (claude-sonnet-4.6)

Why: Net win +100.0% (8W/0T/0L over 8 stimulus vote(s), sign test p=0.004), mean preference +86.7% across 24 paired run(s) — credibly better

Next action: Review overfit evidence.

State: VALID_PASS (credible_preference_improvement)

Gate evidence: n=8; 8W/0T/0L; d=8; p=0.004; net +100.0%

Overfit: High (score 0.59)

Repeated-run reliability (not used by the gate): 24 paired runs (23W/0T/1L).

✅ Improved — grade-tests (gpt-5.6-luna)

Why: Net win +100.0% (8W/0T/0L over 8 stimulus vote(s), sign test p=0.004), mean preference +68.3% across 24 paired run(s) — credibly better

Next action: Review overfit evidence.

State: VALID_PASS (credible_preference_improvement)

Gate evidence: n=8; 8W/0T/0L; d=8; p=0.004; net +100.0%

Overfit: Moderate (score 0.39)

Repeated-run reliability (not used by the gate): 24 paired runs (23W/1T/0L).

🔍 Full Results - all metrics and investigation details

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[github-actions]

❌ Evaluation did not complete successfully (the evaluate job reported failure). Check the workflow run logs, then comment /evaluate f71cddd4a77ee327b2d5ddb0411034107be8dbbc to retry this exact commit.

12 partial result file(s) were preserved for diagnosis but were not consolidated because the full matrix did not complete.

[github-actions]

📊 Skill Evaluation Results

2 model/skill results across 1 skill and 2 models — ✅ 2 improved, ➖ 0 not proven improved, ⚠️ 0 invalid or underpowered, 📉 0 preference losses (report only).

Measurement identity: evaluated commit bce81f3cbd14b2d77de3b9dd252e17d71993db0b; 2 judge models.

Measurement health: 2 expected / 2 observed / 2 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.

A result passes only when the aggregate net win across distinct-stimulus votes is at least 20% and an exact one-sided sign-test result of p ≤ 0.05. Repeated runs measure reliability only.

Skill	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
grade-tests	claude-sonnet-4.6	✅ Improved	n=8; 8W/0T/0L; d=8; p=0.004; net +100.0%	🟡 0.35	—	Review overfit evidence.
grade-tests	gpt-5.6-luna	✅ Improved	n=8; 8W/0T/0L; d=8; p=0.004; net +100.0%	🟡 0.44	—	Review overfit evidence.

ℹ️ How to read this report

• ✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
• ➖ Not proven improved — the result is valid but did not pass both gates. This is not automatically a regression.
• ⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the skill.
• 📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
• Gate evidence — n distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, and net win. The p value applies to one model/skill result; no matrix-wide multiple-comparison correction is applied.
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
• Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
• Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

✅ Improved — grade-tests (claude-sonnet-4.6)

Why: Net win +100.0% (8W/0T/0L over 8 stimulus vote(s), sign test p=0.004), mean preference +81.7% across 24 paired run(s) — credibly better

Next action: Review overfit evidence.

State: VALID_PASS (credible_preference_improvement)

Gate evidence: n=8; 8W/0T/0L; d=8; p=0.004; net +100.0%

Overfit: Moderate (score 0.35)

Repeated-run reliability (not used by the gate): 24 paired runs (23W/0T/1L).

✅ Improved — grade-tests (gpt-5.6-luna)

Why: Net win +100.0% (8W/0T/0L over 8 stimulus vote(s), sign test p=0.004), mean preference +72.5% across 24 paired run(s) — credibly better

Next action: Review overfit evidence.

State: VALID_PASS (credible_preference_improvement)

Gate evidence: n=8; 8W/0T/0L; d=8; p=0.004; net +100.0%

Overfit: Moderate (score 0.44)

Repeated-run reliability (not used by the gate): 24 paired runs (24W/0T/0L).

🔍 Full Results - all metrics and investigation details

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[github-actions]

❌ Evaluation did not complete successfully (the evaluate job reported failure). Check the workflow run logs, then comment /evaluate f71cddd4a77ee327b2d5ddb0411034107be8dbbc to retry this exact commit.

39 partial result file(s) were preserved for diagnosis but were not consolidated because the full matrix did not complete.