Improve GPT-focused dotnet-test eval outcomes - #1122
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The updated coverage-analysis frontmatter description has a small but important wording inconsistency that could misroute or mis-explain CRAP/refactoring-safety behavior.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Lite
Findings: 1
New issues introduced by this change (1)
| Severity | Finding |
|---|---|
 |
plugins/dotnet-test/skills/coverage-analysis/SKILL.md — In the frontmatter description, the wording "calculating CRAP unless risk was requested" is… |
What changed in this PR
This PR tunes dotnet-test skill guidance and evaluation fixtures to address specific GPT-family correctness gaps (severity calibration, deterministic validation, and scoped migration semantics) while keeping changes focused on measurable eval outcomes.
Changes:
- Strengthens eval determinism by adding build + non-executing discovery validation for tagging scenarios and a new DateTime.Now-local-semantics migration scenario (with build verification).
- Refines skill instructions around severity calibration, scope boundaries for semantics-preserving migrations, and default scan scope when a path isn’t provided.
- Trims and tightens
coverage-analysisrouting/description text to recover menu capacity (with minor wording consistency still needed).
| File | Description |
|---|---|
| tests/dotnet-test/test-tagging/eval.yaml | Adds deterministic build + --list-tests discovery validation and related rubric expectations for NUnit tagging. |
| tests/dotnet-test/migrate-static-to-wrapper/eval.yaml | Adds a new predeclared scenario to preserve local calendar semantics when migrating DateTime.Now, including scope/build graders. |
| plugins/dotnet-test/skills/test-tagging/SKILL.md | Adds a verification step and updates validation checklist to prefer non-executing discovery when available. |
| plugins/dotnet-test/skills/test-anti-patterns/SKILL.md | Calibrates severity guidance for demonstrated order-dependent failures and strengthens transformation-oracle guidance. |
| plugins/dotnet-test/skills/migrate-static-to-wrapper/SKILL.md | Clarifies when “intentional” call sites may be migrated (only when explicitly requested with semantics preserved). |
| plugins/dotnet-test/skills/detect-static-dependencies/SKILL.md | Makes target path optional and clarifies default scan scope is the current workspace. |
| plugins/dotnet-test/skills/coverage-analysis/SKILL.md | Compresses routing description text; one sentence needs wording alignment to avoid ambiguity about when CRAP is computed. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Skill Coverage Report
Uncovered:
|
There was a problem hiding this comment.
Copilot review overview
🟢 Approval recommended
The changes are consistent with existing eval/skill patterns in the repo and add deterministic validation without introducing schema or formatting issues in the modified files.
Review tier: Lite
Findings: None
Issues resolved since last review (1)
| Severity | Finding |
|---|---|
|
plugins/dotnet-test/skills/coverage-analysis/SKILL.md — In the frontmatter description, the wording "calculating CRAP unless risk was requested" is… View resolved comment |
There was a problem hiding this comment.
Copilot review overview
🟢 Approval recommended
The changes are consistent with existing eval/schema patterns in the repo and primarily strengthen deterministic validation and clarify skill guidance without introducing risky code paths.
Review tier: Lite
Findings: None
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Inactive
📊 Skill Evaluation Results10 model/skill results across 5 skills and 2 models — ✅ 5 improved, ➖ 3 not proven improved, Measurement identity: evaluated commit Measurement health: 10 expected / 10 observed / 10 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots. Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven. A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of
ℹ️ How to read this report
⛔ Activation contract failed — coverage-analysis (claude-sonnet-4.6)Why: Net win +75.0% (7W/0T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.035), mean preference +25.4% across 26 paired run(s), 5 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill) Next action: Narrow skill routing so the listed off-target scenarios stay dormant. State: Gate evidence: n=8; 7W/0T/1L; d=8; p=0.035; net +75.0%; 5 dormancy excluded Warnings: Dormancy contract: 1 unexpected activation(s) Overfit: Moderate (score 0.22) Repeated-run reliability (not used by the gate): 26 paired runs (16W/6T/4L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ⛔ Activation contract failed — coverage-analysis (gpt-5.6-luna)Why: Net win +50.0% (4W/4T/0L over 8 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +12.3% across 26 paired run(s), 5 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill) Next action: Narrow skill routing so the listed off-target scenarios stay dormant. State: Gate evidence: n=8; 4W/4T/0L; d=4; p=0.063; net +50.0%; 5 dormancy excluded Warnings: Dormancy contract: 1 unexpected activation(s) Overfit: Low (score 0.15) Repeated-run reliability (not used by the gate): 26 paired runs (11W/12T/3L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — detect-static-dependencies (gpt-5.6-luna)Why: Net win +50.0% (4W/4T/0L over 8 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +20.0% across 8 paired run(s) — not credible — 4 of 8 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment. State: Gate evidence: n=8; 4W/4T/0L; d=4; p=0.063; net +50.0% Warnings: Activation: isolated 7/8; plugin 8/8 Overfit: Low (score 0.16) Repeated-run reliability (not used by the gate): 8 paired runs (4W/4T/0L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — test-anti-patterns (claude-sonnet-4.6)Why: Net win +62.5% (6W/1T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +28.9% across 9 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.063 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=8; 6W/1T/1L; d=7; p=0.063; net +62.5%; 1 dormancy excluded Overfit: Moderate (score 0.23) Repeated-run reliability (not used by the gate): 9 paired runs (6W/2T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — test-anti-patterns (gpt-5.6-luna)Why: Net win +50.0% (5W/2T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.109), mean preference +22.2% across 9 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.109 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=8; 5W/2T/1L; d=6; p=0.109; net +50.0%; 1 dormancy excluded Overfit: Moderate (score 0.23) Repeated-run reliability (not used by the gate): 9 paired runs (6W/2T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — detect-static-dependencies (claude-sonnet-4.6)Why: Net win +75.0% (7W/0T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.035), mean preference +45.0% across 8 paired run(s) — credibly better Next action: Review overfit evidence. State: Gate evidence: n=8; 7W/0T/1L; d=8; p=0.035; net +75.0% Overfit: Moderate (score 0.36) Repeated-run reliability (not used by the gate): 8 paired runs (7W/0T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — migrate-static-to-wrapper (claude-sonnet-4.6)Why: Net win +70.0% (8W/1T/1L over 10 preference-eligible stimulus vote(s), sign test p=0.020), mean preference +52.0% across 10 paired run(s) — credibly better Next action: Review overfit evidence. State: Gate evidence: n=10; 8W/1T/1L; d=9; p=0.020; net +70.0% Overfit: Moderate (score 0.32) Repeated-run reliability (not used by the gate): 10 paired runs (8W/1T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — test-tagging (claude-sonnet-4.6)Why: Net win +100.0% (8W/0T/0L over 8 preference-eligible stimulus vote(s), sign test p=0.004), mean preference +50.0% across 12 paired run(s), 4 dormancy stimulus/stimuli excluded from preference — credibly better Next action: Review overfit evidence. State: Gate evidence: n=8; 8W/0T/0L; d=8; p=0.004; net +100.0%; 4 dormancy excluded Overfit: Moderate (score 0.37) Repeated-run reliability (not used by the gate): 12 paired runs (10W/1T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. Routine passing details for 2 results are in Full Results. 🔍 Full Results - all metrics and investigation details
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
GitHub Actions
Inactive
There was a problem hiding this comment.
Copilot review overview
🟢 Approval recommended
The changes are consistent, schema-aligned, and primarily strengthen eval determinism and skill guidance without introducing risky runtime behavior.
Review tier: Lite
Findings: None
Issues resolved since last review (1)
| Severity | Finding |
|---|---|
 |
tests/dotnet-test/migrate-static-to-wrapper/eval.yaml — The grader that generates .eval-csharp-lines only strips full-line // comments. That allows a… View resolved comment |
📊 Skill Evaluation Results10 model/skill results across 5 skills and 2 models — ✅ 3 improved, ➖ 3 not proven improved, Measurement identity: evaluated commit Measurement health: 10 expected / 10 observed / 10 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots. Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven. A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of
ℹ️ How to read this report
⛔ Activation contract failed — coverage-analysis (claude-sonnet-4.6)Why: Net win +87.5% (7W/1T/0L over 8 preference-eligible stimulus vote(s), sign test p=0.008), mean preference +25.4% across 26 paired run(s), 5 dormancy stimulus/stimuli excluded from preference — activation contract failed (2 explicit dormancy scenario(s) activated the isolated target skill) Next action: Narrow skill routing so the listed off-target scenarios stay dormant. State: Gate evidence: n=8; 7W/1T/0L; d=7; p=0.008; net +87.5%; 5 dormancy excluded Warnings: Dormancy contract: 2 unexpected activation(s) Overfit: Moderate (score 0.23) Repeated-run reliability (not used by the gate): 26 paired runs (15W/8T/3L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ⛔ Activation contract failed — coverage-analysis (gpt-5.6-luna)Why: Net win +87.5% (7W/1T/0L over 8 preference-eligible stimulus vote(s), sign test p=0.008), mean preference +18.5% across 26 paired run(s), 5 dormancy stimulus/stimuli excluded from preference — activation contract failed (2 explicit dormancy scenario(s) activated the isolated target skill) Next action: Narrow skill routing so the listed off-target scenarios stay dormant. State: Gate evidence: n=8; 7W/1T/0L; d=7; p=0.008; net +87.5%; 5 dormancy excluded Warnings: Dormancy contract: 2 unexpected activation(s); Activation: isolated 7/8; plugin 7/8 Overfit: Low (score 0.14) Repeated-run reliability (not used by the gate): 26 paired runs (14W/10T/2L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ⛔ Activation contract failed — detect-static-dependencies (claude-sonnet-4.6)Why: Net win +100.0% (7W/0T/0L over 7 preference-eligible stimulus vote(s), sign test p=0.008), mean preference +45.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill) Next action: Narrow skill routing so the listed off-target scenarios stay dormant. State: Gate evidence: n=7; 7W/0T/0L; d=7; p=0.008; net +100.0%; 1 dormancy excluded Warnings: Dormancy contract: 1 unexpected activation(s) Overfit: Moderate (score 0.25) Repeated-run reliability (not used by the gate): 8 paired runs (7W/0T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ⛔ Activation contract failed — test-tagging (claude-sonnet-4.6)Why: Net win +87.5% (7W/1T/0L over 8 preference-eligible stimulus vote(s), sign test p=0.008), mean preference +43.3% across 12 paired run(s), 4 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill) Next action: Narrow skill routing so the listed off-target scenarios stay dormant. State: Gate evidence: n=8; 7W/1T/0L; d=7; p=0.008; net +87.5%; 4 dormancy excluded Warnings: Dormancy contract: 1 unexpected activation(s) Overfit: Moderate (score 0.35) Repeated-run reliability (not used by the gate): 12 paired runs (9W/1T/2L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — detect-static-dependencies (gpt-5.6-luna)Why: Net win +42.9% (5W/0T/2L over 7 preference-eligible stimulus vote(s), sign test p=0.227), mean preference +10.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.227 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=7; 5W/0T/2L; d=7; p=0.227; net +42.9%; 1 dormancy excluded Overfit: Low (score 0.15) Repeated-run reliability (not used by the gate): 8 paired runs (5W/0T/3L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — test-anti-patterns (claude-sonnet-4.6)Why: Net win +62.5% (6W/1T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +40.0% across 9 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.063 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=8; 6W/1T/1L; d=7; p=0.063; net +62.5%; 1 dormancy excluded Overfit: Moderate (score 0.47) Repeated-run reliability (not used by the gate): 9 paired runs (7W/1T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — test-tagging (gpt-5.6-luna)Why: Net win +62.5% (6W/1T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +30.0% across 12 paired run(s), 4 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.063 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=8; 6W/1T/1L; d=7; p=0.063; net +62.5%; 4 dormancy excluded Overfit: Low (score 0.18) Repeated-run reliability (not used by the gate): 12 paired runs (7W/4T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — migrate-static-to-wrapper (claude-sonnet-4.6)Why: Net win +90.0% (9W/1T/0L over 10 preference-eligible stimulus vote(s), sign test p=0.002), mean preference +36.0% across 10 paired run(s) — credibly better Next action: Fix activation gaps; Review overfit evidence. State: Gate evidence: n=10; 9W/1T/0L; d=9; p=0.002; net +90.0% Warnings: Activation: isolated 10/10; plugin 9/10 Overfit: Moderate (score 0.38) Repeated-run reliability (not used by the gate): 10 paired runs (9W/1T/0L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — test-anti-patterns (gpt-5.6-luna)Why: Net win +75.0% (6W/2T/0L over 8 preference-eligible stimulus vote(s), sign test p=0.016), mean preference +26.7% across 9 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — credibly better Next action: Fix activation gaps; Review overfit evidence. State: Gate evidence: n=8; 6W/2T/0L; d=6; p=0.016; net +75.0%; 1 dormancy excluded Warnings: Activation: isolated 8/8; plugin 7/8 Overfit: Moderate (score 0.22) Repeated-run reliability (not used by the gate): 9 paired runs (6W/3T/0L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. Routine passing details for 1 result are in Full Results. 🔍 Full Results - all metrics and investigation details
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
📊 Skill Evaluation Results10 model/skill results across 5 skills and 2 models — ✅ 5 improved, ➖ 3 not proven improved, Measurement identity: evaluated commit Measurement health: 10 expected / 10 observed / 10 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots. Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven. A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of
ℹ️ How to read this report
⛔ Activation contract failed — coverage-analysis (gpt-5.6-luna)Why: Net win +37.5% (5W/1T/2L over 8 preference-eligible stimulus vote(s), sign test p=0.227), mean preference +13.1% across 26 paired run(s), 5 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill) Next action: Narrow skill routing so the listed off-target scenarios stay dormant. State: Gate evidence: n=8; 5W/1T/2L; d=7; p=0.227; net +37.5%; 5 dormancy excluded Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 6/8; plugin 7/8 Overfit: Low (score 0.11) Repeated-run reliability (not used by the gate): 26 paired runs (11W/11T/4L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ⛔ Activation contract failed — detect-static-dependencies (claude-sonnet-4.6)Why: Net win +100.0% (7W/0T/0L over 7 preference-eligible stimulus vote(s), sign test p=0.008), mean preference +45.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill) Next action: Narrow skill routing so the listed off-target scenarios stay dormant. State: Gate evidence: n=7; 7W/0T/0L; d=7; p=0.008; net +100.0%; 1 dormancy excluded Warnings: Dormancy contract: 1 unexpected activation(s) Overfit: Low (score 0.20) Repeated-run reliability (not used by the gate): 8 paired runs (7W/0T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — coverage-analysis (claude-sonnet-4.6)Why: Net win +37.5% (4W/3T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.188), mean preference +15.4% across 26 paired run(s), 5 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.188 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=8; 4W/3T/1L; d=5; p=0.188; net +37.5%; 5 dormancy excluded Overfit: Moderate (score 0.39) Repeated-run reliability (not used by the gate): 26 paired runs (11W/8T/7L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — test-anti-patterns (gpt-5.6-luna)Why: Net win +50.0% (5W/2T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.109), mean preference +22.2% across 9 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.109 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=8; 5W/2T/1L; d=6; p=0.109; net +50.0%; 1 dormancy excluded Overfit: Moderate (score 0.24) Repeated-run reliability (not used by the gate): 9 paired runs (6W/2T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — test-tagging (gpt-5.6-luna)Why: Net win +62.5% (6W/1T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +36.7% across 12 paired run(s), 4 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.063 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=8; 6W/1T/1L; d=7; p=0.063; net +62.5%; 4 dormancy excluded Overfit: Low (score 0.16) Repeated-run reliability (not used by the gate): 12 paired runs (6W/5T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — migrate-static-to-wrapper (claude-sonnet-4.6)Why: Net win +80.0% (8W/2T/0L over 10 preference-eligible stimulus vote(s), sign test p=0.004), mean preference +50.0% across 10 paired run(s) — credibly better Next action: Review overfit evidence. State: Gate evidence: n=10; 8W/2T/0L; d=8; p=0.004; net +80.0% Overfit: Moderate (score 0.28) Repeated-run reliability (not used by the gate): 10 paired runs (8W/2T/0L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — test-anti-patterns (claude-sonnet-4.6)Why: Net win +75.0% (6W/2T/0L over 8 preference-eligible stimulus vote(s), sign test p=0.016), mean preference +33.3% across 9 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — credibly better Next action: Review overfit evidence. State: Gate evidence: n=8; 6W/2T/0L; d=6; p=0.016; net +75.0%; 1 dormancy excluded Overfit: Moderate (score 0.34) Repeated-run reliability (not used by the gate): 9 paired runs (6W/3T/0L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — test-tagging (claude-sonnet-4.6)Why: Net win +75.0% (7W/0T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.035), mean preference +35.0% across 12 paired run(s), 4 dormancy stimulus/stimuli excluded from preference — credibly better Next action: Review overfit evidence. State: Gate evidence: n=8; 7W/0T/1L; d=8; p=0.035; net +75.0%; 4 dormancy excluded Overfit: High (score 0.54) Repeated-run reliability (not used by the gate): 12 paired runs (8W/2T/2L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. Routine passing details for 2 results are in Full Results. 🔍 Full Results - all metrics and investigation details
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62ad695-3f5f-482c-a4e6-6b47c654e366
There was a problem hiding this comment.
Copilot review overview
🟢 Approval recommended
The updates are confined to eval specs, skill documentation, fixtures, and synced manifest versions, and the new graders align with the skill-validator schema/contracts.
Review tier: Lite
Findings: None
|
/evaluate aab180f |
📊 Skill Evaluation Results10 model/skill results across 5 skills and 2 models — ✅ 4 improved, ➖ 2 not proven improved, Measurement identity: evaluated commit Measurement health: 10 expected / 10 observed / 10 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots. Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven. A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of
ℹ️ How to read this report
⛔ Activation contract failed — coverage-analysis (claude-sonnet-4.6)Why: Net win +62.5% (6W/1T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +31.5% across 26 paired run(s), 5 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill) Next action: Narrow skill routing so the listed off-target scenarios stay dormant. State: Gate evidence: n=8; 6W/1T/1L; d=7; p=0.063; net +62.5%; 5 dormancy excluded Warnings: Dormancy contract: 1 unexpected activation(s) Overfit: Moderate (score 0.31) Repeated-run reliability (not used by the gate): 26 paired runs (18W/3T/5L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ⛔ Activation contract failed — coverage-analysis (gpt-5.6-luna)Why: Net win +50.0% (6W/0T/2L over 8 preference-eligible stimulus vote(s), sign test p=0.145), mean preference +15.4% across 26 paired run(s), 5 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill) Next action: Narrow skill routing so the listed off-target scenarios stay dormant. State: Gate evidence: n=8; 6W/0T/2L; d=8; p=0.145; net +50.0%; 5 dormancy excluded Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 6/8; plugin 7/8 Overfit: Low (score 0.12) Repeated-run reliability (not used by the gate): 26 paired runs (14W/8T/4L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ⛔ Activation contract failed — detect-static-dependencies (claude-sonnet-4.6)Why: Net win +100.0% (7W/0T/0L over 7 preference-eligible stimulus vote(s), sign test p=0.008), mean preference +30.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill) Next action: Narrow skill routing so the listed off-target scenarios stay dormant. State: Gate evidence: n=7; 7W/0T/0L; d=7; p=0.008; net +100.0%; 1 dormancy excluded Warnings: Dormancy contract: 1 unexpected activation(s) Overfit: High (score 0.54) Repeated-run reliability (not used by the gate): 8 paired runs (7W/0T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ⛔ Activation contract failed — test-anti-patterns (gpt-5.6-luna)Why: Net win +75.0% (6W/2T/0L over 8 preference-eligible stimulus vote(s), sign test p=0.016), mean preference +22.2% across 9 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill) Next action: Narrow skill routing so the listed off-target scenarios stay dormant. State: Gate evidence: n=8; 6W/2T/0L; d=6; p=0.016; net +75.0%; 1 dormancy excluded Warnings: Dormancy contract: 1 unexpected activation(s) Overfit: Low (score 0.13) Repeated-run reliability (not used by the gate): 9 paired runs (6W/2T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — detect-static-dependencies (gpt-5.6-luna)Why: Net win +42.9% (5W/0T/2L over 7 preference-eligible stimulus vote(s), sign test p=0.227), mean preference +10.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.227 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=7; 5W/0T/2L; d=7; p=0.227; net +42.9%; 1 dormancy excluded Overfit: Low (score 0.10) Repeated-run reliability (not used by the gate): 8 paired runs (5W/0T/3L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — test-tagging (gpt-5.6-luna)Why: Net win +50.0% (5W/2T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.109), mean preference +25.0% across 12 paired run(s), 4 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.109 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=8; 5W/2T/1L; d=6; p=0.109; net +50.0%; 4 dormancy excluded Overfit: Low (score 0.14) Repeated-run reliability (not used by the gate): 12 paired runs (5W/5T/2L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — migrate-static-to-wrapper (claude-sonnet-4.6)Why: Net win +90.0% (9W/1T/0L over 10 preference-eligible stimulus vote(s), sign test p=0.002), mean preference +42.0% across 10 paired run(s) — credibly better Next action: Review overfit evidence. State: Gate evidence: n=10; 9W/1T/0L; d=9; p=0.002; net +90.0% Overfit: Moderate (score 0.34) Repeated-run reliability (not used by the gate): 10 paired runs (9W/1T/0L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — test-anti-patterns (claude-sonnet-4.6)Why: Net win +87.5% (7W/1T/0L over 8 preference-eligible stimulus vote(s), sign test p=0.008), mean preference +37.8% across 9 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — credibly better Next action: Review overfit evidence. State: Gate evidence: n=8; 7W/1T/0L; d=7; p=0.008; net +87.5%; 1 dormancy excluded Overfit: Moderate (score 0.32) Repeated-run reliability (not used by the gate): 9 paired runs (7W/2T/0L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — test-tagging (claude-sonnet-4.6)Why: Net win +100.0% (8W/0T/0L over 8 preference-eligible stimulus vote(s), sign test p=0.004), mean preference +51.7% across 12 paired run(s), 4 dormancy stimulus/stimuli excluded from preference — credibly better Next action: Review overfit evidence. State: Gate evidence: n=8; 8W/0T/0L; d=8; p=0.004; net +100.0%; 4 dormancy excluded Overfit: Moderate (score 0.46) Repeated-run reliability (not used by the gate): 12 paired runs (9W/2T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. Routine passing details for 1 result are in Full Results. 🔍 Full Results - all metrics and investigation details
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
📊 Skill Evaluation Results10 model/skill results across 5 skills and 2 models — ✅ 7 improved, ➖ 2 not proven improved, Measurement identity: evaluated commit Measurement health: 10 expected / 10 observed / 10 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots. Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven. A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of
ℹ️ How to read this report
⛔ Activation contract failed — detect-static-dependencies (claude-sonnet-4.6)Why: Net win +100.0% (7W/0T/0L over 7 preference-eligible stimulus vote(s), sign test p=0.008), mean preference +37.5% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill) Next action: Narrow skill routing so the listed off-target scenarios stay dormant. State: Gate evidence: n=7; 7W/0T/0L; d=7; p=0.008; net +100.0%; 1 dormancy excluded Warnings: Dormancy contract: 1 unexpected activation(s) Overfit: Moderate (score 0.40) Repeated-run reliability (not used by the gate): 8 paired runs (7W/0T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — coverage-analysis (claude-sonnet-4.6)Why: Net win +62.5% (6W/1T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +30.8% across 26 paired run(s), 5 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.063 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=8; 6W/1T/1L; d=7; p=0.063; net +62.5%; 5 dormancy excluded Overfit: Moderate (score 0.42) Repeated-run reliability (not used by the gate): 26 paired runs (17W/6T/3L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — test-anti-patterns (gpt-5.6-luna)Why: Net win +37.5% (4W/3T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.188), mean preference +13.3% across 9 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.188 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=8; 4W/3T/1L; d=5; p=0.188; net +37.5%; 1 dormancy excluded Overfit: Low (score 0.08) Repeated-run reliability (not used by the gate): 9 paired runs (4W/4T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — coverage-analysis (gpt-5.6-luna)Why: Net win +62.5% (5W/3T/0L over 8 preference-eligible stimulus vote(s), sign test p=0.031), mean preference +12.3% across 26 paired run(s), 5 dormancy stimulus/stimuli excluded from preference — credibly better Next action: Fix activation gaps. State: Gate evidence: n=8; 5W/3T/0L; d=5; p=0.031; net +62.5%; 5 dormancy excluded Warnings: Activation: isolated 7/8; plugin 7/8 Overfit: Low (score 0.15) Repeated-run reliability (not used by the gate): 26 paired runs (9W/16T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — migrate-static-to-wrapper (claude-sonnet-4.6)Why: Net win +70.0% (8W/1T/1L over 10 preference-eligible stimulus vote(s), sign test p=0.020), mean preference +40.0% across 10 paired run(s) — credibly better Next action: Review overfit evidence. State: Gate evidence: n=10; 8W/1T/1L; d=9; p=0.020; net +70.0% Overfit: Moderate (score 0.31) Repeated-run reliability (not used by the gate): 10 paired runs (8W/1T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — test-tagging (claude-sonnet-4.6)Why: Net win +100.0% (8W/0T/0L over 8 preference-eligible stimulus vote(s), sign test p=0.004), mean preference +55.0% across 12 paired run(s), 4 dormancy stimulus/stimuli excluded from preference — credibly better Next action: Review overfit evidence. State: Gate evidence: n=8; 8W/0T/0L; d=8; p=0.004; net +100.0%; 4 dormancy excluded Overfit: High (score 0.56) Repeated-run reliability (not used by the gate): 12 paired runs (10W/1T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. Routine passing details for 4 results are in Full Results. 🔍 Full Results - all metrics and investigation details
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62ad695-3f5f-482c-a4e6-6b47c654e366
|
/evaluate cf3a78e |
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Two SKILL.md files introduce fallback shell-reader guidance without explicitly failing closed for content-exclusion/policy/workspace-boundary errors, which could encourage policy-violating behavior.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Lite
Findings: 1
New issues introduced by this change (1)
| Severity | Finding |
|---|---|
|
plugins/dotnet-test/skills/detect-static-dependencies/SKILL.md — The fallback guidance suggests using shell readers/grep after a file read fails, but it doesn’t… |
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
plugins/dotnet-test/skills/coverage-analysis/SKILL.md:90
- This section recommends falling back to shell text/XML readers when the normal viewer fails, but it doesn’t state that this must not be used to bypass content-exclusion, permission/policy, or workspace-boundary errors. Please make the fail-closed behavior explicit so the skill doesn’t encourage policy-violating fallback reads.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62ad695-3f5f-482c-a4e6-6b47c654e366
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62ad695-3f5f-482c-a4e6-6b47c654e366
There was a problem hiding this comment.
Copilot review overview
🟢 Approval recommended
The changes are cohesive, low-risk (docs/evals/fixtures), and I did not find schema, consistency, or broken-reference issues in the modified regions.
Review tier: Lite
Findings: None
Issues resolved since last review (1)
| Severity | Finding |
|---|---|
|
plugins/dotnet-test/skills/detect-static-dependencies/SKILL.md — The fallback guidance suggests using shell readers/grep after a file read fails, but it doesn’t… View resolved comment |
|
/evaluate 53310db |
There was a problem hiding this comment.
Copilot review overview
🟢 Approval recommended
The changes are internally consistent, remove stale references, and add deterministic validation to evals without introducing obvious schema or build issues in the updated fixtures/manifests.
Review tier: Lite
Findings: None
📊 Skill Evaluation Results10 model/skill results across 5 skills and 2 models — ✅ 4 improved, ➖ 3 not proven improved, Measurement identity: evaluated commit Measurement health: 10 expected / 10 observed / 10 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots. Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven. A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of
ℹ️ How to read this report
⛔ Activation contract failed — coverage-analysis (gpt-5.6-luna)Why: Net win +75.0% (7W/0T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.035), mean preference +15.4% across 26 paired run(s), 5 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill) Next action: Narrow skill routing so the listed off-target scenarios stay dormant. State: Gate evidence: n=8; 7W/0T/1L; d=8; p=0.035; net +75.0%; 5 dormancy excluded Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 7/8; plugin 7/8 Overfit: Low (score 0.15) Repeated-run reliability (not used by the gate): 26 paired runs (13W/10T/3L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ⛔ Activation contract failed — detect-static-dependencies (claude-sonnet-4.6)Why: Net win +100.0% (7W/0T/0L over 7 preference-eligible stimulus vote(s), sign test p=0.008), mean preference +45.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill) Next action: Narrow skill routing so the listed off-target scenarios stay dormant. State: Gate evidence: n=7; 7W/0T/0L; d=7; p=0.008; net +100.0%; 1 dormancy excluded Warnings: Dormancy contract: 1 unexpected activation(s) Overfit: Low (score 0.19) Repeated-run reliability (not used by the gate): 8 paired runs (7W/0T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ⛔ Activation contract failed — test-anti-patterns (gpt-5.6-luna)Why: Net win +50.0% (5W/2T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.109), mean preference +17.8% across 9 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill) Next action: Narrow skill routing so the listed off-target scenarios stay dormant. State: Gate evidence: n=8; 5W/2T/1L; d=6; p=0.109; net +50.0%; 1 dormancy excluded Warnings: Dormancy contract: 1 unexpected activation(s) Overfit: Low (score 0.09) Repeated-run reliability (not used by the gate): 9 paired runs (5W/3T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — coverage-analysis (claude-sonnet-4.6)Why: Net win +50.0% (5W/2T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.109), mean preference +16.9% across 26 paired run(s), 5 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.109 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=8; 5W/2T/1L; d=6; p=0.109; net +50.0%; 5 dormancy excluded Overfit: Moderate (score 0.39) Repeated-run reliability (not used by the gate): 26 paired runs (13W/8T/5L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — detect-static-dependencies (gpt-5.6-luna)Why: Net win +57.1% (5W/1T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.109), mean preference +15.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.109 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=7; 5W/1T/1L; d=6; p=0.109; net +57.1%; 1 dormancy excluded Overfit: Low (score 0.17) Repeated-run reliability (not used by the gate): 8 paired runs (5W/1T/2L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — test-anti-patterns (claude-sonnet-4.6)Why: Net win +62.5% (6W/1T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +28.9% across 9 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.063 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=8; 6W/1T/1L; d=7; p=0.063; net +62.5%; 1 dormancy excluded Overfit: Low (score 0.18) Repeated-run reliability (not used by the gate): 9 paired runs (6W/2T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — migrate-static-to-wrapper (claude-sonnet-4.6)Why: Net win +80.0% (8W/2T/0L over 10 preference-eligible stimulus vote(s), sign test p=0.004), mean preference +44.0% across 10 paired run(s) — credibly better Next action: Review overfit evidence. State: Gate evidence: n=10; 8W/2T/0L; d=8; p=0.004; net +80.0% Overfit: Moderate (score 0.29) Repeated-run reliability (not used by the gate): 10 paired runs (8W/2T/0L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — test-tagging (claude-sonnet-4.6)Why: Net win +100.0% (8W/0T/0L over 8 preference-eligible stimulus vote(s), sign test p=0.004), mean preference +48.3% across 12 paired run(s), 4 dormancy stimulus/stimuli excluded from preference — credibly better Next action: Review overfit evidence. State: Gate evidence: n=8; 8W/0T/0L; d=8; p=0.004; net +100.0%; 4 dormancy excluded Overfit: High (score 0.52) Repeated-run reliability (not used by the gate): 12 paired runs (8W/3T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. Routine passing details for 2 results are in Full Results. 🔍 Full Results - all metrics and investigation details
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
📊 Skill Evaluation Results10 model/skill results across 5 skills and 2 models — ✅ 5 improved, ➖ 4 not proven improved, Measurement identity: evaluated commit Measurement health: 10 expected / 10 observed / 10 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots. Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven. A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of
ℹ️ How to read this report
⛔ Activation contract failed — detect-static-dependencies (claude-sonnet-4.6)Why: Net win +100.0% (7W/0T/0L over 7 preference-eligible stimulus vote(s), sign test p=0.008), mean preference +57.5% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill) Next action: Narrow skill routing so the listed off-target scenarios stay dormant. State: Gate evidence: n=7; 7W/0T/0L; d=7; p=0.008; net +100.0%; 1 dormancy excluded Warnings: Dormancy contract: 1 unexpected activation(s) Overfit: Moderate (score 0.46) Repeated-run reliability (not used by the gate): 8 paired runs (7W/1T/0L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — coverage-analysis (claude-sonnet-4.6)Why: Net win +50.0% (6W/0T/2L over 8 preference-eligible stimulus vote(s), sign test p=0.145), mean preference +24.6% across 26 paired run(s), 5 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.145 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=8; 6W/0T/2L; d=8; p=0.145; net +50.0%; 5 dormancy excluded Overfit: Moderate (score 0.29) Repeated-run reliability (not used by the gate): 26 paired runs (14W/8T/4L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — coverage-analysis (gpt-5.6-luna)Why: Net win +25.0% (4W/2T/2L over 8 preference-eligible stimulus vote(s), sign test p=0.344), mean preference +9.2% across 26 paired run(s), 5 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.344 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=8; 4W/2T/2L; d=6; p=0.344; net +25.0%; 5 dormancy excluded Warnings: Activation: isolated 7/8; plugin 7/8 Overfit: Low (score 0.14) Repeated-run reliability (not used by the gate): 26 paired runs (9W/14T/3L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — test-anti-patterns (claude-sonnet-4.6)Why: Net win +50.0% (5W/2T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.109), mean preference +26.7% across 9 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.109 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=8; 5W/2T/1L; d=6; p=0.109; net +50.0%; 1 dormancy excluded Overfit: Moderate (score 0.29) Repeated-run reliability (not used by the gate): 9 paired runs (5W/2T/2L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — test-anti-patterns (gpt-5.6-luna)Why: Net win +62.5% (6W/1T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +28.9% across 9 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.063 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=8; 6W/1T/1L; d=7; p=0.063; net +62.5%; 1 dormancy excluded Overfit: Moderate (score 0.24) Repeated-run reliability (not used by the gate): 9 paired runs (6W/2T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — detect-static-dependencies (gpt-5.6-luna)Why: Net win +71.4% (5W/2T/0L over 7 preference-eligible stimulus vote(s), sign test p=0.031), mean preference +25.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — credibly better Next action: Fix activation gaps. State: Gate evidence: n=7; 5W/2T/0L; d=5; p=0.031; net +71.4%; 1 dormancy excluded Warnings: Activation: isolated 7/7; plugin 6/7 Overfit: Low (score 0.14) Repeated-run reliability (not used by the gate): 8 paired runs (5W/3T/0L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — migrate-static-to-wrapper (claude-sonnet-4.6)Why: Net win +80.0% (9W/0T/1L over 10 preference-eligible stimulus vote(s), sign test p=0.011), mean preference +44.0% across 10 paired run(s) — credibly better Next action: Review overfit evidence. State: Gate evidence: n=10; 9W/0T/1L; d=10; p=0.011; net +80.0% Overfit: Moderate (score 0.31) Repeated-run reliability (not used by the gate): 10 paired runs (9W/0T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — test-tagging (claude-sonnet-4.6)Why: Net win +100.0% (8W/0T/0L over 8 preference-eligible stimulus vote(s), sign test p=0.004), mean preference +55.0% across 12 paired run(s), 4 dormancy stimulus/stimuli excluded from preference — credibly better Next action: Review overfit evidence. State: Gate evidence: n=8; 8W/0T/0L; d=8; p=0.004; net +100.0%; 4 dormancy excluded Overfit: Moderate (score 0.38) Repeated-run reliability (not used by the gate): 12 paired runs (10W/1T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. Routine passing details for 2 results are in Full Results. 🔍 Full Results - all metrics and investigation details
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
📊 Skill Evaluation Results10 model/skill results across 5 skills and 2 models — ✅ 8 improved, ➖ 1 not proven improved, Measurement identity: evaluated commit Measurement health: 10 expected / 10 observed / 10 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots. Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven. A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of
ℹ️ How to read this report
⛔ Activation contract failed — detect-static-dependencies (claude-sonnet-4.6)Why: Net win +71.4% (6W/0T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +35.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill) Next action: Narrow skill routing so the listed off-target scenarios stay dormant. State: Gate evidence: n=7; 6W/0T/1L; d=7; p=0.063; net +71.4%; 1 dormancy excluded Warnings: Dormancy contract: 1 unexpected activation(s) Overfit: Moderate (score 0.46) Repeated-run reliability (not used by the gate): 8 paired runs (6W/0T/2L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — test-anti-patterns (gpt-5.6-luna)Why: Net win +37.5% (4W/3T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.188), mean preference +13.3% across 9 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.188 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=8; 4W/3T/1L; d=5; p=0.188; net +37.5%; 1 dormancy excluded Overfit: Low (score 0.08) Repeated-run reliability (not used by the gate): 9 paired runs (4W/4T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — coverage-analysis (claude-sonnet-4.6)Why: Net win +75.0% (7W/0T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.035), mean preference +26.2% across 26 paired run(s), 5 dormancy stimulus/stimuli excluded from preference — credibly better Next action: Review overfit evidence. State: Gate evidence: n=8; 7W/0T/1L; d=8; p=0.035; net +75.0%; 5 dormancy excluded Overfit: Moderate (score 0.39) Repeated-run reliability (not used by the gate): 26 paired runs (16W/5T/5L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — coverage-analysis (gpt-5.6-luna)Why: Net win +75.0% (6W/2T/0L over 8 preference-eligible stimulus vote(s), sign test p=0.016), mean preference +13.8% across 26 paired run(s), 5 dormancy stimulus/stimuli excluded from preference — credibly better Next action: Fix activation gaps. State: Gate evidence: n=8; 6W/2T/0L; d=6; p=0.016; net +75.0%; 5 dormancy excluded Warnings: Activation: isolated 7/8; plugin 7/8 Overfit: Low (score 0.14) Repeated-run reliability (not used by the gate): 26 paired runs (11W/13T/2L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — migrate-static-to-wrapper (claude-sonnet-4.6)Why: Net win +80.0% (8W/2T/0L over 10 preference-eligible stimulus vote(s), sign test p=0.004), mean preference +44.0% across 10 paired run(s) — credibly better Next action: Review overfit evidence. State: Gate evidence: n=10; 8W/2T/0L; d=8; p=0.004; net +80.0% Overfit: Moderate (score 0.32) Repeated-run reliability (not used by the gate): 10 paired runs (8W/2T/0L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — test-anti-patterns (claude-sonnet-4.6)Why: Net win +87.5% (7W/1T/0L over 8 preference-eligible stimulus vote(s), sign test p=0.008), mean preference +35.6% across 9 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — credibly better Next action: Review overfit evidence. State: Gate evidence: n=8; 7W/1T/0L; d=7; p=0.008; net +87.5%; 1 dormancy excluded Overfit: Moderate (score 0.25) Repeated-run reliability (not used by the gate): 9 paired runs (8W/1T/0L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — test-tagging (claude-sonnet-4.6)Why: Net win +100.0% (8W/0T/0L over 8 preference-eligible stimulus vote(s), sign test p=0.004), mean preference +45.0% across 12 paired run(s), 4 dormancy stimulus/stimuli excluded from preference — credibly better Next action: Review overfit evidence. State: Gate evidence: n=8; 8W/0T/0L; d=8; p=0.004; net +100.0%; 4 dormancy excluded Overfit: High (score 0.56) Repeated-run reliability (not used by the gate): 12 paired runs (8W/2T/2L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — test-tagging (gpt-5.6-luna)Why: Net win +75.0% (6W/2T/0L over 8 preference-eligible stimulus vote(s), sign test p=0.016), mean preference +40.0% across 12 paired run(s), 4 dormancy stimulus/stimuli excluded from preference — credibly better Next action: Review overfit evidence. State: Gate evidence: n=8; 6W/2T/0L; d=6; p=0.016; net +75.0%; 4 dormancy excluded Overfit: Moderate (score 0.22) Repeated-run reliability (not used by the gate): 12 paired runs (6W/6T/0L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. Routine passing details for 2 results are in Full Results. 🔍 Full Results - all metrics and investigation details
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62ad695-3f5f-482c-a4e6-6b47c654e366
|
/evaluate 243bac5 |
There was a problem hiding this comment.
Copilot review overview
🟢 Approval recommended
The changes are coherent, low-risk (docs/evals/fixtures), and include consistent manifest version synchronization while strengthening deterministic validation in targeted scenarios.
Review tier: Lite
Findings: None
|
❌ Evaluation did not complete successfully (the evaluate job reported 8 partial result file(s) were preserved for diagnosis but were not consolidated because the full matrix did not complete. |
📊 Skill Evaluation Results10 model/skill results across 5 skills and 2 models — ✅ 4 improved, ➖ 4 not proven improved, Measurement identity: evaluated commit Measurement health: 10 expected / 10 observed / 10 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots. Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven. A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of
ℹ️ How to read this report
⛔ Activation contract failed — coverage-analysis (gpt-5.6-luna)Why: Net win +50.0% (6W/0T/2L over 8 preference-eligible stimulus vote(s), sign test p=0.145), mean preference +12.3% across 26 paired run(s), 5 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill) Next action: Narrow skill routing so the listed off-target scenarios stay dormant. State: Gate evidence: n=8; 6W/0T/2L; d=8; p=0.145; net +50.0%; 5 dormancy excluded Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 7/8; plugin 7/8 Overfit: Low (score 0.13) Repeated-run reliability (not used by the gate): 26 paired runs (11W/12T/3L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ⛔ Activation contract failed — detect-static-dependencies (claude-sonnet-4.6)Why: Net win +71.4% (6W/0T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +27.5% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill) Next action: Narrow skill routing so the listed off-target scenarios stay dormant. State: Gate evidence: n=7; 6W/0T/1L; d=7; p=0.063; net +71.4%; 1 dormancy excluded Warnings: Dormancy contract: 1 unexpected activation(s) Overfit: Moderate (score 0.24) Repeated-run reliability (not used by the gate): 8 paired runs (6W/0T/2L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — coverage-analysis (claude-sonnet-4.6)Why: Net win +37.5% (5W/1T/2L over 8 preference-eligible stimulus vote(s), sign test p=0.227), mean preference +23.1% across 26 paired run(s), 5 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.227 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=8; 5W/1T/2L; d=7; p=0.227; net +37.5%; 5 dormancy excluded Overfit: Moderate (score 0.30) Repeated-run reliability (not used by the gate): 26 paired runs (15W/5T/6L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — detect-static-dependencies (gpt-5.6-luna)Why: Net win +57.1% (5W/1T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.109), mean preference +22.5% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.109 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=7; 5W/1T/1L; d=6; p=0.109; net +57.1%; 1 dormancy excluded Overfit: Low (score 0.15) Repeated-run reliability (not used by the gate): 8 paired runs (5W/1T/2L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — test-anti-patterns (claude-sonnet-4.6)Why: Net win +50.0% (5W/2T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.109), mean preference +28.9% across 9 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.109 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=8; 5W/2T/1L; d=6; p=0.109; net +50.0%; 1 dormancy excluded Overfit: Moderate (score 0.27) Repeated-run reliability (not used by the gate): 9 paired runs (6W/2T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — test-tagging (gpt-5.6-luna)Why: Net win +62.5% (6W/1T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +40.0% across 12 paired run(s), 4 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.063 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=8; 6W/1T/1L; d=7; p=0.063; net +62.5%; 4 dormancy excluded Overfit: Low (score 0.16) Repeated-run reliability (not used by the gate): 12 paired runs (7W/4T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — migrate-static-to-wrapper (claude-sonnet-4.6)Why: Net win +80.0% (8W/2T/0L over 10 preference-eligible stimulus vote(s), sign test p=0.004), mean preference +50.0% across 10 paired run(s) — credibly better Next action: Review overfit evidence. State: Gate evidence: n=10; 8W/2T/0L; d=8; p=0.004; net +80.0% Overfit: Moderate (score 0.28) Repeated-run reliability (not used by the gate): 10 paired runs (8W/2T/0L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — test-tagging (claude-sonnet-4.6)Why: Net win +100.0% (8W/0T/0L over 8 preference-eligible stimulus vote(s), sign test p=0.004), mean preference +65.0% across 12 paired run(s), 4 dormancy stimulus/stimuli excluded from preference — credibly better Next action: Review overfit evidence. State: Gate evidence: n=8; 8W/0T/0L; d=8; p=0.004; net +100.0%; 4 dormancy excluded Overfit: Moderate (score 0.34) Repeated-run reliability (not used by the gate): 12 paired runs (12W/0T/0L). Routine passing details for 2 results are in Full Results. 🔍 Full Results - all metrics and investigation details
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
📊 Skill Evaluation Results10 model/skill results across 5 skills and 2 models — ✅ 6 improved, ➖ 2 not proven improved, Measurement identity: evaluated commit Measurement health: 10 expected / 10 observed / 10 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots. Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven. A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of
ℹ️ How to read this report
⛔ Activation contract failed — coverage-analysis (gpt-5.6-luna)Why: Net win +75.0% (6W/2T/0L over 8 preference-eligible stimulus vote(s), sign test p=0.016), mean preference +23.8% across 26 paired run(s), 5 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill) Next action: Narrow skill routing so the listed off-target scenarios stay dormant. State: Gate evidence: n=8; 6W/2T/0L; d=6; p=0.016; net +75.0%; 5 dormancy excluded Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 7/8; plugin 7/8 Overfit: Low (score 0.15) Repeated-run reliability (not used by the gate): 26 paired runs (15W/10T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ⛔ Activation contract failed — detect-static-dependencies (claude-sonnet-4.6)Why: Net win +100.0% (7W/0T/0L over 7 preference-eligible stimulus vote(s), sign test p=0.008), mean preference +50.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill) Next action: Narrow skill routing so the listed off-target scenarios stay dormant. State: Gate evidence: n=7; 7W/0T/0L; d=7; p=0.008; net +100.0%; 1 dormancy excluded Warnings: Dormancy contract: 1 unexpected activation(s) Overfit: Moderate (score 0.34) Repeated-run reliability (not used by the gate): 8 paired runs (7W/1T/0L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — coverage-analysis (claude-sonnet-4.6)Why: Net win +37.5% (5W/1T/2L over 8 preference-eligible stimulus vote(s), sign test p=0.227), mean preference +14.6% across 26 paired run(s), 5 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.227 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=8; 5W/1T/2L; d=7; p=0.227; net +37.5%; 5 dormancy excluded Overfit: Moderate (score 0.36) Repeated-run reliability (not used by the gate): 26 paired runs (11W/9T/6L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — test-tagging (gpt-5.6-luna)Why: Net win +62.5% (6W/1T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +35.0% across 12 paired run(s), 4 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.063 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=8; 6W/1T/1L; d=7; p=0.063; net +62.5%; 4 dormancy excluded Overfit: Low (score 0.14) Repeated-run reliability (not used by the gate): 12 paired runs (7W/4T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — detect-static-dependencies (gpt-5.6-luna)Why: Net win +71.4% (5W/2T/0L over 7 preference-eligible stimulus vote(s), sign test p=0.031), mean preference +27.5% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — credibly better Next action: Fix activation gaps. State: Gate evidence: n=7; 5W/2T/0L; d=5; p=0.031; net +71.4%; 1 dormancy excluded Warnings: Activation: isolated 6/7; plugin 7/7 Overfit: Low (score 0.15) Repeated-run reliability (not used by the gate): 8 paired runs (5W/2T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — migrate-static-to-wrapper (claude-sonnet-4.6)Why: Net win +80.0% (9W/0T/1L over 10 preference-eligible stimulus vote(s), sign test p=0.011), mean preference +56.0% across 10 paired run(s) — credibly better Next action: Review overfit evidence. State: Gate evidence: n=10; 9W/0T/1L; d=10; p=0.011; net +80.0% Overfit: Moderate (score 0.38) Repeated-run reliability (not used by the gate): 10 paired runs (9W/0T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — test-anti-patterns (claude-sonnet-4.6)Why: Net win +75.0% (7W/0T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.035), mean preference +44.4% across 9 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — credibly better Next action: Review overfit evidence. State: Gate evidence: n=8; 7W/0T/1L; d=8; p=0.035; net +75.0%; 1 dormancy excluded Overfit: Moderate (score 0.31) Repeated-run reliability (not used by the gate): 9 paired runs (8W/0T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — test-tagging (claude-sonnet-4.6)Why: Net win +100.0% (8W/0T/0L over 8 preference-eligible stimulus vote(s), sign test p=0.004), mean preference +48.3% across 12 paired run(s), 4 dormancy stimulus/stimuli excluded from preference — credibly better Next action: Review overfit evidence. State: Gate evidence: n=8; 8W/0T/0L; d=8; p=0.004; net +100.0%; 4 dormancy excluded Overfit: High (score 0.59) Repeated-run reliability (not used by the gate): 12 paired runs (9W/1T/2L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. Routine passing details for 2 results are in Full Results. 🔍 Full Results - all metrics and investigation details
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62ad695-3f5f-482c-a4e6-6b47c654e366
|
/evaluate efbf069 |
There was a problem hiding this comment.
Copilot review overview
🟢 Approval recommended
The changes are consistent, internally coherent (including fixture/supporting updates), and I found no concrete defects or dangling references in the modified evals/skill docs.
Review tier: Lite
Findings: None
📊 Skill Evaluation Results10 model/skill results across 5 skills and 2 models — ✅ 8 improved, ➖ 2 not proven improved, Measurement identity: evaluated commit Measurement health: 10 expected / 10 observed / 10 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots. Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven. A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of
ℹ️ How to read this report
➖ Not proven improved — coverage-analysis (claude-sonnet-4.6)Why: Net win +50.0% (6W/0T/2L over 8 preference-eligible stimulus vote(s), sign test p=0.145), mean preference +20.8% across 24 paired run(s), 4 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.145 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=8; 6W/0T/2L; d=8; p=0.145; net +50.0%; 4 dormancy excluded Overfit: Moderate (score 0.29) Repeated-run reliability (not used by the gate): 24 paired runs (13W/6T/5L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — test-anti-patterns (gpt-5.6-luna)Why: Net win +62.5% (6W/1T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +26.7% across 9 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.063 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=8; 6W/1T/1L; d=7; p=0.063; net +62.5%; 1 dormancy excluded Overfit: Moderate (score 0.24) Repeated-run reliability (not used by the gate): 9 paired runs (7W/1T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — coverage-analysis (gpt-5.6-luna)Why: Net win +75.0% (6W/2T/0L over 8 preference-eligible stimulus vote(s), sign test p=0.016), mean preference +18.3% across 24 paired run(s), 4 dormancy stimulus/stimuli excluded from preference — credibly better Next action: Fix activation gaps. State: Gate evidence: n=8; 6W/2T/0L; d=6; p=0.016; net +75.0%; 4 dormancy excluded Warnings: Activation: isolated 7/8; plugin 7/8 Overfit: Low (score 0.13) Repeated-run reliability (not used by the gate): 24 paired runs (12W/8T/4L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — detect-static-dependencies (claude-sonnet-4.6)Why: Net win +100.0% (7W/0T/0L over 7 preference-eligible stimulus vote(s), sign test p=0.008), mean preference +45.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — credibly better Next action: Review overfit evidence. State: Gate evidence: n=7; 7W/0T/0L; d=7; p=0.008; net +100.0%; 1 dormancy excluded Overfit: Moderate (score 0.26) Repeated-run reliability (not used by the gate): 8 paired runs (7W/0T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — migrate-static-to-wrapper (claude-sonnet-4.6)Why: Net win +90.0% (9W/1T/0L over 10 preference-eligible stimulus vote(s), sign test p=0.002), mean preference +48.0% across 10 paired run(s) — credibly better Next action: Review overfit evidence. State: Gate evidence: n=10; 9W/1T/0L; d=9; p=0.002; net +90.0% Overfit: Moderate (score 0.30) Repeated-run reliability (not used by the gate): 10 paired runs (9W/1T/0L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — test-anti-patterns (claude-sonnet-4.6)Why: Net win +87.5% (7W/1T/0L over 8 preference-eligible stimulus vote(s), sign test p=0.008), mean preference +37.8% across 9 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — credibly better Next action: Review overfit evidence. State: Gate evidence: n=8; 7W/1T/0L; d=7; p=0.008; net +87.5%; 1 dormancy excluded Overfit: Moderate (score 0.42) Repeated-run reliability (not used by the gate): 9 paired runs (7W/2T/0L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — test-tagging (claude-sonnet-4.6)Why: Net win +100.0% (8W/0T/0L over 8 preference-eligible stimulus vote(s), sign test p=0.004), mean preference +55.0% across 12 paired run(s), 4 dormancy stimulus/stimuli excluded from preference — credibly better Next action: Review overfit evidence. State: Gate evidence: n=8; 8W/0T/0L; d=8; p=0.004; net +100.0%; 4 dormancy excluded Overfit: Moderate (score 0.41) Repeated-run reliability (not used by the gate): 12 paired runs (10W/1T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. Routine passing details for 3 results are in Full Results. 🔍 Full Results - all metrics and investigation details
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
Summary
Retained GPT-family evaluation evidence showed that several
dotnet-testskills remained tie-limited or lost on specific correctness details after the broader optimization work. This change targets those measured gaps without padding repeated runs or rewriting already-passing skills.DateTime.Nowsemantics, exact UTC-call scope, and a successful build.coverage-analysisrouting text to recover model-menu capacity.run-tests, preserving repository overlays and platform-specific runner selection while keepingcoverage-analysisfocused on interpretation.Related issue
N/A
Validation
python eng/eval-quality/check_eval_quality.py- passed with existing repository warnings only.skill-validator check --plugin .\plugins\dotnet-test- passed for 22 skills and 10 agents.detect-static-dependenciesC# fixtures build successfully.eng\version\Test-Sync-PluginVersions.ps1- passed 65 assertions.git diff --check- passed.coverage-analysisanddetect-static-dependencies; later transcript-driven changes target the remaining migration, tagging, and audit losses caused by recoverable workspace tool failures.Checklist
eng/known-domains.txtfor any new external domains referenced by skill content.