Stabilize skill menu and evaluation retries

[AbhitejJohn]

Summary

• reduce the dotnet-test rendered skill-menu footprint from 15,063 to 14,598 characters by tightening only the test-tagging routing description
• retry transient session.idle executor timeouts once for required baseline and isolated-skilled arms
• preserve all successful first-attempt evidence and merge only unique, eval-scoped retry records with matching stable shardKey values
• retain the existing fail-closed measurementInvalidEvalCount === 0 gate for persistent, systemic, non-timeout, missing, duplicate, or ambiguous comparison evidence

Root causes

PR #1051 expanded several routing descriptions and moved dotnet-test from 14,421 rendered characters to 15,063, above the unchanged Copilot runtime limit of 15,000. PR #1059 did not cause that regression.

In full-corpus runs 32989467349 and 32992064149, every expected result file was written, but one required skilled-arm trial timed out waiting for session.idle. The paired baseline record then had no treatment match, so the adapter correctly classified the eval as unmatched_trajectories and measurement-invalid. Exact-commit retries moved the timeout to a different plugin/model job, which showed a transient executor failure rather than dashboard or corpus accounting failure.

Stability behavior

Recovery is narrow and bounded:

• only baseline and isolated-skilled session.idle timeout records qualify
• each affected eval/variant group is rerun once, with at most three groups per matrix leg
• successful first-attempt records are never replaced
• retry pairing requires a unique evalFile and shardKey on both attempts
• merged JSONL is replaced atomically
• retry watchdog or command failure falls through to adaptation, where unresolved evidence still fails closed
• the optional whole-plugin telemetry arm remains report-only and is not retried

Validation

• full skill-validator check: 98 skills, 16 agents, 16 plugins passed
• dotnet-test validator check passed at 14,598 / 15,000 rendered characters
• 57 Vally adapter and fault-injection tests passed
• 19 evaluation workflow behavior tests passed
• pinned actionlint 1.7.7 passed for .github/workflows/evaluation-run.yml
• multi-model review completed with Claude, Gemini, GPT, and Grok reviewers; findings were reconciled and no blockers remain

[github-actions]

Skill Coverage Report

	Plugin	Skill	Covered	Coverage
✅	dotnet-test	test-tagging	27/29	93.1%

Uncovered: dotnet-test/test-tagging

• [CodePattern] [negative] (line 211)
• [CodePattern] [boundary] (line 211)

[Copilot]

Pull request overview

This pull request stabilizes evaluation runs by adding a narrowly-scoped retry for transient Vally executor session.idle timeouts (baseline + isolated-skilled only) while keeping the measurement gate fail-closed, and reduces the dotnet-test skill-menu footprint by tightening the test-tagging routing description.

Changes:

• Tighten plugins/dotnet-test/skills/test-tagging/SKILL.md description to reduce rendered menu size under the 15,000 character runtime limit.
• Add eng/vally-adapter/retry-executor-timeouts.mjs (+ tests) to rerun only affected eval/variant groups once and merge only recovered slots.
• Wire the recovery step into .github/workflows/evaluation-run.yml and update investigation docs/tests to reflect the new retry behavior.

Show a summary per file

File	Description
plugins/dotnet-test/skills/test-tagging/SKILL.md	Shortens routing description to reduce skill-menu footprint.
eng/vally-adapter/retry-executor-timeouts.mjs	Implements bounded retry + atomic merge of recovered executor-timeout records.
eng/vally-adapter/retry-executor-timeouts.test.mjs	Adds unit + CLI-level coverage for retry grouping and merge behavior.
eng/vally-adapter/README.md	Documents the new executor-timeout retry step in the pipeline.
eng/vally-adapter/InvestigatingResults.md	Adds guidance for diagnosing executor-timeout retries and artifacts.
eng/skill-validator/src/docs/InvestigatingResults.md	Notes the new Vally retry behavior in the legacy investigation doc header.
eng/evaluation/test_token_failover.py	Updates workflow-behavior assertions to require the retry step to run before adaptation.
.github/workflows/evaluation-run.yml	Adds the retry step to rerun/merge transient executor timeouts before adaptation.

Review details

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

• Files reviewed: 8/8 changed files
• Comments generated: 2
• Review effort level: Lite

[github-actions]

👋 @AbhitejJohn — this PR has 2 unresolved review thread(s). When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the no-stale label to silence further pings.)

[AbhitejJohn]

@Evangelink, could you please review this stability change when you have time?

[Copilot]

Review details

• Files reviewed: 9/9 changed files
• Comments generated: 0 new
• Review effort level: Lite

[github-actions]

📊 Skill Evaluation Results

2 model/skill results across 1 skill and 2 models — ✅ 1 improved, ➖ 1 not proven improved, ⚠️ 0 invalid or underpowered, 📉 0 preference losses (report only).

Measurement identity: evaluated commit 2ec7b18a3784fb1ae76cca7e3c5c681495f1f819; 2 judge models.

Measurement health: 2 expected / 2 observed / 2 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.

A result passes only when the aggregate net win across distinct-stimulus votes is at least 20% and an exact one-sided sign-test result of p ≤ 0.05. Repeated runs measure reliability only.

Skill	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
test-tagging	claude-sonnet-4.6	✅ Improved	n=12; 9W/2T/1L; d=10; p=0.011; net +66.7%	🟡 0.48	—	Review overfit evidence.
test-tagging	gpt-5.6-luna	➖ Not proven improved	n=12; 6W/5T/1L; d=7; p=0.063; net +41.7%	✅ 0.15	—	Inspect tied or lost stimuli and fix inconsistent skill behavior.

ℹ️ How to read this report

• ✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
• ➖ Not proven improved — the result is valid but did not pass both gates. This is not automatically a regression.
• ⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the skill.
• 📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
• Gate evidence — n distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, and net win. The p value applies to one model/skill result; no matrix-wide multiple-comparison correction is applied.
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
• Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
• Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

➖ Not proven improved — test-tagging (gpt-5.6-luna)

Why: Net win +41.7% (6W/5T/1L over 12 stimulus vote(s), sign test p=0.063), mean preference +31.7% across 12 paired run(s) — not credible (sign test p=0.063 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=12; 6W/5T/1L; d=7; p=0.063; net +41.7%

Overfit: Low (score 0.15)

Repeated-run reliability (not used by the gate): 12 paired runs (6W/5T/1L).

Weak or warning scenarios:

Scenario	Net win	Δ Pref	Runs (W/T/L)
= Audit test distribution without modifying files	+0.0%	+0.0%	0/1/0
= Classify Go tests where no trait attribute mechanism exists	+0.0%	+0.0%	0/1/0
= Decline request to write new tests	+0.0%	+0.0%	0/1/0
= Stay dormant for behavioral gap analysis	+0.0%	+0.0%	0/1/0
= Stay dormant for line and branch coverage diagnosis	+0.0%	+0.0%	0/1/0
▼ Stay dormant for one-member CRAP analysis	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

• Audit test distribution without modifying files: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

✅ Improved — test-tagging (claude-sonnet-4.6)

Why: Net win +66.7% (9W/2T/1L over 12 stimulus vote(s), sign test p=0.011), mean preference +46.7% across 12 paired run(s) — credibly better

Next action: Review overfit evidence.

State: VALID_PASS (credible_preference_improvement)

Gate evidence: n=12; 9W/2T/1L; d=10; p=0.011; net +66.7%

Overfit: Moderate (score 0.48)

Repeated-run reliability (not used by the gate): 12 paired runs (9W/2T/1L).

Weak or warning scenarios:

Scenario	Net win	Δ Pref	Runs (W/T/L)
▼ Decline request to write new tests	-100.0%	-40.0%	0/0/1
= Stay dormant for behavioral gap analysis	+0.0%	+0.0%	0/1/0
= Stay dormant for one-member CRAP analysis	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Decline request to write new tests: A is substantially more grounded in the actual supplied PaymentGateway implementation and candidly scopes the missing webhook capability. B's broader-looking suite is undermined by dependencies on nonexistent members and unprovided behavioral conventions, likely making it unco...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔍 Full Results - all metrics and investigation details

To investigate non-passing or warning results, paste this to your AI coding agent:

For PR 1072 in dotnet/skills, download eval artifacts with gh run download 33051990918 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/2ec7b18a3784fb1ae76cca7e3c5c681495f1f819/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[Copilot]

Review details

• Files reviewed: 9/9 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

plugins/dotnet-test/skills/test-tagging/SKILL.md:9

• The frontmatter description is what the runtime uses for skill routing, and repo guidance says sibling skills should be partitioned with explicit matching exclusions. This shortened DO NOT USE clause no longer names the owning sibling skills (e.g., test-anti-patterns, coverage-analysis, run-tests, crap-score, test-gap-analysis), which can make routing ambiguous even though the body text is detailed.

  Classifies existing tests by standard traits and reports their distribution.
  MUST USE to categorize/tag/label tests, compare happy vs error paths, audit
  the test mix, or describe coverage shape by test type. Read bodies when names
  mislead. Apply canonical attributes; otherwise report only. DO NOT USE for
  test-quality audits, executed coverage or CRAP, behavioral gaps, writing

• Files reviewed: 9/9 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Evangelink]

/evaluate

[github-actions]

📊 Skill Evaluation Results

2 model/skill results across 1 skill and 2 models — ✅ 1 improved, ➖ 1 not proven improved, ⚠️ 0 invalid or underpowered, 📉 0 preference losses (report only).

Measurement identity: evaluated commit bf2879ca0da40ea9bed024fd30436310e9c9fca6; 2 judge models.

Measurement health: 2 expected / 2 observed / 2 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.

A result passes only when the aggregate net win across distinct-stimulus votes is at least 20% and an exact one-sided sign-test result of p ≤ 0.05. Repeated runs measure reliability only.

Skill	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
test-tagging	claude-sonnet-4.6	✅ Improved	n=12; 9W/3T/0L; d=9; p=0.002; net +75.0%	🟡 0.38	—	Review overfit evidence.
test-tagging	gpt-5.6-luna	➖ Not proven improved	n=12; 5W/6T/1L; d=6; p=0.109; net +33.3%	✅ 0.16	—	Inspect tied or lost stimuli and fix inconsistent skill behavior.

ℹ️ How to read this report

• ✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
• ➖ Not proven improved — the result is valid but did not pass both gates. This is not automatically a regression.
• ⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the skill.
• 📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
• Gate evidence — n distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, and net win. The p value applies to one model/skill result; no matrix-wide multiple-comparison correction is applied.
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
• Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
• Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

➖ Not proven improved — test-tagging (gpt-5.6-luna)

Why: Net win +33.3% (5W/6T/1L over 12 stimulus vote(s), sign test p=0.109), mean preference +28.3% across 12 paired run(s) — not credible (sign test p=0.109 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=12; 5W/6T/1L; d=6; p=0.109; net +33.3%

Overfit: Low (score 0.16)

Repeated-run reliability (not used by the gate): 12 paired runs (5W/6T/1L).

Weak or warning scenarios:

Scenario	Net win	Δ Pref	Runs (W/T/L)
= Accurately classify NUnit tests with misleading method names	+0.0%	+0.0%	0/1/0
▼ Audit test distribution without modifying files	-100.0%	-40.0%	0/0/1
= Classify Go tests where no trait attribute mechanism exists	+0.0%	+0.0%	0/1/0
= Decline request to write new tests	+0.0%	+0.0%	0/1/0
= Stay dormant for behavioral gap analysis	+0.0%	+0.0%	0/1/0
= Stay dormant for line and branch coverage diagnosis	+0.0%	+0.0%	0/1/0
= Stay dormant for one-member CRAP analysis	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Accurately classify NUnit tests with misleading method names: Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

✅ Improved — test-tagging (claude-sonnet-4.6)

Why: Net win +75.0% (9W/3T/0L over 12 stimulus vote(s), sign test p=0.002), mean preference +60.0% across 12 paired run(s) — credibly better

Next action: Review overfit evidence.

State: VALID_PASS (credible_preference_improvement)

Gate evidence: n=12; 9W/3T/0L; d=9; p=0.002; net +75.0%

Overfit: Moderate (score 0.38)

Repeated-run reliability (not used by the gate): 12 paired runs (9W/3T/0L).

Weak or warning scenarios:

Scenario	Net win	Δ Pref	Runs (W/T/L)
= Stay dormant for behavioral gap analysis	+0.0%	+0.0%	0/1/0
= Stay dormant for line and branch coverage diagnosis	+0.0%	+0.0%	0/1/0
= Stay dormant for one-member CRAP analysis	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Stay dormant for behavioral gap analysis: Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔍 Full Results - all metrics and investigation details

To investigate non-passing or warning results, paste this to your AI coding agent:

For PR 1072 in dotnet/skills, download eval artifacts with gh run download 33068749475 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/bf2879ca0da40ea9bed024fd30436310e9c9fca6/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[github-actions]

📊 Skill Evaluation Results

2 model/skill results across 1 skill and 2 models — ✅ 2 improved, ➖ 0 not proven improved, ⚠️ 0 invalid or underpowered, 📉 0 preference losses (report only).

Measurement identity: evaluated commit bf2879ca0da40ea9bed024fd30436310e9c9fca6; 2 judge models.

Measurement health: 2 expected / 2 observed / 2 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.

A result passes only when the aggregate net win across distinct-stimulus votes is at least 20% and an exact one-sided sign-test result of p ≤ 0.05. Repeated runs measure reliability only.

Skill	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
test-tagging	claude-sonnet-4.6	✅ Improved	n=12; 11W/0T/1L; d=12; p=0.003; net +83.3%	🟡 0.44	—	Review overfit evidence.
test-tagging	gpt-5.6-luna	✅ Improved	n=12; 7W/4T/1L; d=8; p=0.035; net +50.0%	✅ 0.14	—	None.

ℹ️ How to read this report

• ✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
• ➖ Not proven improved — the result is valid but did not pass both gates. This is not automatically a regression.
• ⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the skill.
• 📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
• Gate evidence — n distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, and net win. The p value applies to one model/skill result; no matrix-wide multiple-comparison correction is applied.
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
• Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
• Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

✅ Improved — test-tagging (claude-sonnet-4.6)

Why: Net win +83.3% (11W/0T/1L over 12 stimulus vote(s), sign test p=0.003), mean preference +63.3% across 12 paired run(s) — credibly better

Next action: Review overfit evidence.

State: VALID_PASS (credible_preference_improvement)

Gate evidence: n=12; 11W/0T/1L; d=12; p=0.003; net +83.3%

Overfit: Moderate (score 0.44)

Repeated-run reliability (not used by the gate): 12 paired runs (11W/0T/1L).

Weak or warning scenarios:

Scenario	Net win	Δ Pref	Runs (W/T/L)
▼ Stay dormant for one-member CRAP analysis	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

• Stay dormant for one-member CRAP analysis: Both final calculations and practical recommendations are substantially the same and correct. A is better overall because it stays on task without the irrelevant test-tagging detour; however, the final-result quality gap is limited since B recovers and gives a useful answer.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

Routine passing details for 1 result are in Full Results.

🔍 Full Results - all metrics and investigation details

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions