Harden eval verdict correctness and failure accounting

[AbhitejJohn]

Why this is needed

The old gate could mistake three different things for skill quality:

1. Repeated runs of one task were pooled as if they were independent tasks.
2. A judge timeout or disabled organization could be counted as a skill loss or disappear as a missing result.
3. A statistically significant but tiny preference difference could pass, while aggregate completion data could mix deterministic checks with LLM grades.

This PR makes the inference unit, failure accounting, and verdict contract explicit.

Example for #986: repeated runs manufactured significance

Assume an eval has 4 distinct stimuli and runs: 3. The treatment wins all 12 paired runs.

• Old pooled calculation: 12W/0L, exact one-sided sign-test p = 1/4096, so the eval could pass.
• Correct task-level calculation: each stimulus supplies one majority-direction vote, so the record is 4W/0L, p = 1/16 = 0.0625. It cannot pass at alpha = 0.05.

The 12 runs show repeatability on four tasks. They do not create 12 independent task samples. This PR therefore makes four stimulus votes authoritative and keeps the 12 paired runs as reliability evidence.

Example for #909: one judge timeout is not a regression

Assume five comparison slots. Four judge calls succeed, but one returns Timeout after 120000ms waiting for session.idle.

This PR:

1. Identifies the failed slot by (stimulusName, trialIndex).
2. Retries the comparison once.
3. Freezes all successful first-attempt judgments.
4. Uses retry output only for the failed slot.
5. Returns INVALID_INCONCLUSIVE with a classified cause if recovery fails.
6. Requires an explicit result for every selected eval, so missing or unexpected output cannot disappear.

A disabled organization, rate limit, service failure, missing result, malformed report, or empty run now fails closed. None can become VALID_PASS, VALID_NO_CHANGE, or VALID_REGRESSION.

Example for #970: significance alone is not enough

Assume 100 distinct stimuli produce 5W/95T/0L.

• The exact sign test conditions on the five non-ties, so p = 1/32 = 0.03125.
• The practical net win is only (5 - 0) / 100 = 5%.

Statistical significance says the direction is unlikely under a 50/50 null among non-ties. It does not say the effect is large enough to matter. This PR also requires an absolute task-level net win of at least 20%, so this sparse result does not pass.

The completion hard gate is not enabled. Vally's aggregate passed value can combine weighted deterministic and LLM graders. Treating that mixed value as objective completion could manufacture a P0 regression. VALID_REGRESSION stays reserved until the harness can prove deterministic grader provenance.

Decision flow

flowchart LR
    A[Distinct stimulus] --> B[Repeated Vally runs]
    B --> C[Pair by stimulusName and trialIndex]
    C --> D{Judge slot failed?}
    D -- Yes --> E[Retry once; freeze successes]
    D -- No --> F[Run-level W/T/L]
    E --> F
    F --> G[One majority-direction vote per stimulus]
    F --> H[Reliability evidence only]
    G --> I[Exact sign test at alpha 0.05]
    G --> J[Absolute net-win floor at 20%]
    I --> K[Verdict state]
    J --> K
    H --> L[Report pass rate, retries, and flakiness]

Loading

What is implemented

Issue	State	Result
#986 — independent inference unit	Done	One majority-direction vote per distinct stimulus. Repeated runs cannot create or hide significance.
#986 — authoring floor	Done	New and expanded evals need at least five distinct stimuli. Extra runs cannot clear the floor.
#986 — current underpowered debt	Deferred	The 48 grandfathered evals below five distinct stimuli remain a separate repair project, as requested.
#909 — judge-side transient failures	Done	One targeted retry, stable slot identity, frozen successes, classified failure provenance, and fail-closed verdicts.
#909 — result-set integrity	Done	Expected, observed, missing, unexpected, invalid, and written results are reconciled exactly.
#970 — statistical gate	Done	Exact one-sided sign test at alpha = 0.05, based on distinct stimulus votes.
#970 — practical net-win floor	Done	A pass also needs abs((wins - losses) / stimulusVotes) >= 0.20.
#970 — objective completion primitive	Partial	The required deterministic provenance contract is defined and mixed aggregate completion remains telemetry-only.
#970 — objective completion hard gate	Deferred	VALID_REGRESSION stays reserved until the harness supplies trusted spec-to-result grader identity.
Coverage suites/tags	Deferred	Coverage management is separate from verdict correctness and is not mixed into this PR.

Vally guidance versus repository policy

Vally documents that:

• a stimulus is a test case;
• repeated runs measure pass rate, pass@k, pass^k, and flakiness;
• three runs are a practical CI recommendation and five to ten runs are an outer-loop recommendation;
• comparisons pair trajectories by stimulus name and trial index;
• grader taxonomy distinguishes deterministic static/complex-static graders from non-deterministic llm graders.

Vally does not prescribe a distinct-stimulus minimum, a sign-test alpha, or a practical net-win floor. Those are repository policies:

• alpha = 0.05;
• five distinct stimuli as the smallest eligibility floor at which any exact sign-test record can pass;
• a 20% practical net-win floor.

Five is not a general power target. Under a no-tie model, exact calculations show that the sign test alone reaches 80% power at about 158 discordant votes for a 60% conditional win rate, 37 for 70%, 18 for 80%, and 8 for 90%. The 20% practical floor is intentionally binding at a 60% conditional win rate: at 158 votes the combined gate passes about 52% of records and approaches 50% as the sample grows. The gate is designed to certify effects above its practical threshold. Eval authors must choose breadth from the smallest effect they need to detect; they must not treat five as "well powered."

Official references:

• How It Works
• Scoring and multi-trial metrics
• Grader taxonomy
• vally compare

Objective completion contract

A future VALID_REGRESSION hard gate must have all of these properties:

1. The eval spec declares which graders are objective completion graders.
2. Each declaration has a stable identity that maps uniquely to one result.
3. Only a frozen allowlist of Vally static and complex-static grader types is eligible.
4. Each paired arm yields pass, fail, or unknown; missing, duplicate, ambiguous, or LLM-backed evidence becomes unknown.
5. Every selected completion grader is present for both arms.
6. A hard regression requires conclusive paired objective losses plus the repository's statistical and practical thresholds.

Current Vally comparison output exposes aggregate pass transitions, but not a trusted unique mapping from each eval-spec grader declaration to each result. The PR reports those transitions but does not overstate them as objective proof.

Output and compatibility

• Result schema version 3 separates authoritative scenarioEvidence from non-gating comparisonTrialEvidence.
• stimulusVoteCount and minCredibleStimuli are the canonical fields.
• trialCount and minCredibleTrials remain compatibility aliases for current consumers.
• The branch now includes Vally CLI 0.13 from current main.
• A live Vally 0.13 raw comparison produced six required and unique (stimulusName, trialIndex) keys, with no missing index and no duplicate slot.
• Vally 0.13 all-errored comparisons, which write a report and then exit nonzero, are handled and tested.

PR result UI and trusted validator handoff

The old PR comment could show the same skill twice without naming the model, label a positive but unproven result with a red cross, say "zero objective regressions" while that gate was disabled, and call all non-passes failures. It also hid exact result accounting and judge retry health.

The new comment:

• reports model/skill results separately from unique skills;
• names the execution model, evaluated commit, and judge identity;
• distinguishes Improved, Not proven improved, Invalid / inconclusive, and Preference loss (report only);
• keeps Overfit in the main table;
• shows expected / observed / written / missing / unexpected / invalid result accounting and recovered / unresolved judge slots;
• shows gate evidence on the correct unit: distinct-stimulus n, W/T/L, discordant votes, one-result p, and aggregate net win;
• gives a cause-specific next action and limits PR details to non-passing or warning-bearing results; and
• states that the objective completion gate is not enabled instead of claiming that zero objective regressions were proven.

For example, 7W/0T/2L over nine stimuli has a strong +55.6% net win but p=0.090. It now appears as ➖ Not proven improved, with the two losing stimuli and the next action, rather than as a generic red failure.

The validator transport also no longer depends on a writable cache. The trusted prepare-validator job builds or restores one archive before PR content is checked out, uploads it as a same-run artifact, and every matrix job downloads that exact archive. The cache remains an optimization only. This removes the issue-comment path where cache save failed and every matrix job then tried to rebuild against the restricted evaluation package sources.

Validation

• 47 adapter and report tests.
• 25 eval-quality self-tests.
• 15 evaluation workflow behavior tests.
• The full 99-spec quality scan.
• Workflow structure validation with actionlint 1.7.7.
• Deterministic fault injection for:
  • session.idle timeout;
  • organization-disabled judge;
  • successful failed-slot recovery;
  • persistent retry failure;
  • missing and unexpected eval results;
  • malformed comparison output;
  • missing or duplicate comparison slot identity;
  • duplicate stimulus names at authoring time;
  • an entirely invalid empty run.
• A live Vally 0.13 comparison on persisted trajectories, plus a schema-version-3 adapter run.
• The final default-profile matrix passed all eight current shard/model legs in run 32206910590, bound to exact commit 780c999546a4007bde9742f511c29adb65c67cd8.
• Its artifacts contain eight schema-version-3 verdicts: all are conclusive and none are underpowered.
• Exact result accounting was 8 expected / 8 observed / 8 written, with zero missing, unexpected, or invalid evals.
• Across all raw Vally 0.13 baseline/skilled reports, every trialIndex was present and non-negative, each (stimulusName, trialIndex) key was unique, and paired key sets had zero differences.
• Every verdict has scenarioEvidence.gateEligible=true, comparisonTrialEvidence.gateEligible=false, zero unmatched trials, and zero errors.
• The final UI and validator-handoff run 32232320378 passed on exact commit 0434d9e58437e6beda3f27814cf9bed996eb57c9.
• Its trusted producer uploaded one same-run validator archive. All eight matrix legs downloaded and extracted that archive; no matrix job used a cache or fallback build.
• Its final artifacts again contain eight schema-version-3 verdicts with exact 8 expected / 8 observed / 8 written accounting, zero missing, unexpected, invalid, underpowered, or unresolved-error results.
• Across 276 raw baseline/skilled Vally 0.13 trials, every trialIndex was present and non-negative, every arm key was unique, and paired key sets had zero differences.
• The rendered PR comment identifies four skills across two models, includes model-specific gate evidence and Overfit, states that the objective gate is disabled, and gives repair actions for each non-passing or warning result.

Related issues

• [skills-eval] Eval-instrument validity: underpowered/biased evals manufacture false P0 regressions #986
• [skills-eval] Judge-side transient failures (disabled PAT + session.idle timeout) corrupt reliability verdicts #909
• Tighten the eval gate: net-win floor, completion-regression hard gate, and power/independence fixes #970

[github-actions]

📊 Skill Evaluation Results

70 skill(s) evaluated — ✅ 6 improved, ❌ 22 no credible change, 🔻 0 objective regressions, 📉 0 preference losses (report only).

⚠️ 42 could not be judged: 40 underpowered — the eval has fewer trials than any result needs to reach p ≤ 0.05, so no verdict was possible. This is the eval's size, not a skill regression; fix it by adding scenarios or raising defaults.runs; 2 inconclusive — the comparison didn't complete (errored, unmatched, or self-contradictory trials).

A skill passes only on a credible net win over baseline: more wins than losses, by an exact one-sided sign test at p ≤ 0.05. LLM preference losses are reported separately from objective completion regressions.

Skill	Result	Net win	p	Δ Pref	W/T/L	Quality	Baseline	Overfit	Skills Loaded
dotnet-maui-doctor	✅	+100.0%	0.004	+77.5%	8/0/0	4.6/5	3.5/5	🔴 0.53	⚠️ 8/8 · 7/8 (plugin)
dotnet-maui-doctor	✅	+75.0%	0.016	+75.0%	6/2/0	4.8/5	3.5/5	🔴 0.54	8/8 · 8/8 (plugin)
dotnet-maui-doctor	✅	+100.0%	0.004	+77.5%	8/0/0	4.7/5	3.4/5	🟡 0.28	8/8 · 8/8 (plugin)
dotnet-maui-doctor	✅	+75.0%	0.035	+67.5%	7/0/1	4.7/5	3.6/5	🟡 0.28	8/8 · 8/8 (plugin)
dotnet-maui-doctor	⚠️	+85.7%	0.016	+60.0%	6/1/0	3.4/5	2.9/5	🟡 0.37	⚠️ 4/8 · 3/8 (plugin)
maui-app-lifecycle	⚠️	+25.0%	0.500	+10.0%	2/1/1	5.0/5	3.7/5	🔴 0.57	4/4 · 4/4 (plugin)
maui-app-lifecycle	⚠️	+100.0%	0.063	+70.0%	4/0/0	5.0/5	2.9/5	🔴 0.52	4/4 · 4/4 (plugin)
maui-app-lifecycle	⚠️	+100.0%	0.063	+55.0%	4/0/0	3.8/5	3.2/5	🟡 0.24	⚠️ 4/4 · 1/4 (plugin)
maui-app-lifecycle	⚠️	+100.0%	0.063	+40.0%	4/0/0	3.6/5	3.5/5	🟡 0.21	4/4 · 4/4 (plugin)
maui-app-lifecycle	⚠️	+100.0%	0.063	+40.0%	4/0/0	2.6/5	2.2/5	🔴 0.61	⚠️ 3/4 · 4/4 (plugin)
maui-collectionview	❌	+20.0%	0.500	+8.0%	1/4/0	5.0/5	5.0/5	🟡 0.45	5/5 · 5/5 (plugin)
maui-collectionview	❌	+20.0%	0.500	+8.0%	2/2/1	5.0/5	4.9/5	🟡 0.36	5/5 · 5/5 (plugin)
maui-collectionview	❌	+60.0%	0.125	+24.0%	3/2/0	5.0/5	4.9/5	✅ 0.07	⚠️ 4/5 · 2/5 (plugin)
maui-collectionview	❌	+80.0%	0.063	+32.0%	4/1/0	4.9/5	5.0/5	✅ 0.07	5/5 · 5/5 (plugin)
maui-collectionview	❌	+20.0%	0.500	+8.0%	2/2/1	4.9/5	4.5/5	✅ 0.12	⚠️ 1/5 · 2/5 (plugin)
maui-data-binding	⚠️	+25.0%	0.500	+10.0%	2/1/1	4.8/5	4.4/5	🟡 0.48	4/4 · 4/4 (plugin)
maui-data-binding	⚠️	+75.0%	0.125	+30.0%	3/1/0	4.9/5	4.0/5	🟡 0.50	4/4 · 4/4 (plugin)
maui-data-binding	⚠️	+0.0%	0.687	+0.0%	2/0/2	4.8/5	4.0/5	✅ 0.17	⚠️ 2/4 · 1/4 (plugin)
maui-data-binding	⚠️	+50.0%	0.312	+20.0%	3/0/1	4.9/5	4.0/5	✅ 0.16	4/4 · 4/4 (plugin)
maui-data-binding	⚠️	+25.0%	0.500	+10.0%	1/3/0	4.4/5	4.0/5	🔴 0.51	⚠️ 1/4 · 1/4 (plugin)
maui-dependency-injection	❌	+80.0%	0.063	+56.0%	4/1/0	4.8/5	4.1/5	🟡 0.46	5/5 · 5/5 (plugin)
maui-dependency-injection	❌	+60.0%	0.188	+36.0%	4/0/1	4.7/5	4.4/5	🔴 0.51	5/5 · 5/5 (plugin)
maui-dependency-injection	❌	+60.0%	0.188	+36.0%	4/0/1	4.6/5	4.1/5	✅ 0.11	⚠️ 4/5 · 2/5 (plugin)
maui-dependency-injection	⚠️	+50.0%	0.312	+20.0%	3/0/1	5.0/5	4.8/5	✅ 0.11	5/5 · 5/5 (plugin)
maui-dependency-injection	❌	+20.0%	0.500	+20.0%	2/2/1	4.4/5	4.3/5	🔴 0.53	⚠️ 3/5 · 1/5 (plugin)
maui-safe-area	⚠️	+100.0%	0.063	+85.0%	4/0/0	4.9/5	3.4/5	🟡 0.38	4/4 · 4/4 (plugin)
maui-safe-area	⚠️	+100.0%	0.063	+100.0%	4/0/0	5.0/5	1.8/5	🟡 0.48	4/4 · 4/4 (plugin)
maui-safe-area	⚠️	+100.0%	0.063	+85.0%	4/0/0	4.8/5	3.1/5	🟡 0.20	4/4 · 4/4 (plugin)
maui-safe-area	⚠️	+75.0%	0.125	+45.0%	3/1/0	5.0/5	3.3/5	🟡 0.21	4/4 · 4/4 (plugin)
maui-safe-area	⚠️	+100.0%	0.063	+70.0%	4/0/0	3.6/5	1.6/5	🟡 0.40	⚠️ 1/4 · 2/4 (plugin)
maui-shell-navigation	✅	+100.0%	0.031	+64.0%	5/0/0	5.0/5	4.6/5	🔴 0.65	5/5 · 5/5 (plugin)
maui-shell-navigation	❌	+80.0%	0.063	+44.0%	4/1/0	5.0/5	3.9/5	🟡 0.40	5/5 · 5/5 (plugin)
maui-shell-navigation	❌	+60.0%	0.125	+36.0%	3/2/0	4.8/5	4.3/5	✅ 0.14	⚠️ 3/5 · 3/5 (plugin)
maui-shell-navigation	❌	+80.0%	0.063	+32.0%	4/1/0	5.0/5	4.8/5	✅ 0.11	5/5 · 5/5 (plugin)
maui-shell-navigation	❌	+40.0%	0.312	+52.0%	3/1/1	4.4/5	3.7/5	🔴 0.64	⚠️ 2/5 · 3/5 (plugin)
maui-theming	❌	+60.0%	0.188	+24.0%	4/0/1	5.0/5	4.8/5	🔴 0.55	5/5 · 5/5 (plugin)
maui-theming	❌	+60.0%	0.188	+36.0%	4/0/1	5.0/5	4.7/5	🔴 0.57	5/5 · 5/5 (plugin)
maui-theming	❌	+40.0%	0.312	+28.0%	3/1/1	4.6/5	4.5/5	✅ 0.10	⚠️ 3/5 · 3/5 (plugin)
maui-theming	✅	+100.0%	0.031	+64.0%	5/0/0	5.0/5	4.1/5	✅ 0.15	5/5 · 5/5 (plugin)
maui-theming	❌	+40.0%	0.312	+16.0%	3/1/1	4.5/5	4.1/5	🔴 0.71	⚠️ 3/5 · 2/5 (plugin)
template-authoring	⚠️	+100.0%	0.250	+40.0%	2/0/0	4.9/5	4.4/5	🟡 0.38	2/2 · 2/2 (plugin)
template-authoring	⚠️	+100.0%	0.250	+70.0%	2/0/0	5.0/5	3.9/5	🟡 0.45	2/2 · 2/2 (plugin)
template-authoring	⚠️	+50.0%	0.500	+20.0%	1/1/0	2.4/5	1.7/5	✅ 0.18	2/2 · 2/2 (plugin)
template-authoring	⚠️	+100.0%	0.250	+40.0%	2/0/0	4.8/5	4.1/5	✅ 0.16	⚠️ 1/2 · 2/2 (plugin)
template-authoring	⚠️	+100.0%	0.250	+40.0%	2/0/0	4.4/5	3.8/5	🟡 0.42	⚠️ 1/2 · 2/2 (plugin)
template-comparison	⚠️	+33.3%	0.500	+13.3%	2/0/1	5.0/5	4.9/5	🟡 0.45	3/3 · 3/3 (plugin)
template-comparison	⚠️	+66.7%	0.250	+26.7%	2/1/0	4.9/5	4.9/5	🟡 0.32	3/3 · 3/3 (plugin)
template-comparison	⚠️	+66.7%	0.250	+46.7%	2/1/0	4.8/5	4.4/5	✅ 0.08	⚠️ 3/3 · 1/3 (plugin)
template-comparison	⚠️	+100.0%	0.125	+40.0%	3/0/0	5.0/5	5.0/5	✅ 0.09	3/3 · 3/3 (plugin)
template-comparison	⚠️	+33.3%	0.500	+13.3%	1/2/0	4.9/5	4.9/5	🟡 0.37	⚠️ 1/3 · 1/3 (plugin)
template-discovery	❌	+20.0%	0.500	+20.0%	3/0/2	4.9/5	4.8/5	🟡 0.47	5/5 · 5/5 (plugin)
template-discovery	❌	+40.0%	0.312	+40.0%	3/1/1	4.8/5	4.7/5	🔴 0.51	5/5 · 5/5 (plugin)
template-discovery	❌	+80.0%	0.063	+44.0%	4/1/0	4.4/5	4.0/5	✅ 0.19	⚠️ 5/5 · 4/5 (plugin)
template-discovery	❌	+80.0%	0.063	+32.0%	4/1/0	4.8/5	4.9/5	🟡 0.30	5/5 · 5/5 (plugin)
template-discovery	❌	+60.0%	0.188	+12.0%	4/0/1	4.6/5	4.5/5	🟡 0.47	⚠️ 4/5 · 3/5 (plugin)
template-instantiation	⚠️	-100.0%	0.500	-40.0%	0/0/1	5.0/5	3.8/5	🟡 0.31	1/1 · 1/1 (plugin)
template-instantiation	⚠️	+100.0%	0.500	+40.0%	1/0/0	3.8/5	5.0/5	🟡 0.31	1/1 · 1/1 (plugin)
template-instantiation	⚠️	+100.0%	0.500	+100.0%	1/0/0	5.0/5	2.5/5	✅ 0.12	1/1 · 1/1 (plugin)
template-instantiation	⚠️	+100.0%	0.500	+40.0%	1/0/0	3.8/5	3.8/5	✅ 0.12	1/1 · 1/1 (plugin)
template-instantiation	⚠️	+0.0%	1.000	+0.0%	0/1/0	3.8/5	5.0/5	🟡 0.34	⚠️ 1/1 · 0/1 (plugin)
template-smart-defaults	⚠️	-50.0%	0.312	-20.0%	1/0/3	5.0/5	4.2/5	🟡 0.48	4/4 · 4/4 (plugin)
template-smart-defaults	⚠️	+75.0%	0.125	+45.0%	3/1/0	5.0/5	3.8/5	🟡 0.46	4/4 · 4/4 (plugin)
template-smart-defaults	⚠️	-50.0%	0.250	-20.0%	0/2/2	4.0/5	4.0/5	✅ 0.17	⚠️ 0/4 · 0/4 (plugin)
template-smart-defaults	⚠️	+50.0%	0.312	+20.0%	3/0/1	5.0/5	4.0/5	✅ 0.16	4/4 · 4/4 (plugin)
template-smart-defaults	⚠️	+0.0%	0.750	+0.0%	1/2/1	4.5/5	4.1/5	🟡 0.40	⚠️ 2/4 · 1/4 (plugin)
template-validation	⚠️	+100.0%	0.250	+70.0%	2/0/0	4.4/5	2.9/5	🟡 0.41	2/2 · 2/2 (plugin)
template-validation	⚠️	+100.0%	0.250	+70.0%	2/0/0	4.4/5	2.7/5	🟡 0.43	2/2 · 2/2 (plugin)
template-validation	⚠️	-50.0%	0.500	-20.0%	0/1/1	0.8/5	1.5/5	✅ 0.14	⚠️ 2/2 · 1/2 (plugin)
template-validation	⚠️	+100.0%	0.250	+100.0%	2/0/0	4.6/5	2.2/5	✅ 0.14	2/2 · 2/2 (plugin)
template-validation	⚠️	+100.0%	0.250	+70.0%	2/0/0	4.2/5	1.6/5	🟡 0.33	2/2 · 2/2 (plugin)

ℹ️ Column legend

• Net win — (wins − losses) / trials for skilled vs baseline, judged head-to-head by vally compare. This is the effect the gate decides on.
• p — one-sided exact sign test over the discordant (non-tie) trials. A skill passes only at p ≤ 0.05, which needs at least 5 winning trials.
• Δ Pref — the same comparison weighted by how decisive each win was (much-better ±100%, slightly-better ±40%). Reported for triage only: weighting the statistic by magnitude made a skill fail for winning harder, which is why the gate deliberately ignores this column.
• W/T/L — wins / ties / losses across trials.
• ⚠️ — the gate withheld a verdict. Either the eval has fewer trials than any result needs to reach p ≤ 0.05 (underpowered — the skill was never actually measured, so this is not a regression; add scenarios or raise defaults.runs), or the comparison didn't complete.
• 🔻 — an objective completion regression. This state is reserved for objective completion evidence.
• 📉 — a credible LLM preference loss. It is report-only and is not an objective completion regression.
• Quality / Baseline — mean absolute judge score 0–5 (skilled isolated vs skill-free control).
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) with its score.
• Skills Loaded — of the scenarios that expect activation, how many actually activated / that total (plugin run shown when present); ⚠️ marks a scenario that expected activation but didn't activate.

⚠️ dotnet-maui-doctor — details

State: INVALID_INCONCLUSIVE (unmatched_trajectories)

Reason: Net win +85.7% (6W/1T/0L over 7 trial(s), sign test p=0.016), mean preference +60.0%, 1 unmatched — inconclusive (unmatched trajectories)

Effective scenarios (report only): 7 (6W/1T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
= Determine required Android SDK packages for specific .NET version	+0.0%	+0.0%	0/0/0
▲ Diagnose non-Microsoft JDK causing build failure	+100.0%	+100.0%	1/0/0
▲ Fix stale MAUI workloads after SDK update	+100.0%	+100.0%	1/0/0
= Guardrail against workload update and repair	+0.0%	+0.0%	0/1/0
▲ Plan Linux MAUI environment for Android	+100.0%	+40.0%	1/0/0
▲ Plan complete MAUI setup on Windows	+100.0%	+40.0%	1/0/0
▲ Plan macOS MAUI setup with Xcode	+100.0%	+100.0%	1/0/0
▲ Prevent incorrect JAVA_HOME configuration	+100.0%	+40.0%	1/0/0

⚠️ maui-app-lifecycle — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +25.0% (2W/1T/1L over 4 trial(s), sign test p=0.500), mean preference +10.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (2W/1T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▼ Avoid legacy Xamarin.Forms lifecycle methods	-100.0%	-40.0%	0/0/1
= Platform-specific lifecycle mapping	+0.0%	+0.0%	0/1/0
▲ Save and restore state on background	+100.0%	+40.0%	1/0/0
▲ Window lifecycle event subscription	+100.0%	+40.0%	1/0/0

⚠️ maui-app-lifecycle — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (4W/0T/0L over 4 trial(s), sign test p=0.063), mean preference +70.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (4W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Avoid legacy Xamarin.Forms lifecycle methods	+100.0%	+40.0%	1/0/0
▲ Platform-specific lifecycle mapping	+100.0%	+40.0%	1/0/0
▲ Save and restore state on background	+100.0%	+100.0%	1/0/0
▲ Window lifecycle event subscription	+100.0%	+100.0%	1/0/0

⚠️ maui-app-lifecycle — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (4W/0T/0L over 4 trial(s), sign test p=0.063), mean preference +55.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (4W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Avoid legacy Xamarin.Forms lifecycle methods	+100.0%	+40.0%	1/0/0
▲ Platform-specific lifecycle mapping	+100.0%	+40.0%	1/0/0
▲ Save and restore state on background	+100.0%	+100.0%	1/0/0
▲ Window lifecycle event subscription	+100.0%	+40.0%	1/0/0

⚠️ maui-app-lifecycle — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (4W/0T/0L over 4 trial(s), sign test p=0.063), mean preference +40.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (4W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Avoid legacy Xamarin.Forms lifecycle methods	+100.0%	+40.0%	1/0/0
▲ Platform-specific lifecycle mapping	+100.0%	+40.0%	1/0/0
▲ Save and restore state on background	+100.0%	+40.0%	1/0/0
▲ Window lifecycle event subscription	+100.0%	+40.0%	1/0/0

⚠️ maui-app-lifecycle — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (4W/0T/0L over 4 trial(s), sign test p=0.063), mean preference +40.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (4W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Avoid legacy Xamarin.Forms lifecycle methods	+100.0%	+40.0%	1/0/0
▲ Platform-specific lifecycle mapping	+100.0%	+40.0%	1/0/0
▲ Save and restore state on background	+100.0%	+40.0%	1/0/0
▲ Window lifecycle event subscription	+100.0%	+40.0%	1/0/0

❌ maui-collectionview — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +20.0% (1W/4T/0L over 5 trial(s), sign test p=0.500), mean preference +8.0% — not credible — 4 of 5 trial(s) tied, leaving only 1 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)

Effective scenarios (report only): 5 (1W/4T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Avoid ListView and ViewCell mistakes	+100.0%	+40.0%	1/0/0
= Basic CollectionView with data binding and DataTemplate	+0.0%	+0.0%	0/1/0
= Grid layout with CollectionView	+0.0%	+0.0%	0/1/0
= ItemSizingStrategy placement for uniform items	+0.0%	+0.0%	0/1/0
= Selection and pull-to-refresh with CollectionView	+0.0%	+0.0%	0/1/0

❌ maui-collectionview — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +20.0% (2W/2T/1L over 5 trial(s), sign test p=0.500), mean preference +8.0% — not credible — 2 of 5 trial(s) tied, leaving only 3 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)

Effective scenarios (report only): 5 (2W/2T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
= Avoid ListView and ViewCell mistakes	+0.0%	+0.0%	0/1/0
▼ Basic CollectionView with data binding and DataTemplate	-100.0%	-40.0%	0/0/1
= Grid layout with CollectionView	+0.0%	+0.0%	0/1/0
▲ ItemSizingStrategy placement for uniform items	+100.0%	+40.0%	1/0/0
▲ Selection and pull-to-refresh with CollectionView	+100.0%	+40.0%	1/0/0

❌ maui-collectionview — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +60.0% (3W/2T/0L over 5 trial(s), sign test p=0.125), mean preference +24.0% — not credible — 2 of 5 trial(s) tied, leaving only 3 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)

Effective scenarios (report only): 5 (3W/2T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
= Avoid ListView and ViewCell mistakes	+0.0%	+0.0%	0/1/0
= Basic CollectionView with data binding and DataTemplate	+0.0%	+0.0%	0/1/0
▲ Grid layout with CollectionView	+100.0%	+40.0%	1/0/0
▲ ItemSizingStrategy placement for uniform items	+100.0%	+40.0%	1/0/0
▲ Selection and pull-to-refresh with CollectionView	+100.0%	+40.0%	1/0/0

❌ maui-collectionview — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +80.0% (4W/1T/0L over 5 trial(s), sign test p=0.063), mean preference +32.0% — not credible — 1 of 5 trial(s) tied, leaving only 4 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)

Effective scenarios (report only): 5 (4W/1T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
= Avoid ListView and ViewCell mistakes	+0.0%	+0.0%	0/1/0
▲ Basic CollectionView with data binding and DataTemplate	+100.0%	+40.0%	1/0/0
▲ Grid layout with CollectionView	+100.0%	+40.0%	1/0/0
▲ ItemSizingStrategy placement for uniform items	+100.0%	+40.0%	1/0/0
▲ Selection and pull-to-refresh with CollectionView	+100.0%	+40.0%	1/0/0

❌ maui-collectionview — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +20.0% (2W/2T/1L over 5 trial(s), sign test p=0.500), mean preference +8.0% — not credible — 2 of 5 trial(s) tied, leaving only 3 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)

Effective scenarios (report only): 5 (2W/2T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Avoid ListView and ViewCell mistakes	+100.0%	+40.0%	1/0/0
▼ Basic CollectionView with data binding and DataTemplate	-100.0%	-40.0%	0/0/1
= Grid layout with CollectionView	+0.0%	+0.0%	0/1/0
▲ ItemSizingStrategy placement for uniform items	+100.0%	+40.0%	1/0/0
= Selection and pull-to-refresh with CollectionView	+0.0%	+0.0%	0/1/0

⚠️ maui-data-binding — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +25.0% (2W/1T/1L over 4 trial(s), sign test p=0.500), mean preference +10.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (2W/1T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▼ Create and use an IValueConverter	-100.0%	-40.0%	0/0/1
▲ Diagnose missing BindingContext and non-compiled bindings	+100.0%	+40.0%	1/0/0
= Implement MVVM ViewModel with ObservableObject	+0.0%	+0.0%	0/1/0
▲ Set up compiled bindings with x:DataType on a page	+100.0%	+40.0%	1/0/0

⚠️ maui-data-binding — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +75.0% (3W/1T/0L over 4 trial(s), sign test p=0.125), mean preference +30.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (3W/1T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
= Create and use an IValueConverter	+0.0%	+0.0%	0/1/0
▲ Diagnose missing BindingContext and non-compiled bindings	+100.0%	+40.0%	1/0/0
▲ Implement MVVM ViewModel with ObservableObject	+100.0%	+40.0%	1/0/0
▲ Set up compiled bindings with x:DataType on a page	+100.0%	+40.0%	1/0/0

⚠️ maui-data-binding — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +0.0% (2W/0T/2L over 4 trial(s), sign test p=0.687), mean preference +0.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (2W/0T/2L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▼ Create and use an IValueConverter	-100.0%	-40.0%	0/0/1
▲ Diagnose missing BindingContext and non-compiled bindings	+100.0%	+40.0%	1/0/0
▼ Implement MVVM ViewModel with ObservableObject	-100.0%	-40.0%	0/0/1
▲ Set up compiled bindings with x:DataType on a page	+100.0%	+40.0%	1/0/0

⚠️ maui-data-binding — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +50.0% (3W/0T/1L over 4 trial(s), sign test p=0.312), mean preference +20.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (3W/0T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Create and use an IValueConverter	+100.0%	+40.0%	1/0/0
▲ Diagnose missing BindingContext and non-compiled bindings	+100.0%	+40.0%	1/0/0
▼ Implement MVVM ViewModel with ObservableObject	-100.0%	-40.0%	0/0/1
▲ Set up compiled bindings with x:DataType on a page	+100.0%	+40.0%	1/0/0

⚠️ maui-data-binding — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +25.0% (1W/3T/0L over 4 trial(s), sign test p=0.500), mean preference +10.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (1W/3T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
= Create and use an IValueConverter	+0.0%	+0.0%	0/1/0
= Diagnose missing BindingContext and non-compiled bindings	+0.0%	+0.0%	0/1/0
= Implement MVVM ViewModel with ObservableObject	+0.0%	+0.0%	0/1/0
▲ Set up compiled bindings with x:DataType on a page	+100.0%	+40.0%	1/0/0

❌ maui-dependency-injection — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +80.0% (4W/1T/0L over 5 trial(s), sign test p=0.063), mean preference +56.0% — not credible — 1 of 5 trial(s) tied, leaving only 4 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)

Effective scenarios (report only): 5 (4W/1T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
= Avoid AddScoped pitfall in MAUI	+0.0%	+0.0%	0/1/0
▲ Diagnose a page whose injected dependencies are missing	+100.0%	+100.0%	1/0/0
▲ Platform-specific service registration with fallback	+100.0%	+100.0%	1/0/0
▲ Register services with correct lifetimes in MauiProgram.cs	+100.0%	+40.0%	1/0/0
▲ Shell navigation auto-resolves DI-registered pages	+100.0%	+40.0%	1/0/0

❌ maui-dependency-injection — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +60.0% (4W/0T/1L over 5 trial(s), sign test p=0.188), mean preference +36.0% — not credible (sign test p=0.188 > 0.05)

Effective scenarios (report only): 5 (4W/0T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Avoid AddScoped pitfall in MAUI	+100.0%	+40.0%	1/0/0
▲ Diagnose a page whose injected dependencies are missing	+100.0%	+100.0%	1/0/0
▲ Platform-specific service registration with fallback	+100.0%	+40.0%	1/0/0
▼ Register services with correct lifetimes in MauiProgram.cs	-100.0%	-40.0%	0/0/1
▲ Shell navigation auto-resolves DI-registered pages	+100.0%	+40.0%	1/0/0

❌ maui-dependency-injection — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +60.0% (4W/0T/1L over 5 trial(s), sign test p=0.188), mean preference +36.0% — not credible (sign test p=0.188 > 0.05)

Effective scenarios (report only): 5 (4W/0T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▼ Avoid AddScoped pitfall in MAUI	-100.0%	-40.0%	0/0/1
▲ Diagnose a page whose injected dependencies are missing	+100.0%	+100.0%	1/0/0
▲ Platform-specific service registration with fallback	+100.0%	+40.0%	1/0/0
▲ Register services with correct lifetimes in MauiProgram.cs	+100.0%	+40.0%	1/0/0
▲ Shell navigation auto-resolves DI-registered pages	+100.0%	+40.0%	1/0/0

⚠️ maui-dependency-injection — details

State: INVALID_INCONCLUSIVE (unmatched_trajectories)

Reason: Net win +50.0% (3W/0T/1L over 4 trial(s), sign test p=0.312), mean preference +20.0%, 1 unmatched — inconclusive (unmatched trajectories)

Effective scenarios (report only): 4 (3W/0T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Avoid AddScoped pitfall in MAUI	+100.0%	+40.0%	1/0/0
▲ Diagnose a page whose injected dependencies are missing	+100.0%	+40.0%	1/0/0
= Platform-specific service registration with fallback	+0.0%	+0.0%	0/0/0
▼ Register services with correct lifetimes in MauiProgram.cs	-100.0%	-40.0%	0/0/1
▲ Shell navigation auto-resolves DI-registered pages	+100.0%	+40.0%	1/0/0

❌ maui-dependency-injection — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +20.0% (2W/2T/1L over 5 trial(s), sign test p=0.500), mean preference +20.0% — not credible — 2 of 5 trial(s) tied, leaving only 3 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)

Effective scenarios (report only): 5 (2W/2T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
= Avoid AddScoped pitfall in MAUI	+0.0%	+0.0%	0/1/0
▲ Diagnose a page whose injected dependencies are missing	+100.0%	+100.0%	1/0/0
= Platform-specific service registration with fallback	+0.0%	+0.0%	0/1/0
▲ Register services with correct lifetimes in MauiProgram.cs	+100.0%	+40.0%	1/0/0
▼ Shell navigation auto-resolves DI-registered pages	-100.0%	-40.0%	0/0/1

⚠️ maui-safe-area — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (4W/0T/0L over 4 trial(s), sign test p=0.063), mean preference +85.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (4W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Avoid deprecated UseSafeArea in new .NET 10 project	+100.0%	+40.0%	1/0/0
▲ Edge-to-edge layout with SafeAreaEdges	+100.0%	+100.0%	1/0/0
▲ Handle notch and status bar safe areas on iOS	+100.0%	+100.0%	1/0/0
▲ Keyboard avoidance with safe area for chat UI	+100.0%	+100.0%	1/0/0

⚠️ maui-safe-area — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (4W/0T/0L over 4 trial(s), sign test p=0.063), mean preference +100.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (4W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Avoid deprecated UseSafeArea in new .NET 10 project	+100.0%	+100.0%	1/0/0
▲ Edge-to-edge layout with SafeAreaEdges	+100.0%	+100.0%	1/0/0
▲ Handle notch and status bar safe areas on iOS	+100.0%	+100.0%	1/0/0
▲ Keyboard avoidance with safe area for chat UI	+100.0%	+100.0%	1/0/0

⚠️ maui-safe-area — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (4W/0T/0L over 4 trial(s), sign test p=0.063), mean preference +85.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (4W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Avoid deprecated UseSafeArea in new .NET 10 project	+100.0%	+40.0%	1/0/0
▲ Edge-to-edge layout with SafeAreaEdges	+100.0%	+100.0%	1/0/0
▲ Handle notch and status bar safe areas on iOS	+100.0%	+100.0%	1/0/0
▲ Keyboard avoidance with safe area for chat UI	+100.0%	+100.0%	1/0/0

⚠️ maui-safe-area — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +75.0% (3W/1T/0L over 4 trial(s), sign test p=0.125), mean preference +45.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (3W/1T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Avoid deprecated UseSafeArea in new .NET 10 project	+100.0%	+40.0%	1/0/0
= Edge-to-edge layout with SafeAreaEdges	+0.0%	+0.0%	0/1/0
▲ Handle notch and status bar safe areas on iOS	+100.0%	+40.0%	1/0/0
▲ Keyboard avoidance with safe area for chat UI	+100.0%	+100.0%	1/0/0

⚠️ maui-safe-area — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (4W/0T/0L over 4 trial(s), sign test p=0.063), mean preference +70.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (4W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Avoid deprecated UseSafeArea in new .NET 10 project	+100.0%	+100.0%	1/0/0
▲ Edge-to-edge layout with SafeAreaEdges	+100.0%	+40.0%	1/0/0
▲ Handle notch and status bar safe areas on iOS	+100.0%	+100.0%	1/0/0
▲ Keyboard avoidance with safe area for chat UI	+100.0%	+40.0%	1/0/0

❌ maui-shell-navigation — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +80.0% (4W/1T/0L over 5 trial(s), sign test p=0.063), mean preference +44.0% — not credible — 1 of 5 trial(s) tied, leaving only 4 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)

Effective scenarios (report only): 5 (4W/1T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
= Diagnose common Shell navigation mistakes	+0.0%	+0.0%	0/1/0
▲ Handle back navigation and unsaved changes guard	+100.0%	+100.0%	1/0/0
▲ Navigate with parameters using GoToAsync	+100.0%	+40.0%	1/0/0
▲ Set up Shell navigation with tabs and flyout	+100.0%	+40.0%	1/0/0
▲ Stable routes for deep linking into tabs	+100.0%	+40.0%	1/0/0

❌ maui-shell-navigation — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +60.0% (3W/2T/0L over 5 trial(s), sign test p=0.125), mean preference +36.0% — not credible — 2 of 5 trial(s) tied, leaving only 3 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)

Effective scenarios (report only): 5 (3W/2T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Diagnose common Shell navigation mistakes	+100.0%	+40.0%	1/0/0
▲ Handle back navigation and unsaved changes guard	+100.0%	+100.0%	1/0/0
▲ Navigate with parameters using GoToAsync	+100.0%	+40.0%	1/0/0
= Set up Shell navigation with tabs and flyout	+0.0%	+0.0%	0/1/0
= Stable routes for deep linking into tabs	+0.0%	+0.0%	0/1/0

❌ maui-shell-navigation — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +80.0% (4W/1T/0L over 5 trial(s), sign test p=0.063), mean preference +32.0% — not credible — 1 of 5 trial(s) tied, leaving only 4 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)

Effective scenarios (report only): 5 (4W/1T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
= Diagnose common Shell navigation mistakes	+0.0%	+0.0%	0/1/0
▲ Handle back navigation and unsaved changes guard	+100.0%	+40.0%	1/0/0
▲ Navigate with parameters using GoToAsync	+100.0%	+40.0%	1/0/0
▲ Set up Shell navigation with tabs and flyout	+100.0%	+40.0%	1/0/0
▲ Stable routes for deep linking into tabs	+100.0%	+40.0%	1/0/0

❌ maui-shell-navigation — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +40.0% (3W/1T/1L over 5 trial(s), sign test p=0.312), mean preference +52.0% — not credible — 1 of 5 trial(s) tied, leaving only 4 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)

Effective scenarios (report only): 5 (3W/1T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Diagnose common Shell navigation mistakes	+100.0%	+100.0%	1/0/0
▼ Handle back navigation and unsaved changes guard	-100.0%	-40.0%	0/0/1
▲ Navigate with parameters using GoToAsync	+100.0%	+100.0%	1/0/0
= Set up Shell navigation with tabs and flyout	+0.0%	+0.0%	0/1/0
▲ Stable routes for deep linking into tabs	+100.0%	+100.0%	1/0/0

❌ maui-theming — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +60.0% (4W/0T/1L over 5 trial(s), sign test p=0.188), mean preference +24.0% — not credible (sign test p=0.188 > 0.05)

Effective scenarios (report only): 5 (4W/0T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Add light/dark mode support using AppThemeBinding	+100.0%	+40.0%	1/0/0
▲ Avoid common theming mistakes	+100.0%	+40.0%	1/0/0
▼ Create custom themes with ResourceDictionary switching	-100.0%	-40.0%	0/0/1
▲ Detect and respond to system theme changes	+100.0%	+40.0%	1/0/0
▲ Swap theme dictionaries without destroying app styles	+100.0%	+40.0%	1/0/0

❌ maui-theming — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +60.0% (4W/0T/1L over 5 trial(s), sign test p=0.188), mean preference +36.0% — not credible (sign test p=0.188 > 0.05)

Effective scenarios (report only): 5 (4W/0T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Add light/dark mode support using AppThemeBinding	+100.0%	+40.0%	1/0/0
▼ Avoid common theming mistakes	-100.0%	-40.0%	0/0/1
▲ Create custom themes with ResourceDictionary switching	+100.0%	+100.0%	1/0/0
▲ Detect and respond to system theme changes	+100.0%	+40.0%	1/0/0
▲ Swap theme dictionaries without destroying app styles	+100.0%	+40.0%	1/0/0

❌ maui-theming — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +40.0% (3W/1T/1L over 5 trial(s), sign test p=0.312), mean preference +28.0% — not credible — 1 of 5 trial(s) tied, leaving only 4 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)

Effective scenarios (report only): 5 (3W/1T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Add light/dark mode support using AppThemeBinding	+100.0%	+40.0%	1/0/0
▲ Avoid common theming mistakes	+100.0%	+100.0%	1/0/0
▲ Create custom themes with ResourceDictionary switching	+100.0%	+40.0%	1/0/0
= Detect and respond to system theme changes	+0.0%	+0.0%	0/1/0
▼ Swap theme dictionaries without destroying app styles	-100.0%	-40.0%	0/0/1

❌ maui-theming — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +40.0% (3W/1T/1L over 5 trial(s), sign test p=0.312), mean preference +16.0% — not credible — 1 of 5 trial(s) tied, leaving only 4 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)

Effective scenarios (report only): 5 (3W/1T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Add light/dark mode support using AppThemeBinding	+100.0%	+40.0%	1/0/0
= Avoid common theming mistakes	+0.0%	+0.0%	0/1/0
▲ Create custom themes with ResourceDictionary switching	+100.0%	+40.0%	1/0/0
▼ Detect and respond to system theme changes	-100.0%	-40.0%	0/0/1
▲ Swap theme dictionaries without destroying app styles	+100.0%	+40.0%	1/0/0

⚠️ template-authoring — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (2W/0T/0L over 2 trial(s), sign test p=0.250), mean preference +40.0% — underpowered (2 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 2 (2W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Create template from existing project	+100.0%	+40.0%	1/0/0
▲ Validate a template.json file	+100.0%	+40.0%	1/0/0

⚠️ template-authoring — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (2W/0T/0L over 2 trial(s), sign test p=0.250), mean preference +70.0% — underpowered (2 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 2 (2W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Create template from existing project	+100.0%	+40.0%	1/0/0
▲ Validate a template.json file	+100.0%	+100.0%	1/0/0

⚠️ template-authoring — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +50.0% (1W/1T/0L over 2 trial(s), sign test p=0.500), mean preference +20.0% — underpowered (2 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 2 (1W/1T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Create template from existing project	+100.0%	+40.0%	1/0/0
= Validate a template.json file	+0.0%	+0.0%	0/1/0

⚠️ template-authoring — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (2W/0T/0L over 2 trial(s), sign test p=0.250), mean preference +40.0% — underpowered (2 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 2 (2W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Create template from existing project	+100.0%	+40.0%	1/0/0
▲ Validate a template.json file	+100.0%	+40.0%	1/0/0

⚠️ template-authoring — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (2W/0T/0L over 2 trial(s), sign test p=0.250), mean preference +40.0% — underpowered (2 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 2 (2W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Create template from existing project	+100.0%	+40.0%	1/0/0
▲ Validate a template.json file	+100.0%	+40.0%	1/0/0

⚠️ template-comparison — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +33.3% (2W/0T/1L over 3 trial(s), sign test p=0.500), mean preference +13.3% — underpowered (3 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 3 (2W/0T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Choose between blazor and blazorwasm	+100.0%	+40.0%	1/0/0
▼ Compare webapi vs webapp side by side	-100.0%	-40.0%	0/0/1
▲ Decide which template fits a background processing scenario	+100.0%	+40.0%	1/0/0

⚠️ template-comparison — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +66.7% (2W/1T/0L over 3 trial(s), sign test p=0.250), mean preference +26.7% — underpowered (3 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 3 (2W/1T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
= Choose between blazor and blazorwasm	+0.0%	+0.0%	0/1/0
▲ Compare webapi vs webapp side by side	+100.0%	+40.0%	1/0/0
▲ Decide which template fits a background processing scenario	+100.0%	+40.0%	1/0/0

⚠️ template-comparison — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +66.7% (2W/1T/0L over 3 trial(s), sign test p=0.250), mean preference +46.7% — underpowered (3 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 3 (2W/1T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
= Choose between blazor and blazorwasm	+0.0%	+0.0%	0/1/0
▲ Compare webapi vs webapp side by side	+100.0%	+100.0%	1/0/0
▲ Decide which template fits a background processing scenario	+100.0%	+40.0%	1/0/0

⚠️ template-comparison — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (3W/0T/0L over 3 trial(s), sign test p=0.125), mean preference +40.0% — underpowered (3 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 3 (3W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Choose between blazor and blazorwasm	+100.0%	+40.0%	1/0/0
▲ Compare webapi vs webapp side by side	+100.0%	+40.0%	1/0/0
▲ Decide which template fits a background processing scenario	+100.0%	+40.0%	1/0/0

⚠️ template-comparison — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +33.3% (1W/2T/0L over 3 trial(s), sign test p=0.500), mean preference +13.3% — underpowered (3 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 3 (1W/2T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
= Choose between blazor and blazorwasm	+0.0%	+0.0%	0/1/0
▲ Compare webapi vs webapp side by side	+100.0%	+40.0%	1/0/0
= Decide which template fits a background processing scenario	+0.0%	+0.0%	0/1/0

❌ template-discovery — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +20.0% (3W/0T/2L over 5 trial(s), sign test p=0.500), mean preference +20.0% — not credible (sign test p=0.500 > 0.05)

Effective scenarios (report only): 5 (3W/0T/2L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Find template for web API project	+100.0%	+100.0%	1/0/0
▼ Inspect template parameters and compare choices	-100.0%	-40.0%	0/0/1
▲ Preview project creation with dry run	+100.0%	+40.0%	1/0/0
▼ Resolve ambiguous project intent to multiple candidates	-100.0%	-40.0%	0/0/1
▲ Search NuGet for specialized template	+100.0%	+40.0%	1/0/0

❌ template-discovery — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +40.0% (3W/1T/1L over 5 trial(s), sign test p=0.312), mean preference +40.0% — not credible — 1 of 5 trial(s) tied, leaving only 4 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)

Effective scenarios (report only): 5 (3W/1T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Find template for web API project	+100.0%	+100.0%	1/0/0
▲ Inspect template parameters and compare choices	+100.0%	+40.0%	1/0/0
= Preview project creation with dry run	+0.0%	+0.0%	0/1/0
▲ Resolve ambiguous project intent to multiple candidates	+100.0%	+100.0%	1/0/0
▼ Search NuGet for specialized template	-100.0%	-40.0%	0/0/1

❌ template-discovery — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +80.0% (4W/1T/0L over 5 trial(s), sign test p=0.063), mean preference +44.0% — not credible — 1 of 5 trial(s) tied, leaving only 4 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)

Effective scenarios (report only): 5 (4W/1T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Find template for web API project	+100.0%	+40.0%	1/0/0
▲ Inspect template parameters and compare choices	+100.0%	+40.0%	1/0/0
▲ Preview project creation with dry run	+100.0%	+100.0%	1/0/0
▲ Resolve ambiguous project intent to multiple candidates	+100.0%	+40.0%	1/0/0
= Search NuGet for specialized template	+0.0%	+0.0%	0/1/0

❌ template-discovery — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +80.0% (4W/1T/0L over 5 trial(s), sign test p=0.063), mean preference +32.0% — not credible — 1 of 5 trial(s) tied, leaving only 4 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)

Effective scenarios (report only): 5 (4W/1T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Find template for web API project	+100.0%	+40.0%	1/0/0
= Inspect template parameters and compare choices	+0.0%	+0.0%	0/1/0
▲ Preview project creation with dry run	+100.0%	+40.0%	1/0/0
▲ Resolve ambiguous project intent to multiple candidates	+100.0%	+40.0%	1/0/0
▲ Search NuGet for specialized template	+100.0%	+40.0%	1/0/0

❌ template-discovery — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +60.0% (4W/0T/1L over 5 trial(s), sign test p=0.188), mean preference +12.0% — not credible (sign test p=0.188 > 0.05)

Effective scenarios (report only): 5 (4W/0T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Find template for web API project	+100.0%	+40.0%	1/0/0
▲ Inspect template parameters and compare choices	+100.0%	+40.0%	1/0/0
▲ Preview project creation with dry run	+100.0%	+40.0%	1/0/0
▲ Resolve ambiguous project intent to multiple candidates	+100.0%	+40.0%	1/0/0
▼ Search NuGet for specialized template	-100.0%	-100.0%	0/0/1

⚠️ template-instantiation — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win -100.0% (0W/0T/1L over 1 trial(s), sign test p=0.500), mean preference -40.0% — underpowered (1 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 1 (0W/0T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▼ Create a console application	-100.0%	-40.0%	0/0/1

⚠️ template-instantiation — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (1W/0T/0L over 1 trial(s), sign test p=0.500), mean preference +40.0% — underpowered (1 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 1 (1W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Create a console application	+100.0%	+40.0%	1/0/0

⚠️ template-instantiation — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (1W/0T/0L over 1 trial(s), sign test p=0.500), mean preference +100.0% — underpowered (1 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 1 (1W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Create a console application	+100.0%	+100.0%	1/0/0

⚠️ template-instantiation — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (1W/0T/0L over 1 trial(s), sign test p=0.500), mean preference +40.0% — underpowered (1 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 1 (1W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Create a console application	+100.0%	+40.0%	1/0/0

⚠️ template-instantiation — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +0.0% (0W/1T/0L over 1 trial(s), sign test p=1.000), mean preference +0.0% — underpowered (1 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 1 (0W/1T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
= Create a console application	+0.0%	+0.0%	0/1/0

⚠️ template-smart-defaults — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win -50.0% (1W/0T/3L over 4 trial(s), sign test p=0.312), mean preference -20.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (1W/0T/3L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▼ AOT implies a compatible framework	-100.0%	-40.0%	0/0/1
▼ Auth implies HTTPS stays enabled	-100.0%	-40.0%	0/0/1
▼ Controllers exclude the minimal-API flag	-100.0%	-40.0%	0/0/1
▲ Never override an explicit user value	+100.0%	+40.0%	1/0/0

⚠️ template-smart-defaults — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +75.0% (3W/1T/0L over 4 trial(s), sign test p=0.125), mean preference +45.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (3W/1T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ AOT implies a compatible framework	+100.0%	+40.0%	1/0/0
▲ Auth implies HTTPS stays enabled	+100.0%	+40.0%	1/0/0
= Controllers exclude the minimal-API flag	+0.0%	+0.0%	0/1/0
▲ Never override an explicit user value	+100.0%	+100.0%	1/0/0

⚠️ template-smart-defaults — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win -50.0% (0W/2T/2L over 4 trial(s), sign test p=0.250), mean preference -20.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (0W/2T/2L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
= AOT implies a compatible framework	+0.0%	+0.0%	0/1/0
▼ Auth implies HTTPS stays enabled	-100.0%	-40.0%	0/0/1
= Controllers exclude the minimal-API flag	+0.0%	+0.0%	0/1/0
▼ Never override an explicit user value	-100.0%	-40.0%	0/0/1

⚠️ template-smart-defaults — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +50.0% (3W/0T/1L over 4 trial(s), sign test p=0.312), mean preference +20.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (3W/0T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ AOT implies a compatible framework	+100.0%	+40.0%	1/0/0
▲ Auth implies HTTPS stays enabled	+100.0%	+40.0%	1/0/0
▼ Controllers exclude the minimal-API flag	-100.0%	-40.0%	0/0/1
▲ Never override an explicit user value	+100.0%	+40.0%	1/0/0

⚠️ template-smart-defaults — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +0.0% (1W/2T/1L over 4 trial(s), sign test p=0.750), mean preference +0.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (1W/2T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ AOT implies a compatible framework	+100.0%	+40.0%	1/0/0
▼ Auth implies HTTPS stays enabled	-100.0%	-40.0%	0/0/1
= Controllers exclude the minimal-API flag	+0.0%	+0.0%	0/1/0
= Never override an explicit user value	+0.0%	+0.0%	0/1/0

⚠️ template-validation — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (2W/0T/0L over 2 trial(s), sign test p=0.250), mean preference +70.0% — underpowered (2 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 2 (2W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Validate correct template and suggest improvements	+100.0%	+100.0%	1/0/0
▲ Validate template with multiple errors	+100.0%	+40.0%	1/0/0

⚠️ template-validation — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (2W/0T/0L over 2 trial(s), sign test p=0.250), mean preference +70.0% — underpowered (2 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 2 (2W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Validate correct template and suggest improvements	+100.0%	+100.0%	1/0/0
▲ Validate template with multiple errors	+100.0%	+40.0%	1/0/0

⚠️ template-validation — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win -50.0% (0W/1T/1L over 2 trial(s), sign test p=0.500), mean preference -20.0% — underpowered (2 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 2 (0W/1T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▼ Validate correct template and suggest improvements	-100.0%	-40.0%	0/0/1
= Validate template with multiple errors	+0.0%	+0.0%	0/1/0

⚠️ template-validation — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (2W/0T/0L over 2 trial(s), sign test p=0.250), mean preference +100.0% — underpowered (2 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 2 (2W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Validate correct template and suggest improvements	+100.0%	+100.0%	1/0/0
▲ Validate template with multiple errors	+100.0%	+100.0%	1/0/0

Per-scenario details for 7 skill(s) were omitted to keep this comment under GitHub's 65,536-character limit — open the job's step summary or Full Results for the complete breakdown.

🔍 Full Results - additional metrics and failure investigation steps

To investigate failures, paste this to your AI coding agent:

For PR 1027 in dotnet/skills, download eval artifacts with gh run download 32085934523 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/24d0a0697fe5c868928d452ca11856a9268a24e2/eng/vally-adapter/InvestigatingResults.md and follow it to analyze the results.json files. Diagnose each failure, suggest fixes to the eval.yaml and skill content, and tell me what to fix first.

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[github-actions]

📊 Skill Evaluation Results

70 skill(s) evaluated — ✅ 6 improved, ❌ 22 no credible change, 🔻 0 objective regressions, 📉 0 preference losses (report only).

⚠️ 42 could not be judged: 40 underpowered — the eval has fewer trials than any result needs to reach p ≤ 0.05, so no verdict was possible. This is the eval's size, not a skill regression; fix it by adding scenarios or raising defaults.runs; 2 inconclusive — the comparison didn't complete (errored, unmatched, or self-contradictory trials).

A skill passes only on a credible net win over baseline: more wins than losses, by an exact one-sided sign test at p ≤ 0.05. LLM preference losses are reported separately from objective completion regressions.

Skill	Result	Net win	p	Δ Pref	W/T/L	Quality	Baseline	Overfit	Skills Loaded
dotnet-maui-doctor	✅	+100.0%	0.004	+77.5%	8/0/0	4.6/5	3.5/5	🔴 0.53	⚠️ 8/8 · 7/8 (plugin)
dotnet-maui-doctor	✅	+75.0%	0.016	+75.0%	6/2/0	4.8/5	3.5/5	🔴 0.54	8/8 · 8/8 (plugin)
dotnet-maui-doctor	✅	+100.0%	0.004	+77.5%	8/0/0	4.7/5	3.4/5	🟡 0.28	8/8 · 8/8 (plugin)
dotnet-maui-doctor	✅	+75.0%	0.035	+67.5%	7/0/1	4.7/5	3.6/5	🟡 0.28	8/8 · 8/8 (plugin)
dotnet-maui-doctor	⚠️	+85.7%	0.016	+60.0%	6/1/0	3.4/5	2.9/5	🟡 0.37	⚠️ 4/8 · 3/8 (plugin)
maui-app-lifecycle	⚠️	+25.0%	0.500	+10.0%	2/1/1	5.0/5	3.7/5	🔴 0.57	4/4 · 4/4 (plugin)
maui-app-lifecycle	⚠️	+100.0%	0.063	+70.0%	4/0/0	5.0/5	2.9/5	🔴 0.52	4/4 · 4/4 (plugin)
maui-app-lifecycle	⚠️	+100.0%	0.063	+55.0%	4/0/0	3.8/5	3.2/5	🟡 0.24	⚠️ 4/4 · 1/4 (plugin)
maui-app-lifecycle	⚠️	+100.0%	0.063	+40.0%	4/0/0	3.6/5	3.5/5	🟡 0.21	4/4 · 4/4 (plugin)
maui-app-lifecycle	⚠️	+100.0%	0.063	+40.0%	4/0/0	2.6/5	2.2/5	🔴 0.61	⚠️ 3/4 · 4/4 (plugin)
maui-collectionview	❌	+20.0%	0.500	+8.0%	1/4/0	5.0/5	5.0/5	🟡 0.45	5/5 · 5/5 (plugin)
maui-collectionview	❌	+20.0%	0.500	+8.0%	2/2/1	5.0/5	4.9/5	🟡 0.36	5/5 · 5/5 (plugin)
maui-collectionview	❌	+60.0%	0.125	+24.0%	3/2/0	5.0/5	4.9/5	✅ 0.07	⚠️ 4/5 · 2/5 (plugin)
maui-collectionview	❌	+80.0%	0.063	+32.0%	4/1/0	4.9/5	5.0/5	✅ 0.07	5/5 · 5/5 (plugin)
maui-collectionview	❌	+20.0%	0.500	+8.0%	2/2/1	4.9/5	4.5/5	✅ 0.12	⚠️ 1/5 · 2/5 (plugin)
maui-data-binding	⚠️	+25.0%	0.500	+10.0%	2/1/1	4.8/5	4.4/5	🟡 0.48	4/4 · 4/4 (plugin)
maui-data-binding	⚠️	+75.0%	0.125	+30.0%	3/1/0	4.9/5	4.0/5	🟡 0.50	4/4 · 4/4 (plugin)
maui-data-binding	⚠️	+0.0%	0.687	+0.0%	2/0/2	4.8/5	4.0/5	✅ 0.17	⚠️ 2/4 · 1/4 (plugin)
maui-data-binding	⚠️	+50.0%	0.312	+20.0%	3/0/1	4.9/5	4.0/5	✅ 0.16	4/4 · 4/4 (plugin)
maui-data-binding	⚠️	+25.0%	0.500	+10.0%	1/3/0	4.4/5	4.0/5	🔴 0.51	⚠️ 1/4 · 1/4 (plugin)
maui-dependency-injection	❌	+80.0%	0.063	+56.0%	4/1/0	4.8/5	4.1/5	🟡 0.46	5/5 · 5/5 (plugin)
maui-dependency-injection	❌	+60.0%	0.188	+36.0%	4/0/1	4.7/5	4.4/5	🔴 0.51	5/5 · 5/5 (plugin)
maui-dependency-injection	❌	+60.0%	0.188	+36.0%	4/0/1	4.6/5	4.1/5	✅ 0.11	⚠️ 4/5 · 2/5 (plugin)
maui-dependency-injection	⚠️	+50.0%	0.312	+20.0%	3/0/1	5.0/5	4.8/5	✅ 0.11	5/5 · 5/5 (plugin)
maui-dependency-injection	❌	+20.0%	0.500	+20.0%	2/2/1	4.4/5	4.3/5	🔴 0.53	⚠️ 3/5 · 1/5 (plugin)
maui-safe-area	⚠️	+100.0%	0.063	+85.0%	4/0/0	4.9/5	3.4/5	🟡 0.38	4/4 · 4/4 (plugin)
maui-safe-area	⚠️	+100.0%	0.063	+100.0%	4/0/0	5.0/5	1.8/5	🟡 0.48	4/4 · 4/4 (plugin)
maui-safe-area	⚠️	+100.0%	0.063	+85.0%	4/0/0	4.8/5	3.1/5	🟡 0.20	4/4 · 4/4 (plugin)
maui-safe-area	⚠️	+75.0%	0.125	+45.0%	3/1/0	5.0/5	3.3/5	🟡 0.21	4/4 · 4/4 (plugin)
maui-safe-area	⚠️	+100.0%	0.063	+70.0%	4/0/0	3.6/5	1.6/5	🟡 0.40	⚠️ 1/4 · 2/4 (plugin)
maui-shell-navigation	✅	+100.0%	0.031	+64.0%	5/0/0	5.0/5	4.6/5	🔴 0.65	5/5 · 5/5 (plugin)
maui-shell-navigation	❌	+80.0%	0.063	+44.0%	4/1/0	5.0/5	3.9/5	🟡 0.40	5/5 · 5/5 (plugin)
maui-shell-navigation	❌	+60.0%	0.125	+36.0%	3/2/0	4.8/5	4.3/5	✅ 0.14	⚠️ 3/5 · 3/5 (plugin)
maui-shell-navigation	❌	+80.0%	0.063	+32.0%	4/1/0	5.0/5	4.8/5	✅ 0.11	5/5 · 5/5 (plugin)
maui-shell-navigation	❌	+40.0%	0.312	+52.0%	3/1/1	4.4/5	3.7/5	🔴 0.64	⚠️ 2/5 · 3/5 (plugin)
maui-theming	❌	+60.0%	0.188	+24.0%	4/0/1	5.0/5	4.8/5	🔴 0.55	5/5 · 5/5 (plugin)
maui-theming	❌	+60.0%	0.188	+36.0%	4/0/1	5.0/5	4.7/5	🔴 0.57	5/5 · 5/5 (plugin)
maui-theming	❌	+40.0%	0.312	+28.0%	3/1/1	4.6/5	4.5/5	✅ 0.10	⚠️ 3/5 · 3/5 (plugin)
maui-theming	✅	+100.0%	0.031	+64.0%	5/0/0	5.0/5	4.1/5	✅ 0.15	5/5 · 5/5 (plugin)
maui-theming	❌	+40.0%	0.312	+16.0%	3/1/1	4.5/5	4.1/5	🔴 0.71	⚠️ 3/5 · 2/5 (plugin)
template-authoring	⚠️	+100.0%	0.250	+40.0%	2/0/0	4.9/5	4.4/5	🟡 0.38	2/2 · 2/2 (plugin)
template-authoring	⚠️	+100.0%	0.250	+70.0%	2/0/0	5.0/5	3.9/5	🟡 0.45	2/2 · 2/2 (plugin)
template-authoring	⚠️	+50.0%	0.500	+20.0%	1/1/0	2.4/5	1.7/5	✅ 0.18	2/2 · 2/2 (plugin)
template-authoring	⚠️	+100.0%	0.250	+40.0%	2/0/0	4.8/5	4.1/5	✅ 0.16	⚠️ 1/2 · 2/2 (plugin)
template-authoring	⚠️	+100.0%	0.250	+40.0%	2/0/0	4.4/5	3.8/5	🟡 0.42	⚠️ 1/2 · 2/2 (plugin)
template-comparison	⚠️	+33.3%	0.500	+13.3%	2/0/1	5.0/5	4.9/5	🟡 0.45	3/3 · 3/3 (plugin)
template-comparison	⚠️	+66.7%	0.250	+26.7%	2/1/0	4.9/5	4.9/5	🟡 0.32	3/3 · 3/3 (plugin)
template-comparison	⚠️	+66.7%	0.250	+46.7%	2/1/0	4.8/5	4.4/5	✅ 0.08	⚠️ 3/3 · 1/3 (plugin)
template-comparison	⚠️	+100.0%	0.125	+40.0%	3/0/0	5.0/5	5.0/5	✅ 0.09	3/3 · 3/3 (plugin)
template-comparison	⚠️	+33.3%	0.500	+13.3%	1/2/0	4.9/5	4.9/5	🟡 0.37	⚠️ 1/3 · 1/3 (plugin)
template-discovery	❌	+20.0%	0.500	+20.0%	3/0/2	4.9/5	4.8/5	🟡 0.47	5/5 · 5/5 (plugin)
template-discovery	❌	+40.0%	0.312	+40.0%	3/1/1	4.8/5	4.7/5	🔴 0.51	5/5 · 5/5 (plugin)
template-discovery	❌	+80.0%	0.063	+44.0%	4/1/0	4.4/5	4.0/5	✅ 0.19	⚠️ 5/5 · 4/5 (plugin)
template-discovery	❌	+80.0%	0.063	+32.0%	4/1/0	4.8/5	4.9/5	🟡 0.30	5/5 · 5/5 (plugin)
template-discovery	❌	+60.0%	0.188	+12.0%	4/0/1	4.6/5	4.5/5	🟡 0.47	⚠️ 4/5 · 3/5 (plugin)
template-instantiation	⚠️	-100.0%	0.500	-40.0%	0/0/1	5.0/5	3.8/5	🟡 0.31	1/1 · 1/1 (plugin)
template-instantiation	⚠️	+100.0%	0.500	+40.0%	1/0/0	3.8/5	5.0/5	🟡 0.31	1/1 · 1/1 (plugin)
template-instantiation	⚠️	+100.0%	0.500	+100.0%	1/0/0	5.0/5	2.5/5	✅ 0.12	1/1 · 1/1 (plugin)
template-instantiation	⚠️	+100.0%	0.500	+40.0%	1/0/0	3.8/5	3.8/5	✅ 0.12	1/1 · 1/1 (plugin)
template-instantiation	⚠️	+0.0%	1.000	+0.0%	0/1/0	3.8/5	5.0/5	🟡 0.34	⚠️ 1/1 · 0/1 (plugin)
template-smart-defaults	⚠️	-50.0%	0.312	-20.0%	1/0/3	5.0/5	4.2/5	🟡 0.48	4/4 · 4/4 (plugin)
template-smart-defaults	⚠️	+75.0%	0.125	+45.0%	3/1/0	5.0/5	3.8/5	🟡 0.46	4/4 · 4/4 (plugin)
template-smart-defaults	⚠️	-50.0%	0.250	-20.0%	0/2/2	4.0/5	4.0/5	✅ 0.17	⚠️ 0/4 · 0/4 (plugin)
template-smart-defaults	⚠️	+50.0%	0.312	+20.0%	3/0/1	5.0/5	4.0/5	✅ 0.16	4/4 · 4/4 (plugin)
template-smart-defaults	⚠️	+0.0%	0.750	+0.0%	1/2/1	4.5/5	4.1/5	🟡 0.40	⚠️ 2/4 · 1/4 (plugin)
template-validation	⚠️	+100.0%	0.250	+70.0%	2/0/0	4.4/5	2.9/5	🟡 0.41	2/2 · 2/2 (plugin)
template-validation	⚠️	+100.0%	0.250	+70.0%	2/0/0	4.4/5	2.7/5	🟡 0.43	2/2 · 2/2 (plugin)
template-validation	⚠️	-50.0%	0.500	-20.0%	0/1/1	0.8/5	1.5/5	✅ 0.14	⚠️ 2/2 · 1/2 (plugin)
template-validation	⚠️	+100.0%	0.250	+100.0%	2/0/0	4.6/5	2.2/5	✅ 0.14	2/2 · 2/2 (plugin)
template-validation	⚠️	+100.0%	0.250	+70.0%	2/0/0	4.2/5	1.6/5	🟡 0.33	2/2 · 2/2 (plugin)

ℹ️ Column legend

• Net win — (wins − losses) / trials for skilled vs baseline, judged head-to-head by vally compare. This is the effect the gate decides on.
• p — one-sided exact sign test over the discordant (non-tie) trials. A skill passes only at p ≤ 0.05, which needs at least 5 winning trials.
• Δ Pref — the same comparison weighted by how decisive each win was (much-better ±100%, slightly-better ±40%). Reported for triage only: weighting the statistic by magnitude made a skill fail for winning harder, which is why the gate deliberately ignores this column.
• W/T/L — wins / ties / losses across trials.
• ⚠️ — the gate withheld a verdict. Either the eval has fewer trials than any result needs to reach p ≤ 0.05 (underpowered — the skill was never actually measured, so this is not a regression; add scenarios or raise defaults.runs), or the comparison didn't complete.
• 🔻 — an objective completion regression. This state is reserved for objective completion evidence.
• 📉 — a credible LLM preference loss. It is report-only and is not an objective completion regression.
• Quality / Baseline — mean absolute judge score 0–5 (skilled isolated vs skill-free control).
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) with its score.
• Skills Loaded — of the scenarios that expect activation, how many actually activated / that total (plugin run shown when present); ⚠️ marks a scenario that expected activation but didn't activate.

⚠️ dotnet-maui-doctor — details

State: INVALID_INCONCLUSIVE (unmatched_trajectories)

Reason: Net win +85.7% (6W/1T/0L over 7 trial(s), sign test p=0.016), mean preference +60.0%, 1 unmatched — inconclusive (unmatched trajectories)

Effective scenarios (report only): 7 (6W/1T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
= Determine required Android SDK packages for specific .NET version	+0.0%	+0.0%	0/0/0
▲ Diagnose non-Microsoft JDK causing build failure	+100.0%	+100.0%	1/0/0
▲ Fix stale MAUI workloads after SDK update	+100.0%	+100.0%	1/0/0
= Guardrail against workload update and repair	+0.0%	+0.0%	0/1/0
▲ Plan Linux MAUI environment for Android	+100.0%	+40.0%	1/0/0
▲ Plan complete MAUI setup on Windows	+100.0%	+40.0%	1/0/0
▲ Plan macOS MAUI setup with Xcode	+100.0%	+100.0%	1/0/0
▲ Prevent incorrect JAVA_HOME configuration	+100.0%	+40.0%	1/0/0

⚠️ maui-app-lifecycle — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +25.0% (2W/1T/1L over 4 trial(s), sign test p=0.500), mean preference +10.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (2W/1T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▼ Avoid legacy Xamarin.Forms lifecycle methods	-100.0%	-40.0%	0/0/1
= Platform-specific lifecycle mapping	+0.0%	+0.0%	0/1/0
▲ Save and restore state on background	+100.0%	+40.0%	1/0/0
▲ Window lifecycle event subscription	+100.0%	+40.0%	1/0/0

⚠️ maui-app-lifecycle — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (4W/0T/0L over 4 trial(s), sign test p=0.063), mean preference +70.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (4W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Avoid legacy Xamarin.Forms lifecycle methods	+100.0%	+40.0%	1/0/0
▲ Platform-specific lifecycle mapping	+100.0%	+40.0%	1/0/0
▲ Save and restore state on background	+100.0%	+100.0%	1/0/0
▲ Window lifecycle event subscription	+100.0%	+100.0%	1/0/0

⚠️ maui-app-lifecycle — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (4W/0T/0L over 4 trial(s), sign test p=0.063), mean preference +55.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (4W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Avoid legacy Xamarin.Forms lifecycle methods	+100.0%	+40.0%	1/0/0
▲ Platform-specific lifecycle mapping	+100.0%	+40.0%	1/0/0
▲ Save and restore state on background	+100.0%	+100.0%	1/0/0
▲ Window lifecycle event subscription	+100.0%	+40.0%	1/0/0

⚠️ maui-app-lifecycle — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (4W/0T/0L over 4 trial(s), sign test p=0.063), mean preference +40.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (4W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Avoid legacy Xamarin.Forms lifecycle methods	+100.0%	+40.0%	1/0/0
▲ Platform-specific lifecycle mapping	+100.0%	+40.0%	1/0/0
▲ Save and restore state on background	+100.0%	+40.0%	1/0/0
▲ Window lifecycle event subscription	+100.0%	+40.0%	1/0/0

⚠️ maui-app-lifecycle — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (4W/0T/0L over 4 trial(s), sign test p=0.063), mean preference +40.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (4W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Avoid legacy Xamarin.Forms lifecycle methods	+100.0%	+40.0%	1/0/0
▲ Platform-specific lifecycle mapping	+100.0%	+40.0%	1/0/0
▲ Save and restore state on background	+100.0%	+40.0%	1/0/0
▲ Window lifecycle event subscription	+100.0%	+40.0%	1/0/0

❌ maui-collectionview — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +20.0% (1W/4T/0L over 5 trial(s), sign test p=0.500), mean preference +8.0% — not credible — 4 of 5 trial(s) tied, leaving only 1 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)

Effective scenarios (report only): 5 (1W/4T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Avoid ListView and ViewCell mistakes	+100.0%	+40.0%	1/0/0
= Basic CollectionView with data binding and DataTemplate	+0.0%	+0.0%	0/1/0
= Grid layout with CollectionView	+0.0%	+0.0%	0/1/0
= ItemSizingStrategy placement for uniform items	+0.0%	+0.0%	0/1/0
= Selection and pull-to-refresh with CollectionView	+0.0%	+0.0%	0/1/0

❌ maui-collectionview — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +20.0% (2W/2T/1L over 5 trial(s), sign test p=0.500), mean preference +8.0% — not credible — 2 of 5 trial(s) tied, leaving only 3 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)

Effective scenarios (report only): 5 (2W/2T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
= Avoid ListView and ViewCell mistakes	+0.0%	+0.0%	0/1/0
▼ Basic CollectionView with data binding and DataTemplate	-100.0%	-40.0%	0/0/1
= Grid layout with CollectionView	+0.0%	+0.0%	0/1/0
▲ ItemSizingStrategy placement for uniform items	+100.0%	+40.0%	1/0/0
▲ Selection and pull-to-refresh with CollectionView	+100.0%	+40.0%	1/0/0

❌ maui-collectionview — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +60.0% (3W/2T/0L over 5 trial(s), sign test p=0.125), mean preference +24.0% — not credible — 2 of 5 trial(s) tied, leaving only 3 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)

Effective scenarios (report only): 5 (3W/2T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
= Avoid ListView and ViewCell mistakes	+0.0%	+0.0%	0/1/0
= Basic CollectionView with data binding and DataTemplate	+0.0%	+0.0%	0/1/0
▲ Grid layout with CollectionView	+100.0%	+40.0%	1/0/0
▲ ItemSizingStrategy placement for uniform items	+100.0%	+40.0%	1/0/0
▲ Selection and pull-to-refresh with CollectionView	+100.0%	+40.0%	1/0/0

❌ maui-collectionview — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +80.0% (4W/1T/0L over 5 trial(s), sign test p=0.063), mean preference +32.0% — not credible — 1 of 5 trial(s) tied, leaving only 4 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)

Effective scenarios (report only): 5 (4W/1T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
= Avoid ListView and ViewCell mistakes	+0.0%	+0.0%	0/1/0
▲ Basic CollectionView with data binding and DataTemplate	+100.0%	+40.0%	1/0/0
▲ Grid layout with CollectionView	+100.0%	+40.0%	1/0/0
▲ ItemSizingStrategy placement for uniform items	+100.0%	+40.0%	1/0/0
▲ Selection and pull-to-refresh with CollectionView	+100.0%	+40.0%	1/0/0

❌ maui-collectionview — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +20.0% (2W/2T/1L over 5 trial(s), sign test p=0.500), mean preference +8.0% — not credible — 2 of 5 trial(s) tied, leaving only 3 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)

Effective scenarios (report only): 5 (2W/2T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Avoid ListView and ViewCell mistakes	+100.0%	+40.0%	1/0/0
▼ Basic CollectionView with data binding and DataTemplate	-100.0%	-40.0%	0/0/1
= Grid layout with CollectionView	+0.0%	+0.0%	0/1/0
▲ ItemSizingStrategy placement for uniform items	+100.0%	+40.0%	1/0/0
= Selection and pull-to-refresh with CollectionView	+0.0%	+0.0%	0/1/0

⚠️ maui-data-binding — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +25.0% (2W/1T/1L over 4 trial(s), sign test p=0.500), mean preference +10.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (2W/1T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▼ Create and use an IValueConverter	-100.0%	-40.0%	0/0/1
▲ Diagnose missing BindingContext and non-compiled bindings	+100.0%	+40.0%	1/0/0
= Implement MVVM ViewModel with ObservableObject	+0.0%	+0.0%	0/1/0
▲ Set up compiled bindings with x:DataType on a page	+100.0%	+40.0%	1/0/0

⚠️ maui-data-binding — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +75.0% (3W/1T/0L over 4 trial(s), sign test p=0.125), mean preference +30.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (3W/1T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
= Create and use an IValueConverter	+0.0%	+0.0%	0/1/0
▲ Diagnose missing BindingContext and non-compiled bindings	+100.0%	+40.0%	1/0/0
▲ Implement MVVM ViewModel with ObservableObject	+100.0%	+40.0%	1/0/0
▲ Set up compiled bindings with x:DataType on a page	+100.0%	+40.0%	1/0/0

⚠️ maui-data-binding — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +0.0% (2W/0T/2L over 4 trial(s), sign test p=0.687), mean preference +0.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (2W/0T/2L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▼ Create and use an IValueConverter	-100.0%	-40.0%	0/0/1
▲ Diagnose missing BindingContext and non-compiled bindings	+100.0%	+40.0%	1/0/0
▼ Implement MVVM ViewModel with ObservableObject	-100.0%	-40.0%	0/0/1
▲ Set up compiled bindings with x:DataType on a page	+100.0%	+40.0%	1/0/0

⚠️ maui-data-binding — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +50.0% (3W/0T/1L over 4 trial(s), sign test p=0.312), mean preference +20.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (3W/0T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Create and use an IValueConverter	+100.0%	+40.0%	1/0/0
▲ Diagnose missing BindingContext and non-compiled bindings	+100.0%	+40.0%	1/0/0
▼ Implement MVVM ViewModel with ObservableObject	-100.0%	-40.0%	0/0/1
▲ Set up compiled bindings with x:DataType on a page	+100.0%	+40.0%	1/0/0

⚠️ maui-data-binding — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +25.0% (1W/3T/0L over 4 trial(s), sign test p=0.500), mean preference +10.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (1W/3T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
= Create and use an IValueConverter	+0.0%	+0.0%	0/1/0
= Diagnose missing BindingContext and non-compiled bindings	+0.0%	+0.0%	0/1/0
= Implement MVVM ViewModel with ObservableObject	+0.0%	+0.0%	0/1/0
▲ Set up compiled bindings with x:DataType on a page	+100.0%	+40.0%	1/0/0

❌ maui-dependency-injection — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +80.0% (4W/1T/0L over 5 trial(s), sign test p=0.063), mean preference +56.0% — not credible — 1 of 5 trial(s) tied, leaving only 4 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)

Effective scenarios (report only): 5 (4W/1T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
= Avoid AddScoped pitfall in MAUI	+0.0%	+0.0%	0/1/0
▲ Diagnose a page whose injected dependencies are missing	+100.0%	+100.0%	1/0/0
▲ Platform-specific service registration with fallback	+100.0%	+100.0%	1/0/0
▲ Register services with correct lifetimes in MauiProgram.cs	+100.0%	+40.0%	1/0/0
▲ Shell navigation auto-resolves DI-registered pages	+100.0%	+40.0%	1/0/0

❌ maui-dependency-injection — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +60.0% (4W/0T/1L over 5 trial(s), sign test p=0.188), mean preference +36.0% — not credible (sign test p=0.188 > 0.05)

Effective scenarios (report only): 5 (4W/0T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Avoid AddScoped pitfall in MAUI	+100.0%	+40.0%	1/0/0
▲ Diagnose a page whose injected dependencies are missing	+100.0%	+100.0%	1/0/0
▲ Platform-specific service registration with fallback	+100.0%	+40.0%	1/0/0
▼ Register services with correct lifetimes in MauiProgram.cs	-100.0%	-40.0%	0/0/1
▲ Shell navigation auto-resolves DI-registered pages	+100.0%	+40.0%	1/0/0

❌ maui-dependency-injection — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +60.0% (4W/0T/1L over 5 trial(s), sign test p=0.188), mean preference +36.0% — not credible (sign test p=0.188 > 0.05)

Effective scenarios (report only): 5 (4W/0T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▼ Avoid AddScoped pitfall in MAUI	-100.0%	-40.0%	0/0/1
▲ Diagnose a page whose injected dependencies are missing	+100.0%	+100.0%	1/0/0
▲ Platform-specific service registration with fallback	+100.0%	+40.0%	1/0/0
▲ Register services with correct lifetimes in MauiProgram.cs	+100.0%	+40.0%	1/0/0
▲ Shell navigation auto-resolves DI-registered pages	+100.0%	+40.0%	1/0/0

⚠️ maui-dependency-injection — details

State: INVALID_INCONCLUSIVE (unmatched_trajectories)

Reason: Net win +50.0% (3W/0T/1L over 4 trial(s), sign test p=0.312), mean preference +20.0%, 1 unmatched — inconclusive (unmatched trajectories)

Effective scenarios (report only): 4 (3W/0T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Avoid AddScoped pitfall in MAUI	+100.0%	+40.0%	1/0/0
▲ Diagnose a page whose injected dependencies are missing	+100.0%	+40.0%	1/0/0
= Platform-specific service registration with fallback	+0.0%	+0.0%	0/0/0
▼ Register services with correct lifetimes in MauiProgram.cs	-100.0%	-40.0%	0/0/1
▲ Shell navigation auto-resolves DI-registered pages	+100.0%	+40.0%	1/0/0

❌ maui-dependency-injection — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +20.0% (2W/2T/1L over 5 trial(s), sign test p=0.500), mean preference +20.0% — not credible — 2 of 5 trial(s) tied, leaving only 3 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)

Effective scenarios (report only): 5 (2W/2T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
= Avoid AddScoped pitfall in MAUI	+0.0%	+0.0%	0/1/0
▲ Diagnose a page whose injected dependencies are missing	+100.0%	+100.0%	1/0/0
= Platform-specific service registration with fallback	+0.0%	+0.0%	0/1/0
▲ Register services with correct lifetimes in MauiProgram.cs	+100.0%	+40.0%	1/0/0
▼ Shell navigation auto-resolves DI-registered pages	-100.0%	-40.0%	0/0/1

⚠️ maui-safe-area — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (4W/0T/0L over 4 trial(s), sign test p=0.063), mean preference +85.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (4W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Avoid deprecated UseSafeArea in new .NET 10 project	+100.0%	+40.0%	1/0/0
▲ Edge-to-edge layout with SafeAreaEdges	+100.0%	+100.0%	1/0/0
▲ Handle notch and status bar safe areas on iOS	+100.0%	+100.0%	1/0/0
▲ Keyboard avoidance with safe area for chat UI	+100.0%	+100.0%	1/0/0

⚠️ maui-safe-area — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (4W/0T/0L over 4 trial(s), sign test p=0.063), mean preference +100.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (4W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Avoid deprecated UseSafeArea in new .NET 10 project	+100.0%	+100.0%	1/0/0
▲ Edge-to-edge layout with SafeAreaEdges	+100.0%	+100.0%	1/0/0
▲ Handle notch and status bar safe areas on iOS	+100.0%	+100.0%	1/0/0
▲ Keyboard avoidance with safe area for chat UI	+100.0%	+100.0%	1/0/0

⚠️ maui-safe-area — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (4W/0T/0L over 4 trial(s), sign test p=0.063), mean preference +85.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (4W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Avoid deprecated UseSafeArea in new .NET 10 project	+100.0%	+40.0%	1/0/0
▲ Edge-to-edge layout with SafeAreaEdges	+100.0%	+100.0%	1/0/0
▲ Handle notch and status bar safe areas on iOS	+100.0%	+100.0%	1/0/0
▲ Keyboard avoidance with safe area for chat UI	+100.0%	+100.0%	1/0/0

⚠️ maui-safe-area — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +75.0% (3W/1T/0L over 4 trial(s), sign test p=0.125), mean preference +45.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (3W/1T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Avoid deprecated UseSafeArea in new .NET 10 project	+100.0%	+40.0%	1/0/0
= Edge-to-edge layout with SafeAreaEdges	+0.0%	+0.0%	0/1/0
▲ Handle notch and status bar safe areas on iOS	+100.0%	+40.0%	1/0/0
▲ Keyboard avoidance with safe area for chat UI	+100.0%	+100.0%	1/0/0

⚠️ maui-safe-area — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (4W/0T/0L over 4 trial(s), sign test p=0.063), mean preference +70.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (4W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Avoid deprecated UseSafeArea in new .NET 10 project	+100.0%	+100.0%	1/0/0
▲ Edge-to-edge layout with SafeAreaEdges	+100.0%	+40.0%	1/0/0
▲ Handle notch and status bar safe areas on iOS	+100.0%	+100.0%	1/0/0
▲ Keyboard avoidance with safe area for chat UI	+100.0%	+40.0%	1/0/0

❌ maui-shell-navigation — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +80.0% (4W/1T/0L over 5 trial(s), sign test p=0.063), mean preference +44.0% — not credible — 1 of 5 trial(s) tied, leaving only 4 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)

Effective scenarios (report only): 5 (4W/1T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
= Diagnose common Shell navigation mistakes	+0.0%	+0.0%	0/1/0
▲ Handle back navigation and unsaved changes guard	+100.0%	+100.0%	1/0/0
▲ Navigate with parameters using GoToAsync	+100.0%	+40.0%	1/0/0
▲ Set up Shell navigation with tabs and flyout	+100.0%	+40.0%	1/0/0
▲ Stable routes for deep linking into tabs	+100.0%	+40.0%	1/0/0

❌ maui-shell-navigation — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +60.0% (3W/2T/0L over 5 trial(s), sign test p=0.125), mean preference +36.0% — not credible — 2 of 5 trial(s) tied, leaving only 3 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)

Effective scenarios (report only): 5 (3W/2T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Diagnose common Shell navigation mistakes	+100.0%	+40.0%	1/0/0
▲ Handle back navigation and unsaved changes guard	+100.0%	+100.0%	1/0/0
▲ Navigate with parameters using GoToAsync	+100.0%	+40.0%	1/0/0
= Set up Shell navigation with tabs and flyout	+0.0%	+0.0%	0/1/0
= Stable routes for deep linking into tabs	+0.0%	+0.0%	0/1/0

❌ maui-shell-navigation — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +80.0% (4W/1T/0L over 5 trial(s), sign test p=0.063), mean preference +32.0% — not credible — 1 of 5 trial(s) tied, leaving only 4 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)

Effective scenarios (report only): 5 (4W/1T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
= Diagnose common Shell navigation mistakes	+0.0%	+0.0%	0/1/0
▲ Handle back navigation and unsaved changes guard	+100.0%	+40.0%	1/0/0
▲ Navigate with parameters using GoToAsync	+100.0%	+40.0%	1/0/0
▲ Set up Shell navigation with tabs and flyout	+100.0%	+40.0%	1/0/0
▲ Stable routes for deep linking into tabs	+100.0%	+40.0%	1/0/0

❌ maui-shell-navigation — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +40.0% (3W/1T/1L over 5 trial(s), sign test p=0.312), mean preference +52.0% — not credible — 1 of 5 trial(s) tied, leaving only 4 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)

Effective scenarios (report only): 5 (3W/1T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Diagnose common Shell navigation mistakes	+100.0%	+100.0%	1/0/0
▼ Handle back navigation and unsaved changes guard	-100.0%	-40.0%	0/0/1
▲ Navigate with parameters using GoToAsync	+100.0%	+100.0%	1/0/0
= Set up Shell navigation with tabs and flyout	+0.0%	+0.0%	0/1/0
▲ Stable routes for deep linking into tabs	+100.0%	+100.0%	1/0/0

❌ maui-theming — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +60.0% (4W/0T/1L over 5 trial(s), sign test p=0.188), mean preference +24.0% — not credible (sign test p=0.188 > 0.05)

Effective scenarios (report only): 5 (4W/0T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Add light/dark mode support using AppThemeBinding	+100.0%	+40.0%	1/0/0
▲ Avoid common theming mistakes	+100.0%	+40.0%	1/0/0
▼ Create custom themes with ResourceDictionary switching	-100.0%	-40.0%	0/0/1
▲ Detect and respond to system theme changes	+100.0%	+40.0%	1/0/0
▲ Swap theme dictionaries without destroying app styles	+100.0%	+40.0%	1/0/0

❌ maui-theming — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +60.0% (4W/0T/1L over 5 trial(s), sign test p=0.188), mean preference +36.0% — not credible (sign test p=0.188 > 0.05)

Effective scenarios (report only): 5 (4W/0T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Add light/dark mode support using AppThemeBinding	+100.0%	+40.0%	1/0/0
▼ Avoid common theming mistakes	-100.0%	-40.0%	0/0/1
▲ Create custom themes with ResourceDictionary switching	+100.0%	+100.0%	1/0/0
▲ Detect and respond to system theme changes	+100.0%	+40.0%	1/0/0
▲ Swap theme dictionaries without destroying app styles	+100.0%	+40.0%	1/0/0

❌ maui-theming — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +40.0% (3W/1T/1L over 5 trial(s), sign test p=0.312), mean preference +28.0% — not credible — 1 of 5 trial(s) tied, leaving only 4 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)

Effective scenarios (report only): 5 (3W/1T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Add light/dark mode support using AppThemeBinding	+100.0%	+40.0%	1/0/0
▲ Avoid common theming mistakes	+100.0%	+100.0%	1/0/0
▲ Create custom themes with ResourceDictionary switching	+100.0%	+40.0%	1/0/0
= Detect and respond to system theme changes	+0.0%	+0.0%	0/1/0
▼ Swap theme dictionaries without destroying app styles	-100.0%	-40.0%	0/0/1

❌ maui-theming — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +40.0% (3W/1T/1L over 5 trial(s), sign test p=0.312), mean preference +16.0% — not credible — 1 of 5 trial(s) tied, leaving only 4 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)

Effective scenarios (report only): 5 (3W/1T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Add light/dark mode support using AppThemeBinding	+100.0%	+40.0%	1/0/0
= Avoid common theming mistakes	+0.0%	+0.0%	0/1/0
▲ Create custom themes with ResourceDictionary switching	+100.0%	+40.0%	1/0/0
▼ Detect and respond to system theme changes	-100.0%	-40.0%	0/0/1
▲ Swap theme dictionaries without destroying app styles	+100.0%	+40.0%	1/0/0

⚠️ template-authoring — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (2W/0T/0L over 2 trial(s), sign test p=0.250), mean preference +40.0% — underpowered (2 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 2 (2W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Create template from existing project	+100.0%	+40.0%	1/0/0
▲ Validate a template.json file	+100.0%	+40.0%	1/0/0

⚠️ template-authoring — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (2W/0T/0L over 2 trial(s), sign test p=0.250), mean preference +70.0% — underpowered (2 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 2 (2W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Create template from existing project	+100.0%	+40.0%	1/0/0
▲ Validate a template.json file	+100.0%	+100.0%	1/0/0

⚠️ template-authoring — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +50.0% (1W/1T/0L over 2 trial(s), sign test p=0.500), mean preference +20.0% — underpowered (2 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 2 (1W/1T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Create template from existing project	+100.0%	+40.0%	1/0/0
= Validate a template.json file	+0.0%	+0.0%	0/1/0

⚠️ template-authoring — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (2W/0T/0L over 2 trial(s), sign test p=0.250), mean preference +40.0% — underpowered (2 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 2 (2W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Create template from existing project	+100.0%	+40.0%	1/0/0
▲ Validate a template.json file	+100.0%	+40.0%	1/0/0

⚠️ template-authoring — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (2W/0T/0L over 2 trial(s), sign test p=0.250), mean preference +40.0% — underpowered (2 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 2 (2W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Create template from existing project	+100.0%	+40.0%	1/0/0
▲ Validate a template.json file	+100.0%	+40.0%	1/0/0

⚠️ template-comparison — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +33.3% (2W/0T/1L over 3 trial(s), sign test p=0.500), mean preference +13.3% — underpowered (3 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 3 (2W/0T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Choose between blazor and blazorwasm	+100.0%	+40.0%	1/0/0
▼ Compare webapi vs webapp side by side	-100.0%	-40.0%	0/0/1
▲ Decide which template fits a background processing scenario	+100.0%	+40.0%	1/0/0

⚠️ template-comparison — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +66.7% (2W/1T/0L over 3 trial(s), sign test p=0.250), mean preference +26.7% — underpowered (3 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 3 (2W/1T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
= Choose between blazor and blazorwasm	+0.0%	+0.0%	0/1/0
▲ Compare webapi vs webapp side by side	+100.0%	+40.0%	1/0/0
▲ Decide which template fits a background processing scenario	+100.0%	+40.0%	1/0/0

⚠️ template-comparison — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +66.7% (2W/1T/0L over 3 trial(s), sign test p=0.250), mean preference +46.7% — underpowered (3 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 3 (2W/1T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
= Choose between blazor and blazorwasm	+0.0%	+0.0%	0/1/0
▲ Compare webapi vs webapp side by side	+100.0%	+100.0%	1/0/0
▲ Decide which template fits a background processing scenario	+100.0%	+40.0%	1/0/0

⚠️ template-comparison — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (3W/0T/0L over 3 trial(s), sign test p=0.125), mean preference +40.0% — underpowered (3 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 3 (3W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Choose between blazor and blazorwasm	+100.0%	+40.0%	1/0/0
▲ Compare webapi vs webapp side by side	+100.0%	+40.0%	1/0/0
▲ Decide which template fits a background processing scenario	+100.0%	+40.0%	1/0/0

⚠️ template-comparison — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +33.3% (1W/2T/0L over 3 trial(s), sign test p=0.500), mean preference +13.3% — underpowered (3 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 3 (1W/2T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
= Choose between blazor and blazorwasm	+0.0%	+0.0%	0/1/0
▲ Compare webapi vs webapp side by side	+100.0%	+40.0%	1/0/0
= Decide which template fits a background processing scenario	+0.0%	+0.0%	0/1/0

❌ template-discovery — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +20.0% (3W/0T/2L over 5 trial(s), sign test p=0.500), mean preference +20.0% — not credible (sign test p=0.500 > 0.05)

Effective scenarios (report only): 5 (3W/0T/2L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Find template for web API project	+100.0%	+100.0%	1/0/0
▼ Inspect template parameters and compare choices	-100.0%	-40.0%	0/0/1
▲ Preview project creation with dry run	+100.0%	+40.0%	1/0/0
▼ Resolve ambiguous project intent to multiple candidates	-100.0%	-40.0%	0/0/1
▲ Search NuGet for specialized template	+100.0%	+40.0%	1/0/0

❌ template-discovery — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +40.0% (3W/1T/1L over 5 trial(s), sign test p=0.312), mean preference +40.0% — not credible — 1 of 5 trial(s) tied, leaving only 4 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)

Effective scenarios (report only): 5 (3W/1T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Find template for web API project	+100.0%	+100.0%	1/0/0
▲ Inspect template parameters and compare choices	+100.0%	+40.0%	1/0/0
= Preview project creation with dry run	+0.0%	+0.0%	0/1/0
▲ Resolve ambiguous project intent to multiple candidates	+100.0%	+100.0%	1/0/0
▼ Search NuGet for specialized template	-100.0%	-40.0%	0/0/1

❌ template-discovery — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +80.0% (4W/1T/0L over 5 trial(s), sign test p=0.063), mean preference +44.0% — not credible — 1 of 5 trial(s) tied, leaving only 4 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)

Effective scenarios (report only): 5 (4W/1T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Find template for web API project	+100.0%	+40.0%	1/0/0
▲ Inspect template parameters and compare choices	+100.0%	+40.0%	1/0/0
▲ Preview project creation with dry run	+100.0%	+100.0%	1/0/0
▲ Resolve ambiguous project intent to multiple candidates	+100.0%	+40.0%	1/0/0
= Search NuGet for specialized template	+0.0%	+0.0%	0/1/0

❌ template-discovery — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +80.0% (4W/1T/0L over 5 trial(s), sign test p=0.063), mean preference +32.0% — not credible — 1 of 5 trial(s) tied, leaving only 4 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)

Effective scenarios (report only): 5 (4W/1T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Find template for web API project	+100.0%	+40.0%	1/0/0
= Inspect template parameters and compare choices	+0.0%	+0.0%	0/1/0
▲ Preview project creation with dry run	+100.0%	+40.0%	1/0/0
▲ Resolve ambiguous project intent to multiple candidates	+100.0%	+40.0%	1/0/0
▲ Search NuGet for specialized template	+100.0%	+40.0%	1/0/0

❌ template-discovery — details

State: VALID_NO_CHANGE (no_credible_preference_change)

Reason: Net win +60.0% (4W/0T/1L over 5 trial(s), sign test p=0.188), mean preference +12.0% — not credible (sign test p=0.188 > 0.05)

Effective scenarios (report only): 5 (4W/0T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Find template for web API project	+100.0%	+40.0%	1/0/0
▲ Inspect template parameters and compare choices	+100.0%	+40.0%	1/0/0
▲ Preview project creation with dry run	+100.0%	+40.0%	1/0/0
▲ Resolve ambiguous project intent to multiple candidates	+100.0%	+40.0%	1/0/0
▼ Search NuGet for specialized template	-100.0%	-100.0%	0/0/1

⚠️ template-instantiation — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win -100.0% (0W/0T/1L over 1 trial(s), sign test p=0.500), mean preference -40.0% — underpowered (1 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 1 (0W/0T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▼ Create a console application	-100.0%	-40.0%	0/0/1

⚠️ template-instantiation — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (1W/0T/0L over 1 trial(s), sign test p=0.500), mean preference +40.0% — underpowered (1 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 1 (1W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Create a console application	+100.0%	+40.0%	1/0/0

⚠️ template-instantiation — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (1W/0T/0L over 1 trial(s), sign test p=0.500), mean preference +100.0% — underpowered (1 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 1 (1W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Create a console application	+100.0%	+100.0%	1/0/0

⚠️ template-instantiation — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (1W/0T/0L over 1 trial(s), sign test p=0.500), mean preference +40.0% — underpowered (1 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 1 (1W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Create a console application	+100.0%	+40.0%	1/0/0

⚠️ template-instantiation — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +0.0% (0W/1T/0L over 1 trial(s), sign test p=1.000), mean preference +0.0% — underpowered (1 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 1 (0W/1T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
= Create a console application	+0.0%	+0.0%	0/1/0

⚠️ template-smart-defaults — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win -50.0% (1W/0T/3L over 4 trial(s), sign test p=0.312), mean preference -20.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (1W/0T/3L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▼ AOT implies a compatible framework	-100.0%	-40.0%	0/0/1
▼ Auth implies HTTPS stays enabled	-100.0%	-40.0%	0/0/1
▼ Controllers exclude the minimal-API flag	-100.0%	-40.0%	0/0/1
▲ Never override an explicit user value	+100.0%	+40.0%	1/0/0

⚠️ template-smart-defaults — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +75.0% (3W/1T/0L over 4 trial(s), sign test p=0.125), mean preference +45.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (3W/1T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ AOT implies a compatible framework	+100.0%	+40.0%	1/0/0
▲ Auth implies HTTPS stays enabled	+100.0%	+40.0%	1/0/0
= Controllers exclude the minimal-API flag	+0.0%	+0.0%	0/1/0
▲ Never override an explicit user value	+100.0%	+100.0%	1/0/0

⚠️ template-smart-defaults — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win -50.0% (0W/2T/2L over 4 trial(s), sign test p=0.250), mean preference -20.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (0W/2T/2L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
= AOT implies a compatible framework	+0.0%	+0.0%	0/1/0
▼ Auth implies HTTPS stays enabled	-100.0%	-40.0%	0/0/1
= Controllers exclude the minimal-API flag	+0.0%	+0.0%	0/1/0
▼ Never override an explicit user value	-100.0%	-40.0%	0/0/1

⚠️ template-smart-defaults — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +50.0% (3W/0T/1L over 4 trial(s), sign test p=0.312), mean preference +20.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (3W/0T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ AOT implies a compatible framework	+100.0%	+40.0%	1/0/0
▲ Auth implies HTTPS stays enabled	+100.0%	+40.0%	1/0/0
▼ Controllers exclude the minimal-API flag	-100.0%	-40.0%	0/0/1
▲ Never override an explicit user value	+100.0%	+40.0%	1/0/0

⚠️ template-smart-defaults — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +0.0% (1W/2T/1L over 4 trial(s), sign test p=0.750), mean preference +0.0% — underpowered (4 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 4 (1W/2T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ AOT implies a compatible framework	+100.0%	+40.0%	1/0/0
▼ Auth implies HTTPS stays enabled	-100.0%	-40.0%	0/0/1
= Controllers exclude the minimal-API flag	+0.0%	+0.0%	0/1/0
= Never override an explicit user value	+0.0%	+0.0%	0/1/0

⚠️ template-validation — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (2W/0T/0L over 2 trial(s), sign test p=0.250), mean preference +70.0% — underpowered (2 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 2 (2W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Validate correct template and suggest improvements	+100.0%	+100.0%	1/0/0
▲ Validate template with multiple errors	+100.0%	+40.0%	1/0/0

⚠️ template-validation — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (2W/0T/0L over 2 trial(s), sign test p=0.250), mean preference +70.0% — underpowered (2 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 2 (2W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Validate correct template and suggest improvements	+100.0%	+100.0%	1/0/0
▲ Validate template with multiple errors	+100.0%	+40.0%	1/0/0

⚠️ template-validation — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win -50.0% (0W/1T/1L over 2 trial(s), sign test p=0.500), mean preference -20.0% — underpowered (2 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 2 (0W/1T/1L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▼ Validate correct template and suggest improvements	-100.0%	-40.0%	0/0/1
= Validate template with multiple errors	+0.0%	+0.0%	0/1/0

⚠️ template-validation — details

State: INVALID_INCONCLUSIVE (underpowered)

Reason: Net win +100.0% (2W/0T/0L over 2 trial(s), sign test p=0.250), mean preference +100.0% — underpowered (2 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Effective scenarios (report only): 2 (2W/0T/0L).

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Validate correct template and suggest improvements	+100.0%	+100.0%	1/0/0
▲ Validate template with multiple errors	+100.0%	+100.0%	1/0/0

Per-scenario details for 7 skill(s) were omitted to keep this comment under GitHub's 65,536-character limit — open the job's step summary or Full Results for the complete breakdown.

🔍 Full Results - additional metrics and failure investigation steps

To investigate failures, paste this to your AI coding agent:

For PR 1027 in dotnet/skills, download eval artifacts with gh run download 32085934523 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/24d0a0697fe5c868928d452ca11856a9268a24e2/eng/vally-adapter/InvestigatingResults.md and follow it to analyze the results.json files. Diagnose each failure, suggest fixes to the eval.yaml and skill content, and tell me what to fix first.

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[github-actions]

📊 Skill Evaluation Results

5 skill(s) evaluated — ✅ 2 improved, ❌ 3 no credible change, 🔻 0 regressed.

A skill passes only on a credible net win over baseline: more wins than losses, by an exact one-sided sign test at p ≤ 0.05.

Skill	Result	Net win	p	Δ Pref	W/T/L	Quality	Baseline	Overfit	Skills Loaded
coverage-analysis	❌	+38.9%	0.059	+38.9%	11/3/4	4.5/5	3.4/5	🟡 0.23	⚠️ 6/9 · 7/9 (plugin)
coverage-analysis	❌	+27.8%	0.151	+11.1%	10/3/5	4.6/5	3.9/5	✅ 0.16	⚠️ 6/9 · 8/9 (plugin)
find-untested-sources	✅	+100.0%	0.000	+60.0%	18/0/0	4.8/5	3.8/5	🟡 0.37	6/6 · 6/6 (plugin)
find-untested-sources	✅	+72.2%	0.000	+42.2%	13/5/0	4.5/5	3.6/5	🟡 0.24	6/6 · 6/6 (plugin)
generate-testability-wrappers	❌	+40.0%	0.073	+40.0%	9/3/3	4.8/5	2.9/5	🟡 0.41	4/4 · 4/4 (plugin)

ℹ️ Column legend

• Net win — (wins − losses) / trials for skilled vs baseline, judged head-to-head by vally compare. This is the effect the gate decides on.
• p — one-sided exact sign test over the discordant (non-tie) trials. A skill passes only at p ≤ 0.05, which needs at least 5 winning trials.
• Δ Pref — the same comparison weighted by how decisive each win was (much-better ±100%, slightly-better ±40%). Reported for triage only: weighting the statistic by magnitude made a skill fail for winning harder, which is why the gate deliberately ignores this column.
• W/T/L — wins / ties / losses across trials.
• ⚠️ — the gate withheld a verdict. Either the eval has fewer trials than any result needs to reach p ≤ 0.05 (underpowered — the skill was never actually measured, so this is not a regression; add scenarios or raise defaults.runs), or the comparison didn't complete.
• 🔻 — a credible regression: the losses themselves clear the same bar the gate uses for wins.
• Quality / Baseline — mean absolute judge score 0–5 (skilled isolated vs skill-free control).
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) with its score.
• Skills Loaded — of the scenarios that expect activation, how many actually activated / that total (plugin run shown when present); ⚠️ marks a scenario that expected activation but didn't activate.

❌ coverage-analysis — details

Reason: Net win +38.9% (11W/3T/4L over 18 trial(s), sign test p=0.059), mean preference +38.9% — not credible (sign test p=0.059 > 0.05)

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Analyse a CI Cobertura report without re-running tests or installing tools	+100.0%	+40.0%	2/0/0
▲ Coverage plateau diagnosis	+50.0%	+20.0%	1/1/0
▼ Distinguish partially covered branches from covered lines	-50.0%	-20.0%	0/1/1
= Keep MTP coverage arguments behind the separator in SDK 10 VSTest mode	+0.0%	+30.0%	1/0/1
▲ Preserve a classic packages.config project when coverage data is absent	+100.0%	+100.0%	2/0/0
▲ Project-wide coverage analysis with existing Cobertura data	+100.0%	+100.0%	2/0/0
▼ Reconcile a coverage target spread across several members	-100.0%	-40.0%	0/0/2
▲ Refactoring safety assessment from coverage data	+100.0%	+100.0%	2/0/0
▲ Run coverage from scratch without existing data	+50.0%	+20.0%	1/1/0

❌ coverage-analysis — details

Reason: Net win +27.8% (10W/3T/5L over 18 trial(s), sign test p=0.151), mean preference +11.1% — not credible (sign test p=0.151 > 0.05)

Scenario	Net win	Δ Pref	Trials (W/T/L)
= Analyse a CI Cobertura report without re-running tests or installing tools	+0.0%	+0.0%	1/0/1
▲ Coverage plateau diagnosis	+100.0%	+40.0%	2/0/0
▼ Distinguish partially covered branches from covered lines	-50.0%	-20.0%	0/1/1
▼ Keep MTP coverage arguments behind the separator in SDK 10 VSTest mode	-50.0%	-20.0%	0/1/1
▲ Preserve a classic packages.config project when coverage data is absent	+50.0%	+20.0%	1/1/0
▲ Project-wide coverage analysis with existing Cobertura data	+100.0%	+40.0%	2/0/0
▼ Reconcile a coverage target spread across several members	-100.0%	-40.0%	0/0/2
▲ Refactoring safety assessment from coverage data	+100.0%	+40.0%	2/0/0
▲ Run coverage from scratch without existing data	+100.0%	+40.0%	2/0/0

❌ generate-testability-wrappers — details

Reason: Net win +40.0% (9W/3T/3L over 15 trial(s), sign test p=0.073), mean preference +40.0% — not credible (sign test p=0.073 > 0.05)

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Decline wrapper generation for already-abstracted code	+100.0%	+100.0%	3/0/0
▲ Generate TimeProvider adoption for DateTime.UtcNow	+33.3%	+13.3%	1/2/0
▲ Generate custom Environment wrapper	+33.3%	+13.3%	2/0/1
▼ Make time controllable in a library that has no DI container	-66.7%	-26.7%	0/1/2
▲ Recommend System.IO.Abstractions for file system calls	+100.0%	+100.0%	3/0/0

Per-scenario details for 2 skill(s) were omitted to keep this comment under GitHub's 65,536-character limit — open the job's step summary or Full Results for the complete breakdown.

🔍 Full Results - additional metrics and failure investigation steps

To investigate failures, paste this to your AI coding agent:

For PR 1027 in dotnet/skills, download eval artifacts with gh run download 32281833213 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/4a833351b67d525402713c947d8d20bd9941ecbf/eng/vally-adapter/InvestigatingResults.md and follow it to analyze the results.json files. Diagnose each failure, suggest fixes to the eval.yaml and skill content, and tell me what to fix first.

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[JanKrivanek]

This PR also requires an absolute task-level net win of at least 20%, so this sparse result does not pass.

Might this happen to be too strict for some niche skills? Should there be option to override it?

[JanKrivanek]

Overall - looks good to go

[github-actions]

📊 Skill Evaluation Results

7 skill(s) evaluated — ✅ 4 improved, ❌ 3 no credible change, 🔻 0 regressed.

A skill passes only on a credible net win over baseline: more wins than losses, by an exact one-sided sign test at p ≤ 0.05.

Skill	Result	Net win	p	Δ Pref	W/T/L	Quality	Baseline	Overfit	Skills Loaded
coverage-analysis	❌	+38.9%	0.059	+38.9%	11/3/4	4.5/5	3.4/5	🟡 0.23	⚠️ 6/9 · 7/9 (plugin)
coverage-analysis	❌	+27.8%	0.151	+11.1%	10/3/5	4.6/5	3.9/5	✅ 0.16	⚠️ 6/9 · 8/9 (plugin)
find-untested-sources	✅	+100.0%	0.000	+60.0%	18/0/0	4.8/5	3.8/5	🟡 0.37	6/6 · 6/6 (plugin)
find-untested-sources	✅	+72.2%	0.000	+42.2%	13/5/0	4.5/5	3.6/5	🟡 0.24	6/6 · 6/6 (plugin)
generate-testability-wrappers	❌	+40.0%	0.073	+40.0%	9/3/3	4.8/5	2.9/5	🟡 0.41	4/4 · 4/4 (plugin)
grade-tests	✅	+88.9%	0.000	+85.6%	17/0/1	3.9/5	1.3/5	🔴 0.62	6/6 · 6/6 (plugin)
grade-tests	✅	+94.4%	0.000	+87.8%	17/1/0	4.8/5	1.8/5	🟡 0.49	6/6 · 6/6 (plugin)

ℹ️ Column legend

• Net win — (wins − losses) / trials for skilled vs baseline, judged head-to-head by vally compare. This is the effect the gate decides on.
• p — one-sided exact sign test over the discordant (non-tie) trials. A skill passes only at p ≤ 0.05, which needs at least 5 winning trials.
• Δ Pref — the same comparison weighted by how decisive each win was (much-better ±100%, slightly-better ±40%). Reported for triage only: weighting the statistic by magnitude made a skill fail for winning harder, which is why the gate deliberately ignores this column.
• W/T/L — wins / ties / losses across trials.
• ⚠️ — the gate withheld a verdict. Either the eval has fewer trials than any result needs to reach p ≤ 0.05 (underpowered — the skill was never actually measured, so this is not a regression; add scenarios or raise defaults.runs), or the comparison didn't complete.
• 🔻 — a credible regression: the losses themselves clear the same bar the gate uses for wins.
• Quality / Baseline — mean absolute judge score 0–5 (skilled isolated vs skill-free control).
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) with its score.
• Skills Loaded — of the scenarios that expect activation, how many actually activated / that total (plugin run shown when present); ⚠️ marks a scenario that expected activation but didn't activate.

❌ coverage-analysis — details

Reason: Net win +38.9% (11W/3T/4L over 18 trial(s), sign test p=0.059), mean preference +38.9% — not credible (sign test p=0.059 > 0.05)

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Analyse a CI Cobertura report without re-running tests or installing tools	+100.0%	+40.0%	2/0/0
▲ Coverage plateau diagnosis	+50.0%	+20.0%	1/1/0
▼ Distinguish partially covered branches from covered lines	-50.0%	-20.0%	0/1/1
= Keep MTP coverage arguments behind the separator in SDK 10 VSTest mode	+0.0%	+30.0%	1/0/1
▲ Preserve a classic packages.config project when coverage data is absent	+100.0%	+100.0%	2/0/0
▲ Project-wide coverage analysis with existing Cobertura data	+100.0%	+100.0%	2/0/0
▼ Reconcile a coverage target spread across several members	-100.0%	-40.0%	0/0/2
▲ Refactoring safety assessment from coverage data	+100.0%	+100.0%	2/0/0
▲ Run coverage from scratch without existing data	+50.0%	+20.0%	1/1/0

❌ coverage-analysis — details

Reason: Net win +27.8% (10W/3T/5L over 18 trial(s), sign test p=0.151), mean preference +11.1% — not credible (sign test p=0.151 > 0.05)

Scenario	Net win	Δ Pref	Trials (W/T/L)
= Analyse a CI Cobertura report without re-running tests or installing tools	+0.0%	+0.0%	1/0/1
▲ Coverage plateau diagnosis	+100.0%	+40.0%	2/0/0
▼ Distinguish partially covered branches from covered lines	-50.0%	-20.0%	0/1/1
▼ Keep MTP coverage arguments behind the separator in SDK 10 VSTest mode	-50.0%	-20.0%	0/1/1
▲ Preserve a classic packages.config project when coverage data is absent	+50.0%	+20.0%	1/1/0
▲ Project-wide coverage analysis with existing Cobertura data	+100.0%	+40.0%	2/0/0
▼ Reconcile a coverage target spread across several members	-100.0%	-40.0%	0/0/2
▲ Refactoring safety assessment from coverage data	+100.0%	+40.0%	2/0/0
▲ Run coverage from scratch without existing data	+100.0%	+40.0%	2/0/0

❌ generate-testability-wrappers — details

Reason: Net win +40.0% (9W/3T/3L over 15 trial(s), sign test p=0.073), mean preference +40.0% — not credible (sign test p=0.073 > 0.05)

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Decline wrapper generation for already-abstracted code	+100.0%	+100.0%	3/0/0
▲ Generate TimeProvider adoption for DateTime.UtcNow	+33.3%	+13.3%	1/2/0
▲ Generate custom Environment wrapper	+33.3%	+13.3%	2/0/1
▼ Make time controllable in a library that has no DI container	-66.7%	-26.7%	0/1/2
▲ Recommend System.IO.Abstractions for file system calls	+100.0%	+100.0%	3/0/0

Per-scenario details for 4 skill(s) were omitted to keep this comment under GitHub's 65,536-character limit — open the job's step summary or Full Results for the complete breakdown.

🔍 Full Results - additional metrics and failure investigation steps

To investigate failures, paste this to your AI coding agent:

For PR 1027 in dotnet/skills, download eval artifacts with gh run download 32281833213 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/4a833351b67d525402713c947d8d20bd9941ecbf/eng/vally-adapter/InvestigatingResults.md and follow it to analyze the results.json files. Diagnose each failure, suggest fixes to the eval.yaml and skill content, and tell me what to fix first.

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[AbhitejJohn]

@JanKrivanek The 20% floor is repository policy rather than a Vally default, and I did not add a per-eval override. It does not make small or niche evals harder to pass: for every instrument from 5 through 25 stimuli, any W/T/L record that can pass the exact one-sided sign test already has net win >=20%. The existing exhaustive boundary test proves this. The first record newly rejected by the floor is 5W/21T/0L at 26 stimuli: p = 0.03125, but net win is only 5/26 = 19.2%. An override would therefore matter only for larger instruments with statistically credible but sparse effects, which is the false-positive class the floor is meant to stop; it would also allow threshold tuning after results are known. I documented this boundary in 00998ca4e. If later evidence shows 20% is wrong, I recommend changing one predeclared, versioned repository policy for all evals rather than adding result-specific exceptions.

[Copilot]

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

.github/workflows/evaluation-run.yml:168

• The pinned SHA for actions/upload-artifact is annotated as v4.6.2 here, but the same SHA is used elsewhere in this workflow with a v7 comment. This inconsistency can confuse audits and future upgrades; align the version comment with the actual action version you intend to pin.

This issue also appears on line 433 of the same file.

      - name: Upload trusted skill-validator archive
        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v4.6.2

.github/workflows/evaluation-run.yml:436

• The pinned SHA for actions/download-artifact is annotated as v4.3.0 here, but the same SHA is referenced elsewhere in the repo as v8.x. Keeping the inline version comment accurate avoids confusion during dependency reviews.

      - name: Download trusted skill-validator archive
        if: steps.find-evals.outputs.has_evals == 'true'
        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v4.3.0
        with:

• Files reviewed: 28/28 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

• Files reviewed: 28/28 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

• Files reviewed: 28/28 changed files
• Comments generated: 2
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

eng/vally-adapter/consolidate.mjs:498

• In the main markdown table, Skill and Model cells are only passed through td() (pipe/newline escaping) and are not HTML-escaped. Because these values originate from results produced against evaluated content, a crafted name containing </& could inject HTML and break or spoof the PR comment rendering. Escape skillName and model with html() before table rendering (while keeping td() for table-safety).

    const common = [
      verdict.skillName,
      verdict.model,
      resultLabel(verdict),
      gateEvidence(verdict),
    ];

• Files reviewed: 28/28 changed files
• Comments generated: 0 new
• Review effort level: Lite

[github-actions]

📊 Skill Evaluation Results

5 skill(s) evaluated — ✅ 4 improved, ❌ 0 no credible change, 🔻 0 regressed.

⚠️ 1 could not be judged: 1 inconclusive — the comparison didn't complete (errored, unmatched, or self-contradictory trials).

A skill passes only on a credible net win over baseline: more wins than losses, by an exact one-sided sign test at p ≤ 0.05.

Skill	Result	Net win	p	Δ Pref	W/T/L	Quality	Baseline	Overfit	Skills Loaded
coverage-analysis	✅	+66.7%	0.002	+53.3%	14/2/2	4.4/5	3.4/5	🟡 0.28	⚠️ 6/9 · 7/9 (plugin)
generate-testability-wrappers	✅	+53.3%	0.019	+45.3%	10/3/2	4.7/5	2.8/5	🟡 0.41	4/4 · 4/4 (plugin)
generate-testability-wrappers	✅	+60.0%	0.011	+48.0%	11/2/2	4.0/5	2.1/5	✅ 0.17	4/4 · 4/4 (plugin)
grade-tests	✅	+88.9%	0.000	+82.2%	17/0/1	3.8/5	1.6/5	🔴 0.58	6/6 · 6/6 (plugin)
grade-tests	⚠️	+100.0%	0.000	+89.4%	17/0/0	4.9/5	1.6/5	🔴 0.62	6/6 · 6/6 (plugin)

ℹ️ Column legend

• Net win — (wins − losses) / trials for skilled vs baseline, judged head-to-head by vally compare. This is the effect the gate decides on.
• p — one-sided exact sign test over the discordant (non-tie) trials. A skill passes only at p ≤ 0.05, which needs at least 5 winning trials.
• Δ Pref — the same comparison weighted by how decisive each win was (much-better ±100%, slightly-better ±40%). Reported for triage only: weighting the statistic by magnitude made a skill fail for winning harder, which is why the gate deliberately ignores this column.
• W/T/L — wins / ties / losses across trials.
• ⚠️ — the gate withheld a verdict. Either the eval has fewer trials than any result needs to reach p ≤ 0.05 (underpowered — the skill was never actually measured, so this is not a regression; add scenarios or raise defaults.runs), or the comparison didn't complete.
• 🔻 — a credible regression: the losses themselves clear the same bar the gate uses for wins.
• Quality / Baseline — mean absolute judge score 0–5 (skilled isolated vs skill-free control).
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) with its score.
• Skills Loaded — of the scenarios that expect activation, how many actually activated / that total (plugin run shown when present); ⚠️ marks a scenario that expected activation but didn't activate.

⚠️ grade-tests — details

Reason: Net win +100.0% (17W/0T/0L over 17 trial(s), sign test p=0.000), mean preference +89.4%, 1 unmatched — inconclusive (unmatched trajectories)

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Ask for a bounded list instead of grading the workspace	+100.0%	+100.0%	3/0/0
▲ Grade C# tests against available production code	+100.0%	+100.0%	3/0/0
▲ Grade Go table-driven tests without misreading the loop as branching	+100.0%	+100.0%	3/0/0
▲ Grade pytest test methods using the same rubric	+100.0%	+60.0%	3/0/0
▲ Grade tests when the production code under test is unavailable	+100.0%	+100.0%	3/0/0
▲ Keep a 62-test grading report readable as a PR comment	+100.0%	+70.0%	2/0/0

Per-scenario details for 4 skill(s) were omitted to keep this comment under GitHub's 65,536-character limit — open the job's step summary or Full Results for the complete breakdown.

🔍 Full Results - additional metrics and failure investigation steps

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[Copilot]

Review details

• Files reviewed: 28/28 changed files
• Comments generated: 0 new
• Review effort level: Lite

[github-actions]

📊 Skill Evaluation Results

5 skill(s) evaluated — ✅ 4 improved, ❌ 0 no credible change, 🔻 0 regressed.

⚠️ 1 could not be judged: 1 inconclusive — the comparison didn't complete (errored, unmatched, or self-contradictory trials).

A skill passes only on a credible net win over baseline: more wins than losses, by an exact one-sided sign test at p ≤ 0.05.

Skill	Result	Net win	p	Δ Pref	W/T/L	Quality	Baseline	Overfit	Skills Loaded
find-untested-sources	✅	+88.9%	0.000	+55.6%	17/0/1	4.8/5	3.9/5	🟡 0.41	6/6 · 6/6 (plugin)
generate-testability-wrappers	✅	+66.7%	0.003	+50.7%	11/3/1	4.8/5	2.6/5	🟡 0.46	4/4 · 4/4 (plugin)
generate-testability-wrappers	⚠️	+71.4%	0.003	+50.0%	11/2/1	4.0/5	2.2/5	✅ 0.16	4/4 · 4/4 (plugin)
grade-tests	✅	+94.4%	0.000	+84.4%	17/1/0	3.7/5	1.3/5	🔴 0.53	6/6 · 6/6 (plugin)
grade-tests	✅	+100.0%	0.000	+93.3%	18/0/0	4.8/5	1.8/5	🟡 0.50	6/6 · 6/6 (plugin)

ℹ️ Column legend

• Net win — (wins − losses) / trials for skilled vs baseline, judged head-to-head by vally compare. This is the effect the gate decides on.
• p — one-sided exact sign test over the discordant (non-tie) trials. A skill passes only at p ≤ 0.05, which needs at least 5 winning trials.
• Δ Pref — the same comparison weighted by how decisive each win was (much-better ±100%, slightly-better ±40%). Reported for triage only: weighting the statistic by magnitude made a skill fail for winning harder, which is why the gate deliberately ignores this column.
• W/T/L — wins / ties / losses across trials.
• ⚠️ — the gate withheld a verdict. Either the eval has fewer trials than any result needs to reach p ≤ 0.05 (underpowered — the skill was never actually measured, so this is not a regression; add scenarios or raise defaults.runs), or the comparison didn't complete.
• 🔻 — a credible regression: the losses themselves clear the same bar the gate uses for wins.
• Quality / Baseline — mean absolute judge score 0–5 (skilled isolated vs skill-free control).
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) with its score.
• Skills Loaded — of the scenarios that expect activation, how many actually activated / that total (plugin run shown when present); ⚠️ marks a scenario that expected activation but didn't activate.

⚠️ generate-testability-wrappers — details

Reason: Net win +71.4% (11W/2T/1L over 14 trial(s), sign test p=0.003), mean preference +50.0%, 1 unmatched — inconclusive (unmatched trajectories)

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Decline wrapper generation for already-abstracted code	+100.0%	+100.0%	2/0/0
▲ Generate TimeProvider adoption for DateTime.UtcNow	+33.3%	+13.3%	2/0/1
▲ Generate custom Environment wrapper	+33.3%	+13.3%	1/2/0
▲ Make time controllable in a library that has no DI container	+100.0%	+40.0%	3/0/0
▲ Recommend System.IO.Abstractions for file system calls	+100.0%	+100.0%	3/0/0

Per-scenario details for 4 skill(s) were omitted to keep this comment under GitHub's 65,536-character limit — open the job's step summary or Full Results for the complete breakdown.

🔍 Full Results - additional metrics and failure investigation steps

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[Copilot]

Review details

• Files reviewed: 28/28 changed files
• Comments generated: 0 new
• Review effort level: Lite

[github-actions]

📊 Skill Evaluation Results

6 skill(s) evaluated — ✅ 6 improved, ❌ 0 no credible change, 🔻 0 regressed.

A skill passes only on a credible net win over baseline: more wins than losses, by an exact one-sided sign test at p ≤ 0.05.

Skill	Result	Net win	p	Δ Pref	W/T/L	Quality	Baseline	Overfit	Skills Loaded
coverage-analysis	✅	+77.8%	0.000	+34.4%	15/2/1	4.7/5	3.8/5	✅ 0.17	⚠️ 6/9 · 8/9 (plugin)
find-untested-sources	✅	+83.3%	0.000	+50.0%	16/1/1	4.9/5	3.9/5	🟡 0.28	6/6 · 6/6 (plugin)
find-untested-sources	✅	+50.0%	0.011	+16.7%	11/5/2	4.2/5	3.6/5	🟡 0.24	6/6 · 6/6 (plugin)
generate-testability-wrappers	✅	+53.3%	0.029	+49.3%	11/1/3	4.7/5	2.7/5	🟡 0.42	4/4 · 4/4 (plugin)
generate-testability-wrappers	✅	+66.7%	0.003	+42.7%	11/3/1	4.1/5	2.2/5	✅ 0.16	4/4 · 4/4 (plugin)
grade-tests	✅	+94.4%	0.000	+81.1%	17/1/0	4.8/5	1.8/5	🔴 0.55	6/6 · 6/6 (plugin)

ℹ️ Column legend

• Net win — (wins − losses) / trials for skilled vs baseline, judged head-to-head by vally compare. This is the effect the gate decides on.
• p — one-sided exact sign test over the discordant (non-tie) trials. A skill passes only at p ≤ 0.05, which needs at least 5 winning trials.
• Δ Pref — the same comparison weighted by how decisive each win was (much-better ±100%, slightly-better ±40%). Reported for triage only: weighting the statistic by magnitude made a skill fail for winning harder, which is why the gate deliberately ignores this column.
• W/T/L — wins / ties / losses across trials.
• ⚠️ — the gate withheld a verdict. Either the eval has fewer trials than any result needs to reach p ≤ 0.05 (underpowered — the skill was never actually measured, so this is not a regression; add scenarios or raise defaults.runs), or the comparison didn't complete.
• 🔻 — a credible regression: the losses themselves clear the same bar the gate uses for wins.
• Quality / Baseline — mean absolute judge score 0–5 (skilled isolated vs skill-free control).
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) with its score.
• Skills Loaded — of the scenarios that expect activation, how many actually activated / that total (plugin run shown when present); ⚠️ marks a scenario that expected activation but didn't activate.

Per-scenario details for 6 skill(s) were omitted to keep this comment under GitHub's 65,536-character limit — open the job's step summary or Full Results for the complete breakdown.

🔍 Full Results - additional metrics and failure investigation steps

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[github-actions]

📊 Skill Evaluation Results

7 skill(s) evaluated — ✅ 7 improved, ❌ 0 no credible change, 🔻 0 regressed.

A skill passes only on a credible net win over baseline: more wins than losses, by an exact one-sided sign test at p ≤ 0.05.

Skill	Result	Net win	p	Δ Pref	W/T/L	Quality	Baseline	Overfit	Skills Loaded
coverage-analysis	✅	+77.8%	0.000	+34.4%	15/2/1	4.7/5	3.8/5	✅ 0.17	⚠️ 6/9 · 8/9 (plugin)
find-untested-sources	✅	+83.3%	0.000	+50.0%	16/1/1	4.9/5	3.9/5	🟡 0.28	6/6 · 6/6 (plugin)
find-untested-sources	✅	+50.0%	0.011	+16.7%	11/5/2	4.2/5	3.6/5	🟡 0.24	6/6 · 6/6 (plugin)
generate-testability-wrappers	✅	+53.3%	0.029	+49.3%	11/1/3	4.7/5	2.7/5	🟡 0.42	4/4 · 4/4 (plugin)
generate-testability-wrappers	✅	+66.7%	0.003	+42.7%	11/3/1	4.1/5	2.2/5	✅ 0.16	4/4 · 4/4 (plugin)
grade-tests	✅	+72.2%	0.001	+68.9%	15/1/2	4.0/5	1.6/5	🔴 0.58	6/6 · 6/6 (plugin)
grade-tests	✅	+94.4%	0.000	+81.1%	17/1/0	4.8/5	1.8/5	🔴 0.55	6/6 · 6/6 (plugin)

ℹ️ Column legend

• Net win — (wins − losses) / trials for skilled vs baseline, judged head-to-head by vally compare. This is the effect the gate decides on.
• p — one-sided exact sign test over the discordant (non-tie) trials. A skill passes only at p ≤ 0.05, which needs at least 5 winning trials.
• Δ Pref — the same comparison weighted by how decisive each win was (much-better ±100%, slightly-better ±40%). Reported for triage only: weighting the statistic by magnitude made a skill fail for winning harder, which is why the gate deliberately ignores this column.
• W/T/L — wins / ties / losses across trials.
• ⚠️ — the gate withheld a verdict. Either the eval has fewer trials than any result needs to reach p ≤ 0.05 (underpowered — the skill was never actually measured, so this is not a regression; add scenarios or raise defaults.runs), or the comparison didn't complete.
• 🔻 — a credible regression: the losses themselves clear the same bar the gate uses for wins.
• Quality / Baseline — mean absolute judge score 0–5 (skilled isolated vs skill-free control).
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) with its score.
• Skills Loaded — of the scenarios that expect activation, how many actually activated / that total (plugin run shown when present); ⚠️ marks a scenario that expected activation but didn't activate.

Per-scenario details for 7 skill(s) were omitted to keep this comment under GitHub's 65,536-character limit — open the job's step summary or Full Results for the complete breakdown.

🔍 Full Results - additional metrics and failure investigation steps

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[github-actions]

📊 Skill Evaluation Results

8 skill(s) evaluated — ✅ 8 improved, ❌ 0 no credible change, 🔻 0 regressed.

A skill passes only on a credible net win over baseline: more wins than losses, by an exact one-sided sign test at p ≤ 0.05.

Skill	Result	Net win	p	Δ Pref	W/T/L	Quality	Baseline	Overfit	Skills Loaded
coverage-analysis	✅	+66.7%	0.002	+53.3%	14/2/2	4.5/5	3.5/5	🟡 0.25	⚠️ 6/9 · 7/9 (plugin)
coverage-analysis	✅	+77.8%	0.000	+34.4%	15/2/1	4.7/5	3.8/5	✅ 0.17	⚠️ 6/9 · 8/9 (plugin)
find-untested-sources	✅	+83.3%	0.000	+50.0%	16/1/1	4.9/5	3.9/5	🟡 0.28	6/6 · 6/6 (plugin)
find-untested-sources	✅	+50.0%	0.011	+16.7%	11/5/2	4.2/5	3.6/5	🟡 0.24	6/6 · 6/6 (plugin)
generate-testability-wrappers	✅	+53.3%	0.029	+49.3%	11/1/3	4.7/5	2.7/5	🟡 0.42	4/4 · 4/4 (plugin)
generate-testability-wrappers	✅	+66.7%	0.003	+42.7%	11/3/1	4.1/5	2.2/5	✅ 0.16	4/4 · 4/4 (plugin)
grade-tests	✅	+72.2%	0.001	+68.9%	15/1/2	4.0/5	1.6/5	🔴 0.58	6/6 · 6/6 (plugin)
grade-tests	✅	+94.4%	0.000	+81.1%	17/1/0	4.8/5	1.8/5	🔴 0.55	6/6 · 6/6 (plugin)

ℹ️ Column legend

• Net win — (wins − losses) / trials for skilled vs baseline, judged head-to-head by vally compare. This is the effect the gate decides on.
• p — one-sided exact sign test over the discordant (non-tie) trials. A skill passes only at p ≤ 0.05, which needs at least 5 winning trials.
• Δ Pref — the same comparison weighted by how decisive each win was (much-better ±100%, slightly-better ±40%). Reported for triage only: weighting the statistic by magnitude made a skill fail for winning harder, which is why the gate deliberately ignores this column.
• W/T/L — wins / ties / losses across trials.
• ⚠️ — the gate withheld a verdict. Either the eval has fewer trials than any result needs to reach p ≤ 0.05 (underpowered — the skill was never actually measured, so this is not a regression; add scenarios or raise defaults.runs), or the comparison didn't complete.
• 🔻 — a credible regression: the losses themselves clear the same bar the gate uses for wins.
• Quality / Baseline — mean absolute judge score 0–5 (skilled isolated vs skill-free control).
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) with its score.
• Skills Loaded — of the scenarios that expect activation, how many actually activated / that total (plugin run shown when present); ⚠️ marks a scenario that expected activation but didn't activate.

Per-scenario details for 8 skill(s) were omitted to keep this comment under GitHub's 65,536-character limit — open the job's step summary or Full Results for the complete breakdown.

🔍 Full Results - additional metrics and failure investigation steps

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions