Flag reference skills that carry a direct-activation eval (#899)

eng/eval-quality/README.md

@@ -368,7 +368,7 @@ way to raise an eval's trial count, because the fixture already exists —
 `migrate-nullable-references` sits at 3 scenarios with three unreferenced
 fixtures beside it.
 
-### Skills with no eval
+### Skill eval coverage
 
 A skill that ships with `SKILL.md` but has no `tests/<plugin>/<skill>/eval.yaml`
 carries zero evidence of impact.
@@ -385,22 +385,32 @@ and adding such an eval would make the number worse, not better.
 
 The honest coverage for these is **dependency-level**: they are exercised
 through the evals of the skills that load them (for example `run-tests` and
-`mtp-hot-reload` for `platform-detection`, the polyglot analysis skills for
-`test-analysis-extensions`, and `code-testing-agent` for
+`mtp-hot-reload` load `platform-detection` and `filter-syntax`, the polyglot
+analysis skills load `test-analysis-extensions`, and `code-testing-agent` loads
 `code-testing-extensions`), and in the plugin arm, where the whole plugin is
 loaded. Closing this properly needs harness support for declaring a dependency
 in the skilled variant, not a per-skill eval file.
 
-> **`filter-syntax` is the exception, added in #976.** It carries a direct
-> `tests/dotnet-test/filter-syntax/eval.yaml` whose stimuli are ordinary
-> user requests ("one command that runs only the integration tests but leaves
-> out the slow ones"), so the skilled arm is graded on whether the answer
-> carries correct filter syntax rather than on whether the skill self-activated.
-> Whether that produces a *measurable* gap over baseline for a skill the model
-> cannot invoke is still unconfirmed: the evaluation on that PR landed during the
-> PAT-pool outage and reported "no results", so no verdict exists for it yet.
-> Worth reading its first real result before copying the pattern to the other
-> three.
+**A reference skill that already has a direct eval is reported too, and more
+loudly.** The same argument cuts both ways: if the skilled arm cannot reach the
+skill, an eval sitting beside it does not measure the skill — it measures the
+judge comparing baseline to baseline and then labels the result a pass or a
+fail. That is worse than no eval, because no eval is visibly zero evidence
+whereas a fabricated verdict is counted in the plugin's pass rate. The gate
+originally skipped any skill that had an eval, which made the worse case the
+quieter one; it now names them.
+
+> **Two `dotnet-test` reference skills currently carry a direct eval:**
+> `filter-syntax` (added in #976) and `platform-detection` (added in #974).
+> Their stimuli are ordinary user requests ("one command that runs only the
+> integration tests but leaves out the slow ones"), so the intent was to grade
+> the answer on whether it carries the correct syntax rather than on whether the
+> skill self-activated. Whether that can produce a *measurable* gap over baseline
+> for a skill the model cannot invoke is still unconfirmed — the evaluation on
+> #976 landed during the PAT-pool outage and reported "no results", and no
+> cross-family run has covered either eval since. Read a real result before
+> copying the pattern to `code-testing-extensions` or `test-analysis-extensions`;
+> if the gap is zero, retire both evals rather than keep scoring noise.
 
 ### Dormancy guard without an anti-hijack rubric item
 

eng/eval-quality/check_eval_quality.py

@@ -590,13 +590,24 @@ def _is_reference_skill(skill_dir: str) -> bool:
 def report_uncovered() -> None:
     missing = []
     reference = []
+    degenerate = []
     for plugin_dir in sorted(glob.glob("plugins/*")):
         plugin = os.path.basename(plugin_dir)
         evals = {os.path.basename(os.path.dirname(f))
                  for f in glob.glob(f"tests/{plugin}/*/eval.yaml")}
         for skill_dir in sorted(glob.glob(f"{plugin_dir}/skills/*")):
             skill = os.path.basename(skill_dir)
-            if not os.path.isdir(skill_dir) or skill in evals:
+            if not os.path.isdir(skill_dir):
+                continue
+            if skill in evals:
+                # A reference skill that *has* a direct eval is the worse half of
+                # this problem, not the solved half: the same argument that says
+                # such an eval would compare two identical arms says the verdict
+                # it produces is judge noise wearing a pass/fail label. Silence
+                # here is how two of these landed after the reasoning was
+                # written down. No eval is honest; a fabricated verdict is not.
+                if _is_reference_skill(skill_dir):
+                    degenerate.append(f"    {plugin}/{skill} — tests/{plugin}/{skill}/eval.yaml")
                 continue
             if _is_reference_skill(skill_dir):
                 reference.append(f"    {plugin}/{skill}")
@@ -612,6 +623,13 @@ def report_uncovered() -> None:
             f"compare two identical arms. Cover them through the consumers that "
             f"load them:")
         warnings.extend(reference)
+    if degenerate:
+        warnings.append(
+            f"{len(degenerate)} reference skill(s) carry a direct-activation eval — they set "
+            f"`disable-model-invocation: true`, so the model cannot reach the skill in the "
+            f"skilled arm either: the eval scores baseline against baseline and its verdict is "
+            f"judge noise. Retire the eval or cover the skill through a consumer:")
+        warnings.extend(degenerate)
 
 
 def check_floor_agreement() -> None:

eng/eval-quality/selftest_eval_quality.py

@@ -102,6 +102,28 @@ def output_case(label, mutate, expect_substring):
 EV = lambda d: os.path.join(d, "tests", "demo", "widget", "eval.yaml")
 
 
+def silent_case(label, mutate, forbidden_substring):
+    """Assert the gate stays quiet — the other half of every warning's contract.
+
+    A warning that fires on well-formed input is worse than no warning: it
+    trains the team to skim past the whole report. Pairing each `output_case`
+    with this keeps the trigger condition pinned from both sides.
+    """
+    d = scratch()
+    try:
+        mutate(d)
+        subprocess.run(["git", "add", "-A"], cwd=d, capture_output=True, check=True)
+        code, out = run_gate(d)
+        ok = code == 0 and forbidden_substring not in out
+        print(f"  [{'OK ' if ok else 'BAD'}] {label:<52} forbidden={forbidden_substring!r}")
+        if not ok:
+            print(f"        exit={code}")
+            print("        " + out.strip().replace("\n", "\n        ")[:900])
+        return ok
+    finally:
+        shutil.rmtree(d, ignore_errors=True)
+
+
 def clean(d):
     pass
 
@@ -246,6 +268,29 @@ def guard_ok(d):
         )
 
 
+# --- reference skills -------------------------------------------------------
+# `disable-model-invocation: true` hides a skill from the model-facing menu, so
+# the skilled arm cannot reach it either and the eval scores baseline against
+# baseline. The gate used to skip any skill that had an eval, which made the
+# worse case (a fabricated verdict) quieter than the better one (no verdict).
+
+def _write_skill_md(d, *, hidden):
+    path = os.path.join(d, "plugins", "demo", "skills", "widget", "SKILL.md")
+    with open(path, "w") as f:
+        f.write("---\nname: widget\ndescription: Does the thing\n")
+        if hidden:
+            f.write("disable-model-invocation: true\n")
+        f.write("---\n\n# Widget\n")
+
+
+def reference_skill_with_a_direct_eval(d):
+    _write_skill_md(d, hidden=True)
+
+
+def invocable_skill_with_a_direct_eval(d):
+    _write_skill_md(d, hidden=False)
+
+
 # --- statistical power ------------------------------------------------------
 # Trials = scenarios x runs. Below the floor the pass gate cannot reach a
 # credible verdict at any effect size, so a new eval must not land there.
@@ -364,6 +409,12 @@ def unresolvable_base_ref(d):
     case("spec declares both config: and defaults:", config_and_defaults_together, expect_fail=True),
     case("dormancy guard also sets reject_skills", guard_with_reject_skills, expect_fail=True),
     case("well-formed dormancy guard", guard_ok, expect_fail=False),
+    output_case("reference skill carrying a direct-activation eval",
+                reference_skill_with_a_direct_eval,
+                "1 reference skill(s) carry a direct-activation eval"),
+    silent_case("model-invocable skill with a direct eval",
+                invocable_skill_with_a_direct_eval,
+                "carry a direct-activation eval"),
     case("eval below the trial floor", underpowered, expect_fail=True),
     case("below the floor but grandfathered", underpowered_but_allowlisted, expect_fail=False),
     output_case("grandfathered warning reports scenarios x runs",

plugins/dotnet-test/README.md

@@ -73,20 +73,22 @@ For non-.NET languages, use the native coverage tool: `coverage.py`/`pytest-cov`
 | **filter-syntax** *(.NET)* | Test filter syntax reference for VSTest and MTP across all frameworks |
 
 These four set `disable-model-invocation: true`, so the CLI keeps them out of the
-model-facing skill menu and a consumer loads them by name. Three of them
-(`code-testing-extensions`, `test-analysis-extensions`, `platform-detection`)
-deliberately have no `tests/dotnet-test/<skill>/eval.yaml`: the experiment's
-skilled arm loads a single skill, which the model could never invoke here, so
-such an eval would compare two identical arms and score judge noise. They are
-measured through the evals of the skills that load them — `run-tests` and
-`mtp-hot-reload` for `platform-detection`, the polyglot analysis skills and
+model-facing skill menu and a consumer loads them by name. Two of them
+(`code-testing-extensions`, `test-analysis-extensions`) deliberately have no
+`tests/dotnet-test/<skill>/eval.yaml`: the experiment's skilled arm loads a
+single skill, which the model could never invoke here, so such an eval would
+compare two identical arms and score judge noise. They are measured through the
+evals of the skills that load them — the polyglot analysis skills and
 `grade-tests` for `test-analysis-extensions`, and `code-testing-agent` for
 `code-testing-extensions`.
 
-`filter-syntax` is the exception: #976 gave it a direct eval built from ordinary
-user requests, so the answer is graded on carrying the right filter syntax rather
-than on the skill self-activating. That approach has not produced a verdict yet
-(its evaluation landed during the PAT-pool outage). See
+`platform-detection` (#974) and `filter-syntax` (#976) are the exceptions: both
+were given a direct eval built from ordinary user requests, so the answer is
+graded on carrying the right detection or filter syntax rather than on the skill
+self-activating. Neither has produced a verdict yet — `filter-syntax` landed
+during the PAT-pool outage, and no cross-family run has covered either since —
+so whether that grading survives an arm the model cannot reach is still an open
+question. The eval-quality gate reports both until it is answered. See
 `eng/eval-quality/README.md`.
 
 ## Agents