Allow expected-failures entries to name a single check (#406)

A baseline entry names a scenario, but a scenario is many checks:
server-stateless is 29. Excusing the one or two a spec change breaks
means baselining the whole scenario, which stops enforcing the rest.

An entry may now be '<scenario>:<check-id>', in which case every failing
check in that scenario is judged on its own. Bare scenario entries are
unchanged, and YAML parses '- a:b' as a plain string, so every existing
baseline keeps working untouched.

loadExpectedFailures now owns the entry grammar and returns parsed
BaselineEntry values, so evaluateBaseline only matches and cannot throw.
Rejecting a bad entry and interpreting a good one no longer live in
separate layers.

A check id matches all of its occurrences, since ids repeat within a run;
they collapse to one verdict, most-severe first. Absent and SKIPPED are
tolerated, matching how the runner already treats them as green.

Move collapseDuplicateChecks to src/checks/collapse.ts with its tests:
expected-failures.ts needs the reducer and cannot import dpop.ts's auth
server stack. No behavior change.

Reject two configs that previously coerced to an entry matching nothing:
a mapping ('- scenario: check-id', with a space) and an empty list item.

Author: felixweinberger

README.md

@@ -134,6 +134,40 @@ This ensures:
 - CI fails on new regressions (unexpected failures)
 - CI fails when a fix lands but the baseline isn't updated (stale entries)
 
+### Baselining a single check
+
+A scenario is many checks — `server-stateless` alone is over twenty — so baselining the
+whole scenario to excuse one of them stops enforcing the other nineteen. An entry can
+instead name a single check, as `<scenario>:<check-id>`:
+
+```yaml
+server:
+  - tasks-lifecycle # whole scenario may fail
+  - server-stateless:sep-2575-server-implements-discover # only this check may fail
+```
+
+The check id is the left-hand column the runner already prints for each check, so it can
+be copied straight out of a failing run.
+
+With a per-check entry, every failing check in that scenario is judged on its own: the
+named one is excused, and any other failure is still an unexpected regression. The four
+exit-code rules above apply per check rather than per scenario, with one addition — a
+baselined check that is absent or skipped is tolerated, because a scenario that bails on
+a failed prerequisite legitimately never reaches its later checks, and the prerequisite
+reports its own failure anyway.
+
+Two things to know:
+
+- **A check id addresses every occurrence of that id.** Ids repeat within a run (a loop,
+  a retried flow), and the occurrences collapse to one verdict, most-severe first. So
+  baselining a repeated id excuses all of its occurrences — coarser than ideal, still far
+  narrower than baselining the scenario.
+- **Mind the space.** `- scenario:check-id` is a string; `- scenario: check-id` is YAML
+  for a mapping and is rejected with an error.
+
+A scenario cannot be listed both wholesale and per-check — the wholesale entry already
+excuses everything, so the pair is contradictory and is rejected.
+
 ## GitHub Action
 
 This repo provides a composite GitHub Action so SDK repos don't need to write their own conformance scripts.

src/scenarios/client/auth/dpop.test.ts src/checks/collapse.test.tssrc/scenarios/client/auth/dpop.test.ts renamed to src/checks/collapse.test.ts

@@ -1,5 +1,5 @@
-import { collapseDuplicateChecks } from './dpop';
-import type { ConformanceCheck, CheckStatus } from '../../../types';
+import { collapseDuplicateChecks } from './collapse';
+import type { ConformanceCheck, CheckStatus } from '../types';
 
 /** Minimal check factory for the dedupe unit tests. */
 function chk(id: string, status: CheckStatus, tag?: string): ConformanceCheck {
@@ -13,7 +13,7 @@ function chk(id: string, status: CheckStatus, tag?: string): ConformanceCheck {
   };
 }
 
-describe('collapseDuplicateChecks (DPoP nonce-posture shared-check dedupe)', () => {
+describe('collapseDuplicateChecks', () => {
   it('collapses duplicate SUCCESS ids to a single entry', () => {
     const out = collapseDuplicateChecks([
       chk('token-request', 'SUCCESS'),

src/checks/collapse.ts

@@ -0,0 +1,35 @@
+import type { CheckStatus, ConformanceCheck } from '../types';
+
+/**
+ * Collapse duplicate non-INFO check IDs to a single entry, preferring the
+ * MOST-SEVERE occurrence (FAILURE > WARNING > SUCCESS > any other status, e.g.
+ * SKIPPED) so a real failure is never masked. Equal-severity ties keep the LAST
+ * occurrence. Per-request INFO log entries are always kept.
+ *
+ * A check ID is not unique within a run: scenarios re-emit a shared ID when a
+ * flow repeats. The RFC 9449 §8/§9 nonce round-trip re-POSTs /token (challenge
+ * → retry), so the shared token-flow checks (`token-request`, `pkce-*`) are
+ * appended twice; `sep-2575-http-server-meta-invalid-400` is emitted once per
+ * iteration of the `_meta` test-case loop. Collapsing reports each ID once
+ * without hiding a failure recorded on any occurrence, which is what lets an
+ * expected-failures baseline address a check by ID.
+ */
+export function collapseDuplicateChecks(
+  checks: ConformanceCheck[]
+): ConformanceCheck[] {
+  const severity = (s: CheckStatus): number =>
+    s === 'FAILURE' ? 3 : s === 'WARNING' ? 2 : s === 'SUCCESS' ? 1 : 0;
+  // Winning index per non-INFO id: highest severity, ties → last occurrence.
+  const winner = new Map<string, number>();
+  checks.forEach((c, i) => {
+    if (c.status === 'INFO') return;
+    const cur = winner.get(c.id);
+    if (
+      cur === undefined ||
+      severity(c.status) >= severity(checks[cur].status)
+    ) {
+      winner.set(c.id, i);
+    }
+  });
+  return checks.filter((c, i) => c.status === 'INFO' || winner.get(c.id) === i);
+}