feat: add offline maintainer corpus validation

docs/VALIDATE_CORPUS.md

@@ -0,0 +1,101 @@
+# Offline Corpus Validation
+
+`pcf validate-corpus` measures PCF decisions against consented maintainer labels.
+It is an offline evidence tool, not a training command, evaluator tuner, or
+product-validity oracle.
+
+```bash
+pcf validate-corpus consented.jsonl --format json
+pcf validate-corpus consented.csv --format markdown
+pcf validate-corpus - --input-format jsonl --format pretty < consented.jsonl
+```
+
+The command reads one file or standard input and writes only to standard output.
+It makes no network request and does not modify PCF fixtures, benchmarks,
+policies, feedback stores, or evaluator behavior. Use normal shell redirection
+when a local receipt is desired.
+
+## JSONL contract
+
+Each non-empty line is one case:
+
+```json
+{"id":"opaque-001","repository":"owner/repo","policyId":"policy-v1","consent":{"allowedForValidation":true,"reference":"consent-batch-a"},"pcf":{"lane":"review-now","score":91,"nextActor":"maintainer"},"ratings":[{"raterId":"maintainer-a","lane":"repair","nextActor":"reporter"},{"raterId":"maintainer-b","lane":"repair","nextActor":"reporter"}],"timing":{"baselineSeconds":120,"pcfSeconds":80},"provenance":{"dataset":"consented-alpha","caseRef":"alpha-001","policySnapshot":"policy-v1@abc123","collectedAt":"2026-07-13T12:00:00Z"}}
+```
+
+Required boundaries:
+
+- `id` is an opaque case identifier and must be unique.
+- At least one of `repository` or `policyId` identifies the evaluation context.
+- `consent.allowedForValidation` must be `true`, with a non-empty consent
+  reference.
+- PCF and maintainer lanes are exactly `review-now`, `repair`, or `defer`.
+- `pcf.score` is numeric from 0 through 100.
+- Every case has at least one rating; `raterId` values must be unique within the
+  case. IDs are caller assertions and do not prove identity or independence.
+- `provenance.dataset`, `caseRef`, and `policySnapshot` are required.
+- `timing` is optional. When present, `baselineSeconds` must be positive and
+  `pcfSeconds` must be non-negative.
+- Declared identifiers are single-line values capped at 256 characters. A case
+  may contain at most 50 ratings.
+
+## CSV contract
+
+CSV uses one row per rater. Repeated case metadata must be identical:
+
+```text
+id,repository,policyId,dataset,caseRef,policySnapshot,consentReference,consentAllowed,pcfLane,pcfScore,pcfNextActor,raterId,raterLane,raterNextActor,baselineSeconds,pcfSeconds
+```
+
+`consentAllowed` must be the literal `true`. The header must contain exactly the
+listed columns with no duplicates. Standard quoted CSV fields, embedded commas,
+and escaped quotes are parsed locally; identifier values must remain single-line.
+
+## Privacy boundary
+
+The schema intentionally excludes raw third-party material. PCF fails closed if
+any nested object includes fields named `title`, `body`, `text`, `content`,
+`patch`, `diff`, `comment`, `comments`, `raw`, or `payload`.
+
+Receipts expose aggregate metrics plus opaque IDs, repository/policy identifiers,
+approved `caseRef` values, and corpus provenance. They do not expose rater IDs,
+consent references, raw issue or PR content, or local absolute paths.
+
+Local resource limits are 10 MiB per corpus, 10,000 cases per run, and 50
+ratings per case. These bounds protect an offline operator from accidental
+oversized inputs; they are not statistical sufficiency claims.
+
+## Measurement semantics
+
+- A lane is scored only when its maintainer ratings have a strict majority.
+  Ties are reported as `NO_STRICT_MAJORITY` exclusions.
+- The confusion matrix uses PCF lanes as rows and maintainer consensus as
+  columns.
+- Precision and recall are reported for every lane. A missing denominator is
+  `null` in JSON and `n/a` in human receipts.
+- False `review-now` means PCF selected `review-now` while maintainer consensus
+  selected `repair` or `defer`.
+- Inter-rater output includes pairwise agreement and a nominal, multi-rater
+  kappa derived from aggregate lane prevalence. Variable rater counts are
+  allowed.
+- Score calibration uses fixed 20-point bins and compares PCF `review-now`
+  frequency with consensus `review-now` frequency. It is descriptive, not a
+  probabilistic calibration claim.
+- Paired timing reports seconds and percent saved. Negative savings remain
+  visible.
+- The SHA-256 digest covers the exact input bytes. Replaying identical bytes
+  produces the same result object.
+
+## Formats and exit codes
+
+- `--format pretty`: concise terminal receipt.
+- `--format json`: complete machine-readable result.
+- `--format markdown`: aggregate human review receipt.
+- Exit `0`: the declared corpus shape was valid and metrics were produced.
+- Exit `1`: consent, privacy, schema, or integrity validation failed closed.
+- Exit `2`: command usage or format was invalid.
+
+Every successful result retains decision status `INCONCLUSIVE`. Metrics require
+human interpretation, an independently governed study, and a later VERITAS
+decision before any accuracy, maintainer-endorsement, product-validity, or
+release-readiness claim.

package.json

@@ -21,6 +21,7 @@
     "src",
     "fixtures",
     "docs/MCP.md",
+    "docs/VALIDATE_CORPUS.md",
     "docs/WATCHLIST.md",
     "docs/SERIOUS_SCOUT.md",
     "docs/UPSTREAM_CONTRIBUTION_LEDGER.md",
@@ -48,7 +49,7 @@
   ],
   "scripts": {
     "start": "node src/server.mjs",
-    "check": "node --check scripts/run-pr-gate.mjs && node --check src/server.mjs && node --check src/cli.mjs && node --check src/config.mjs && node --check src/core/api.mjs && node --check src/core/adversary.mjs && node --check src/core/author-context.mjs && node --check src/core/ai-contribution-posture.mjs && node --check src/core/behavioral-signals.mjs && node --check src/core/benchmark.mjs && node --check src/core/calibration.mjs && node --check src/core/candidates.mjs && node --check src/core/contribution-drafts.mjs && node --check src/core/contributor-preflight.mjs && node --check src/core/diff-shape.mjs && node --check src/core/evaluator.mjs && node --check src/core/feedback.mjs && node --check src/core/history.mjs && node --check src/core/issue-form-validator.mjs && node --check src/core/lane-schema.mjs && node --check src/core/lane-status.mjs && node --check src/core/lane-store.mjs && node --check src/core/maintainer-stack.mjs && node --check src/core/pilot-proof.mjs && node --check src/core/policy.mjs && node --check src/core/policy-scan.mjs && node --check src/core/patch.mjs && node --check src/core/queue.mjs && node --check src/core/repository-context.mjs && node --check src/core/repro-gate.mjs && node --check src/core/mcp-submission.mjs && node --check src/core/scout.mjs && node --check src/core/semantic-duplicate-assist.mjs && node --check src/core/serious-scout.mjs && node --check src/core/setup.mjs && node --check src/core/setup-guide.mjs && node --check src/core/shielded-posture.mjs && node --check src/core/text-safety.mjs && node --check src/core/vouch-context.mjs && node --check src/core/watchlist.mjs && node --check src/github/client.mjs && node --check src/github/webhook.mjs && node --check src/github/templates.mjs && node --check src/mcp/core.mjs && node --check src/mcp/server.mjs && node --check scripts/mcp-smoke.mjs && node --check scripts/run-adversary.mjs && node --check scripts/run-benchmark.mjs && node --check scripts/run-maintainer-demo.mjs && node --check scripts/run-public-pilot.mjs && node --check scripts/run-large-bench.mjs && node --check scripts/run-serious-scout.mjs && node --check scripts/run-watchlist.mjs && node --check scripts/verify-ci-workflow.mjs && node --check scripts/verify-repo-hygiene.mjs && node --check public/app.js",
+    "check": "node --check scripts/run-pr-gate.mjs && node --check src/server.mjs && node --check src/cli.mjs && node --check src/config.mjs && node --check src/core/api.mjs && node --check src/core/adversary.mjs && node --check src/core/author-context.mjs && node --check src/core/ai-contribution-posture.mjs && node --check src/core/behavioral-signals.mjs && node --check src/core/benchmark.mjs && node --check src/core/calibration.mjs && node --check src/core/candidates.mjs && node --check src/core/contribution-drafts.mjs && node --check src/core/contributor-preflight.mjs && node --check src/core/corpus-validation.mjs && node --check src/core/diff-shape.mjs && node --check src/core/evaluator.mjs && node --check src/core/feedback.mjs && node --check src/core/history.mjs && node --check src/core/issue-form-validator.mjs && node --check src/core/lane-schema.mjs && node --check src/core/lane-status.mjs && node --check src/core/lane-store.mjs && node --check src/core/maintainer-stack.mjs && node --check src/core/pilot-proof.mjs && node --check src/core/policy.mjs && node --check src/core/policy-scan.mjs && node --check src/core/patch.mjs && node --check src/core/queue.mjs && node --check src/core/repository-context.mjs && node --check src/core/repro-gate.mjs && node --check src/core/mcp-submission.mjs && node --check src/core/scout.mjs && node --check src/core/semantic-duplicate-assist.mjs && node --check src/core/serious-scout.mjs && node --check src/core/setup.mjs && node --check src/core/setup-guide.mjs && node --check src/core/shielded-posture.mjs && node --check src/core/text-safety.mjs && node --check src/core/vouch-context.mjs && node --check src/core/watchlist.mjs && node --check src/github/client.mjs && node --check src/github/webhook.mjs && node --check src/github/templates.mjs && node --check src/mcp/core.mjs && node --check src/mcp/server.mjs && node --check scripts/mcp-smoke.mjs && node --check scripts/run-adversary.mjs && node --check scripts/run-benchmark.mjs && node --check scripts/run-maintainer-demo.mjs && node --check scripts/run-public-pilot.mjs && node --check scripts/run-large-bench.mjs && node --check scripts/run-serious-scout.mjs && node --check scripts/run-watchlist.mjs && node --check scripts/verify-ci-workflow.mjs && node --check scripts/verify-repo-hygiene.mjs && node --check public/app.js",
     "mcp": "node src/mcp/server.mjs",
     "mcp:smoke": "node scripts/mcp-smoke.mjs",
     "setup:pilot": "node src/cli.mjs setup",
@@ -80,7 +81,8 @@
     "demo:pr": "node src/cli.mjs evaluate fixtures/pr-unready.json",
     "demo:issue": "node src/cli.mjs evaluate fixtures/issue-unready.json",
     "demo:kernel": "node src/cli.mjs evaluate fixtures/pr-kernel-ready.json --profile kernel-grade",
-    "preflight": "node src/cli.mjs preflight"
+    "preflight": "node src/cli.mjs preflight",
+    "validate:corpus": "node src/cli.mjs validate-corpus"
   },
   "engines": {
     "node": ">=22"

src/cli.mjs

@@ -1,10 +1,17 @@
 #!/usr/bin/env node
 import { readFile } from "node:fs/promises";
+import { basename } from "node:path";
 import { evaluateContribution, renderMarkdownReport } from "./core/evaluator.mjs";
 import { parsePatchSubmission } from "./core/patch.mjs";
 import { normalizeRepositoryFiles } from "./core/policy.mjs";
 import { buildMaintainerQueue } from "./core/queue.mjs";
 import { buildSetupGuide, renderSetupGuideMarkdown, renderSetupGuideText } from "./core/setup-guide.mjs";
+import {
+  CorpusValidationError,
+  renderCorpusValidationMarkdown,
+  renderCorpusValidationSummary,
+  validateCorpusText
+} from "./core/corpus-validation.mjs";
 import { loadConfig } from "./config.mjs";
 
 const args = process.argv.slice(2);
@@ -15,7 +22,7 @@ if (args.length === 0 || args.includes("--help") || args.includes("-h")) {
 }
 
 const command = args[0];
-if (!["evaluate", "evaluate-patch", "queue", "setup", "setup-pilot", "preflight"].includes(command)) {
+if (!["evaluate", "evaluate-patch", "queue", "setup", "setup-pilot", "preflight", "validate-corpus"].includes(command)) {
   console.error(`Unknown command: ${command}`);
   printHelp();
   process.exit(2);
@@ -40,6 +47,44 @@ if (command === "setup" || command === "setup-pilot") {
   process.exit(0);
 }
 
+if (command === "validate-corpus") {
+  const file = args[1];
+  if (!file) {
+    console.error("Missing corpus file.");
+    printHelp();
+    process.exit(2);
+  }
+  const format = readFlag(args, "--format") || "pretty";
+  if (!["pretty", "json", "markdown"].includes(format)) {
+    console.error(`Unsupported format: ${format}. Use pretty, json, or markdown.`);
+    process.exit(2);
+  }
+  const inputFormat = readFlag(args, "--input-format");
+  try {
+    const text = file === "-" ? await readStdin() : await readFile(file, "utf8");
+    const result = validateCorpusText(text, {
+      inputFormat,
+      sourceName: file === "-" ? "stdin" : basename(file)
+    });
+    if (format === "json") {
+      console.log(JSON.stringify(result, null, 2));
+    } else if (format === "markdown") {
+      process.stdout.write(renderCorpusValidationMarkdown(result));
+    } else {
+      process.stdout.write(renderCorpusValidationSummary(result));
+    }
+    process.exit(0);
+  } catch (error) {
+    const message = error instanceof CorpusValidationError
+      ? error.message
+      : error?.code === "ENOENT"
+        ? `Cannot read corpus file '${basename(file)}'.`
+        : "Unexpected corpus validation error.";
+    console.error(`PCF corpus validation failed: ${message}`);
+    process.exit(1);
+  }
+}
+
 const file = args[1];
 if (!file) {
   console.error("Missing input file.");
@@ -215,7 +260,9 @@ function printHelp() {
   node src/cli.mjs evaluate <payload.json> [--format pretty|json|markdown] [--profile standard|kernel-grade]
   node src/cli.mjs evaluate-patch <patch-or-mbox> [--format pretty|json|markdown] [--profile kernel-grade] [--policy policy-files.json]
   node src/cli.mjs preflight <payload.json|patch-or-mbox> [--allow-repair] [--format pretty|json|markdown] [--profile standard|kernel-grade] [--policy policy-files.json]
+  node src/cli.mjs validate-corpus <consented.jsonl|consented.csv|-> [--input-format jsonl|csv] [--format pretty|json|markdown]
   cat queue-payload.json | node src/cli.mjs queue - --format json
 
-Preflight exit codes: 0 = ready to submit, 1 = not ready, 2 = usage error.`);
+Preflight exit codes: 0 = ready to submit, 1 = not ready, 2 = usage error.
+Corpus validation exit codes: 0 = corpus measured, 1 = validation failed closed, 2 = usage error.`);
 }