Context
While executing Feature #295 (F-292 post-merge verification runs, US1/T017/SC-003), the security-analyst agent applied specs/292-output-integrity-cross-sink-refinement/contracts/cross-link-no-emission-contract.md §3's verification procedure and found it dual-defective. Both defects were confirmed by direct execution, not inspection alone.
Defect (a): zero-match ruleId filter
Contract §3 Step 1/3 (and §6 counter-tests, which inherit the same pattern) specify:
jq '.runs[0].results[] | select(.ruleId | startswith("OI-"))' /tmp/oi-pre-292.json
This filter can never match any real SARIF result. Actual ruleId values for output-integrity findings are tachi/ai/llm (per scripts/sarif_common.py::PREFIX_TO_RULE) or tachi/ai/llm-threats (per .claude/skills/tachi-orchestration/references/sarif-specification.md:26, the orchestrator's own live-authoring spec) — never an OI--prefixed string. The actual OI discriminator lives at partialFingerprints["findingId/v1"] (e.g. "OI-1").
Consequence — false-pass by construction: applying this filter to any SARIF file (pre-292 or post-292) yields an empty result set on both sides. diff of two empty sets is empty, which §3 Step 4 reads as "Expected: empty output (byte-identical)" — i.e. the broken procedure reports PASS unconditionally, without ever testing the invariant it claims to verify. This is an empty-vs-empty false-pass.
Defect (b): non-executable invocation
Contract §3 Steps 1 and 3 invoke:
SOURCE_DATE_EPOCH=1700000000 tachi.threat-model examples/agentic-app/architecture.md > /tmp/oi-pre-292.json
tachi.threat-model is a Claude Code skill invoked through the agent — it is not a shell executable and has never produced stdout JSON via redirection. This command cannot run as written in any shell.
Corrected procedure (now committed and verified)
specs/295-f292-verification-runs/contracts/oi-extraction-contract.md §1 documents and this feature's SC-003 verification record (specs/295-f292-verification-runs/sc-003-verification-record.md) exercises the corrected extraction:
jq '[.runs[0].results[] | select(.partialFingerprints["findingId/v1"] // "" | startswith("OI-"))]' <input.sarif>
Applied against the pre-292 anchor (git show 0629fa2~1:examples/agentic-app/sample-report/threats.sarif), this correctly yields exactly {OI-1, OI-2, OI-3, OI-4} (cardinality 4) — a real, non-vacuous extraction. The SC-003 record's false-pass guard demonstration (assembler tool rejecting an empty/malformed findings file with non-zero exit) further confirms the fail-closed discipline the archived contract lacked.
OQ-4: disposition of the archived artifact
Per Feature #295 spec (FR-008), the disposition of specs/292-output-integrity-cross-sink-refinement/contracts/cross-link-no-emission-contract.md — annotate (leave the historical file as-is, add a superseded-by pointer to the corrected contract) vs correct (edit the filter/invocation in place) — is decided in this Issue, not by Feature #295 (F-295's own scope excludes editing archived/detection-tier contracts — no inline fix). Recommendation for consideration: annotate-in-place (add a short "Superseded" notice at the top pointing to specs/295-f292-verification-runs/contracts/oi-extraction-contract.md) preserves the historical record of what F-292 actually shipped with, while directing future readers to the working procedure.
Evidence
- SC-003 verification record:
specs/295-f292-verification-runs/sc-003-verification-record.md (this feature, commit follows this Issue)
- Live confirmation that the broken filter is vacuous:
jq '[.runs[0].results[] | select(.ruleId | startswith("OI-"))]' against the anchor SARIF returns [] (zero results) — every ruleId in that file is tachi/ai/llm-family, never OI--prefixed
.claude/skills/tachi-orchestration/references/sarif-specification.md:26 and scripts/sarif_common.py:36-38 (current canonical ruleId mappings, both tachi/*-family)
Context
While executing Feature #295 (F-292 post-merge verification runs, US1/T017/SC-003), the security-analyst agent applied
specs/292-output-integrity-cross-sink-refinement/contracts/cross-link-no-emission-contract.md§3's verification procedure and found it dual-defective. Both defects were confirmed by direct execution, not inspection alone.Defect (a): zero-match
ruleIdfilterContract §3 Step 1/3 (and §6 counter-tests, which inherit the same pattern) specify:
jq '.runs[0].results[] | select(.ruleId | startswith("OI-"))' /tmp/oi-pre-292.jsonThis filter can never match any real SARIF result. Actual
ruleIdvalues for output-integrity findings aretachi/ai/llm(perscripts/sarif_common.py::PREFIX_TO_RULE) ortachi/ai/llm-threats(per.claude/skills/tachi-orchestration/references/sarif-specification.md:26, the orchestrator's own live-authoring spec) — never anOI--prefixed string. The actual OI discriminator lives atpartialFingerprints["findingId/v1"](e.g."OI-1").Consequence — false-pass by construction: applying this filter to any SARIF file (pre-292 or post-292) yields an empty result set on both sides.
diffof two empty sets is empty, which §3 Step 4 reads as "Expected: empty output (byte-identical)" — i.e. the broken procedure reports PASS unconditionally, without ever testing the invariant it claims to verify. This is an empty-vs-empty false-pass.Defect (b): non-executable invocation
Contract §3 Steps 1 and 3 invoke:
SOURCE_DATE_EPOCH=1700000000 tachi.threat-model examples/agentic-app/architecture.md > /tmp/oi-pre-292.jsontachi.threat-modelis a Claude Code skill invoked through the agent — it is not a shell executable and has never produced stdout JSON via redirection. This command cannot run as written in any shell.Corrected procedure (now committed and verified)
specs/295-f292-verification-runs/contracts/oi-extraction-contract.md§1 documents and this feature's SC-003 verification record (specs/295-f292-verification-runs/sc-003-verification-record.md) exercises the corrected extraction:Applied against the pre-292 anchor (
git show 0629fa2~1:examples/agentic-app/sample-report/threats.sarif), this correctly yields exactly{OI-1, OI-2, OI-3, OI-4}(cardinality 4) — a real, non-vacuous extraction. The SC-003 record's false-pass guard demonstration (assembler tool rejecting an empty/malformed findings file with non-zero exit) further confirms the fail-closed discipline the archived contract lacked.OQ-4: disposition of the archived artifact
Per Feature #295 spec (FR-008), the disposition of
specs/292-output-integrity-cross-sink-refinement/contracts/cross-link-no-emission-contract.md— annotate (leave the historical file as-is, add a superseded-by pointer to the corrected contract) vs correct (edit the filter/invocation in place) — is decided in this Issue, not by Feature #295 (F-295's own scope excludes editing archived/detection-tier contracts — no inline fix). Recommendation for consideration: annotate-in-place (add a short "Superseded" notice at the top pointing tospecs/295-f292-verification-runs/contracts/oi-extraction-contract.md) preserves the historical record of what F-292 actually shipped with, while directing future readers to the working procedure.Evidence
specs/295-f292-verification-runs/sc-003-verification-record.md(this feature, commit follows this Issue)jq '[.runs[0].results[] | select(.ruleId | startswith("OI-"))]'against the anchor SARIF returns[](zero results) — everyruleIdin that file istachi/ai/llm-family, neverOI--prefixed.claude/skills/tachi-orchestration/references/sarif-specification.md:26andscripts/sarif_common.py:36-38(current canonical ruleId mappings, bothtachi/*-family)