Skip to content

defect(292): cross-link-no-emission-contract.md §3/§6 — broken ruleId filter + non-executable invocation #354

Description

@davidmatousek

Context

While executing Feature #295 (F-292 post-merge verification runs, US1/T017/SC-003), the security-analyst agent applied specs/292-output-integrity-cross-sink-refinement/contracts/cross-link-no-emission-contract.md §3's verification procedure and found it dual-defective. Both defects were confirmed by direct execution, not inspection alone.

Defect (a): zero-match ruleId filter

Contract §3 Step 1/3 (and §6 counter-tests, which inherit the same pattern) specify:

jq '.runs[0].results[] | select(.ruleId | startswith("OI-"))' /tmp/oi-pre-292.json

This filter can never match any real SARIF result. Actual ruleId values for output-integrity findings are tachi/ai/llm (per scripts/sarif_common.py::PREFIX_TO_RULE) or tachi/ai/llm-threats (per .claude/skills/tachi-orchestration/references/sarif-specification.md:26, the orchestrator's own live-authoring spec) — never an OI--prefixed string. The actual OI discriminator lives at partialFingerprints["findingId/v1"] (e.g. "OI-1").

Consequence — false-pass by construction: applying this filter to any SARIF file (pre-292 or post-292) yields an empty result set on both sides. diff of two empty sets is empty, which §3 Step 4 reads as "Expected: empty output (byte-identical)" — i.e. the broken procedure reports PASS unconditionally, without ever testing the invariant it claims to verify. This is an empty-vs-empty false-pass.

Defect (b): non-executable invocation

Contract §3 Steps 1 and 3 invoke:

SOURCE_DATE_EPOCH=1700000000 tachi.threat-model examples/agentic-app/architecture.md > /tmp/oi-pre-292.json

tachi.threat-model is a Claude Code skill invoked through the agent — it is not a shell executable and has never produced stdout JSON via redirection. This command cannot run as written in any shell.

Corrected procedure (now committed and verified)

specs/295-f292-verification-runs/contracts/oi-extraction-contract.md §1 documents and this feature's SC-003 verification record (specs/295-f292-verification-runs/sc-003-verification-record.md) exercises the corrected extraction:

jq '[.runs[0].results[] | select(.partialFingerprints["findingId/v1"] // "" | startswith("OI-"))]' <input.sarif>

Applied against the pre-292 anchor (git show 0629fa2~1:examples/agentic-app/sample-report/threats.sarif), this correctly yields exactly {OI-1, OI-2, OI-3, OI-4} (cardinality 4) — a real, non-vacuous extraction. The SC-003 record's false-pass guard demonstration (assembler tool rejecting an empty/malformed findings file with non-zero exit) further confirms the fail-closed discipline the archived contract lacked.

OQ-4: disposition of the archived artifact

Per Feature #295 spec (FR-008), the disposition of specs/292-output-integrity-cross-sink-refinement/contracts/cross-link-no-emission-contract.mdannotate (leave the historical file as-is, add a superseded-by pointer to the corrected contract) vs correct (edit the filter/invocation in place) — is decided in this Issue, not by Feature #295 (F-295's own scope excludes editing archived/detection-tier contracts — no inline fix). Recommendation for consideration: annotate-in-place (add a short "Superseded" notice at the top pointing to specs/295-f292-verification-runs/contracts/oi-extraction-contract.md) preserves the historical record of what F-292 actually shipped with, while directing future readers to the working procedure.

Evidence

  • SC-003 verification record: specs/295-f292-verification-runs/sc-003-verification-record.md (this feature, commit follows this Issue)
  • Live confirmation that the broken filter is vacuous: jq '[.runs[0].results[] | select(.ruleId | startswith("OI-"))]' against the anchor SARIF returns [] (zero results) — every ruleId in that file is tachi/ai/llm-family, never OI--prefixed
  • .claude/skills/tachi-orchestration/references/sarif-specification.md:26 and scripts/sarif_common.py:36-38 (current canonical ruleId mappings, both tachi/*-family)

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions