Current behavior
v0.2.0 ships a structured finding schema (Finding ID, Severity, File, Line, User-visible lie, Evidence: |, Recommended fix, Fix size, Confidence) emitted by both specialists. The orchestrator's aggregator parses these blocks block-by-block and dedups on (normalized File, Line) with a User-visible lie similarity fallback.
Parsing today is done by the orchestrator LLM reading both source reports. It is described in the SKILL as "block-by-block, tolerate minor formatting variance," which is honest but not deterministic. The combined HIGH count is the headline number of the report; an off-by-one due to a parse miss is a credibility hit.
Proposed
Ship a small Python script under scripts/ (or tests/lib/) that:
- Reads
SAFE-FAIL-AUDIT.md and MOCK-STUB-AUDIT.md.
- Parses each
### Finding ID: block into a structured object.
- Tolerates documented variance: extra whitespace, inline vs. pipe-block evidence,
unknown/N/A/missing Line, severity disagreements.
- Emits one combined JSON list of findings, plus a small summary (counts per severity, conflicts).
- Optionally a Markdown renderer for the combined report itself, so the orchestrator's only job becomes "invoke this parser."
Why this matters
- Auditable. Same inputs produce the same combined report. Two runs of the orchestrator on identical source reports can produce subtly different combined files today.
- Testable. Parser ships with its own unit tests against synthetic source reports.
- Lower variance on the headline counts.
Acceptance criteria
scripts/aggregate.py (or similar) parses both source reports, emits a structured combined report, exits non-zero on schema violations the orchestrator should know about.
- Unit tests under
tests/aggregator/ cover: minimum-fields, inline evidence vs. pipe-block evidence, Line: unknown, severity conflict, dedup by (File, Line), dedup by lie-similarity when Line is unknown.
skills/dishonest-code-audit/SKILL.md updated to invoke the parser in step 5, with a fallback to LLM parsing if Python is unavailable in the runtime.
- CHANGELOG: move this entry from
## [Unreleased] > Planned into the next release section.
Constraints
- Stdlib only. The plugin currently has zero runtime dependencies and adding
pyyaml for an arguably-YAML-shaped schema would be net-bad for the install story.
- Permissive on input, strict on output.
Related
Current behavior
v0.2.0 ships a structured finding schema (
Finding ID,Severity,File,Line,User-visible lie,Evidence: |,Recommended fix,Fix size,Confidence) emitted by both specialists. The orchestrator's aggregator parses these blocks block-by-block and dedups on(normalized File, Line)with aUser-visible liesimilarity fallback.Parsing today is done by the orchestrator LLM reading both source reports. It is described in the SKILL as "block-by-block, tolerate minor formatting variance," which is honest but not deterministic. The combined HIGH count is the headline number of the report; an off-by-one due to a parse miss is a credibility hit.
Proposed
Ship a small Python script under
scripts/(ortests/lib/) that:SAFE-FAIL-AUDIT.mdandMOCK-STUB-AUDIT.md.### Finding ID:block into a structured object.unknown/N/A/missing Line, severity disagreements.Why this matters
Acceptance criteria
scripts/aggregate.py(or similar) parses both source reports, emits a structured combined report, exits non-zero on schema violations the orchestrator should know about.tests/aggregator/cover: minimum-fields, inline evidence vs. pipe-block evidence,Line: unknown, severity conflict, dedup by(File, Line), dedup by lie-similarity when Line is unknown.skills/dishonest-code-audit/SKILL.mdupdated to invoke the parser in step 5, with a fallback to LLM parsing if Python is unavailable in the runtime.## [Unreleased] > Plannedinto the next release section.Constraints
pyyamlfor an arguably-YAML-shaped schema would be net-bad for the install story.Related
## [Unreleased] > Plannedentrya02542c, but the underlying determinism gap is what this issue closes)