Skip to content

Deterministic Python parser for structured-finding aggregation #3

Description

@yhyatt

Current behavior

v0.2.0 ships a structured finding schema (Finding ID, Severity, File, Line, User-visible lie, Evidence: |, Recommended fix, Fix size, Confidence) emitted by both specialists. The orchestrator's aggregator parses these blocks block-by-block and dedups on (normalized File, Line) with a User-visible lie similarity fallback.

Parsing today is done by the orchestrator LLM reading both source reports. It is described in the SKILL as "block-by-block, tolerate minor formatting variance," which is honest but not deterministic. The combined HIGH count is the headline number of the report; an off-by-one due to a parse miss is a credibility hit.

Proposed

Ship a small Python script under scripts/ (or tests/lib/) that:

  • Reads SAFE-FAIL-AUDIT.md and MOCK-STUB-AUDIT.md.
  • Parses each ### Finding ID: block into a structured object.
  • Tolerates documented variance: extra whitespace, inline vs. pipe-block evidence, unknown/N/A/missing Line, severity disagreements.
  • Emits one combined JSON list of findings, plus a small summary (counts per severity, conflicts).
  • Optionally a Markdown renderer for the combined report itself, so the orchestrator's only job becomes "invoke this parser."

Why this matters

  • Auditable. Same inputs produce the same combined report. Two runs of the orchestrator on identical source reports can produce subtly different combined files today.
  • Testable. Parser ships with its own unit tests against synthetic source reports.
  • Lower variance on the headline counts.

Acceptance criteria

  • scripts/aggregate.py (or similar) parses both source reports, emits a structured combined report, exits non-zero on schema violations the orchestrator should know about.
  • Unit tests under tests/aggregator/ cover: minimum-fields, inline evidence vs. pipe-block evidence, Line: unknown, severity conflict, dedup by (File, Line), dedup by lie-similarity when Line is unknown.
  • skills/dishonest-code-audit/SKILL.md updated to invoke the parser in step 5, with a fallback to LLM parsing if Python is unavailable in the runtime.
  • CHANGELOG: move this entry from ## [Unreleased] > Planned into the next release section.

Constraints

  • Stdlib only. The plugin currently has zero runtime dependencies and adding pyyaml for an arguably-YAML-shaped schema would be net-bad for the install story.
  • Permissive on input, strict on output.

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions