Skip to content

fix(coverage): bind Standard completion to file review receipts - #829

Draft
soyeon-oai wants to merge 3 commits into
mainfrom
codex/standard-review-receipts
Draft

soyeon-oai wants to merge 3 commits into
mainfrom
codex/standard-review-receipts

Conversation

@soyeon-oai

@soyeon-oai soyeon-oai commented Sep 8, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Standard scans could finish with completed findings while their persisted review count remained zero. Bind review progress to the files selected at scan start, and require every file's review receipt before leaving discovery or sealing the result.

Changes

  • Store a unique repository-relative path and SHA-256 digest per Standard file in migration 42. Preserve explicitly requested files and deduplicate overlapping scopes.
  • Accept completed paths through MCP reviewedFiles and the workbench helper's repeatable --reviewed-file argument. Validate their content, persist receipts transactionally, and derive coverage totals from those rows. Standard scans reject aggregate review counters.
  • Apply the same completion gate to direct and SDK-prepared completion. Update Standard workflow instructions, the custom-validation workflow fingerprint, and completion fixtures. Bump the MCP app to 0.1.160.
  • Preserve explicitly requested mock scans without fabricating reviewed-file counts. Only the saved launch recipe can select this behavior; an artifact's mock marker does not bypass real-scan completion.

Testing

  • Confirmed that discovery transition and completion regression tests failed on the base and passed with the change.
  • Verified duplicate receipts, rejected-batch rollback, changed content, unknown files, overlapping scopes, explicitly requested ignored files, and partial coverage preservation.
  • Full Python suite on the final implementation: 1,116 passed, 5 skipped, 110 subtests passed. Final completion/progress suites: 39 passed.
  • MCP suite: 23 passed before the explicit-mock adjustment. Publication and helper checks: 18 passed.
  • SDK typecheck and formatting, Python Ruff checks, and portable plugin source checks passed. Rebuilt the 123-file bundled plugin from source.
  • The full SDK run at the initial receipt implementation had 2,403 passed, 44 skipped, and 11 failed. Three failures exposed the mock completion incompatibility, now fixed; the mock suite then passed 10/10. Eight failures came from local state/process permissions or injected Node proxy warnings; isolated publication and authentication/release reruns passed 12/12 and 293/293. The entire SDK suite was not rerun after these corrections.
  • CI exposed an installed-package test fixture that did not close its README review. Updated the fixture and verified its built-SDK lifecycle, cancellation, and persisted 1/1/1 coverage locally. Fresh npm installation remains locally blocked: registry requests returned HTTP 403, followed by npm's Exit handler never called! error.
  • Latest-head CI is not fully green: the Windows knowledge-base preflight test exceeded its 120-second limit, followed by a missing temporary repository error. That test and its production path are unchanged in this PR; regression versus runner flake remains unclassified. See the Windows job.

Risk and rollout

In-flight Standard scans created before this migration need to restart because they have no frozen per-file inventory. Existing completed scans remain readable. Receipts record declared review completion for frozen file content; they do not attest to the quality of model reasoning.

Native Desktop and live-model CLI scans of this OSS revision remain follow-up QA. This PR does not publish a release.

Public disclosure review

  • No customer, partner, prospect, or user identities, data, or identifying details are included.
  • No credentials, personal data, private source, scan findings, or nonpublic links or tickets are included.
  • I reviewed the branch name, title, description, commits, changes, comments, logs, screenshots, attachments, and links for public disclosure.

Change impact

The workbench freezes the file list, validates declared reviews against those bytes, and blocks real scan completion while receipts remain open.

flowchart LR
  subgraph column_0["Inputs"]
    direction TB
    node_0["Frozen file inventory"]
    node_1["MCP reviewedFiles"]
  end
  subgraph column_1["Persisted reviews"]
    direction TB
    node_2["Content-bound receipts"]
  end
  subgraph column_2["Completion rule"]
    direction TB
    node_3["All file reviews closed"]
  end
  subgraph column_3["Consumers"]
    direction TB
    node_4["Leave discovery"]
    node_5["Prepare or complete scan"]
  end
  node_0 -->|"binds paths and hashes"| node_2
  node_1 -->|"records paths"| node_2
  node_2 -->|"supplies closed rows"| node_3
  node_3 -->|"permits transition"| node_4
  node_3 -->|"permits finalization"| node_5
  class node_0 changed
  class node_1 changed
  class node_2 changed
  class node_3 changed
  class node_4 affected
  class node_5 affected
  classDef changed fill:#d7f5e5,stroke:#237a4b,color:#111
  classDef affected fill:#e6f0ff,stroke:#3569a8,color:#111
  classDef context fill:#f2f3f5,stroke:#6e7781,color:#111
Loading

Limits: Native Desktop and live-model CLI QA of this OSS revision remain open. · Old in-flight Standard scans without a frozen inventory must restart. · Receipts declare review completion, not reasoning quality; explicit mock launches remain synthetic.

Source evidence (6)

@github-actions github-actions Bot added the bug Something isn't working label Sep 8, 2026
@mldangelo-oai mldangelo-oai changed the title fix(plugin): bind Standard completion to file review receipts fix(coverage): bind Standard completion to file review receipts Oct 8, 2026
@mldangelo-oai mldangelo-oai added area:artifacts Saved scan state, storage, sealing, recovery, history, and artifact integrity. area:targets Repository and path targets, Git discovery, scope, inventories, and snapshots. and removed area:artifacts Saved scan state, storage, sealing, recovery, history, and artifact integrity. area:targets Repository and path targets, Git discovery, scope, inventories, and snapshots. labels Oct 8, 2026

@mldangelo-oai mldangelo-oai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI-assisted review, reviewed by me. Most feedback is suggestions; concrete behavior issues are called out where applicable.

Reviewed 77cc515cc04ea1b1f4e3fde07dba0556fc8c57b7. Freezing the Standard inventory and deriving completion from content-bound file receipts gives the completion gate a concrete source of truth. Transactional receipt batches, duplicate handling, and the distinction between progress display and authoritative completion are valuable parts of this design. One introduced filename-transport issue is inline.

The change's explicit compatibility boundary is reasonable: an older in-flight Standard scan lacks the frozen inventory needed by this contract and must restart, while completed history remains readable. Keep that restart guidance close to the actual recovery error. The gate records review completion against the frozen bytes; it should continue to describe that contract without implying it establishes the quality of model reasoning.

One simplification suggestion is to remove the new 512-receipt batch cap unless a concrete transport or storage limit requires it. The transactional implementation already provides the important integrity behavior; an unexplained count adds coordination work for the caller. Retain the atomic rejection, duplicate/idempotent receipt, and source-drift tests while simplifying that surface.

Validation passed 39 focused Python tests, three source/shipped MCP test scripts, 58 SDK receipt/completion/recovery tests, type/build checks, Ruff, and the historical portable-source checker and its nine tests. The filename issue was reproduced through the shipped stdio MCP server and real Python helper/database; the base completed the same synthetic target, and the equals-form argument closed its receipt. Deep parent and worker completion paths were traced, including resumed workers. No live model, external service, rendered desktop UI, or native Windows execution was used.

...optionalNumberArg("--phase-items-completed", derivedPhaseItemsCompleted),
...optionalArg("--phase-progress-unit", derivedPhaseProgressUnit),
...(serializedPreflightIssues ? ["--preflight-issues-json-stdin"] : []),
...(reviewedFiles ?? []).flatMap((path) => ["--reviewed-file", path]),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve leading-dash filenames when forwarding receipts

For an inventoried top-level file named -source.py, this emits separate arguments --reviewed-file and -source.py. Python argparse treats the filename as another option and reports that --reviewed-file needs an argument. Through the shipped MCP server, the receipt stays at 0/1 and the new mandatory gate prevents validation and completion; prefixing ./ is rejected by canonical-path validation. The same synthetic target completes at the base revision. Forward each value as --reviewed-file=<path>: that control closes the receipt without changing its canonical identity. Add a shipped MCP/helper regression so a valid repository filename cannot make Standard completion impossible.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants