Add an explicit audited OpenCode evaluation runner

[conorbronsdon]

#201's frozen evaluation harness has no provider executor. This optional runner sends its exact plans through OpenCode 1.18.30 with no tools and an explicit observed free-model allowlist.

It checks resolved agent/config, Zen endpoint and SDK, costs, system/user/settings/model/session receipts, and final-text boundaries. Every hook invocation is retained and checked. Failed or interrupted attempts remain visible; resume/import revalidate evidence and never silently retry or overwrite a result. The core harness, case set, metrics, prompts, and human/held-out policy remain unchanged.

Final head: 40c3828edb6e86322bb32071323347117b3c9c50. The last amendment addresses all four late Qodo findings: atomic flushed publication with no overwrite, recovery from interrupted config/plugin setup, one full Git-backed plan validation per run/import instead of repeated per-row Git reads, and absolute executable paths. Malformed existing results are preserved with hashes and quarantined while later tasks continue.

Validation: independent GPT-5.6 Sol high implementation and review; 19 local test suites pass; fake-client end-to-end, tamper, interrupted setup, corrupt-result recovery, and publication-collision regressions pass. The reviewer independently measured exactly six Git invocations for a two-task import. No provider calls were made for the final amendment.

The capped diagnostic completed using the earlier runner commit fdee2ca6c8b93ac5bdda2567d0299dc1558cc12a (runner source SHA-256 80d8eab6aeaeb30e5a0a7f584d1bc3451ac89c025d107eb41b91add395f6aaae). Of 24 calls, 15 had valid outputs and nine MiMo responses failed required text boundaries; all observed cost receipts were zero. The separately preserved first attempt contributed no selected outputs. The final amendment does not replace that execution provenance. Frozen diagnostic summary and commitments.

This is execution infrastructure. #201 remains pending, and rewrite quality has not been established. Zero human judgments have been recorded; the full development run and held-out gate remain incomplete. #295 and #296 remain drafts.

[qodo-free-for-open-source-projects]

PR Summary by Qodo

Add an audited OpenCode rewrite-evaluation runner

✨ Enhancement 🧪 Tests 📝 Documentation 🕐 40+ Minutes

AI Description

• Adds an opt-in OpenCode runner for frozen evaluation plans using approved free Zen models.
• Audits configuration, prompts, parameters, receipts, costs, and final output before importing
 results.
• Preserves failures and validates resumptions with fake-client, tamper, timeout, and configuration
 tests.

Diagram

sequenceDiagram
  actor O as Operator
  participant P as Frozen Plan
  participant R as Audit Runner
  participant C as OpenCode
  participant A as Audit Plugin
  participant Z as Zen API
  participant E as Run Evidence
  participant I as Result Import
  O->>R: Start selected tasks
  R->>P: Validate frozen plan
  R->>C: Verify config and agent
  R->>C: Start fresh session
  C->>A: Apply prompt and params
  C->>Z: Send audited request
  Z-->>C: Return model output
  C-->>R: Return events and session
  R->>E: Retain receipts
  O->>I: Import completed run
  I->>E: Revalidate evidence
  I->>P: Verify provenance
  I-->>O: Write validated results

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Call Zen through the SDK directly

• ➕ Reduces subprocess, CLI, session-export, and plugin-hook complexity.
• ➕ Provides direct control over request construction and response capture.
• ➖ Would not validate OpenCode's resolved agent, configuration, or transport behavior.
• ➖ Requires a separate provider executor and receipt format rather than exercising the intended OpenCode path.

2. Run a containerized OpenCode image

• ➕ Pins the executable and runtime environment more strongly than an operator-supplied path.
• ➕ Reduces variation from host installation and environment state.
• ➖ Adds image-building, distribution, and platform-support overhead.
• ➖ Still requires the audit plugin and retained-receipt validation implemented by this PR.

Recommendation: Keep the explicit OpenCode adapter because the objective is to execute through and audit OpenCode itself, making a direct SDK client strategically mismatched. A pinned container could strengthen reproducibility later, but the current version, source-commit, resolved-config, plugin-hash, and receipt checks are a proportionate opt-in foundation.

Files changed (5) +890 / -0

Enhancement (1) +610 / -0

rewrite-eval-opencode.jsImplement the audited OpenCode evaluation runner +610/-0

Implement the audited OpenCode evaluation runner

• Adds run and import commands pinned to OpenCode 1.18.30 and an allowlist of free Zen models. The runner isolates configuration, disables tools, audits prompt and parameter hooks, retains complete execution evidence, preserves failures, and revalidates all provenance and receipts before import.

scripts/rewrite-eval-opencode.js

Tests (2) +202 / -0

rewrite-eval-opencode.test.jsTest execution, tamper detection, and failure retention +201/-0

Test execution, tamper detection, and failure retention

• Adds a fake OpenCode executable for no-provider-call end-to-end coverage. Tests successful import, repeated hooks, immutable outputs, allowlist enforcement, evidence tampering, timeouts, failure retention, and rejection of unexpected plugins, providers, MCPs, or agent settings.

scripts/rewrite-eval-opencode.test.js

run-tests.jsRegister the OpenCode runner test suite +1/-0

Register the OpenCode runner test suite

• Adds the new fake-client OpenCode transport checks to the default test-suite sequence.

scripts/run-tests.js

Documentation (2) +78 / -0

CHANGELOG.mdDocument the audited OpenCode executor +1/-0

Document the audited OpenCode executor

• Adds a release note describing the opt-in OpenCode executor, its free-model restrictions, evidence validation, and separation from benchmark judgments and release gates.

CHANGELOG.md

README.mdDocument OpenCode execution and evidence requirements +77/-0

Document OpenCode execution and evidence requirements

• Explains runner configuration, supported models and settings, exact prompt replacement, no-tools isolation, run/import commands, and moving-alias limitations. Documents fail-closed resume, failure retention, receipt validation, and diagnostic-only subset policy.

evals/rewrite/README.md

[conorbronsdon]

Independent GPT-5.6 Sol high implementation review cleared runner source SHA-256 80d8eab6aeaeb30e5a0a7f584d1bc3451ac89c025d107eb41b91add395f6aaae after reproducing and fixing import/resume evidence gaps, failure capture, route/config isolation, receipt metadata, and repeated hook validation. Targeted tamper/failure tests and the full 19-suite run pass; all four exact-head CI checks are green.

The fresh, explicitly recorded 24-call diagnostic has completed: 15 validated results and 9 retained malformed-boundary failures. The runner correctly exits nonzero for the incomplete run. No result is repaired, retried automatically, or substituted from the earlier aborted attempt. Independent evidence validation is underway.

This review concerns executor correctness, not semantic rewrite quality. #201 remains open for human adjudication and the complete development/held-out release policy.

[qodo-free-for-open-source-projects]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. Partial results halt every later task ✓ Resolved 🐞 Bug ☼ Reliability

Description

runTask writes result.json directly with writeExclusive, so interruption or an I/O failure can
leave a truncated result that is subsequently treated as a completed result and parsed outside the
task failure handler. Resuming then throws before the remaining tasks run or status.json is
updated, and a write exception can additionally leave both result and failure files permanently
rejected.

Code

scripts/rewrite-eval-opencode.js[R534-535]

+    checkResults(plan, [result]);
+    writeExclusive(resultPath, result);

Evidence

writeExclusive writes directly to the final pathname, while the resume path treats any existing
result as complete and validates it outside the task-level catch. An interrupted or failed write can
therefore leave a malformed result that aborts the whole mapping operation before batch status is
rewritten.

scripts/rewrite-eval-opencode.js[33-40]
scripts/rewrite-eval-opencode.js[447-460]
scripts/rewrite-eval-opencode.js[534-554]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`result.json` can be left partially written, causing every later resume to abort instead of retaining the attempt as failed and continuing with other tasks.
## Fix Focus Areas
- scripts/rewrite-eval-opencode.js[33-41]
- scripts/rewrite-eval-opencode.js[447-460]
- scripts/rewrite-eval-opencode.js[534-546]
## Recommended Fix
Write successful results to a uniquely created temporary file, flush and close it, then atomically install it without replacing an existing result. On resume, distinguish a valid completed result from a malformed partial result, retain the latter as failed evidence without creating a contradictory result/failure pair, and continue processing the batch.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Interrupted setup cannot be resumed ✓ Resolved 🐞 Bug ☼ Reliability

Description

initialize writes runner-config.json and the plugin as separate exclusive operations, but
chooses its resume branch solely from whether the plugin exists. An interruption after the first
write leaves a config-only directory whose next run tries to create the same config again and exits
with EEXIST before any task can proceed.

Code

scripts/rewrite-eval-opencode.js[R278-280]

+  if (!fs.existsSync(pluginPath)) {
+    writeExclusive(path.join(runDir, 'runner-config.json'), config);
+    writeExclusive(pluginPath, pluginSource());

Evidence

The config is created before the plugin with exclusive writes, while only plugin existence controls
whether initialization is considered new or resumed. The README explicitly describes the run
directory as resumable, but the config-only interruption state always re-enters the creation path
and collides with the retained config.

scripts/rewrite-eval-opencode.js[33-35]
scripts/rewrite-eval-opencode.js[275-284]
evals/rewrite/README.md[252-259]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
An interruption between initial metadata writes leaves a run directory that the documented resume operation cannot reopen.
## Fix Focus Areas
- scripts/rewrite-eval-opencode.js[268-284]
## Recommended Fix
Validate each initialization artifact independently when it already exists and create only missing artifacts, or stage the complete initialization and atomically publish it. Preserve the existing equality checks so conflicting configuration or plugin contents still fail closed.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Full imports launch thousands of Git jobs ✓ Resolved 🐞 Bug ➹ Performance

Description

validateTaskEvidence invokes checkResults for every individual task, and checkResults performs
a complete plan validation that rereads all six pinned files through synchronous Git subprocesses.
The intended two-model development plan has 648 tasks, so an import or completed-run resume launches
about 3,888 redundant Git processes before performing another full aggregate validation.

Code

scripts/rewrite-eval-opencode.js[R434-436]

+  const result = read(resultPath);
+  assert.deepEqual(result, expected, `${task.id}: result is not derived from the retained receipts`);
+  checkResults(plan, [result]);

Evidence

The frozen protocol has 36 development cases, three repetitions, and three conditions, producing 648
tasks with the required two models. Every task-level checkResults calls checkPlan, whose three
snapshot checks execute git show once for each of six pinned files; import then calls
checkResults once more for the complete array.

evals/rewrite/protocol.json[3-16]
scripts/rewrite-eval.js[212-242]
scripts/rewrite-eval.js[300-315]
scripts/rewrite-eval.js[352-354]
scripts/rewrite-eval-opencode.js[434-436]
scripts/rewrite-eval-opencode.js[572-578]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Per-task result validation repeatedly performs the same expensive plan and Git provenance checks, making full imports and resumes launch thousands of redundant subprocesses.
## Fix Focus Areas
- scripts/rewrite-eval-opencode.js[382-437]
- scripts/rewrite-eval-opencode.js[551-578]
- scripts/rewrite-eval.js[352-380]
## Recommended Fix
Validate the plan and pinned Git snapshots once when starting run or import, then use an internal result-row validator that assumes the already-validated plan. Validate the collected result set once at the end without repeating Git-backed plan verification for every task.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

View medium (1)

4. Relative executable paths fail to run ✓ Resolved 🐞 Bug ≡ Correctness

Description

checkConfig accepts any nonempty opencode_path, while commands resolve that value under runDir
and the manifest resolves it under the caller's current directory. A relative path can therefore
target different files during execution and provenance recording, or fail immediately even though
configuration validation accepted it.

Code

scripts/rewrite-eval-opencode.js[R51-52]

+  assert(typeof config.opencode_path === 'string' && config.opencode_path, 'runner config opencode_path required');
+  assert.equal(config.opencode_version, '1.18.30', 'this adapter is pinned to OpenCode 1.18.30');

Evidence

Validation checks only that the path is a nonempty string. OpenCode is spawned with the run
directory as its working directory, but path.resolve used for the manifest resolves against the
runner process's current directory; the README's example explicitly requires an absolute path.

scripts/rewrite-eval-opencode.js[47-53]
scripts/rewrite-eval-opencode.js[157-165]
scripts/rewrite-eval-opencode.js[229-240]
scripts/rewrite-eval-opencode.js[286-288]
evals/rewrite/README.md[234-242]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Relative executable paths are accepted even though execution and manifest generation resolve them from different working directories.
## Fix Focus Areas
- scripts/rewrite-eval-opencode.js[47-53]
- scripts/rewrite-eval-opencode.js[229-240]
- scripts/rewrite-eval-opencode.js[286-288]
## Recommended Fix
Require `opencode_path` to satisfy `path.isAbsolute` during configuration validation, matching the documented configuration format. Use that validated absolute value consistently for process execution and manifest recording.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Tip of the day

💡 Did you know, you can start a comment with 'qodo' or '@qodo' to chat about any finding

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

[conorbronsdon]

Final review of 40c3828edb6e86322bb32071323347117b3c9c50: independent GPT-5.6 Sol high review found no remaining blocker, and the reviewed files match this published tree.

Thanks to Qodo for the four concrete recovery/performance findings. All four are addressed: immutable files are flushed and published atomically without overwrite; partial config/plugin setup resumes after validating existing contents; malformed results are hash-bound and quarantined without another provider call; executable paths must be absolute; and Git-backed plan validation happens once per run/import. The independent reviewer measured six Git invocations for a two-task import, with the same run behavior covered by regression tests. Publication-collision and partial-setup cases are included.

The full local suite passes 19/19. CI is running on this exact head. Recommendation: merge execution infrastructure once those checks pass.

The completed diagnostic belongs to the earlier runner commit fdee2ca, as recorded in the linked immutable evidence. This amendment made no provider calls and changes no preserved artifacts. Fifteen valid outputs from 24 calls and zero human judgments do not satisfy #201; #295/#296 remain drafts.