Build a reproducible benchmark for rewrite fidelity, useful edits, and voice preservation

[conorbronsdon]

Prepared by ChatGPT/Codex, incorporating an independent GPT-5.6 Sol review at extra-high reasoning, at @conorbronsdon's request. This is an implementation plan, not a claim that the proposed behavior has already been validated.

Parent plan: #199

Problem

The skill's main promise is useful editing that preserves the source. Detector fixtures and authorship corpus measurements do not establish that promise. The quick-demo correction is tracked separately and supplies an initial regression case.

Scope

• Add a small, reproducible editing evaluation harness and case format. Begin with targeted preservation regressions and expand to a 40–60-case development pilot across social, newsletter/blog, technical documentation, email, and already-good prose.
• Record source claims, protected content, allowed edits, uncertainty/negation, and expected preserve/change decisions. Use observable constraints and acceptable outcomes rather than exact rewritten sentences.
• Compare a pinned current skill baseline, the candidate, and a fixed simple editing prompt on the same editor model and settings. Use at least two editor model families for the release comparison when available, report family-specific results, and do not generalize from one family. Use three repetitions per case/condition for the pilot or document a preregistered disagreement-rerun strategy. Keep a held-out split by document/author; do not tune against held-out results.

Acceptance criteria

• Cases cover invented and removed facts, quantities/units, negation, certainty drift, authentic voice, technical terms, necessary hedges, quoted or attributed text, links, code, tables, and no-op edits.
• The demo regression fails when the unsupported investor/integration claim is added or the supplied dashboard capability disappears.
• Reports keep preservation failures, unnecessary edits, missed justified edits, and blind reader preference separate. Detector score reduction is not the quality target. Unchanged output cannot win solely by preserving everything.
• Record exact skill commit, case-set hash, prompt, provider/model version, settings, tool availability, raw outputs, repetitions, timing, and actual token usage when available; label estimates. Separate editor and judge roles and randomize comparison order. Human review adjudicates subjective or disputed cases.
• Freeze metrics and decision rules before comparative runs. Treat the pilot as diagnostic, not a statistically sufficient public benchmark; report counts and limitations rather than claiming universal improvement. Predefine a held-out release evaluation and per-register regression policy.
• Local fixture/schema checks run without model credentials; paid model runs are explicit. Mechanical checks do not claim to prove semantic fidelity. Preserve privacy/licensing and avoid committing private drafts.

Boundaries and sequencing

This is the baseline for behavioral refactoring. It does not expand or recalibrate the authorship detector corpus, which remains under #70, #71, #72, #85; cross-tool authorship benchmarking stays in #84. The separately tracked demo correction can land before this full pilot.

Initial case matrix

Use 48 cases as a concrete starting point: 12 clean/no-op, 12 clear edits, 12 contextual judgments, and 12 preservation/scope conflicts, spread across supported profiles and modes. Encode atomic source propositions and allowed equivalences. Freeze a held-out split and noninferiority/improvement gates before looking at candidate outputs; inconclusive results do not justify a default rollout.

Linked prerequisites

Seed case: #200.

Repository integration

Edit canonical sources and regenerate shipped copies using the existing sync scripts; do not hand-edit generated artifacts. Run the relevant tests and existing packaging/parity checks. Follow CONTRIBUTING.md's changelog/version policy for user-visible changes. Preserve existing public modes, options, detector type IDs, and machine-readable interfaces unless a compatibility change is explicitly documented and tested.

[conorbronsdon]

Codex implementation update: #267 supplies the offline harness, 48-case pilot, frozen protocol and human-review workflow, based on #266's source-preservation regression. Local full-suite and self-scan checks pass. The tests use synthetic plumbing outputs; they are not a model-performance result.

Keeping this issue open for the remaining experimental work: concrete versioned editor configurations, explicitly executed and archived baseline/candidate/simple runs across two families, three repetitions per case/condition, blind human adjudication, and held-out release review. The harness makes no provider calls. Its no-tools pilot covers prose rewrite/edit decisions, not actual filesystem edits or detect-mode output. Public synthetic holdouts must be replaced if they inform candidate tuning.

[conorbronsdon]

Reopened after merging #267. That PR lands the offline evaluation infrastructure only; selecting provider/version configurations, running and archiving the comparisons, collecting blind human judgments, and applying the held-out release policy remain tracked here.

[conorbronsdon]

Codex evaluation preparation for the authorized #202 candidate. Pinning baseline and corpus/protocol to 7e88455009da2d8f41b314e56ec767763fc6f379 before any editorial changes. A separate GPT-5.6 Sol high agent is checking provider execution, exact prompt/settings recording, raw-output capture, and import/blinding compatibility. Candidate full SHA will be fixed before runs. No paid provider calls are being introduced.

Free OpenCode model access is available; whether it can satisfy the protocol's frozen prompt/settings contract is being verified rather than assumed. Any moving model aliases, provider/harness additions, partial runs, or missing usage metadata will be recorded explicitly. Human judgments remain human judgments; model-assisted checks will be labeled separately. No held-out examples will be used to tune the candidate.

[conorbronsdon]

The first capped cross-family development diagnostic is complete for #295 candidate b46b1c33888bc8cdbd3604fbc4b98280b627c696, with baseline/cases/protocol pinned to 7e88455009da2d8f41b314e56ec767763fc6f379.

Immutable public summary and evidence commitments.

The preregistered subset used four development cases × three conditions × two families × one repetition = 24 calls. All ran: Ling produced 12/12 valid outputs, MiMo 3/12; the other nine MiMo responses failed required final-text boundary validation. All 24 receipts report zero cost and exact no-tool transport. Independent GPT-5.6 Sol high review checked every receipt chain, failure, private-key mapping, and packet. These are validity/transport observations, not semantic-quality conclusions.

An earlier infrastructure attempt was preserved separately after receipt-format assumptions failed; none of its outputs was substituted into this attempt. Raw evidence and the condition key are archived privately to preserve blinding. A human packet contains all 15 valid outputs and blank judgments; incomplete comparison groups are explicitly not ranked.

Gate remains open: zero human judgments; 624 development tasks unrun; nine failed selected tasks; no held-out execution. #295 remains a draft. #296 is a separate later candidate and has not been evaluated by this run. #297 supplies the optional executor; final restart/write edge-case fixes are being reviewed separately, without altering the frozen experiment.

Next experimental work is resolving editor-format reliability under a new explicit plan, completing the required per-family/per-profile repetitions and human adjudication, then running the frozen held-out gate. No default-rollout claim is made.

[conorbronsdon]

Runner #297 has landed as 41d781e487bcc625259f4a5dc7de5fc139ab2c20, after independent GPT-5.6 Sol high review and all four CI checks on final head 40c3828. Thanks to Qodo for the concrete recovery and performance findings; all four are addressed with offline regressions.

GitHub interpreted a negated closing phrase in the PR body as an automatic closure directive at merge. I removed that phrase and reopened this issue: the full development comparison, actual human judgments, and held-out release gate are still pending. #295 and #296 remain drafts. The preserved diagnostic and commitments linked above are unchanged; the final runner amendment made no additional provider calls.

[conorbronsdon]

#296 capped diagnostic update\n\nFrozen comparison:\n\n- baseline #295: �8826be2c03435bc297a2746418663bfa844563c\n- candidate #296: dd160fb\n- corrected plan hash: 695c1109d5c239839be4c197ba506e1a31975f54ac6d0c866ce1d31917d21d04\n- four development cases: clean, clear edit, contextual judgment, preservation conflict\n- two moving free Zen aliases, three conditions, repetition 1 only: 24 selected calls\n- isolated OpenCode 1.18.30 executor, tools disabled and audited\n\nThe first run yielded 11/24 valid results and retained 13 boundary failures. One explicit retry ran only those 13 failed task IDs in a new evidence directory; it recovered 3 and retained 10 failures. Combined distinct validity is 14/24:\n\n- Ling: 11/12 valid\n- MiMo: 3/12 valid\n- failures were missing opening or closing final-text markers, not timeouts\n\nThe canonical importer refuses the incomplete set at the first missing result, as designed. The official blind command also requires the complete frozen task set, so this diagnostic cannot produce a benchmark packet or support rollout.\n\nI assembled a clearly labeled partial, condition-blind human inspection packet for the 14 valid outputs only: 14 unique aliases, no task IDs or condition labels, 10 missing outputs disclosed, and three complete same-case/model triplets eligible for optional preference review. Packet SHA-256: 7A2493F5C427E8FE83F6D53D0C7311203452127A7159CD81CF616709963C2AF8. The private key and raw provider evidence remain local.\n\nBottom line: #296 is mechanically healthy after restacking, but editor reliability and the required complete human/heldout comparison remain open. No release decision is supported.