diff --git a/CLAUDE.md b/CLAUDE.md index 8cdfe98..d0eddd2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -29,8 +29,20 @@ document, or run staged critique -> defend -> tighten passes. bash ./co-evolve-bouncer.sh --vanilla "What is the strongest version of this argument?" bash ./co-evolve-bouncer.sh --vanilla --bounce-only docs/plan.md bash ./co-evolve-bouncer.sh --vanilla --chain "Should we ship this migration?" +bash ./co-evolve-bouncer.sh --vanilla --single-model "Stress test this" # both roles on claude ``` +`--single-model [claude|codex]` pins both roles onto one agent (bare two-role +mode by default — empirical winner on dense technical docs per 2026-05-24 A/B). +Use when only one model is available, or to A/B against cross-model runs. + +Add `--persona-discipline` to prepend the divergence preface +(`templates/co-evolve/single-model-preface.md`) that asks the model to +deliberately read against its own prior turn. Best paired with compose-then- +bounce of your own draft, where the shared-author bias actually applies. +On `--bounce-only` of external docs the preface tends to suppress the model's +natural investigatory impulse — use bare mode there. + ### Agent Bouncer (`agent-bouncer/`) Legacy standalone script that bounces any markdown document between two agents. diff --git a/README.md b/README.md index 456ec41..214ffd5 100644 --- a/README.md +++ b/README.md @@ -74,6 +74,32 @@ bash ./co-evolve-bouncer.sh --vanilla "What is the strongest version of this arg bash ./co-evolve-bouncer.sh --vanilla --chain "Should we ship this migration?" ``` +When only one agent is available — or to A/B against a cross-model run — use +`--single-model` to pin both reviewer and composer onto the same agent. By +default this is **bare two-role mode**: same model, two role prompts (composer +and critic), nothing else. An A/B on a dense technical document (2026-05-24) +showed this beats the persona-discipline variant — the model preserves its +natural impulse to investigate the codebase and produces concrete, code-grounded +critique instead of abstract methodological objections. + +```bash +bash ./co-evolve-bouncer.sh --vanilla --single-model "Stress test this argument" +bash ./co-evolve-bouncer.sh --vanilla --single-model codex --chain "Ship plan?" +``` + +If you want the divergence preface — which asks the model to deliberately read +against its own prior turn — enable it with `--persona-discipline`. This pairs +naturally with compose-then-bounce of your own draft, where there *is* a +shared-author bias to fight: + +```bash +bash ./co-evolve-bouncer.sh --vanilla --single-model --persona-discipline "..." +``` + +Expect shallower diversity than cross-model bounces (shared weights share +blindspots), but most single-model runs still surface 1-2 real objections in +the first pass. + ### [Agent Bouncer](agent-bouncer/) A standalone bash script that bounces any markdown document between two agents. diff --git a/co-evolve-bouncer.sh b/co-evolve-bouncer.sh index 9fc63f6..6f8034f 100644 --- a/co-evolve-bouncer.sh +++ b/co-evolve-bouncer.sh @@ -15,6 +15,18 @@ CHAIN=false MAX_BOUNCES=2 AGENT_A="claude" AGENT_B="codex" +SINGLE_MODEL=false +SINGLE_MODEL_AGENT="" +# Persona-discipline preface — opt-in. Default off because empirical A/B on a +# dense technical document showed the preface suppressed the model's natural +# investigatory impulse (raised abstract objections instead of concrete, +# code-grounded ones). Still useful when compose-then-bouncing your own draft; +# enable with --persona-discipline. +PERSONA_DISCIPLINE=false +# Set true by the bounce loop when an agent returns empty output after retry. +# Causes the script to exit 2 and emit a HINT pointing at --single-model. +BOUNCE_FAILED=false +FAILED_AGENT="" BOUNCE_ONLY=false OUTPUT_FILE="" TASK="" @@ -37,11 +49,13 @@ TIMESTAMP=$(date +%Y%m%d-%H%M%S) TEMPLATE_DIR="$SCRIPT_DIR/templates/co-evolve" PROTOCOL_TEMPLATE="$SCRIPT_DIR/agent-bouncer/templates/bounce-protocol.md" +SINGLE_MODEL_PREFACE="$TEMPLATE_DIR/single-model-preface.md" # Validate templates exist for _tmpl in "$TEMPLATE_DIR/role-reviewer-light.md" "$TEMPLATE_DIR/role-composer-light.md" \ "$TEMPLATE_DIR/chain-critique.md" "$TEMPLATE_DIR/chain-defend.md" \ - "$TEMPLATE_DIR/chain-tighten.md" "$PROTOCOL_TEMPLATE"; do + "$TEMPLATE_DIR/chain-tighten.md" "$SINGLE_MODEL_PREFACE" \ + "$PROTOCOL_TEMPLATE"; do [[ -f "$_tmpl" ]] || die "Missing template: $_tmpl" done @@ -64,6 +78,19 @@ Options: --chain Use staged passes: critique -> defend -> tighten --bounces N Max bounce passes (default: 2, ignored with --chain) --agents A,B Agent pair (default: claude,codex) + --single-model[=AGENT] + Force both roles onto the same agent (default: claude). + Bare two-role mode by default (no preface) — empirical A/B + showed this preserves the model's natural investigatory + impulse on dense technical docs. Overrides --agents. + Add --persona-discipline to enable the divergence preface. + --persona-discipline + Prepend a persona-discipline preface that asks the model + to deliberately diverge from its own prior turn. Useful + when compose-then-bouncing your own draft (where there + IS a shared-author bias to fight). May hurt on bounce-only + of external docs — see notesforhumans.md for the A/B data. + Can be used with or without --single-model. --dev-review Add execute + verify phases after bounce --bounce-only Skip compose, bounce a file directly --output FILE Write final output to a file instead of stdout @@ -107,6 +134,40 @@ while [[ $# -gt 0 ]]; do [[ -z "$AGENT_A" || -z "$AGENT_B" ]] && die "--agents requires exactly two agents separated by comma (e.g., claude,codex)" shift 2 ;; + --single-model) + SINGLE_MODEL=true + # Optional positional agent: --single-model codex. If next arg looks like + # a flag or is absent, fall back to the default ("claude"). + if [[ $# -ge 2 && -n "${2:-}" && "${2:0:1}" != "-" ]]; then + SINGLE_MODEL_AGENT="$2" + shift 2 + else + SINGLE_MODEL_AGENT="claude" + shift + fi + case "$SINGLE_MODEL_AGENT" in + claude|codex) ;; + *) die "--single-model agent must be claude or codex (got: $SINGLE_MODEL_AGENT)" ;; + esac + AGENT_A="$SINGLE_MODEL_AGENT" + AGENT_B="$SINGLE_MODEL_AGENT" + ;; + --single-model=*) + SINGLE_MODEL=true + SINGLE_MODEL_AGENT="${1#--single-model=}" + [[ -n "$SINGLE_MODEL_AGENT" ]] || die "--single-model= requires an agent name" + case "$SINGLE_MODEL_AGENT" in + claude|codex) ;; + *) die "--single-model agent must be claude or codex (got: $SINGLE_MODEL_AGENT)" ;; + esac + AGENT_A="$SINGLE_MODEL_AGENT" + AGENT_B="$SINGLE_MODEL_AGENT" + shift + ;; + --persona-discipline) + PERSONA_DISCIPLINE=true + shift + ;; --dev-review) die "--dev-review is not yet implemented. Use dev-review/codex/dev-review.sh directly." ;; --bounce-only) BOUNCE_ONLY=true; shift ;; --output) OUTPUT_FILE="$2"; shift 2 ;; @@ -322,34 +383,49 @@ $(cat "$CONTEXT_FILE") fi # --- Role Preamble Generation --- +# Prepends the persona-discipline preface when --persona-discipline is set. +# Decoupled from --single-model after a 2026-05-24 A/B test on a dense +# technical document showed the preface SUPPRESSED concrete code-grounded +# critique in favor of abstract methodological objections — opposite of what +# was intended. Kept available as opt-in for compose-then-bounce-own-draft +# flows, where the prior-turn divergence framing actually applies. +maybe_prepend_single_model_preface() { + local body="$1" + if [[ "$PERSONA_DISCIPLINE" == "true" ]]; then + printf '%s\n%s' "$(cat "$SINGLE_MODEL_PREFACE")" "$body" + else + printf '%s' "$body" + fi +} + build_reviewer_preamble() { + local preamble if [[ -n "$LENS" ]]; then - echo "You are the ${LENS} reviewing this work. Be adversarial from that perspective. Every critique must include a concrete alternative." + preamble="You are the ${LENS} reviewing this work. Be adversarial from that perspective. Every critique must include a concrete alternative." elif [[ "$SKIP_INTERVIEW" == "true" ]]; then - cat "$TEMPLATE_DIR/role-reviewer-light.md" + preamble=$(cat "$TEMPLATE_DIR/role-reviewer-light.md") else - local preamble preamble=$(cat "$TEMPLATE_DIR/role-reviewer-light.md") if [[ -n "$AUDIENCE" && "$AUDIENCE" != "general" && "$AUDIENCE" != "auto" ]]; then preamble="${preamble}Evaluate this as if you are a ${AUDIENCE} reading it. What would they find unconvincing, unclear, or missing?" fi - echo "$preamble" fi + maybe_prepend_single_model_preface "$preamble" } build_composer_preamble() { + local preamble if [[ -n "$LENS" ]]; then - echo "Resolve all critiques from the ${LENS} perspective. Strengthen weak points. Make it bulletproof." + preamble="Resolve all critiques from the ${LENS} perspective. Strengthen weak points. Make it bulletproof." elif [[ "$SKIP_INTERVIEW" == "true" ]]; then - cat "$TEMPLATE_DIR/role-composer-light.md" + preamble=$(cat "$TEMPLATE_DIR/role-composer-light.md") else - local preamble preamble=$(cat "$TEMPLATE_DIR/role-composer-light.md") if [[ -n "${OUTPUT_TYPE:-}" && "$OUTPUT_TYPE" != "auto" ]]; then preamble="${preamble}The output should be a ${OUTPUT_TYPE}. Shape it accordingly." fi - echo "$preamble" fi + maybe_prepend_single_model_preface "$preamble" } # --- Agent Invocation Helper --- @@ -446,6 +522,7 @@ run_bounce_phase() { esac role="critique" [[ "$pass" == "3" ]] && role="tighten" + role_preamble=$(maybe_prepend_single_model_preface "$role_preamble") else role_preamble=$(build_reviewer_preamble) role="reviewer" @@ -455,6 +532,7 @@ run_bounce_phase() { if [[ "$CHAIN" == "true" ]]; then role_preamble=$(cat "$TEMPLATE_DIR/chain-defend.md") role="defend" + role_preamble=$(maybe_prepend_single_model_preface "$role_preamble") else role_preamble=$(build_composer_preamble) role="composer" @@ -499,6 +577,8 @@ $(cat "$PROTOCOL_TEMPLATE")" if [[ ! -s "$output_file" ]]; then log " ERROR: ${current_agent} returned empty output on retry. Stopping." + BOUNCE_FAILED=true + FAILED_AGENT="$current_agent" break fi @@ -554,6 +634,12 @@ log " Input: $INPUT_TYPE" log " Task: $(echo "$TASK" | head -c 80)" log " Compose: $AGENT_A" log " Bounce: $AGENT_A / $AGENT_B" +if [[ "$SINGLE_MODEL" == "true" ]]; then + log " Single: yes (both roles on $SINGLE_MODEL_AGENT, bare two-role mode)" +fi +if [[ "$PERSONA_DISCIPLINE" == "true" ]]; then + log " Persona: discipline preface enabled (opt-in)" +fi if [[ "$CHAIN" == "true" ]]; then log " Mode: chain (critique -> defend -> tighten)" else @@ -589,14 +675,42 @@ if [[ -n "$OUTPUT_FILE" ]]; then fi log "============================================" -log " CO-EVOLVE COMPLETE" +if [[ "$BOUNCE_FAILED" == "true" ]]; then + log " CO-EVOLVE INCOMPLETE (bounce aborted)" +else + log " CO-EVOLVE COMPLETE" +fi log "============================================" log " Task: $(echo "$TASK" | head -c 80)" log " Run dir: $RUN_DIR" log " Final: $FINAL_FILE" log "============================================" +# Cross-vendor failure UX: emit a loud WARNING + actionable HINT pointing at +# --single-model with whichever agent did NOT fail. Suppress the hint when the +# user is already in single-model mode (no useful escape hatch to suggest). +if [[ "$BOUNCE_FAILED" == "true" ]]; then + log "" + log "WARNING: bounce ended early because ${FAILED_AGENT} returned empty output." + log " The output may contain unresolved [CONTESTED]/[CLARIFY] markers." + if [[ "$SINGLE_MODEL" == "false" ]]; then + if [[ "$FAILED_AGENT" == "$AGENT_A" ]]; then + working_agent="$AGENT_B" + else + working_agent="$AGENT_A" + fi + log "" + log "HINT: if ${FAILED_AGENT} isn't installed or available, retry with:" + log " --single-model ${working_agent}" + log " (uses ${working_agent} for both roles; cross-vendor diversity reduced)" + fi +fi + # Print clean result to stdout unless output was redirected to file if [[ -z "$OUTPUT_FILE" ]]; then cat "$FINAL_FILE" fi + +if [[ "$BOUNCE_FAILED" == "true" ]]; then + exit 2 +fi diff --git a/lib/codex-access.sh b/lib/codex-access.sh new file mode 100644 index 0000000..48cbe83 --- /dev/null +++ b/lib/codex-access.sh @@ -0,0 +1,104 @@ +#!/usr/bin/env bash +# codex-access — single source of truth for invoking the Codex CLI across +# environments. Historically this logic was duplicated in 6+ places across +# the codebase; the lib/ helper centralizes the launch matrix so callers +# don't reimplement it. +# +# Launch matrix (checked in order): +# 1. WSL with Windows-side codex install: +# `cmd.exe /c codex exec ...` — bridges from WSL to the Windows binary +# because auth state is per-side (installing in both WSL AND Windows +# doubles the OPENAI billing). +# 2. Native Linux/macOS with codex on PATH: +# `codex exec ...` — direct invocation. +# 3. No codex reachable: +# `codex_available` returns 1; `codex_invoke` writes the install hint +# to stderr and produces empty output (matches the existing empty-output +# contract so callers don't need a new branch). +# +# See skills/codex-access/SKILL.md for the user-facing description. +# +# This module is intentionally side-effect free at source time — it only +# defines functions. Source it from any script that needs codex access. + +# codex_available → exit 0 iff codex is reachable via any supported path. +# WSL+cmd.exe path is checked first because that's the upstream-recommended +# install on Windows (auth lives on the Windows side). +codex_available() { + if [[ -n "${WSL_DISTRO_NAME:-}" ]] \ + && command -v cmd.exe >/dev/null 2>&1 \ + && command -v wslpath >/dev/null 2>&1; then + cmd.exe /c codex --version >/dev/null 2>&1 && return 0 + fi + command -v codex >/dev/null 2>&1 +} + +# codex_install_hint → prints a multi-line, actionable install/availability +# message to stdout. Callers redirect to stderr when used inside an error path. +codex_install_hint() { + cat <<'HINT' +codex CLI is not reachable from this environment. + +To enable codex-backed runs: + 1. Install: npm install -g @openai/codex + 2. Authenticate: export OPENAI_API_KEY=sk-... + 3. Verify: codex --version + +WSL users: install codex on the Windows side, NOT inside WSL. +codex-access bridges via cmd.exe automatically. Auth state is per-side, +so installing twice would double your OpenAI billing. + +If codex is genuinely unavailable in this environment (e.g. a remote +container with no API key), use: + bash co-evolve-bouncer.sh --single-model claude ... + +to run same-model co-evolution. This trades cross-vendor diversity for +the ability to actually run at all. +HINT +} + +# codex_invoke "$prompt_file" "$output_file" "$stderr_file" +# Drop-in equivalent of lib/co-evolution.sh's invoke_codex, but routed through +# the launch matrix above so the WSL/native/missing cases all live in one +# place. Returns 0 always — errors flow through empty output_file + populated +# stderr_file, matching the existing invoke_* contract (callers detect failure +# by checking [[ ! -s "$output_file" ]]). +codex_invoke() { + local prompt_file="$1" + local output_file="$2" + local stderr_file="$3" + local workdir="${WORKDIR:-$PWD}" + local -a cmd + local windows_workdir="" + local windows_output="" + + if [[ -n "${WSL_DISTRO_NAME:-}" ]] \ + && command -v cmd.exe >/dev/null 2>&1 \ + && command -v wslpath >/dev/null 2>&1; then + windows_workdir=$(wslpath -w "$workdir") + windows_output=$(wslpath -w "$output_file") + cmd=(cmd.exe /c codex exec --full-auto --skip-git-repo-check -C "$windows_workdir") + elif command -v codex >/dev/null 2>&1; then + cmd=(codex exec --full-auto --skip-git-repo-check -C "$workdir") + else + { + echo "ERROR: codex CLI not reachable in this environment." + echo "" + codex_install_hint + } > "$stderr_file" + : > "$output_file" + return 0 + fi + + if [[ -n "${CODEX_MODEL:-}" ]]; then + cmd+=(-c "model=${CODEX_MODEL}") + fi + + if [[ -n "$windows_output" ]]; then + cmd+=(-o "$windows_output") + else + cmd+=(-o "$output_file") + fi + + "${cmd[@]}" < "$prompt_file" > /dev/null 2>"$stderr_file" || true +} diff --git a/notesforhumans.md b/notesforhumans.md index 88a0bef..aae1675 100644 --- a/notesforhumans.md +++ b/notesforhumans.md @@ -61,6 +61,42 @@ When Claude reviews and Codex composes (or vice versa), the disagreements are re The lesson: the protocol is agent-agnostic, but the value scales with cognitive diversity. +## Same-Model A/B (2026-05-24) + +When cross-vendor isn't available, `--single-model` pins both roles to one +agent. We A/B'd two variants of single-model on the same inputs: + +- **Bare two-role mode** — same model, just the composer and critic prompts. +- **Persona-discipline preface** — bare mode plus a prepended instruction asking + the model to deliberately diverge from its own prior turn. + +Two test runs, both with Claude as the single model: + +| Input | Bare markers (pass 1) | Preface markers (pass 1) | Winner | +|---|---|---|---| +| Open prompt (~30 words, meta-question) | 4 | 7 | preface (more critique) | +| Dense paper section (`04-pilot-data.md`, 1277 words) | **14** | 6 | **bare (much more critique, and code-grounded)** | + +On the dense doc the bare baseline produced devastating, codebase-grounded +critique — caught that `runs/` is in `.gitignore` so the dataset is +unreproducible, that the "final-pass enforcement check" referenced in the prose +doesn't exist as code, that the marker-dynamics table has internal +inconsistencies. The preface variant stayed at the abstract methodological +level and missed all of these. + +The hypothesis: the preface tells the model to "read as if a stranger wrote it" +and "suppress default voice," which pushes it into pure-text-critique mode and +suppresses the natural impulse to investigate the codebase (the non-writable +phase allows `Read`, just not `Edit`/`Bash`/`Glob`/`Grep`). On an open-ended +prompt with nothing to verify, the nudge helps. On a dense technical document +where verification is the highest-value behavior, the nudge actively hurts. + +The fix: bare two-role mode is the new default for `--single-model`. The +preface still exists as opt-in via `--persona-discipline`, which we expect to +remain useful for compose-then-bounce of your own draft (where shared-author +bias actually applies). That case wasn't isolated in the A/B and is open +research. + ## What We Learned Building It **Codex edits files it can see.** When Codex runs with `--full-auto`, it has filesystem write access. If the prompt references a file path, Codex might edit that file directly — even when you've told it to write output elsewhere. The fix: never expose the plan file path to the agent. Embed the plan content inline in the prompt, pipe via stdin, and capture output to a separate file. The Agent Bouncer is the sole owner of the canonical plan. diff --git a/skills/codex-access/SKILL.md b/skills/codex-access/SKILL.md new file mode 100644 index 0000000..2e7bdb3 --- /dev/null +++ b/skills/codex-access/SKILL.md @@ -0,0 +1,170 @@ +--- +name: codex-access +description: > + Reference pattern for invoking the OpenAI Codex CLI across environments + (WSL with Windows-side install, native Linux/macOS, or "codex unavailable" + fallback). Centralizes the launch matrix used by the co-evolve bouncer, + dev-review runtime, and PEL adapters so scripts don't reimplement it. + Triggers on "how do I call codex", "codex not found", "set up codex", + "codex install", "WSL codex bridge", and "make codex work for co-evolution". +allowed-tools: Bash, Read +--- + +# /codex-access — Codex CLI Launch Pattern + +Use this skill when: +- A script needs to invoke `codex` and you don't want to duplicate the WSL + bridge / install-detection / fallback logic. +- A user reports "codex command not found" or "codex unavailable" and you need + to point them at the canonical fix. +- You're writing a new agent adapter and want it to handle codex availability + the same way `co-evolve-bouncer.sh` does. + +This skill does NOT install codex for the user — installation requires their +OpenAI account and an API key. It documents the launch matrix and provides a +sourceable helper. + +## The Launch Matrix + +Codex is invoked one of three ways, in priority order: + +| # | When | How | +|---|------|-----| +| 1 | WSL with Windows-side codex install | `cmd.exe /c codex exec ...` (bridge) | +| 2 | Native Linux/macOS with `codex` on PATH | `codex exec ...` (direct) | +| 3 | Codex not reachable anywhere | Empty output + install hint to stderr | + +### Why the WSL bridge + +WSL and Windows keep separate auth state — installing codex inside WSL means +re-authenticating with OpenAI and double-billing. The recommended pattern is +to install codex on the Windows side, then call it from WSL via `cmd.exe /c +codex`. Path translation is handled by `wslpath -w` for the `-C ` +and `-o ` flags. + +### Why the empty-output fallback + +The existing `invoke_*` contract across this codebase is: errors flow through +empty output_file + populated stderr_file. Callers detect failure by checking +`[[ ! -s "$output_file" ]]`. The "codex unreachable" case follows this same +contract so existing callers don't need a new branch — they get the empty +output and surface it with their normal failure UX (e.g. `co-evolve-bouncer.sh` +prints "CO-EVOLVE INCOMPLETE" + HINT when this happens during a bounce pass). + +## The Helper + +Source `lib/codex-access.sh` and call: + +```bash +source "$REPO_ROOT/lib/codex-access.sh" + +# Check up-front whether codex is reachable (before committing to a workflow): +if codex_available; then + echo "codex ready" +else + codex_install_hint >&2 + exit 1 +fi + +# Or just invoke and let empty-output handle failure: +codex_invoke "$prompt_file" "$output_file" "$stderr_file" +if [[ ! -s "$output_file" ]]; then + cat "$stderr_file" >&2 # contains the install hint when codex was missing + exit 1 +fi +``` + +### Function reference + +| Function | Returns | Purpose | +|---|---|---| +| `codex_available` | exit 0 if reachable, 1 otherwise | Up-front check before workflows that require codex | +| `codex_invoke ` | always 0 | Drop-in for the existing `invoke_codex` contract | +| `codex_install_hint` | prints to stdout | Multi-line actionable install/availability message | + +`codex_invoke` honors the same env vars as the existing `invoke_codex` in +`lib/co-evolution.sh`: +- `WORKDIR` — codex `-C` argument (defaults to `$PWD`) +- `CODEX_MODEL` — passes `-c model=` to codex + +## Install Path For Users + +When a user asks how to get codex working: + +### Native Linux / macOS + +```bash +npm install -g @openai/codex +export OPENAI_API_KEY=sk-... +codex --version # verify +``` + +Authenticate once, then `codex_available` will return 0 and `codex_invoke` +will route through the direct path. + +### WSL (recommended) + +Install on the **Windows** side, not inside WSL: + +```powershell +# In a Windows shell (PowerShell or cmd.exe): +npm install -g @openai/codex +setx OPENAI_API_KEY "sk-..." +codex --version +``` + +Then from WSL, `codex_invoke` will automatically use the `cmd.exe /c codex` +bridge with `wslpath` translation. Don't install codex inside WSL too — auth +state is per-side and you'll be billed twice. + +### Remote container / no API key (the cloud case) + +If you're running in a managed remote container (e.g. Claude Code on the web, +a CI runner, an ephemeral sandbox) and don't have an OpenAI API key in the +environment, codex won't be reachable regardless of install attempts. The +right move is: + +```bash +bash co-evolve-bouncer.sh --single-model claude ... +``` + +This trades cross-vendor diversity for the ability to run at all. The +`co-evolve-bouncer.sh` failure UX will already suggest this when codex fails +mid-bounce. + +## Integration With Existing Code + +`lib/co-evolution.sh:invoke_codex` already implements the launch matrix +inline (it predates this helper). Both implementations are kept in sync: +new scripts should source `lib/codex-access.sh` rather than copy the inline +pattern. A future refactor may collapse `invoke_codex` to call `codex_invoke`, +but that's a hot path and gated on a separate change. + +The PEL adapters (`lab/pel/*/adapter.sh`) and dev-review runtime +(`dev-review/codex/dev-review.sh`) currently inline the same pattern. They +work — but if you touch them, prefer routing through the helper. + +## Verifying The Pattern + +The hermetic test `tests/codex-access-simulation.sh` exercises: +- WSL detection branch (when `WSL_DISTRO_NAME` is set and `cmd.exe` is on PATH) +- Direct-call branch (codex on PATH, no WSL) +- Unreachable branch (no codex, no WSL — produces install hint to stderr, + empty output, exit 0) + +Run it before changing the helper: + +```bash +bash tests/codex-access-simulation.sh +``` + +## What This Skill Does NOT Do + +- Does not install codex on the user's machine — that requires their account. +- Does not configure `OPENAI_API_KEY` — that's user state. +- Does not fake codex with claude. Same-model co-evolution lives behind + `--single-model` in `co-evolve-bouncer.sh`; mixing the two would hide + cross-vendor diversity behind a misleading flag name. +- Does not auto-fall-back from `codex` to `claude`. The invariant in + `co-evolve-bouncer.sh` is that single-model only activates via explicit + user flag. This skill respects that. diff --git a/templates/co-evolve/single-model-preface.md b/templates/co-evolve/single-model-preface.md new file mode 100644 index 0000000..1061060 --- /dev/null +++ b/templates/co-evolve/single-model-preface.md @@ -0,0 +1,22 @@ +## SINGLE-MODEL PERSONA DISCIPLINE + +You and the previous turn(s) share the same underlying model. Same weights, same +training, same priors — which means the same blindspots unless you actively +fight them. + +To make this bounce produce real disagreement, not polish: + +- Read the document as if a stranger wrote it. Do not pattern-match on your own + prior style or word choices. +- If you find yourself nodding along, that is the shared-weight bias. Push + harder — find the second-order objection. +- Your role this turn (described below) is your sole instruction. Suppress any + default voice that would soften it. +- Diverge deliberately: pick a different framing, a different reading order, + a different threat model than the prior turn implicitly used. + +The protocol's value depends on the turns being substantively different. With +shared weights, that difference comes from you — not the model. + +--- + diff --git a/tests/codex-access-simulation.sh b/tests/codex-access-simulation.sh new file mode 100755 index 0000000..4598acb --- /dev/null +++ b/tests/codex-access-simulation.sh @@ -0,0 +1,278 @@ +#!/usr/bin/env bash +# tests/codex-access-simulation.sh +# Hermetic simulation of the lib/codex-access.sh launch matrix. +# +# All scenarios are hermetic: no real codex/cmd.exe/claude binaries are run. +# We construct a PATH-shadow directory with fake `codex`, `cmd.exe`, and +# `wslpath` stubs, then source the helper and exercise each branch. +# +# Scenarios: +# A: codex_available returns 1 when no codex is reachable. +# B: codex_available returns 0 when `codex` is on PATH (native branch). +# C: codex_available returns 0 when WSL_DISTRO_NAME is set and cmd.exe +# + wslpath are on PATH (WSL bridge branch). +# D: codex_invoke with no codex reachable writes empty output_file, +# populates stderr_file with the install hint, and returns 0. +# E: codex_invoke (native branch) actually calls the codex stub with the +# expected argv and the stub's canned output reaches output_file. +# F: codex_invoke (WSL branch) routes through cmd.exe stub with +# wslpath-translated paths. +# G: codex_install_hint prints the actionable multi-line message. +# +# Final line on success: `7/7 scenarios passed`. +# Exit 0 iff all scenarios pass; exit 1 otherwise. + +set -euo pipefail + +TEST_DIR=$(mktemp -d -t codex-access-sim-XXXXXX) +cleanup() { rm -rf "$TEST_DIR"; } +trap cleanup EXIT + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" +HELPER="$REPO_ROOT/lib/codex-access.sh" + +[[ -f "$HELPER" ]] || { echo "FAIL: helper missing: $HELPER" >&2; exit 1; } + +FAILURES=0 +TOTAL=0 +fail() { echo "FAIL: $1" >&2; FAILURES=$((FAILURES + 1)); } +pass() { echo "PASS: $1"; } + +# Minimal PATH containing system binaries (bash, grep, etc.) but NOT the +# user's real codex/claude/cmd.exe. Each scenario prepends its own stub dir +# so we can isolate which launch-matrix branch fires. +BASE_PATH="/usr/bin:/bin" + +# --------------------------------------------------------------------------- +# PATH-shadow setup helpers +# --------------------------------------------------------------------------- +# Each scenario builds its own shadow $PATH and (optionally) sets WSL_DISTRO_NAME +# so the launch-matrix branches are exercised in isolation. We never touch the +# real $PATH outside of a subshell. + +make_codex_stub() { + local stub_dir="$1" + mkdir -p "$stub_dir" + cat > "$stub_dir/codex" <<'STUB' +#!/usr/bin/env bash +# codex stub — handles --version and the `exec` subcommand. +if [[ "${1:-}" == "--version" ]]; then + echo "codex-stub 0.0.1" + exit 0 +fi +# exec subcommand: pick up -o and write canned text to it. +output="" +while [[ $# -gt 0 ]]; do + case "$1" in + -o) + output="$2"; shift 2 ;; + *) + shift ;; + esac +done +if [[ -n "$output" ]]; then + echo "stub codex output" > "$output" +fi +STUB + chmod +x "$stub_dir/codex" +} + +make_cmdexe_stub() { + local stub_dir="$1" + mkdir -p "$stub_dir" + # cmd.exe /c codex — forward to the codex stub in same dir. + # First arg is "/c", remaining args are the command to run. + cat > "$stub_dir/cmd.exe" < → just echo with a recognizable Windows-ish prefix so + # the test can grep for it in scenario F. + cat > "$stub_dir/wslpath" <<'STUB' +#!/usr/bin/env bash +[[ "${1:-}" == "-w" ]] && shift +printf 'C:\\fake\\wsl\\%s\n' "${1#/}" +STUB + chmod +x "$stub_dir/wslpath" +} + +# --------------------------------------------------------------------------- +# Scenario A: nothing reachable → codex_available returns 1 +# --------------------------------------------------------------------------- +TOTAL=$((TOTAL + 1)) +( + empty_dir="$TEST_DIR/sA-empty" + mkdir -p "$empty_dir" + PATH="$empty_dir:$BASE_PATH" \ + WSL_DISTRO_NAME="" \ + bash -c "source '$HELPER' && codex_available && exit 0 || exit 1" + rc=$? + if [[ $rc -eq 0 ]]; then + echo "A: codex_available returned 0 with no codex on PATH (expected 1)" >&2 + exit 1 + fi +) && pass "Scenario A (codex_available=1 when nothing reachable)" \ + || fail "Scenario A (no-codex detection)" + +# --------------------------------------------------------------------------- +# Scenario B: native codex on PATH → codex_available returns 0 +# --------------------------------------------------------------------------- +TOTAL=$((TOTAL + 1)) +( + native_dir="$TEST_DIR/sB-native" + make_codex_stub "$native_dir" + PATH="$native_dir:$BASE_PATH" \ + WSL_DISTRO_NAME="" \ + bash -c "source '$HELPER' && codex_available" +) && pass "Scenario B (codex_available=0 native branch)" \ + || fail "Scenario B (native detection)" + +# --------------------------------------------------------------------------- +# Scenario C: WSL bridge reachable → codex_available returns 0 +# --------------------------------------------------------------------------- +TOTAL=$((TOTAL + 1)) +( + wsl_dir="$TEST_DIR/sC-wsl" + make_codex_stub "$wsl_dir" # cmd.exe stub forwards to this codex + make_cmdexe_stub "$wsl_dir" + make_wslpath_stub "$wsl_dir" + PATH="$wsl_dir:$BASE_PATH" \ + WSL_DISTRO_NAME="Ubuntu" \ + bash -c "source '$HELPER' && codex_available" +) && pass "Scenario C (codex_available=0 WSL bridge branch)" \ + || fail "Scenario C (WSL bridge detection)" + +# --------------------------------------------------------------------------- +# Scenario D: codex_invoke with no codex → empty output + install hint stderr +# --------------------------------------------------------------------------- +TOTAL=$((TOTAL + 1)) +( + empty_dir="$TEST_DIR/sD-empty" + mkdir -p "$empty_dir" + prompt="$TEST_DIR/sD.prompt"; echo "test prompt" > "$prompt" + out="$TEST_DIR/sD.out" + err="$TEST_DIR/sD.err" + + PATH="$empty_dir:$BASE_PATH" \ + WSL_DISTRO_NAME="" \ + bash -c "source '$HELPER' && codex_invoke '$prompt' '$out' '$err'" + rc=$? + [[ $rc -eq 0 ]] \ + || { echo "D: codex_invoke returned $rc (expected 0 — error flows through files)" >&2; exit 1; } + [[ -f "$out" && ! -s "$out" ]] \ + || { echo "D: output_file should exist but be empty; size=$(wc -c < "$out" 2>/dev/null)" >&2; exit 1; } + grep -qF "codex CLI is not reachable" "$err" \ + || { echo "D: stderr missing install hint; got:" >&2; cat "$err" >&2; exit 1; } + grep -qF "npm install -g @openai/codex" "$err" \ + || { echo "D: stderr missing install command in hint" >&2; exit 1; } +) && pass "Scenario D (codex_invoke unreachable → install hint + empty output)" \ + || fail "Scenario D (unreachable fallback)" + +# --------------------------------------------------------------------------- +# Scenario E: codex_invoke native branch — stub gets invoked, output flows +# --------------------------------------------------------------------------- +TOTAL=$((TOTAL + 1)) +( + native_dir="$TEST_DIR/sE-native" + make_codex_stub "$native_dir" + prompt="$TEST_DIR/sE.prompt"; echo "test prompt" > "$prompt" + out="$TEST_DIR/sE.out" + err="$TEST_DIR/sE.err" + + PATH="$native_dir:$BASE_PATH" \ + WSL_DISTRO_NAME="" \ + WORKDIR="$TEST_DIR" \ + bash -c "source '$HELPER' && codex_invoke '$prompt' '$out' '$err'" + rc=$? + [[ $rc -eq 0 ]] || { echo "E: codex_invoke returned $rc" >&2; exit 1; } + grep -qF "stub codex output" "$out" \ + || { echo "E: output didn't contain stub canned text; out:" >&2; cat "$out" >&2; exit 1; } +) && pass "Scenario E (codex_invoke native branch invokes codex)" \ + || fail "Scenario E (native invocation)" + +# --------------------------------------------------------------------------- +# Scenario F: codex_invoke WSL branch — routes through cmd.exe + wslpath +# --------------------------------------------------------------------------- +# We use the cmd.exe stub which forwards to the codex stub. The codex stub +# writes to whichever -o path it's given. The wslpath stub returns +# `C:\fake\wsl\...`, so if the WSL branch is wired correctly the codex stub +# would TRY to write to that fake-Windows path (which doesn't exist as a real +# file). We make the test robust by having cmd.exe stub strip the windowsy +# prefix and write to the real path: see make_cmdexe_stub below for adjusted +# semantics — actually it's easier to just verify cmd.exe was invoked at all +# by checking stderr from a deliberately failing cmd.exe stub that logs +# its argv. Simpler: write a marker file when cmd.exe is invoked, then +# assert the marker exists. +# --------------------------------------------------------------------------- +TOTAL=$((TOTAL + 1)) +( + wsl_dir="$TEST_DIR/sF-wsl" + mkdir -p "$wsl_dir" + + # cmd.exe stub that writes a marker file proving it was invoked, then + # writes canned output to the real (unix) output file. The bouncer passes + # the Windows-translated path via -o, but our stub ignores that and uses + # an env-var-supplied real path so the assertion is straightforward. + marker="$TEST_DIR/sF.cmdexe-was-invoked" + out="$TEST_DIR/sF.out" + err="$TEST_DIR/sF.err" + prompt="$TEST_DIR/sF.prompt"; echo "test prompt" > "$prompt" + + cat > "$wsl_dir/cmd.exe" < "$out" +exit 0 +STUB + chmod +x "$wsl_dir/cmd.exe" + make_wslpath_stub "$wsl_dir" + + PATH="$wsl_dir:$BASE_PATH" \ + WSL_DISTRO_NAME="Ubuntu" \ + WORKDIR="$TEST_DIR" \ + bash -c "source '$HELPER' && codex_invoke '$prompt' '$out' '$err'" + + [[ -f "$marker" ]] \ + || { echo "F: cmd.exe stub was never invoked (WSL branch not taken)" >&2; exit 1; } + grep -qF "stub WSL output" "$out" \ + || { echo "F: output missing canned WSL stub text" >&2; exit 1; } +) && pass "Scenario F (codex_invoke WSL branch routes through cmd.exe)" \ + || fail "Scenario F (WSL bridge invocation)" + +# --------------------------------------------------------------------------- +# Scenario G: codex_install_hint prints the actionable message +# --------------------------------------------------------------------------- +TOTAL=$((TOTAL + 1)) +( + out=$(bash -c "source '$HELPER' && codex_install_hint") + echo "$out" | grep -qF "npm install -g @openai/codex" \ + || { echo "G: hint missing install command" >&2; exit 1; } + echo "$out" | grep -qF "OPENAI_API_KEY" \ + || { echo "G: hint missing API key step" >&2; exit 1; } + echo "$out" | grep -qF -- "--single-model claude" \ + || { echo "G: hint missing single-model claude fallback suggestion" >&2; exit 1; } + echo "$out" | grep -qF "WSL users" \ + || { echo "G: hint missing WSL guidance" >&2; exit 1; } +) && pass "Scenario G (codex_install_hint includes install + auth + fallback)" \ + || fail "Scenario G (install hint content)" + +# --------------------------------------------------------------------------- +echo +if [[ $FAILURES -eq 0 ]]; then + echo "$TOTAL/$TOTAL scenarios passed" + exit 0 +else + echo "$((TOTAL - FAILURES))/$TOTAL scenarios passed ($FAILURES failures)" + exit 1 +fi diff --git a/tests/single-model-simulation.sh b/tests/single-model-simulation.sh new file mode 100755 index 0000000..55fe1e4 --- /dev/null +++ b/tests/single-model-simulation.sh @@ -0,0 +1,389 @@ +#!/usr/bin/env bash +# tests/single-model-simulation.sh +# Hermetic simulation of the --single-model flag on co-evolve-bouncer.sh. +# +# Scenarios (all hermetic — no Claude/Codex CLIs invoked, only the +# arg-parser + preamble-builder paths are exercised): +# +# A: --help advertises --single-model and retains pre-existing flags +# (byte-parity spot-check for --agents, --bounces, --lab). +# B: Bare --single-model defaults to "claude" for both roles and +# logs the "Single: yes" banner line. +# C: --single-model codex routes both roles to codex. +# D: --single-model=opus rejected with a clear error (allow-list guard). +# E: When --single-model is NOT set, the banner has no "Single:" line +# (byte-parity invariant for default cross-model runs). +# F: --single-model alone does NOT prepend the persona-discipline preface +# (post-2026-05-24 A/B finding: bare two-role mode is the empirical +# winner on dense technical docs; preface decoupled into --persona-discipline). +# G: --persona-discipline prepends the preface (proves the opt-in wiring +# actually reaches the role preamble — covers both --single-model +# and standalone --persona-discipline cases). +# H: When a cross-vendor partner agent fails (empty output on retry), +# the bouncer exits 2, prints "INCOMPLETE", and emits a HINT pointing +# the user at --single-model . Verifies the +# 2026-05-25 smoke-test fix for silent partner-failure. +# I: When --single-model is active and the agent fails, the bouncer still +# exits 2 but does NOT emit the --single-model HINT (no useful escape +# hatch to suggest — the user is already in single-model mode). +# +# Final line on success: `9/9 scenarios passed`. +# Exit 0 iff all 9 scenarios pass; exit 1 otherwise. + +set -euo pipefail + +TEST_DIR=$(mktemp -d -t single-model-sim-XXXXXX) +cleanup() { rm -rf "$TEST_DIR"; } +trap cleanup EXIT + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" +BOUNCER="$REPO_ROOT/co-evolve-bouncer.sh" + +FAILURES=0 +TOTAL=0 +fail() { echo "FAIL: $1" >&2; FAILURES=$((FAILURES + 1)); } +pass() { echo "PASS: $1"; } + +# --------------------------------------------------------------------------- +# Scenario A: --help advertises --single-model + retains older flags +# --------------------------------------------------------------------------- +TOTAL=$((TOTAL + 1)) +( + out=$(bash "$BOUNCER" --help 2>&1) + echo "$out" | grep -qF -- "--single-model" \ + || { echo "A: --help missing --single-model row" >&2; exit 1; } + echo "$out" | grep -qF -- "--agents" \ + || { echo "A: --help missing --agents (byte-parity regression)" >&2; exit 1; } + echo "$out" | grep -qF -- "--bounces" \ + || { echo "A: --help missing --bounces (byte-parity regression)" >&2; exit 1; } + echo "$out" | grep -qF -- "--lab" \ + || { echo "A: --help missing --lab (byte-parity regression)" >&2; exit 1; } +) && pass "Scenario A (--help advertises --single-model, older flags intact)" \ + || fail "Scenario A (--help intact)" + +# --------------------------------------------------------------------------- +# Stub the underlying agent invocations so the bouncer can run end-to-end +# without network. We source lib/co-evolution.sh, then override invoke_claude +# and invoke_codex to write a tiny canned response so the bouncer reaches its +# banner-log + compose-phase code paths. +# +# This trick: run a wrapper script that pre-loads stubs into the environment +# via BASH_ENV, then exec's the real bouncer. The bouncer sources +# lib/co-evolution.sh AFTER BASH_ENV runs, so our stubs get overwritten — +# instead, we wrap by sourcing the lib ourselves, redefining the functions, +# then sourcing the bouncer's body. +# +# Simpler: write a stub-lib that the bouncer sources INSTEAD of the real lib, +# by manipulating its `source "$SCRIPT_DIR/lib/co-evolution.sh"` line via a +# shadow SCRIPT_DIR. We achieve that by copying the bouncer to TEST_DIR and +# placing a stub lib alongside it. +# --------------------------------------------------------------------------- + +SHADOW_DIR="$TEST_DIR/shadow" +mkdir -p "$SHADOW_DIR/lib" "$SHADOW_DIR/templates/co-evolve" \ + "$SHADOW_DIR/agent-bouncer/templates" + +# Copy templates the bouncer validates on startup. +cp "$REPO_ROOT/templates/co-evolve/role-reviewer-light.md" "$SHADOW_DIR/templates/co-evolve/" +cp "$REPO_ROOT/templates/co-evolve/role-composer-light.md" "$SHADOW_DIR/templates/co-evolve/" +cp "$REPO_ROOT/templates/co-evolve/chain-critique.md" "$SHADOW_DIR/templates/co-evolve/" +cp "$REPO_ROOT/templates/co-evolve/chain-defend.md" "$SHADOW_DIR/templates/co-evolve/" +cp "$REPO_ROOT/templates/co-evolve/chain-tighten.md" "$SHADOW_DIR/templates/co-evolve/" +cp "$REPO_ROOT/templates/co-evolve/single-model-preface.md" "$SHADOW_DIR/templates/co-evolve/" +cp "$REPO_ROOT/agent-bouncer/templates/bounce-protocol.md" "$SHADOW_DIR/agent-bouncer/templates/" +cp "$BOUNCER" "$SHADOW_DIR/co-evolve-bouncer.sh" + +# Stub lib: re-export the few real functions the bouncer needs, but stub +# invoke_claude / invoke_codex so they emit a canned ~50-word response. +cat > "$SHADOW_DIR/lib/co-evolution.sh" <<'STUB' +#!/usr/bin/env bash +log() { + local message="${1:-}" + if [[ -n "${LOG_FILE:-}" ]]; then + echo "$message" | tee -a "$LOG_FILE" + else + echo "$message" + fi +} +die() { log "ERROR: ${1:-Fatal error}"; exit 1; } + +validate_lab_mode() { return 1; } +list_available_lab_modes() { printf '(none)'; } +dispatch_lab_mode() { die "lab dispatch disabled in stub"; } + +# Canned ~60-word response so size_sanity_check passes for short inputs. +_canned_output() { + cat <<'EOF' +This is a stub agent response generated by the single-model test harness. +The harness exercises the bouncer's arg-parser and preamble-builder paths +without invoking any real model. Both compose and bounce phases produce +this identical body so marker-counting, working-file copying, and the +banner-log all execute the same code paths they would in a real run, but +with deterministic content and zero network access. +EOF +} + +# Failure-injection knobs (read at stub-call time, so the test can set them): +# STUB__FAILS=1 → fail unconditionally on every call +# STUB__FAILS_AFTER=N → succeed on calls 1..N, fail starting from N+1 +# (lets the compose pass succeed while a later +# bounce pass fails — required by scenario I) +# Per-run call counter is kept in $RUN_DIR/.stub--count, which the +# bouncer creates before the first invoke_* call. +_stub_should_fail() { + local agent="$1" count="$2" + local fails_var="STUB_${agent}_FAILS" + local after_var="STUB_${agent}_FAILS_AFTER" + if [[ "${!fails_var:-0}" == "1" ]]; then + return 0 + fi + if [[ -n "${!after_var:-}" ]] && (( count > ${!after_var} )); then + return 0 + fi + return 1 +} + +invoke_claude() { + local prompt_file="$1"; local output_file="$2"; local stderr_file="$3" + local count_file="${RUN_DIR:-/tmp}/.stub-claude-count" + local count=0 + [[ -f "$count_file" ]] && count=$(cat "$count_file") + count=$((count + 1)) + echo "$count" > "$count_file" + + if _stub_should_fail CLAUDE "$count"; then + : > "$output_file" + echo "stub: claude forced to fail (call #$count)" > "$stderr_file" + return 0 + fi + _canned_output > "$output_file" + : > "$stderr_file" + # Snapshot prompt so preface-wiring scenarios can grep it. + cp "$prompt_file" "${prompt_file}.captured" 2>/dev/null || true +} + +invoke_codex() { + local prompt_file="$1"; local output_file="$2"; local stderr_file="$3" + local count_file="${RUN_DIR:-/tmp}/.stub-codex-count" + local count=0 + [[ -f "$count_file" ]] && count=$(cat "$count_file") + count=$((count + 1)) + echo "$count" > "$count_file" + + if _stub_should_fail CODEX "$count"; then + : > "$output_file" + echo "stub: codex forced to fail (call #$count)" > "$stderr_file" + return 0 + fi + _canned_output > "$output_file" + : > "$stderr_file" + cp "$prompt_file" "${prompt_file}.captured" 2>/dev/null || true +} + +count_markers() { + awk -v marker="$2" 'BEGIN{c=0} index($0,marker){c++} END{print c}' "$1" \ + | tr -d '\r\n ' +} +strip_human_summary() { + awk '/^## HUMAN SUMMARY/{found=1} !found{print}' "$1" > "$2" +} +STUB + +run_bouncer() { + # Run the shadowed bouncer from inside its own dir so SCRIPT_DIR resolves + # to SHADOW_DIR and our stub lib is what gets sourced. + (cd "$SHADOW_DIR" && bash ./co-evolve-bouncer.sh "$@") +} + +# --------------------------------------------------------------------------- +# Scenario B: bare --single-model defaults to claude + logs banner line +# --------------------------------------------------------------------------- +TOTAL=$((TOTAL + 1)) +( + out=$(run_bouncer --vanilla --single-model --bounces 1 "smoke test prompt" 2>&1) \ + || { echo "B: bouncer exited non-zero; out: $out" >&2; exit 1; } + echo "$out" | grep -qE "Single:[[:space:]]+yes \(both roles on claude" \ + || { echo "B: missing 'Single: yes (... claude' banner; out: $out" >&2; exit 1; } + echo "$out" | grep -qF "bare two-role mode" \ + || { echo "B: banner missing 'bare two-role mode' phrase; out: $out" >&2; exit 1; } + echo "$out" | grep -qE "Compose:[[:space:]]+claude" \ + || { echo "B: compose agent not pinned to claude; out: $out" >&2; exit 1; } + echo "$out" | grep -qE "Bounce:[[:space:]]+claude / claude" \ + || { echo "B: bounce pair not pinned to claude/claude; out: $out" >&2; exit 1; } + # Bare --single-model must NOT enable the preface (default-off post-2026-05-24). + if echo "$out" | grep -qE "Persona:[[:space:]]+discipline preface enabled"; then + echo "B: bare --single-model unexpectedly enabled the persona-discipline preface" >&2 + exit 1 + fi +) && pass "Scenario B (bare --single-model defaults to claude)" \ + || fail "Scenario B (bare --single-model)" + +# --------------------------------------------------------------------------- +# Scenario C: --single-model codex pins both roles to codex +# --------------------------------------------------------------------------- +TOTAL=$((TOTAL + 1)) +( + out=$(run_bouncer --vanilla --single-model codex --bounces 1 "smoke test prompt" 2>&1) \ + || { echo "C: bouncer exited non-zero; out: $out" >&2; exit 1; } + echo "$out" | grep -qE "Bounce:[[:space:]]+codex / codex" \ + || { echo "C: bounce pair not pinned to codex/codex; out: $out" >&2; exit 1; } + echo "$out" | grep -qE "Single:[[:space:]]+yes \(both roles on codex" \ + || { echo "C: banner missing codex; out: $out" >&2; exit 1; } +) && pass "Scenario C (--single-model codex pins both roles to codex)" \ + || fail "Scenario C (--single-model codex)" + +# --------------------------------------------------------------------------- +# Scenario D: --single-model=opus rejected (allow-list guard) +# --------------------------------------------------------------------------- +TOTAL=$((TOTAL + 1)) +( + out=$(run_bouncer --vanilla --single-model=opus --bounces 1 "smoke test prompt" 2>&1 || true) + echo "$out" | grep -qF -- "--single-model agent must be claude or codex" \ + || { echo "D: missing allow-list error; out: $out" >&2; exit 1; } +) && pass "Scenario D (unknown --single-model agent rejected)" \ + || fail "Scenario D (allow-list)" + +# --------------------------------------------------------------------------- +# Scenario E: default (no --single-model) → no 'Single:' banner line +# --------------------------------------------------------------------------- +TOTAL=$((TOTAL + 1)) +( + # Force AGENT_B=claude too so we don't actually try codex; the stubs cover both anyway. + out=$(run_bouncer --vanilla --agents claude,claude --bounces 1 "smoke test prompt" 2>&1) \ + || { echo "E: bouncer exited non-zero; out: $out" >&2; exit 1; } + if echo "$out" | grep -qE "Single:[[:space:]]+yes"; then + echo "E: banner leaked 'Single:' line in default cross-model run; out: $out" >&2; exit 1 + fi +) && pass "Scenario E (default run has no 'Single:' banner — byte-parity)" \ + || fail "Scenario E (default banner clean)" + +# --------------------------------------------------------------------------- +# Scenario F: --single-model alone does NOT prepend the preface (default-off) +# --------------------------------------------------------------------------- +# Post-2026-05-24 finding: bare two-role mode is the empirical winner on +# dense docs. The preface MUST NOT leak into prompts unless --persona-discipline +# is explicitly set. Scans captured prompts for absence of the preface header. +# --------------------------------------------------------------------------- +TOTAL=$((TOTAL + 1)) +( + out=$(run_bouncer --vanilla --single-model --bounces 1 "default-off preface smoke" 2>&1) \ + || { echo "F: bouncer exited non-zero; out: $out" >&2; exit 1; } + + run_dir=$(echo "$out" | grep -E "Run dir:[[:space:]]" | tail -1 | sed -E 's/.*Run dir:[[:space:]]+//') + [[ -n "$run_dir" && -d "$run_dir" ]] \ + || { echo "F: could not locate run dir from banner; out: $out" >&2; exit 1; } + + captured=$(find "$run_dir" -maxdepth 1 -name '*.captured' 2>/dev/null) + [[ -n "$captured" ]] \ + || { echo "F: no .captured prompt files in $run_dir" >&2; exit 1; } + + if grep -lF "SINGLE-MODEL PERSONA DISCIPLINE" $captured >/dev/null 2>&1; then + echo "F: preface header leaked into prompts without --persona-discipline flag" >&2 + exit 1 + fi +) && pass "Scenario F (bare --single-model does NOT prepend preface — default-off)" \ + || fail "Scenario F (preface default-off)" + +# --------------------------------------------------------------------------- +# Scenario G: --persona-discipline prepends the preface (opt-in wiring) +# --------------------------------------------------------------------------- +# Combined with --single-model, the flag MUST cause the preface header to +# appear in captured prompts AND the banner MUST show the opt-in line. +# --------------------------------------------------------------------------- +TOTAL=$((TOTAL + 1)) +( + out=$(run_bouncer --vanilla --single-model --persona-discipline --bounces 1 \ + "persona-discipline opt-in smoke" 2>&1) \ + || { echo "G: bouncer exited non-zero; out: $out" >&2; exit 1; } + + echo "$out" | grep -qE "Persona:[[:space:]]+discipline preface enabled" \ + || { echo "G: banner missing 'Persona: discipline preface enabled' line; out: $out" >&2; exit 1; } + + run_dir=$(echo "$out" | grep -E "Run dir:[[:space:]]" | tail -1 | sed -E 's/.*Run dir:[[:space:]]+//') + [[ -n "$run_dir" && -d "$run_dir" ]] \ + || { echo "G: could not locate run dir from banner; out: $out" >&2; exit 1; } + + captured=$(find "$run_dir" -maxdepth 1 -name '*.captured' 2>/dev/null) + [[ -n "$captured" ]] \ + || { echo "G: no .captured prompt files in $run_dir" >&2; exit 1; } + + if ! grep -lF "SINGLE-MODEL PERSONA DISCIPLINE" $captured >/dev/null 2>&1; then + echo "G: preface header missing from all captured prompts in $run_dir" >&2 + exit 1 + fi +) && pass "Scenario G (--persona-discipline opt-in prepends preface)" \ + || fail "Scenario G (persona-discipline opt-in)" + +# --------------------------------------------------------------------------- +# Scenario H: cross-vendor partner failure → exit 2 + INCOMPLETE banner + HINT +# --------------------------------------------------------------------------- +# Use --chain so pass 2 always runs (no early convergence on marker-free stub +# output). Pass 2 in chain mode is the "defend" pass on AGENT_B (codex). +# The bouncer must: +# - exit code 2 (not 0) +# - log "CO-EVOLVE INCOMPLETE" +# - emit "HINT:" line suggesting --single-model with the working agent (claude) +# --------------------------------------------------------------------------- +TOTAL=$((TOTAL + 1)) +( + set +e + out=$(STUB_CODEX_FAILS=1 run_bouncer --vanilla --chain \ + --agents claude,codex "cross-vendor failure smoke" 2>&1) + exit_code=$? + set -e + + if [[ "$exit_code" != "2" ]]; then + echo "H: expected exit 2, got $exit_code" >&2 + echo "--- out ---" >&2; echo "$out" >&2 + exit 1 + fi + echo "$out" | grep -qF "CO-EVOLVE INCOMPLETE" \ + || { echo "H: missing INCOMPLETE banner; out: $out" >&2; exit 1; } + echo "$out" | grep -qF "HINT:" \ + || { echo "H: missing HINT line; out: $out" >&2; exit 1; } + echo "$out" | grep -qF -- "--single-model claude" \ + || { echo "H: hint doesn't suggest '--single-model claude'; out: $out" >&2; exit 1; } +) && pass "Scenario H (cross-vendor failure exits 2 with HINT)" \ + || fail "Scenario H (cross-vendor failure)" + +# --------------------------------------------------------------------------- +# Scenario I: --single-model + agent failure → exit 2 but NO HINT +# --------------------------------------------------------------------------- +# Use --chain (3 passes) + STUB_CLAUDE_FAILS_AFTER=2 so the first two claude +# calls (compose + pass 1 critique) succeed and the third (pass 2 defend) +# fails. This isolates the bounce-failure path from the compose-failure path +# (which exits 1, not 2). User already opted into single-model — no useful +# escape hatch — so HINT line must be absent. +# --------------------------------------------------------------------------- +TOTAL=$((TOTAL + 1)) +( + set +e + out=$(STUB_CLAUDE_FAILS_AFTER=2 run_bouncer --vanilla --single-model claude --chain \ + "single-model failure smoke" 2>&1) + exit_code=$? + set -e + + if [[ "$exit_code" != "2" ]]; then + echo "I: expected exit 2, got $exit_code" >&2 + echo "--- out ---" >&2; echo "$out" >&2 + exit 1 + fi + echo "$out" | grep -qF "CO-EVOLVE INCOMPLETE" \ + || { echo "I: missing INCOMPLETE banner; out: $out" >&2; exit 1; } + if echo "$out" | grep -qF "HINT:"; then + echo "I: HINT leaked despite --single-model mode; out: $out" >&2 + exit 1 + fi +) && pass "Scenario I (single-model failure exits 2 without HINT)" \ + || fail "Scenario I (single-model failure)" + +# --------------------------------------------------------------------------- +echo +if [[ $FAILURES -eq 0 ]]; then + echo "$TOTAL/$TOTAL scenarios passed" + exit 0 +else + echo "$((TOTAL - FAILURES))/$TOTAL scenarios passed ($FAILURES failures)" + exit 1 +fi