Add blinded BMADX value study and read-only dependency gate

.github/workflows/ci.yml

@@ -24,8 +24,12 @@ jobs:
         run: |
           python scripts/test_install_bmadx.py
           python scripts/test_install_and_verify_bmadx.py
+          python skill/bmadx/scripts/test_advise_planning_effort.py
           python skill/bmadx/scripts/test_check_codex_compat.py
           python skill/bmadx/scripts/test_sync_bmadx.py
           python benchmark/scripts/test_handoff_packet_schema.py
           python benchmark/scripts/test_run_bmadx_benchmark.py
+          python benchmark/scripts/test_sol_bmadx_ab.py
+          python benchmark/scripts/test_sol_bmadx_causal_canary.py
+          python benchmark/scripts/test_bmadx_value_study.py
           python benchmark/scripts/test_verify_bmadx_performance.py

.gitignore

@@ -2,3 +2,4 @@ __pycache__/
 *.pyc
 .DS_Store
 skill/bmadx/state/*.json
+benchmark/value-study/runs/

CHANGELOG.md

@@ -14,6 +14,33 @@ All notable changes to this repository will be documented here.
   blocks execution.
 - Removed automatic BMAD `check`/`sync` remediation from task startup and made
   installer verification exercise the read-only `X3` gate.
+### Added
+
+- Criteria-based planning-effort advisor that reports `Y/8` signals and may
+  recommend `high`, `xhigh`, `max`, or operator-confirmed `ultra` without
+  changing gear, BMAD gates, execution effort, or Codex configuration.
+- Optional benchmark `--run-label` for isolated raw and summary artifacts.
+- Framework-neutral, checkpointed plain-Sol versus BMADX A/B runner with
+  deterministic interleaving and shared scoring.
+- A/B runner v2 hardening: pinned per-call BMAD fixture, cleared inherited BMAD
+  environment, per-call filesystem isolation and hashes, exact resume manifest,
+  atomic checkpoints, explicit timeout, provenance hashes, and collision-safe names.
+
+### Benchmarks
+
+- GPT-5.6 Sol fixed-effort comparison for `medium`, `high`, and `xhigh` across
+  all groups with two repeats; retained as supplied-route contract conformance
+  because the legacy precomputed gate exposed the expected gear.
+- Blinded 180-case GPT-5.6 Sol A/B found a 13.33-16.67 percentage-point BMADX
+  exact-scorer gain at each tested effort, with higher token and latency cost;
+  external review classifies this as directional assigned-arm evidence only.
+- The preregistered three-arm decision-value study completed 162 Sol generation
+  calls and 325 blinded synthetic judgments. The final panel was unhealthy:
+  GLM 5.2 and Nemotron 3 Ultra missed the `0.70` order-stability threshold, so
+  unblinding and any BMADX value-added claim were blocked.
+- Added schema-only and transport-only retry accounting, exact checkpoint
+  migration after an Ollama Cloud outage, reviewer qualification canaries, and
+  a deterministic pre-unblinding panel gate.
 
 ## [0.3.1] - 2026-07-10
 

README.md

@@ -23,6 +23,9 @@ early testers.
 
 Current public version: `v0.3.1`
 
+Main development line: `v0.3.2-dev`, adding criteria-based planning effort and
+reproducible Sol reasoning comparisons.
+
 GPT-5.6 Sol, Terra, and Luna passed their first subscription-backed healthy
 canaries on 2026-07-10. They remain candidate profiles until repeated healthy
 and degraded BMADX benchmarks pass.
@@ -127,6 +130,19 @@ turn BMADX into a runtime orchestrator. See
 Rule: no public model claim without the same BMADX benchmark gates for routing,
 compactness, red-zone escalation, degraded BMAD behavior, and `X4` rarity.
 
+### Evidence status
+
+The July 16 blinded decision-value study completed 162 Sol generation calls and
+325 synthetic panel judgments, but two of five reviewer families failed the
+preregistered candidate-order stability gate. The run remained blinded and is
+**inconclusive**: it does not support a claim that BMADX is better or worse than
+plain Sol, and it does not establish outcomes for novice builders.
+
+This failure is published because the benchmark is meant to block attractive
+claims when the evaluation panel is not reliable enough. See the
+[full result and trade-offs](docs/bmadx-value-study-results-2026-07-16.md) and
+[machine-readable gate](benchmark/value-study/results/panel-gate-v1.13.json).
+
 ## Help test BMADX
 
 The most useful feedback is not "nice project." It is where BMADX
@@ -287,6 +303,8 @@ More detail:
 - [scripts/install_and_verify_bmadx.py](scripts/install_and_verify_bmadx.py)
 - [scripts/install_bmadx.py](scripts/install_bmadx.py)
 - [benchmark/scripts/run_bmadx_benchmark.py](benchmark/scripts/run_bmadx_benchmark.py)
+- [benchmark/scripts/run_sol_bmadx_ab.py](benchmark/scripts/run_sol_bmadx_ab.py)
+- [Plain Sol vs BMADX A/B](docs/sol-bmadx-ab-2026-07-12.md)
 - [samples/fubar-bundle](samples/fubar-bundle)
 
 ## Contributing

_bmad-output/project-context.md

@@ -1,4 +1,4 @@
-# Project Context — BMADX v0.3.1
+# Project Context — BMADX v0.3.2-dev
 
 ## Product Boundary
 
@@ -20,6 +20,7 @@ Non-negotiable:
 - Audit: `docs/audit-2026-07-10-gpt56.md`
 - Execution plan: `docs/bmadx-v0.3-plan.md`
 - Model guide: `docs/gpt56-model-compatibility.md`
+- Sol reasoning evidence: `docs/sol-reasoning-comparison-2026-07-11.md`
 
 Historical v0.2 artifacts remain evidence, not current instructions.
 
@@ -29,7 +30,7 @@ Done:
 
 - Codex CLI upgraded to `0.144.1`; GPT-5.6 minimum is `0.144.0`.
 - Local catalog exposes Sol, Terra, and Luna with expected reasoning levels.
-- Manifest and public surfaces are aligned on `v0.3.1`.
+- Public release remains `v0.3.1`; main development is `v0.3.2-dev`.
 - Shared model profile policy and compatibility checker added.
 - Benchmark model is explicit and advisor output is model-aware.
 - Performance claims and required coverage are isolated per model.
@@ -40,6 +41,10 @@ Done:
 - Global BMAD synchronized to upstream `v6.10.0`; the live X3 hard gate passes.
 - Subscription-backed healthy canaries pass baseline verification for Sol,
   Terra, and Luna.
+- Sol fixed-effort evidence: `medium=29/30`, `high=30/30`, `xhigh=30/30` on
+  decision quality across all groups with two repeats.
+- Criteria-based planning-effort advisor added with visible `Y/8` evidence and
+  model caps.
 
 Blocked/pending:
 
@@ -74,9 +79,9 @@ expose `ultra` in the observed Codex `0.144.1` catalog.
 
 1. Keep the published `v0.3.x` compatibility scope distinct from model
    promotion.
-2. With explicit quota approval, run repeated same-model fixed/advisor pairs.
-3. Cover both healthy and degraded BMAD profiles.
-4. Stop on any safety-critical under-escalation.
+2. Add a plain-Sol control before claiming causal BMAD/BMADX quality uplift.
+3. Add executable implementation tasks before claiming code-quality uplift.
+4. Validate planning `max/ultra` only on planning-heavy scenarios.
 5. Investigate Sol/Luna total-token stability before recommending efficiency
    gains for light work.
 6. Use same-model claim verification before writing any performance claim.
@@ -86,6 +91,7 @@ expose `ultra` in the observed Codex `0.144.1` catalog.
 ```bash
 python3 scripts/test_install_bmadx.py
 python3 scripts/test_install_and_verify_bmadx.py
+python3 skill/bmadx/scripts/test_advise_planning_effort.py
 python3 skill/bmadx/scripts/test_check_codex_compat.py
 python3 skill/bmadx/scripts/test_sync_bmadx.py
 python3 benchmark/scripts/test_handoff_packet_schema.py
@@ -98,9 +104,9 @@ python3 skill/bmadx/scripts/sync_bmadx.py check --gear X3 --compact
 
 ## Handoff Capsule
 
-- Current state: `v0.3.1` CI patch ready; first healthy canaries remain valid.
-- Done: audit, model policy, checker, runner/verifier hardening, docs, CI, BMAD sync, Sol/Terra/Luna canaries.
-- Next: repeated healthy/degraded same-model evidence matrix.
+- Current state: `v0.3.2-dev` planning advisor and Sol reasoning evidence implemented locally.
+- Done: audit, model policy, checker, runner/verifier hardening, docs, CI, BMAD sync, canaries, Sol medium/high/xhigh comparison.
+- Next: plain-Sol control, implementation-quality tasks, and planning-specific max/ultra evidence.
 - Active approvals: none persisted; live-thread approval is not reusable.
 - Hard stops: do not weaken `X3/X4`, mutate global model config, or publish performance/default-model claims.
 - Rollback: revert the release commit and restore global skill backups from `/private/tmp`.

artifacts/gpt56-pro-ab-review-prompt.md

@@ -0,0 +1,86 @@
+# GPT-5.6 Sol Pro review prompt
+
+You are an independent experimental-design and AI-agent evaluation reviewer.
+Review the attached BMADX benchmark package critically. Do not assume that
+BMADX works, and do not optimize for agreeing with the authors.
+
+## Research question
+
+Does adding BMADX with a healthy BMAD dependency improve GPT-5.6 Sol workflow
+decision quality versus plain GPT-5.6 Sol at fixed `medium`, `high`, and
+`xhigh` reasoning effort?
+
+The experiment contains 180 model calls:
+
+- 15 matched scenarios,
+- 2 repeats,
+- 3 fixed effort levels,
+- plain Sol and BMADX+healthy-BMAD arms,
+- deterministic shuffled interleaving,
+- a shared framework-neutral JSON response contract,
+- no X1-X4 labels or expected route exposed to either arm.
+
+The headline result is a BMADX gain of 13.33-16.67 percentage points in full
+primary-pass rate at each effort, with higher token and latency cost. The
+current recommendation is to keep `high` as the normal consequential-planning
+baseline because BMADX xhigh improved only one additional case out of 30 over
+BMADX high.
+
+## Your job
+
+1. Verify the reported counts, percentages, token deltas, and latency deltas
+   directly from the JSON summary. Report any mismatch.
+2. Audit whether the two arms are genuinely comparable. Pay special attention
+   to prompt differences, isolated `CODEX_HOME` setup, skill availability,
+   healthy-BMAD fixture behavior, ordering, repeat structure, and possible
+   context or temporal contamination.
+3. Audit the neutral scorer. Challenge the process labels, scenario-specific
+   risk ground truth, applicability rules for handoff/goal/loop, strict JSON
+   requirement, variable per-case score maximum, and the definition of safety
+   under-escalation.
+4. Assess treatment fidelity. Codex logs the explicit `$bmadx` treatment and
+   isolated BMADX home but does not expose hidden skill injection as a shell
+   read. Decide how much this weakens causal attribution and propose a robust
+   activation marker or manipulation check.
+5. Check whether the historical precomputed-gate leak is fully understood and
+   whether removing `requested_gear` from future prompts is sufficient.
+6. Evaluate the statistical strength of two repeats over 15 authored
+   scenarios. Use matched reasoning where appropriate. State what can and
+   cannot be generalized; do not manufacture independence between repeated
+   effort cells or scenarios.
+7. Decide whether the evidence supports each of these claims separately:
+   - BMADX+healthy-BMAD improved workflow decisions on this fixed suite.
+   - BMADX itself caused the improvement.
+   - BMAD itself caused the improvement.
+   - high should remain the default over xhigh.
+   - BMADX improves implementation quality.
+8. Design the smallest decisive follow-up:
+   - a third arm that separates BMADX from healthy BMAD,
+   - an activation/manipulation check,
+   - stronger repetitions and stopping rules,
+   - a blinded implementation benchmark using isolated repositories and
+     executable hidden tests.
+9. Identify any code defects in the runner, resume/checkpoint logic, artifact
+   naming, scorer, or tests that could invalidate or distort results.
+
+## Required response structure
+
+Return these sections:
+
+1. **Verdict** - one paragraph with confidence level.
+2. **Verified numbers** - recomputed table for all six cells.
+3. **Findings** - ordered Critical, High, Medium, Low, with exact file/function
+   references from the package.
+4. **Causal claims matrix** - supported, directional only, unsupported, or
+   falsified, with one-sentence justification per claim.
+5. **High vs xhigh decision** - explicit recommendation and why.
+6. **Next benchmark protocol** - concrete arms, prompts, repetitions,
+   randomization, scoring, stop conditions, manipulation checks, and executable
+   implementation tasks.
+7. **Publication wording** - a short paragraph that is accurate enough for a
+   public README or release note.
+
+Be adversarial but practical. Prefer findings that change the decision or next
+experiment. Distinguish observed evidence from inference. If raw logs are
+required to validate a claim not represented in the summary, name the smallest
+specific stratified sample needed rather than requesting all 360 files.