Skip to content

Add blinded BMADX value study and read-only dependency gate - #8

Merged
pdurlej merged 21 commits into
mainfrom
codex/causal-canary-v1
Jul 16, 2026
Merged

Add blinded BMADX value study and read-only dependency gate#8
pdurlej merged 21 commits into
mainfrom
codex/causal-canary-v1

Conversation

@pdurlej

@pdurlej pdurlej commented Jul 12, 2026

Copy link
Copy Markdown
Owner

Summary

  • replace the stateful BMAD startup check with a local read-only capability preflight
  • keep X3/X4 blocked only for a missing or structurally unusable BMAD install
  • add the planning-effort advisor and reproducible Sol comparison harnesses
  • complete the blinded three-arm BMADX decision-value generation and synthetic panel
  • publish the fail-closed panel result, machine-readable gate, and claim boundary
  • remove branch-only raw logs and obsolete stopped ZIPs from the public PR

Dependency gate

  • X3 compact smoke: classification_allowed=true, execution_allowed=true, bmad_status=ok, remediation=[]
  • BMAD release/reference drift and an unwritable state cache are warnings, not startup blockers
  • BMAD synchronization remains explicit operator-requested maintenance
  • the repository version is installed and verified at ~/.codex/skills/bmadx

Decision-value study result

  • 162/162 GPT-5.6 Sol generation calls completed across three blinded arms
  • 325/325 scientific panel judgments completed through five Ollama Cloud families in Pi
  • 329 provider attempts; 3 scientific calls required at least one retry
  • Gemma 4, Mistral Large 3, and Qwen 3.5 passed the frozen health gate
  • GLM 5.2 (0.590909) and Nemotron 3 Ultra (0.651515) missed the preregistered 0.70 preference-order Jaccard threshold
  • final gate: blocked_unhealthy_panel
  • arm mapping was not revealed and no BMADX value-added claim is allowed

This result is inconclusive about whether BMADX is better or worse than plain Sol. It also does not establish productivity, learning, or implementation outcomes for novice builders. The failure is published because an unhealthy evaluation panel must block an attractive claim.

Verification

  • 185 deterministic local tests pass
  • changed benchmark scripts compile under local Python
  • frozen v1.13 runner, analyzer, canary, arm-map, panel, and amendment hashes match the completed-run provenance
  • git diff --check passes
  • GitHub CI targets Python 3.10, 3.12, and 3.14

Evidence

@pdurlej pdurlej changed the title Add blinded cross-model BMADX value study Add blinded BMADX value study and read-only dependency gate Jul 16, 2026
codex added 2 commits July 17, 2026 00:54
# Conflicts:
#	CHANGELOG.md
#	docs/bmadx-v0.3-plan.md
#	scripts/test_install_and_verify_bmadx.py
@pdurlej
pdurlej marked this pull request as ready for review July 16, 2026 22:59
@pdurlej
pdurlej merged commit 62b2997 into main Jul 16, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant