feat: agentic-end-to-end-testing skill + spec-derived scenario cards#1931
Draft
obra wants to merge 23 commits into
Draft
feat: agentic-end-to-end-testing skill + spec-derived scenario cards#1931obra wants to merge 23 commits into
obra wants to merge 23 commits into
Conversation
<date> inside the innerHTML string literal parses as an empty HTML element in the browser rather than literal text if left unsubstituted. Use the house bracket convention instead.
…disclosure doc The predicate-keyed at-skill-start trigger is replaced by an unconditional offer to the human after the final whole-branch review, before finishing-a-development-branch. The procedure (spec discovery, author/checker/runner flow, fix-wave rules) moves to spec-derived-e2e.md; SKILL.md keeps the offer, a flowchart node, and the Integration reference. Micro-tested 6/6: four controllers at the post-review point all offer before finishing (including under wrap-up-efficiently pressure with the human away); two accepted-offer controllers follow the disclosure doc with the checker run by the controller, one fix subagent, and a fix-diff review gate. Deviation recorded in the design spec.
This was referenced Jul 5, 2026
This was referenced Jul 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Who is submitting this PR? (required)
What problem are you trying to solve?
Three related failures observed across real sessions:
reports, assertions weakened until they pass, stale builds tested as if
they were the branch. The patterns that prevent this (scenario cards,
disposable verification runners, proof movies/documents, honest-failure
reporting) existed only in a personal dotfiles skill and scattered
session history, not in core.
they built rather than what was requested. Nothing ties e2e assertions
back to the spec that predates the implementation.
found a real product bug while verifying it also fixed it mid-run,
citing ambient norms — same prompt, second worker declined. Ambient
norms do not decide finder-vs-fixer consistently; the boundary must be
explicit.
What does this PR change?
Adds the
agentic-end-to-end-testingskill (decision core + runner prompttemplate + driving recipes for browser/CLI-TUI/computer-use + proof-movie
recipes + a
check-cards-against-specchecker with a 23-assertion testharness). Extends brainstorming with a conditional "E2E scenario cards"
table for specs that add or change user-visible behavior, and gives
subagent-driven-development a pre-finish offer of spec-derived e2e
verification (procedure in a progressive-disclosure doc). Design specs and
plans are committed under
docs/superpowers/.Is this change appropriate for the core library?
Yes: agent-agnostic, project-agnostic verification methodology. No
third-party service integration; the only script is bash + POSIX tools.
The eval scenarios developed alongside it live in the separate
superpowers-evals repo (branch
agentic-e2e-scenarios), not in core.What alternatives did you consider?
rejected by maintainer decision: repeatability ships in-skill (checker
script + RED/GREEN discipline); eval scenarios live in superpowers-evals.
the falsification table locks at spec time, card expansion happens
post-implementation under a verbatim-lift rule the experiment showed
agents follow (4/4 cells verbatim).
mandate per agent; the agents that find issues are never responsible for
issues being solved.
Does this PR contain multiple unrelated changes?
It contains one skill plus its integrations. The brainstorming table, the
checker, and the SDD offer all exist to feed or consume the new skill's
scenario cards — none stands alone without it.
Existing PRs
spec-writing skills — adjacent but different layer: this PR is about
live agentic execution with evidence discipline and spec-to-card
traceability, not spec authoring); feat: add harness-engineering skill #531 (closed harness-validation
skill, different scope)
Environment tested
New harness support (required if this PR adds a new harness)
N/A — no new harness.
Evaluation
agentic-end-to-end-testingskill from theagentic-scenario / verification-runner / proof-movie patterns developed
in prior sessions; later extended with "what about writing scenarios
based on a superpowers brainstorming spec?"
(2 scenarios × bare/skill arms) against real fixture apps; a 4-worker
card-authoring drift experiment; 9/9 positive + clean negative-gate
micro-tests for the brainstorming predicate (the first predicate wording
failed the negative gate 0/4 and was re-keyed on that evidence); 6/6
micro-tests for the SDD offer; 23-assertion checker harness plus
adversarial probe fixtures.
looser workflows; skill arms read the new SKILL.md, loaded supporting
files on demand, produced per-assertion PASS/FAIL tables with concrete
evidence, reported failures honestly without touching product code, and
one arm dispatched a disposable runner with the template filled
correctly (honesty clause verbatim, evidence re-read gate observed in
the event stream).
Rigor
superpowers:writing-skillsandcompleted adversarial pressure testing (paste results below)
rationalizations, "human partner" language) without extensive evals
showing the change is an improvement
Adversarial results (summary): RED baselines recorded before every GREEN
edit; reviewer probes found two real checker false-passes (whole-file
falsification match; level-1 heading not terminating the Expected section)
— both fixed with regression tests; task reviewers forensically verified
RED/GREEN claims against session transcripts and caught one
misrepresented baseline, which was corrected; brainstorming/SDD wording
was micro-tested against control arms and pressure scenarios (including a
wrap-up-efficiently-while-human-is-away arm) before landing.
Human review
Draft PR: opened for @obra's full-diff review; box intentionally unchecked
until that review happens.