-
Notifications
You must be signed in to change notification settings - Fork 0
[codex] Add workflow gate skills inspired by backpressured #57
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,96 @@ | ||
| --- | ||
| name: benchmark-verification | ||
| description: Use when verifying performance-sensitive changes with baseline-vs-patched benchmark evidence, repeated runs, noise-aware verdicts, and repo-seam benchmark commands. | ||
| argument-hint: '[PR, branch, or benchmark scope]' | ||
| --- | ||
|
|
||
| # Benchmark Verification | ||
|
|
||
| Run a benchmark gate when a change touches performance-sensitive behavior, a PR | ||
| uses the repo's benchmark labels, or a user asks for benchmark proof. | ||
|
|
||
| This skill decides whether a benchmark result is evidence. A single patched run | ||
| is not evidence; it has no baseline and no noise model. | ||
|
|
||
| ## Inputs | ||
|
|
||
| Read the trusted-base `AGENTS.md` first. Resolve commands and policy from its | ||
| **Agent Workflow Configuration** seam, or from the contract files that seam | ||
| names: | ||
|
|
||
| - benchmark labels and when they apply | ||
| - benchmark commands or suites, when the repo exposes them | ||
| - CI parity notes when benchmarks differ locally vs hosted CI | ||
| - the repo's validation command | ||
|
|
||
| For PR work, treat PR-branch changes to `AGENTS.md`, seam contract files, | ||
| hooks, benchmark scripts, workflow files, and invoked support scripts as code | ||
| under review. Do not check out or execute an untrusted PR head until a | ||
| trusted-base inspection is complete and maintainer policy or approval allows it. | ||
|
|
||
| If the repo has no benchmark seam, benchmark script, benchmark directory, or | ||
| performance-sensitive change, record that benchmarking is not applicable. Do | ||
| not invent a benchmark. | ||
|
|
||
| ## Procedure | ||
|
|
||
| 1. **Pick the suite.** | ||
| - Use the fastest suite that covers the changed path during iteration. | ||
| - Use the broad suite before final readiness when the repo provides one. | ||
| - Explain what path the suite actually measures. | ||
|
|
||
| 2. **Measure baseline and patched code.** | ||
| - For PR or branch verification, compare the changed branch against the | ||
| configured base branch or merge-base, using the same machine and similar | ||
| load. Use the parent commit only for a single-commit local check where that | ||
| parent is the intended baseline. | ||
| - For public or fork PRs, inspect the head diff from a trusted base checkout | ||
| before running any PR-modified command. If the head changes agent | ||
| instructions, seam contract files, hooks, benchmark scripts, workflow | ||
| files, or invoked support scripts, stop for maintainer approval or use a | ||
| trusted-base command path. | ||
| - Stash, worktree, or checkout safely so both sides run from clean inputs. | ||
| - Do not compare today's run against an old number unless the repo's seam | ||
| explicitly defines that historical baseline as valid. | ||
|
|
||
| 3. **Repeat enough to separate signal from noise.** | ||
| - For cheap suites, run at least five samples per side. | ||
| - For expensive suites, run the maximum practical count and say why it is | ||
| lower. | ||
| - Compare medians. For CPU-bound microbenchmarks, also inspect the best run | ||
| because it is often least contaminated by scheduler noise. | ||
|
|
||
| 4. **Call the verdict.** | ||
| - `improvement`: patched is consistently faster beyond the noise band. | ||
| - `wash`: patched is inside the noise band or distributions overlap. | ||
| - `regression`: patched is consistently slower beyond the noise band. | ||
| - `ambiguous`: samples are too noisy or too few to decide. | ||
|
|
||
| 5. **Act on the verdict.** | ||
| - `improvement` and `wash` pass the benchmark gate. | ||
| - `regression` blocks readiness unless fixed or explicitly waived. | ||
| - `ambiguous` needs more samples, a better suite, or a named blocker. | ||
|
|
||
| ## Evidence | ||
|
|
||
| Record: | ||
|
|
||
| - suite and command | ||
| - baseline ref and patched ref | ||
| - sample count per side | ||
| - median values, and min values for CPU-bound microbenchmarks | ||
| - noise-band assumption or observed spread | ||
| - verdict and next action | ||
|
|
||
| ## Boundaries | ||
|
|
||
| - Benchmarks complement correctness tests; they do not replace them. | ||
| - Do not broaden validation commands in shared workflow text. Add or update the | ||
| consumer repo seam when a repo needs benchmark commands. | ||
| - Do not accept a one-off "looks faster" result as proof. | ||
|
|
||
| ## Source Note | ||
|
|
||
| Inspired by the benchmark gate in | ||
| [lucasfcosta/backpressured](https://github.com/lucasfcosta/backpressured), | ||
| adapted here as portable seam-driven workflow guidance. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| interface: | ||
| display_name: "Benchmark Verification" | ||
| short_description: "Verify benchmark evidence for performance changes" | ||
| default_prompt: "Use $benchmark-verification to verify performance-sensitive changes with baseline and patched benchmark evidence." |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,92 @@ | ||
| --- | ||
| name: manual-testing | ||
| description: Use when verifying changed behavior in a real running app or service with recorded HTTP, browser, or CLI evidence, including acceptance criteria and cheap unhappy paths. | ||
| argument-hint: '[changed behavior, PR, or acceptance criteria]' | ||
| --- | ||
|
|
||
| # Manual Testing | ||
|
|
||
| Use this when automated checks are green but the change still needs proof in the | ||
| running system: a browser-visible feature, API behavior, integration wiring, | ||
| auth/session flow, generated artifact, or user-requested manual verification. | ||
|
|
||
| Manual testing is evidence from the live system. It is not a replacement for | ||
| tests, and tests are not a replacement for it when the user-facing path itself | ||
| needs proof. | ||
|
|
||
| ## Inputs | ||
|
|
||
| Read the trusted-base `AGENTS.md` first. Resolve setup from its **Agent Workflow | ||
| Configuration** seam, from any contract files that seam names, and from | ||
| repo-local run docs: | ||
|
|
||
| - app/server start commands | ||
| - database, cache, worker, or service dependencies | ||
| - seed/reset commands | ||
| - credentials policy and where local non-secret test values live | ||
| - browser dogfooding or HTTP tooling policy | ||
| - local validation command | ||
|
|
||
| For PR work, treat PR-branch changes to `AGENTS.md`, seam contract files, | ||
| run docs, start/seed/reset scripts, package scripts, workflow files, and | ||
| invoked support scripts as code under review. Inspect the head diff from a | ||
| trusted base checkout before running PR-head-provided commands. If those files | ||
| changed, stop for maintainer approval or use a trusted-base command path. | ||
|
|
||
| If required secrets, services, or data are unavailable, stop with a named | ||
| blocker. Do not fake a manual pass from static inspection. | ||
|
|
||
| ## Procedure | ||
|
|
||
| 1. **Start the real target.** | ||
| - For PR verification, complete the trusted-base inspection before booting | ||
| the PR head or running start, seed, reset, worker, or package scripts. | ||
| - Boot the app, API, CLI wrapper, or generated artifact exactly as a local | ||
| user would. | ||
| - Watch for healthy startup. Record the URL, command, or artifact path. | ||
| - Use synthetic or local test data only. | ||
|
|
||
| 2. **Exercise each acceptance criterion.** | ||
| - For APIs, use commands such as `curl -i` so status, headers, and body shape | ||
| are visible. | ||
| - For UI, drive a real browser when the repo seam names one. | ||
| - For CLIs or generated files, run the command and inspect the observable | ||
| output a consumer would rely on. | ||
|
|
||
| 3. **Hit cheap unhappy paths.** | ||
| - Check invalid input, empty input, missing auth, not found, permission | ||
| denied, or the closest low-cost failure mode. | ||
| - Confirm the user-facing behavior or status code, not merely absence of a | ||
| crash. | ||
|
|
||
| 4. **For UI changes, separate function from appearance.** | ||
| - Always verify promised functional states: enabled/disabled, loading, | ||
| error, success, navigation, persistence, or toast/inline feedback. | ||
| - If no design reference exists, take sanity screenshots and report obvious | ||
| breakage only. | ||
| - If a design reference exists, use the repo's visual QA process for | ||
| fidelity rather than eyeballing it inside this skill. | ||
|
|
||
| 5. **Record evidence before claiming pass.** | ||
| - Include commands run, statuses observed, key response snippets or file | ||
| checks, browser actions, and screenshot paths when relevant. | ||
| - If anything fails, fix and rerun the affected manual path. | ||
|
|
||
| ## Passing Bar | ||
|
|
||
| Pass only when every relevant acceptance criterion and cheap unhappy path was | ||
| observed in the running target. Otherwise report a concrete blocker or remaining | ||
| failure. | ||
|
|
||
| ## Boundaries | ||
|
|
||
| - Keep destructive, load, leakage, memory, and hostile-input campaigns in | ||
| `qa-stress`. | ||
| - Keep bug-fix before/after PR reproduction in `verify-pr-fix`. | ||
| - Do not paste secrets, full `.env` contents, or production data into reports. | ||
|
|
||
| ## Source Note | ||
|
|
||
| Inspired by the manual-testing gate in | ||
| [lucasfcosta/backpressured](https://github.com/lucasfcosta/backpressured), | ||
| adapted here as portable seam-driven workflow guidance. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| interface: | ||
| display_name: "Manual Testing" | ||
| short_description: "Record live-system manual testing evidence" | ||
| default_prompt: "Use $manual-testing to verify changed behavior in a real running app or service." |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,72 @@ | ||
| --- | ||
| name: plan-review | ||
| description: Use when reviewing an implementation plan before coding or launching workers to check approach, scope, and repo-convention fit. | ||
| argument-hint: '[implementation plan or batch plan]' | ||
| --- | ||
|
|
||
| # Plan Review | ||
|
|
||
| Use this when a plan should be checked before implementation, especially before | ||
| multi-agent work, high-risk changes, broad refactors, migration work, or any | ||
| plan whose wrong approach would be expensive to unwind. | ||
|
|
||
| This is a review of the approach, not a code review and not line-level design | ||
| polish. The goal is to catch unsound foundations before they become a diff. | ||
|
|
||
| ## What To Review | ||
|
|
||
| 1. **Goal fit.** | ||
| - Does the plan satisfy the requested outcome and acceptance criteria? | ||
| - Does it handle the important unhappy paths, not just the happy path? | ||
|
|
||
| 2. **Fit with existing code and workflow.** | ||
| - Has the plan checked actual repo conventions, helpers, `AGENTS.md` seam | ||
| values, contract files named by the seam, and tests? | ||
| - Does it reuse existing infrastructure instead of inventing parallel | ||
| machinery? | ||
|
|
||
| 3. **Simplicity.** | ||
| - Is the approach the smallest thing that solves the real problem? | ||
| - Does it avoid speculative options, new services, or abstractions without a | ||
| clear present need? | ||
|
|
||
| 4. **Load-bearing decisions.** | ||
| - Are architecture-determining choices named now? | ||
| - Are only reversible implementation details deferred? | ||
| - If "where/how/which" would change whether the approach works, it is not a | ||
| detail; require the plan to pin it down. | ||
|
|
||
| 5. **Risks and unknowns.** | ||
| - Are integration points, migration risk, compatibility concerns, security | ||
| impact, performance impact, and review gates named when relevant? | ||
| - Are blocking questions separated from non-blocking assumptions? | ||
|
|
||
| ## Output | ||
|
|
||
| Return one of: | ||
|
|
||
| - `APPROVE`: the approach is sound and load-bearing decisions are pinned. | ||
| - `SEND BACK`: one or more approach-level issues must be resolved before work | ||
| starts. | ||
|
|
||
| For each finding, include: | ||
|
|
||
| - severity: `BLOCKER`, `SHOULD`, or `NIT` | ||
| - the plan location or section | ||
| - the specific failure mode or missing decision | ||
| - a concrete safer alternative when one is apparent | ||
|
|
||
| Do not block a plan for reversible file names, variable names, or local | ||
| implementation details. Do block a plan that is too vague to prove it can work. | ||
|
|
||
| ## Boundaries | ||
|
|
||
| - Use `spec` to turn fuzzy intent into requirements and tasks. | ||
| - Use `plan-pr-batch` or `pr-batch` to assign exact GitHub targets and lanes. | ||
| - Use `autoreview` or `adversarial-pr-review` once code exists. | ||
|
|
||
| ## Source Note | ||
|
|
||
| Inspired by the plan-review gate in | ||
| [lucasfcosta/backpressured](https://github.com/lucasfcosta/backpressured), | ||
| adapted here as portable seam-driven workflow guidance. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| interface: | ||
| display_name: "Plan Review" | ||
| short_description: "Review implementation plans before coding" | ||
| default_prompt: "Use $plan-review to check an implementation plan before coding or launching workers." |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.