Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .github/workflows/validate.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,11 @@ jobs:
m = re.match(r'^---\n(.*?)\n---\n', text, re.S)
if not m:
print(f"FAIL {skill}: no YAML frontmatter"); fail = True; continue
fm = m.group(1)
if fm != fm.strip() or re.search(r'(?m)^[ \t]*$', fm):
print(f"FAIL {skill}: blank line inside frontmatter"); fail = True
for field in ('name', 'description'):
if not re.search(rf'^{field}:', m.group(1), re.M):
if not re.search(rf'^{field}:', fm, re.M):
print(f"FAIL {skill}: frontmatter missing {field}"); fail = True
sys.exit(1 if fail else 0)
EOF
Expand Down
22 changes: 22 additions & 0 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,3 +147,25 @@ Next up:
- `weekly-review` / `decision-log` (✨) — speculative `journal` siblings once `worklog` has proven out.
- `triage-issues` (♻️) — the other issue-intake candidate, still on the list.
- **Migration** — onboard a real repo (pepper or obsidian-gemini) to prove the whole suite end-to-end.

## Settled decisions (don't re-litigate)

Things that came up, got decided with a reason, and shouldn't be reopened without new information.

- **`code-review` and `typescript-patterns.md` / `review-checklist.md` stay deleted** (dropped in
#14). Claude Code ships `/code-review`, so the skill was redundant *and* the name collided. The
two reference docs went with it, and the recurring suggestion is to restore them as seed material
for `bootstrap`'s `coding.md` scaffold. **Don't.** `coding.md` is what `audit-architecture` and
any reviewer read as *"the rules this repo wrote down"*; seeding it with ~400 lines of generic
TypeScript craft would make the audits enforce generic practice — the exact boundary #15 drew
when it handed dangerous-code-patterns to `/security-review` and kept only repo-specific
invariants. Generic practice belongs to the built-in reviewer; `guidelines/*.md` is for what a
general reviewer *cannot* know. Both files remain in git history if that judgement ever changes.
- **No documentation site.** `homepage`/`repository` point at the repo and the per-plugin READMEs
(#19). 1 of 13 official plugins sets those fields at all, the repo already ships its docs as
Markdown, and a generated site would add a sync surface with no checker — the failure mode #10
and #12 exist to prevent.
- **`auto-dev` may exceed the 5,000-word skill guideline.** It sits at ~5,200 and the remainder is
the tick state machine plus the safety invariants and hard prohibitions. Those must be in front
of the model on an unattended run that writes code; moving them to satisfy a word count would
optimise the metric against its purpose.
83 changes: 10 additions & 73 deletions plugins/audits/skills/audit-architecture/SKILL.md

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Scheduling — audit-architecture

This skill is **not** part of the `daily-update` meta-skill, because `daily-update` bundles its work into one PR and this skill explicitly opens many. Schedule it as its own slot (e.g. nightly at 2am local time) via the `schedule` skill. The schedule should invoke this skill directly; there is no autonomous-prompt variant — pass a literal `/audit-architecture` or equivalent.

If the user is running short on `/schedule` slots and wants to combine with `daily-update`, the right consolidation is to have this skill run *first*, produce its PRs/issues, and then let `daily-update` run its own one-PR sweep on top — but they remain logically separate runs from the maintainer's point of view.

**Model tier:** DRY/abstraction judgment, invariant drift, and PR-vs-issue routing are judgment-heavy — schedule this on the **`capable`** tier (a smaller model mis-routes and over-files). See [`../../../references/model-tiers.md`](../../../references/model-tiers.md).
2 changes: 1 addition & 1 deletion plugins/audits/skills/audit-security/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ dependency you never touched, a secret committed months ago, a permissive defaul
| Category | Typical severity | Detection | Default routing |
| --- | --- | --- | --- |
| **Vulnerable dependencies (CVEs)** | per advisory | Run the ecosystem auditor if present (see language blocks). Each advisory = one finding: package, installed version, fixed version, CVE/GHSA id, severity. | **PR** if a non-breaking patch/minor bump fixes it (bump the pin + lockfile); **issue** if the fix needs a major/breaking bump or there's no fix yet. |
| **Committed secrets / credentials** | **Critical** | Run a secret scanner if present (`gitleaks detect`, `trufflehog`); otherwise grep heuristics over the tree **and git history** for high-entropy strings and known token shapes (`AKIA…`, `ghp_…`, `sk-…`, `-----BEGIN … PRIVATE KEY-----`, `xox[baprs]-…`, JWT triples, `password|secret|api_key\s*[:=]\s*["'][^"']+`). | **Issue (redacted) + alert the user directly.** Never a "fix" PR — see [Handling secrets safely](#handling-secrets-safely). |
| **Committed secrets / credentials** | **Critical** | Run a secret scanner if present (`gitleaks detect`, `trufflehog`); otherwise grep heuristics over the tree **and git history** for high-entropy strings and known token shapes (`AKIA…`, `ghp_…`, `sk-…`, `-----BEGIN … PRIVATE KEY-----`, `xox[baprs]-…`, JWT triples, `password\|secret\|api_key\s*[:=]\s*["'][^"']+`). | **Issue (redacted) + alert the user directly.** Never a "fix" PR — see [Handling secrets safely](#handling-secrets-safely). |
| **Hardcoded config / permissive defaults** | Medium | Read for in-source credentials that should be env/secret-managed, `DEBUG=True` in shippable config, CORS `*` with credentials, overly broad file modes, auth disabled in non-test code. Cross-check against `config.guidelines`. | **Issue** — usually a judgment/ownership call. |

You're not limited to this table **within these three categories** — a logged secret, a credential
Expand Down
36 changes: 5 additions & 31 deletions plugins/audits/skills/audit-tests/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,34 +82,9 @@ Route each the same way: mechanical and behavior-preserving → PR, judgment-hea

## Language-specific mechanics

The smells above are language-agnostic. The exact detection commands and the framework names depend on `config.language`. Apply the equivalent for whichever language the config declares; read `config.guidelines.testing` for the conventions that are specific to this repo's runner.

### When `config.language` is `python`

Framework: pytest + `unittest.mock`. Treat `config.paths.tests` as the grep root.

- **Mocked collaborators / sessions:** `grep -rnE "(MagicMock|AsyncMock|Mock)\(" <tests>` and read for `patch("...")` / `patch.object(...)`. Cross-reference each against the "what may not be mocked" rule in `config.guidelines.testing` (e.g. mocking the DB session instead of the real test-DB fixture).
- **Weak assertions / broad raises:** `grep -rn "pytest.raises(Exception)\|pytest.raises(BaseException)" <tests>`; scan for assertion-free test bodies.
- **Flaky:** `grep -rn "time\.sleep\|asyncio\.sleep" <tests>`; scan for unfrozen `datetime.now()`/`date.today()`/`random` in result-asserting tests (the fix is `freezegun`/`freeze_time` or injecting the value).
- **Skip/xfail rot:** `grep -rn "@pytest.mark.skip\|@pytest.mark.xfail\|pytest.skip(" <tests>` — flag any without `reason=`, and any `xfail` that now passes (`--runxfail` reports XPASS).
- **Redundant framework boilerplate:** if `config.guidelines.testing` says the repo runs pytest-asyncio in `asyncio_mode = "auto"`, then `@pytest.mark.asyncio` is a no-op — `grep -rn "@pytest.mark.asyncio" <tests>` and remove. Don't assume auto-mode; confirm it in the guidelines first.
- **Parametrize:** the duplication target is `@pytest.mark.parametrize`.
- **Slow tests:** `<config.commands.test> --durations=25 -q`.
- **Coverage:** `config.commands.coverage` typically emits `coverage.json` (branch coverage on); read it for covered-lines-but-uncovered-branches.

### When `config.language` is `typescript`

Framework: the repo's test runner (Jest or Vitest — check `config.commands.test`). Apply the equivalent of each python check.

- **Mocked collaborators:** `grep -rnE "(jest|vi)\.(mock|fn|spyOn)\(" <tests>` and read each. Cross-reference against the "what may not be mocked" rule in `config.guidelines.testing`. Spying on / mocking a module you own that could run for real is the same smell as the python session-mock case.
- **Weak assertions / broad throws:** flag `expect(...).toThrow()` with no error matcher, and test bodies with **no `expect(...)`** at all. Tighten `toThrow()` to a specific error type/message.
- **Flaky:** `grep -rnE "setTimeout|new Promise\(.*setTimeout" <tests>` for real-time waits; unfrozen `Date.now()`/`Math.random()` in result-asserting tests. The fix is **fake timers** (`jest.useFakeTimers()` / `vi.useFakeTimers()` and `setSystemTime`) or injecting the value — the direct analog of freezing the clock.
- **Skip/only rot:** `grep -rnE "\.(skip|only|todo)\(|xit\(|xdescribe\(" <tests>`. A stray `.only` is a real smell — it silently disables every other test in the file; a `.skip`/`xit` without a comment reason is rot.
- **Redundant framework boilerplate:** per `config.guidelines.testing` — leftover `.only`, redundant `async` wrappers, etc.
- **Parametrize:** the duplication target is `it.each` / `test.each` / `describe.each`.
- **Golden-snapshot noise:** oversized or volatile `toMatchSnapshot()` / inline snapshots that re-bless on every change.
- **Slow tests:** the runner's slow-test reporting (Jest `--verbose` timings; Vitest's slow-test reporter).
- **Coverage:** `config.commands.coverage` typically emits `coverage-summary.json` / lcov; read it for uncovered branches in already-tested files.
The greps and tool flags for each language live in
[`references/language-mechanics.md`](references/language-mechanics.md) — read the block matching
`config.language`. What counts as a finding stays in the table above.

## What it does NOT look for

Expand Down Expand Up @@ -348,9 +323,8 @@ A healthy suite produces **0 findings on most runs** — that's the steady state

## When integrated with scheduling

Schedule this as its own slot (a few times a day is fine given the silent-on-clean + low-cap design), invoking it directly (`/audit-tests` or equivalent) — there is no autonomous-prompt variant. It is intentionally **separate** from both `daily-update` (which bundles its work into one PR; this skill opens discrete ones) and `audit-architecture` (which owns the source side). Running both audits is fine; they don't overlap and each dedups against its own label/branch prefix.

**Model tier:** "is this mock decorative? is this assertion actually weak?" is judgment — schedule on **`capable`**, or on a **`mid`** rung if the repo defines one (this runs several times a day, so the cost trade is real). See [`../../references/model-tiers.md`](../../references/model-tiers.md).
Cadence, `daily-update` relationship, and model tier are in
[`references/scheduling.md`](references/scheduling.md).

## Related skills

Expand Down
34 changes: 34 additions & 0 deletions plugins/audits/skills/audit-tests/references/language-mechanics.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# Language-specific mechanics — audit-tests

The concrete greps and tool flags behind the category table. **Run only the block matching
`config.language`.** The table in SKILL.md decides what counts as a finding; this file is how to
find it in this language.

The smells above are language-agnostic. The exact detection commands and the framework names depend on `config.language`. Apply the equivalent for whichever language the config declares; read `config.guidelines.testing` for the conventions that are specific to this repo's runner.

## When `config.language` is `python`

Framework: pytest + `unittest.mock`. Treat `config.paths.tests` as the grep root.

- **Mocked collaborators / sessions:** `grep -rnE "(MagicMock|AsyncMock|Mock)\(" <tests>` and read for `patch("...")` / `patch.object(...)`. Cross-reference each against the "what may not be mocked" rule in `config.guidelines.testing` (e.g. mocking the DB session instead of the real test-DB fixture).
- **Weak assertions / broad raises:** `grep -rn "pytest.raises(Exception)\|pytest.raises(BaseException)" <tests>`; scan for assertion-free test bodies.
- **Flaky:** `grep -rn "time\.sleep\|asyncio\.sleep" <tests>`; scan for unfrozen `datetime.now()`/`date.today()`/`random` in result-asserting tests (the fix is `freezegun`/`freeze_time` or injecting the value).
- **Skip/xfail rot:** `grep -rn "@pytest.mark.skip\|@pytest.mark.xfail\|pytest.skip(" <tests>` — flag any without `reason=`, and any `xfail` that now passes (`--runxfail` reports XPASS).
- **Redundant framework boilerplate:** if `config.guidelines.testing` says the repo runs pytest-asyncio in `asyncio_mode = "auto"`, then `@pytest.mark.asyncio` is a no-op — `grep -rn "@pytest.mark.asyncio" <tests>` and remove. Don't assume auto-mode; confirm it in the guidelines first.
- **Parametrize:** the duplication target is `@pytest.mark.parametrize`.
- **Slow tests:** `<config.commands.test> --durations=25 -q`.
- **Coverage:** `config.commands.coverage` typically emits `coverage.json` (branch coverage on); read it for covered-lines-but-uncovered-branches.

## When `config.language` is `typescript`

Framework: the repo's test runner (Jest or Vitest — check `config.commands.test`). Apply the equivalent of each python check.

- **Mocked collaborators:** `grep -rnE "(jest|vi)\.(mock|fn|spyOn)\(" <tests>` and read each. Cross-reference against the "what may not be mocked" rule in `config.guidelines.testing`. Spying on / mocking a module you own that could run for real is the same smell as the python session-mock case.
- **Weak assertions / broad throws:** flag `expect(...).toThrow()` with no error matcher, and test bodies with **no `expect(...)`** at all. Tighten `toThrow()` to a specific error type/message.
- **Flaky:** `grep -rnE "setTimeout|new Promise\(.*setTimeout" <tests>` for real-time waits; unfrozen `Date.now()`/`Math.random()` in result-asserting tests. The fix is **fake timers** (`jest.useFakeTimers()` / `vi.useFakeTimers()` and `setSystemTime`) or injecting the value — the direct analog of freezing the clock.
- **Skip/only rot:** `grep -rnE "\.(skip|only|todo)\(|xit\(|xdescribe\(" <tests>`. A stray `.only` is a real smell — it silently disables every other test in the file; a `.skip`/`xit` without a comment reason is rot.
- **Redundant framework boilerplate:** per `config.guidelines.testing` — leftover `.only`, redundant `async` wrappers, etc.
- **Parametrize:** the duplication target is `it.each` / `test.each` / `describe.each`.
- **Golden-snapshot noise:** oversized or volatile `toMatchSnapshot()` / inline snapshots that re-bless on every change.
- **Slow tests:** the runner's slow-test reporting (Jest `--verbose` timings; Vitest's slow-test reporter).
- **Coverage:** `config.commands.coverage` typically emits `coverage-summary.json` / lcov; read it for uncovered branches in already-tested files.
5 changes: 5 additions & 0 deletions plugins/audits/skills/audit-tests/references/scheduling.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# Scheduling — audit-tests

Schedule this as its own slot (a few times a day is fine given the silent-on-clean + low-cap design), invoking it directly (`/audit-tests` or equivalent) — there is no autonomous-prompt variant. It is intentionally **separate** from both `daily-update` (which bundles its work into one PR; this skill opens discrete ones) and `audit-architecture` (which owns the source side). Running both audits is fine. Ownership is separate — source vs tests — and each dedups against its own label/branch prefix, but they are **not** independent: this skill skips any file touched by an open `arch-*` PR (SKILL.md, step 3 and "What not to do"), so an architecture PR in flight defers the test audit on that file rather than racing it.

**Model tier:** "is this mock decorative? is this assertion actually weak?" is judgment — schedule on **`capable`**, or on a **`mid`** rung if the repo defines one (this runs several times a day, so the cost trade is real). See [`../../../references/model-tiers.md`](../../../references/model-tiers.md).
28 changes: 8 additions & 20 deletions plugins/auto-dev/skills/auto-dev/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -255,26 +255,13 @@ file before posting. Every comment's first line is `config.autoDev.marker`.

## Exit report

Every tick ends by printing a structured report — it is the run's summary output (the scheduled task surfaces it; an interactive run shows it inline):

```text
auto-dev tick — <ISO timestamp>
step executed: <0-failed | 1-reconcile | 2-pr-advance | 3-build | 4-triage | 5-idle>
open auto PRs (<count>/<config.autoDev.maxPrsInFlight>): #<n> (<status>), … | none
actions:
- #123: asked 2 clarifying questions → needs-info
- #145: plan approved by reply → ready
- #151: proposed parking (design fork is the maintainer's call) → needs-info
- #152: maintainer replied "park it" → parked
- PR #210: fixed 2 CodeRabbit findings, replied to 4 threads, pushed <sha>
- PR #212: no external review after 60m — self-reviewed, fixed 1 finding (<sha>), posted fallback review
- #160: built approved plan → PR #211 (labeled auto:pr); verified via /verify (drove the new CLI flag, observed expected output) → marked ready
- #163: built approved plan → PR #212 (draft); behavioral verification not run in sandbox (needs a live DB) — flagged for manual check
blocked on human:
- PR #210 awaiting review/merge
- #145 ready to build once #210 merges
errors: <none | details>
```
Every tick ends by printing a structured report — the scheduled task surfaces it, an interactive
run shows it inline. Format and a worked example:
[`references/exit-report.md`](references/exit-report.md). `step executed:` names the **terminal**
outcome — the one step that did the tick's work — including `5-idle` when nothing had work. It is
not a checklist: steps 2–4 fall through when they have nothing to do, and the step-0 PR-label
restamp runs before the numbered flow regardless. Record restamps and failures under `actions:` /
`errors:`, not by adding step lines. A tick that did nothing still prints a report.

## What not to do

Expand Down Expand Up @@ -304,6 +291,7 @@ Each is pointed at from the step that needs it; this is the index.
- [`references/comment-formats.md`](references/comment-formats.md) — plan, question, and park-proposal templates.
- [`references/pr-labeling.md`](references/pr-labeling.md) — why the `auto:pr` label exists and how its failure modes are handled.
- [`references/scheduling.md`](references/scheduling.md) — cadence, overlap/races, model tier. For whoever schedules the task, not for the tick.
- [`references/exit-report.md`](references/exit-report.md) — the structured report every tick prints. Read when writing the report, not while deciding what to do.

## Related skills

Expand Down
25 changes: 25 additions & 0 deletions plugins/auto-dev/skills/auto-dev/references/exit-report.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# Exit report format — auto-dev

The structured summary every tick prints. Read this when writing the report at the end of a run;
the tick logic itself never needs it.

Every tick ends by printing a structured report — it is the run's summary output (the scheduled task surfaces it; an interactive run shows it inline):

```text
auto-dev tick — <ISO timestamp>
step executed: <0-failed | 1-reconcile | 2-pr-advance | 3-build | 4-triage | 5-idle>
Comment thread
coderabbitai[bot] marked this conversation as resolved.
open auto PRs (<count>/<config.autoDev.maxPrsInFlight>): #<n> (<status>), … | none
actions:
- #123: asked 2 clarifying questions → needs-info
- #145: plan approved by reply → ready
- #151: proposed parking (design fork is the maintainer's call) → needs-info
- #152: maintainer replied "park it" → parked
- PR #210: fixed 2 CodeRabbit findings, replied to 4 threads, pushed <sha>
- PR #212: no external review after 60m — self-reviewed, fixed 1 finding (<sha>), posted fallback review
- #160: built approved plan → PR #211 (labeled auto:pr); verified via /verify (drove the new CLI flag, observed expected output) → marked ready
- #163: built approved plan → PR #212 (draft); behavioral verification not run in sandbox (needs a live DB) — flagged for manual check
blocked on human:
- PR #210 awaiting review/merge
- #145 ready to build once #210 merges
errors: <none | details>
```
Loading
Loading