-
Notifications
You must be signed in to change notification settings - Fork 1
chore: the audit tail — frontmatter nit, skill sizes, and settled decisions #20
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
12 commits
Select commit
Hold shift + click to select a range
aa3c446
fix(worklog): drop the stray blank line inside the frontmatter
allenhutchison 6ed651d
ci: reject blank lines inside skill frontmatter
allenhutchison 69a5629
refactor(audit-architecture): move language detection into references…
allenhutchison 595521b
refactor(audit-tests): move language mechanics into references/, 4,58…
allenhutchison 2855b4c
refactor(auto-dev): move the exit-report format into references/, 5,3…
allenhutchison e85806a
docs(roadmap): record the settled decisions so they stop resurfacing
allenhutchison ad19c22
fix(ci): catch whitespace-only blank lines in frontmatter
allenhutchison 125d173
fix(audit-architecture): finish the extraction — one home for languag…
allenhutchison 99bc099
fix(audit-tests): state the cross-audit rule the scheduling note denied
allenhutchison 8ea2480
fix(auto-dev): clarify that a tick reports one step, and index exit-r…
allenhutchison f402b0c
fix(audit-architecture): stop requiring await on session.add, and fin…
allenhutchison 0705379
fix(auto-dev): define step executed as the terminal outcome
allenhutchison File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
82 changes: 82 additions & 0 deletions
82
plugins/audits/skills/audit-architecture/references/language-detection.md
Large diffs are not rendered by default.
Oops, something went wrong.
7 changes: 7 additions & 0 deletions
7
plugins/audits/skills/audit-architecture/references/scheduling.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| # Scheduling — audit-architecture | ||
|
|
||
| This skill is **not** part of the `daily-update` meta-skill, because `daily-update` bundles its work into one PR and this skill explicitly opens many. Schedule it as its own slot (e.g. nightly at 2am local time) via the `schedule` skill. The schedule should invoke this skill directly; there is no autonomous-prompt variant — pass a literal `/audit-architecture` or equivalent. | ||
|
|
||
| If the user is running short on `/schedule` slots and wants to combine with `daily-update`, the right consolidation is to have this skill run *first*, produce its PRs/issues, and then let `daily-update` run its own one-PR sweep on top — but they remain logically separate runs from the maintainer's point of view. | ||
|
|
||
| **Model tier:** DRY/abstraction judgment, invariant drift, and PR-vs-issue routing are judgment-heavy — schedule this on the **`capable`** tier (a smaller model mis-routes and over-files). See [`../../../references/model-tiers.md`](../../../references/model-tiers.md). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
34 changes: 34 additions & 0 deletions
34
plugins/audits/skills/audit-tests/references/language-mechanics.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| # Language-specific mechanics — audit-tests | ||
|
|
||
| The concrete greps and tool flags behind the category table. **Run only the block matching | ||
| `config.language`.** The table in SKILL.md decides what counts as a finding; this file is how to | ||
| find it in this language. | ||
|
|
||
| The smells above are language-agnostic. The exact detection commands and the framework names depend on `config.language`. Apply the equivalent for whichever language the config declares; read `config.guidelines.testing` for the conventions that are specific to this repo's runner. | ||
|
|
||
| ## When `config.language` is `python` | ||
|
|
||
| Framework: pytest + `unittest.mock`. Treat `config.paths.tests` as the grep root. | ||
|
|
||
| - **Mocked collaborators / sessions:** `grep -rnE "(MagicMock|AsyncMock|Mock)\(" <tests>` and read for `patch("...")` / `patch.object(...)`. Cross-reference each against the "what may not be mocked" rule in `config.guidelines.testing` (e.g. mocking the DB session instead of the real test-DB fixture). | ||
| - **Weak assertions / broad raises:** `grep -rn "pytest.raises(Exception)\|pytest.raises(BaseException)" <tests>`; scan for assertion-free test bodies. | ||
| - **Flaky:** `grep -rn "time\.sleep\|asyncio\.sleep" <tests>`; scan for unfrozen `datetime.now()`/`date.today()`/`random` in result-asserting tests (the fix is `freezegun`/`freeze_time` or injecting the value). | ||
| - **Skip/xfail rot:** `grep -rn "@pytest.mark.skip\|@pytest.mark.xfail\|pytest.skip(" <tests>` — flag any without `reason=`, and any `xfail` that now passes (`--runxfail` reports XPASS). | ||
| - **Redundant framework boilerplate:** if `config.guidelines.testing` says the repo runs pytest-asyncio in `asyncio_mode = "auto"`, then `@pytest.mark.asyncio` is a no-op — `grep -rn "@pytest.mark.asyncio" <tests>` and remove. Don't assume auto-mode; confirm it in the guidelines first. | ||
| - **Parametrize:** the duplication target is `@pytest.mark.parametrize`. | ||
| - **Slow tests:** `<config.commands.test> --durations=25 -q`. | ||
| - **Coverage:** `config.commands.coverage` typically emits `coverage.json` (branch coverage on); read it for covered-lines-but-uncovered-branches. | ||
|
|
||
| ## When `config.language` is `typescript` | ||
|
|
||
| Framework: the repo's test runner (Jest or Vitest — check `config.commands.test`). Apply the equivalent of each python check. | ||
|
|
||
| - **Mocked collaborators:** `grep -rnE "(jest|vi)\.(mock|fn|spyOn)\(" <tests>` and read each. Cross-reference against the "what may not be mocked" rule in `config.guidelines.testing`. Spying on / mocking a module you own that could run for real is the same smell as the python session-mock case. | ||
| - **Weak assertions / broad throws:** flag `expect(...).toThrow()` with no error matcher, and test bodies with **no `expect(...)`** at all. Tighten `toThrow()` to a specific error type/message. | ||
| - **Flaky:** `grep -rnE "setTimeout|new Promise\(.*setTimeout" <tests>` for real-time waits; unfrozen `Date.now()`/`Math.random()` in result-asserting tests. The fix is **fake timers** (`jest.useFakeTimers()` / `vi.useFakeTimers()` and `setSystemTime`) or injecting the value — the direct analog of freezing the clock. | ||
| - **Skip/only rot:** `grep -rnE "\.(skip|only|todo)\(|xit\(|xdescribe\(" <tests>`. A stray `.only` is a real smell — it silently disables every other test in the file; a `.skip`/`xit` without a comment reason is rot. | ||
| - **Redundant framework boilerplate:** per `config.guidelines.testing` — leftover `.only`, redundant `async` wrappers, etc. | ||
| - **Parametrize:** the duplication target is `it.each` / `test.each` / `describe.each`. | ||
| - **Golden-snapshot noise:** oversized or volatile `toMatchSnapshot()` / inline snapshots that re-bless on every change. | ||
| - **Slow tests:** the runner's slow-test reporting (Jest `--verbose` timings; Vitest's slow-test reporter). | ||
| - **Coverage:** `config.commands.coverage` typically emits `coverage-summary.json` / lcov; read it for uncovered branches in already-tested files. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| # Scheduling — audit-tests | ||
|
|
||
| Schedule this as its own slot (a few times a day is fine given the silent-on-clean + low-cap design), invoking it directly (`/audit-tests` or equivalent) — there is no autonomous-prompt variant. It is intentionally **separate** from both `daily-update` (which bundles its work into one PR; this skill opens discrete ones) and `audit-architecture` (which owns the source side). Running both audits is fine. Ownership is separate — source vs tests — and each dedups against its own label/branch prefix, but they are **not** independent: this skill skips any file touched by an open `arch-*` PR (SKILL.md, step 3 and "What not to do"), so an architecture PR in flight defers the test audit on that file rather than racing it. | ||
|
|
||
| **Model tier:** "is this mock decorative? is this assertion actually weak?" is judgment — schedule on **`capable`**, or on a **`mid`** rung if the repo defines one (this runs several times a day, so the cost trade is real). See [`../../../references/model-tiers.md`](../../../references/model-tiers.md). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
25 changes: 25 additions & 0 deletions
25
plugins/auto-dev/skills/auto-dev/references/exit-report.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| # Exit report format — auto-dev | ||
|
|
||
| The structured summary every tick prints. Read this when writing the report at the end of a run; | ||
| the tick logic itself never needs it. | ||
|
|
||
| Every tick ends by printing a structured report — it is the run's summary output (the scheduled task surfaces it; an interactive run shows it inline): | ||
|
|
||
| ```text | ||
| auto-dev tick — <ISO timestamp> | ||
| step executed: <0-failed | 1-reconcile | 2-pr-advance | 3-build | 4-triage | 5-idle> | ||
| open auto PRs (<count>/<config.autoDev.maxPrsInFlight>): #<n> (<status>), … | none | ||
| actions: | ||
| - #123: asked 2 clarifying questions → needs-info | ||
| - #145: plan approved by reply → ready | ||
| - #151: proposed parking (design fork is the maintainer's call) → needs-info | ||
| - #152: maintainer replied "park it" → parked | ||
| - PR #210: fixed 2 CodeRabbit findings, replied to 4 threads, pushed <sha> | ||
| - PR #212: no external review after 60m — self-reviewed, fixed 1 finding (<sha>), posted fallback review | ||
| - #160: built approved plan → PR #211 (labeled auto:pr); verified via /verify (drove the new CLI flag, observed expected output) → marked ready | ||
| - #163: built approved plan → PR #212 (draft); behavioral verification not run in sandbox (needs a live DB) — flagged for manual check | ||
| blocked on human: | ||
| - PR #210 awaiting review/merge | ||
| - #145 ready to build once #210 merges | ||
| errors: <none | details> | ||
| ``` | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.