Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .claude/skills/review-prs/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@ The review pipeline minimizes LLM token usage by pushing all heavy data through

**Proposed best practices.** With `review.proposeBestPractices`, the validator may add one entry to `proposals` for a rule the review showed the project lacks and no check could decide. After posting, `collect-results.py --auto` files it as an issue in `project.issueRepository`, titled `Best practice: <rule>`, through `file-proposals.py`: at most one per PR and three per run, nothing filed when it matches an existing rule heading or an issue that already proposes it (or came from the same PR), every field cleaned of mentions and foreign links, and a failure reported and never raised. Without `--auto` the proposals are only printed. Only a PR that has a validator can draw one: a PR with no candidates and no open threads is clean and gets none.

**Labels.** With `review.labelPrs`, the first review that reads a PR also labels it with the review it needs (`pr/risk/network-calls`, `pr/tests-only`, and so on), and no later review labels it again: see [docs/pr-labels.md](../../../docs/pr-labels.md). Paths decide some labels in `prepare-review.py`, with no model. When the profile lists labels a path cannot decide, `prepare-review.py` writes one more prompt for the PR, `label_prompt`, which one detect subagent answers by choosing from the profile's list. `collect-results.py` adds the labels after posting, and records the PR in `_labelled` in the cache. Nothing is applied until every label the profile names exists in the PR repository, and until then no label prompt is written.

The main LLM session only orchestrates: run scripts, read a small manifest, launch subagents with tiny prompts, run the collector. It never sees diffs, rule text, or violation details.

---
Expand Down Expand Up @@ -83,6 +85,7 @@ Read the manifest file at `{work_dir}/manifest.json`. It contains:
- `files_reviewed` of `files_total`: how many of the PR's files this run reviews
- `subagent_prompts`: array of entries with `prompt_file` and `results_file` paths (NOT prompt text)
- `summary_prompt`: `null`, or one more entry with `prompt_file` and `results_file` for the subagent that describes the PR. Present only for a project whose profile sets `review.verdict`
- `labels`: `null`, or `decided` (labels the changed paths gave) and `judged` (labels the label subagent may choose from). `label_prompt`: `null`, or one more entry with `prompt_file` and `results_file` for that subagent. Both are `null` for a PR already labelled, and for a project with no `review.labelPrs`
- **`cached_prs`**: PRs not reviewed this run — already reviewed at this commit, or no file changed since the last review (handled by the prepare script — just log results)
- **`errors`**: per-PR errors encountered during preparation

Expand Down Expand Up @@ -112,6 +115,8 @@ Every prompt file fits in one Read call. When a check's diff does not fit alongs

A PR's `summary_prompt`, when it is not `null`, gets a subagent in the same way, with the same `subagent_type`. It is not one of the detect prompts: `select-candidates.py` never reads it, and a PR whose summary subagent failed is still reviewed, with no description section.

A PR's `label_prompt`, when it is not `null`, gets a subagent in the same way too. `select-candidates.py` never reads it either. A PR whose label subagent wrote nothing usable is reviewed as usual and gets no labels, and the next review of it asks again.

**Launch ALL detect subagents across ALL PRs in a single message** so they run concurrently.

**CRITICAL: Launch ALL subagents — no exceptions.** The prepare script already filtered documents by file type. Every entry in `subagent_prompts` MUST get a subagent. Do NOT skip any.
Expand Down Expand Up @@ -154,6 +159,8 @@ The script handles everything. Reviews are inline comments only, and never carry

For **interactive mode** (no `--auto`): before running collect-results.py, read each validator's results from `{work_dir}/pr_{number}/validated.json`, present each violation to the user for approval, write only approved violations back to that file, then run collect-results.py.

With `--auto` the collector then adds each PR's labels and records it as labelled. Without `--auto` it only prints the labels it would add, and records nothing.

Read the script's stderr — it contains the summary. Print it for cron logs.

### Summary of what consumes LLM tokens
Expand All @@ -166,6 +173,8 @@ Read the script's stderr — it contains the summary. Print it for cron logs.
| Read manifest, launch subagents | **~50 tokens per subagent** | Main LLM session (Sonnet on the schedules) |
| Rule-checking the diff | **Subagent tokens** | Detect subagents (Sonnet), one per rule chunk plus one for bugs |
| Checking candidates against the source | **Subagent tokens** | Validate subagents (Opus), one per PR with candidates |
| Decide labels from paths, add labels | **Zero** | `prepare-review.py`, `collect-results.py` |
| Choose the labels a path cannot decide | **Subagent tokens, once per PR** | Label subagent (Sonnet), one per PR not yet labelled |
| Collect results, post, approve | **Zero** | `collect-results.py` + `post-review.py` |

---
Expand Down
15 changes: 15 additions & 0 deletions .claude/skills/review-prs/collect-results.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
BOT_DIR = os.path.normpath(os.path.join(SCRIPT_DIR, "..", "..", ".."))

sys.path.insert(0, os.path.join(BOT_DIR, "scripts"))
from lib import pr_labels
from lib.load_config import (
get_config,
load_config,
Expand Down Expand Up @@ -411,6 +412,18 @@ def file_best_practice_proposals(pr_results, pr_repo, issue_repo, bp_dir, dry_ru
log(f"WARNING: best-practice proposals not filed: {e}")


def apply_labels(manifest, pr_repo, dry_run):
"""Label each PR whose manifest entry carries a label plan, once.

Independent of the review: a PR whose review was left for the next run is
labelled all the same, and one whose labels cannot be applied stays
unrecorded and is planned again.
"""
cache_path = os.path.join(BOT_DIR, ".ignore", "review-prs-cache.json")
for pr in manifest.get("prs", []):
pr_labels.label_pr(pr_repo, pr, cache_path, dry_run, log)


def cleanup_worktrees(manifest):
"""Remove git worktrees created by prepare-review.py for each PR."""
target_repo_path = manifest.get("target_repo_path", "")
Expand Down Expand Up @@ -574,6 +587,8 @@ def main():
dry_run=not auto_mode,
)

apply_labels(manifest, pr_repo, dry_run=not auto_mode)

cleanup_worktrees(manifest)

sys.exit(result.returncode)
Expand Down
41 changes: 37 additions & 4 deletions .claude/skills/review-prs/prepare-review.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,13 +38,15 @@

import importlib.util

from lib import pr_labels
from lib.load_config import (
load_config,
load_profile,
require_config,
resolve_docs_dir,
resolve_target_repo,
review_guidance,
review_label_prs,
review_link_rules,
review_propose_practices,
review_verdict,
Expand Down Expand Up @@ -94,6 +96,7 @@
REVIEW_GUIDANCE = review_guidance(load_profile(_config, _BOT_DIR))
REVIEW_SUMMARY = review_verdict(load_profile(_config, _BOT_DIR))
REVIEW_LINK_RULES = review_link_rules(load_profile(_config, _BOT_DIR))
REVIEW_LABEL_PRS = review_label_prs(load_profile(_config, _BOT_DIR))
REVIEW_PROPOSE_PRACTICES = review_propose_practices(load_profile(_config, _BOT_DIR))


Expand Down Expand Up @@ -1946,8 +1949,13 @@ def process_pr(
auto_mode=False,
prior_hashes=None,
approved_hashes=None,
label_cfg=None,
):
"""Write the detect prompts for one PR. Returns (manifest entry, error)."""
"""Write the detect prompts for one PR. Returns (manifest entry, error).

`label_cfg` is the label configuration when this PR has not been labelled
yet, and None otherwise.
"""
pr_number = pr["number"]
pr_title = pr["title"]
head_sha = pr["headRefOid"]
Expand Down Expand Up @@ -2294,6 +2302,20 @@ def write_parts(kind, base_id, paths, omit, build, chunk=None):
"body_file": body_file,
}

labels = label_prompt = None
if label_cfg:
# The whole PR, not the files changed since the last review: the labels
# describe the PR and are decided once.
labels, label_prompt = pr_labels.plan_pr(
label_cfg,
pr_number,
pr_title,
list(sections),
join_sections(sections.values()),
pr_work_dir,
worktree_path or TARGET_REPO_PATH,
)

pr_result = {
"number": pr_number,
"title": pr_title,
Expand All @@ -2308,6 +2330,8 @@ def write_parts(kind, base_id, paths, omit, build, chunk=None):
"thread_resolution": thread_resolution,
"subagent_prompts": subagent_prompts,
"summary_prompt": summary_prompt,
"labels": labels,
"label_prompt": label_prompt,
"worktree_path": worktree_path,
"source_path": worktree_path or TARGET_REPO_PATH,
"file_hashes_file": hashes_file,
Expand Down Expand Up @@ -2337,10 +2361,14 @@ def write_parts(kind, base_id, paths, omit, build, chunk=None):
def pr_needs_model(entry):
"""Whether a manifest entry has anything for a detector or the validator.

An entry with a detect prompt or a summary prompt does. One with open bot
threads does too, because the validator rules on them.
An entry with a detect prompt, a summary prompt or a label prompt does. One
with open bot threads does too, because the validator rules on them.
"""
if entry.get("subagent_prompts") or entry.get("summary_prompt"):
if (
entry.get("subagent_prompts")
or entry.get("summary_prompt")
or entry.get("label_prompt")
):
return True
return bool(entry.get("open_threads_file"))

Expand Down Expand Up @@ -2555,6 +2583,10 @@ def pr_entry(pr):

log("\n".join(progress_lines))

label_cfgs = pr_labels.configs_for_run(
REVIEW_LABEL_PRS, [p["number"] for p in prs_to_process], cache, PR_REPO, log
)

prune_stale_work_dirs()

# Create work directory for prompt/result files
Expand All @@ -2579,6 +2611,7 @@ def pr_entry(pr):
auto_mode,
prior_hashes_for(pr, prior_files),
approved_files.get(str(pr["number"])),
label_cfg=label_cfgs.get(pr["number"]),
): pr
for pr in prs_to_process
}
Expand Down
56 changes: 56 additions & 0 deletions docs/pr-labels.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Pull request labels

`/review-prs` labels a pull request with the review it needs, so a person or a policy can decide
from the labels how much human review the PR gets. The labels are named per project in
`review.labelPrs` of its [profile](../projects/README.md), and the code is the same for every
project.

## When a PR is labelled

A PR is labelled once, by the first review that reads it. The review cache records it in
`_labelled`, and no later push, re-review or `full` run labels it again. A label a person removed
stays removed. A PR first reviewed before a project's `review.labelPrs` was added is labelled the
next time a review reads it.

PRs the review skips are not labelled: a non-member's PR, which the org gate drops before anything
else reads it, a PR the bot already approved, and an uplift.

Nothing is applied until every label the profile names exists in the PR repository. Creating a
label needs write access, which the reviewer account may not have, so a person creates them with
`gh label create`. While one is missing, each run logs which, writes no label prompt, runs no
model and records no PR, so no PR loses its turn.

## How a label is decided

| Kind | Decided by | Model |
| --- | --- | --- |
| `rules` | the changed file paths, as globs | none |
| `judged` | one subagent that reads the diff and chooses from the profile's list | once per PR |

A glob matches the whole path and `*` crosses `/`. A glob with no `/` matches the file name, so
`DEPS` and `*_unittest.cc` apply at any depth. `any` applies the label when some changed file
matches, `only` when every changed file does, and `exclude` withholds it when some changed file
matches.

The diff is written by the PR's author, so three limits keep it from deciding anything:

- The subagent chooses from the profile's list. A label it names that is not on the list is
ignored, and nothing it writes reaches a command line.
- A judged label only raises scrutiny. A label that would lower it, such as `pr/no-behavior-change`,
is not judged by a model: a PR could be written to earn it. It stays a person's label until a
check that does not read the diff can decide it.
- Labels are only added.

## The labels of a project

A project's labels, what decides each, and the `gh label create` commands for them are in the
"Pull request labels" section of `projects/<name>/docs/labels.md`. The profile is the source of
the labels, and a test checks that the project's doc names every label and has a command for it.
The shipped profiles start every label with `pr/`, so none of them is a label the repository
already had, and a test holds them to it.

## Reading what it did

A run logs one `LABELS:` line per PR: the model's reason for each judged label, then what was
added. Without `--auto` it logs what it would add and records nothing. When a person removes a
judged label, the reason in the log says why the model chose it.
2 changes: 1 addition & 1 deletion projects/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ projects/<name>/
| `testTargets` | Maps a suite (`unit`, `browser`) and a location to a test binary. `local` is a test defined in the target repo, `upstream` one inherited from the surrounding checkout. |
| `uiPaths` | Repo-relative paths whose changes a person can look at. A change under one of them has to show the screen it produces, and `check-pr-body.py --diff-base` errors when the body shows none — see [Showing a terminal screen](../docs/pr-descriptions.md#showing-a-terminal-screen). A path with no `*`, `?` or `[` matches everything beneath it. Omit the key for a project with no interface, and no screen is asked for. |
| `localization` | Where this project keeps the words a person reads, so `check-untranslated.py --diff-base` can name the sentences a branch wrote into the source instead. `paths` and `exempt` read like `uiPaths`; `catalog` is the reference catalog and `how` the instruction quoted back to the author. The rest is the language's own spelling, and there are no built-in defaults worth relying on: `quotes` (default `"`), `comments` (default `//`), `developer` — calls whose strings are for whoever is debugging, where a trailing `*` is a prefix and a trailing `!` or `(` is matched as written — and `testsBegin`, a regex for the line below which a file is test code. Omit the key and nothing is checked; the check says so rather than passing quietly. |
| `review` | How `/review-prs` reviews this project's pull requests, beyond the best-practices rules. `guidance` is a string or a list of strings of review text; when present, one extra detect subagent applies it to each PR, reading the files it names from the PR's source tree, and the validator judges its findings against it. Its findings are severity `high` with no rule link, so a PR that trips one is not approved. `requestChanges: true` submits a review that has a finding above nit severity, or an earlier comment still open, as GitHub's `REQUEST_CHANGES` (a `COMMENT` review on a PR the bot opened itself), and approves a clean one, with none of what `verdict` adds: the review is the inline comments and the event, with no recommendation header, summary or description check. `verdict: true` makes a posted review open with its recommendation, "Recommendation: approve" on the approval and "Recommendation: request changes" on a review with a high or medium finding or an earlier comment still open, and "Recommendation: comments only" on one with only nits. The first is submitted as GitHub's `REQUEST_CHANGES` (a `COMMENT` review on a PR the bot opened itself), the others as `COMMENT`. Under the recommendation the review says what the pull request does, gives numbered steps for trying it by hand when its description has none, and lists the parts of the description shape in [docs/pr-descriptions.md](../docs/pr-descriptions.md) it lacks (the `User impact:` line, the problem, reproduction, fix and test plan sections). Below those it carries a collapsed "How this review reached its recommendation" section that says, in plain sentences, how many files and rules were read, which passes ran, and how many flagged problems the second reader kept. `checks` is a list of plain-language sentences, one per thing `guidance` looks for, that the section prints under "This project's own criteria"; without it the section names that pass but lists nothing. `linkRules: true` has the validator name the rule a finding from the bug or project-guidance pass breaks, as a link appended to the comment (the link is checked against the docs, and it does not change how findings are deduplicated). `proposeBestPractices: true` lets the validator describe one rule the project lacks, which an automatic run files as an issue titled `Best practice: ...` in the issue repository (one per PR, three per run, skipped when a rule or an earlier issue already says it); it never edits the docs. Omit the key and the review runs the rules and the bug pass only, with an empty review body. |
| `review` | How `/review-prs` reviews this project's pull requests, beyond the best-practices rules. `guidance` is a string or a list of strings of review text; when present, one extra detect subagent applies it to each PR, reading the files it names from the PR's source tree, and the validator judges its findings against it. Its findings are severity `high` with no rule link, so a PR that trips one is not approved. `requestChanges: true` submits a review that has a finding above nit severity, or an earlier comment still open, as GitHub's `REQUEST_CHANGES` (a `COMMENT` review on a PR the bot opened itself), and approves a clean one, with none of what `verdict` adds: the review is the inline comments and the event, with no recommendation header, summary or description check. `verdict: true` makes a posted review open with its recommendation, "Recommendation: approve" on the approval and "Recommendation: request changes" on a review with a high or medium finding or an earlier comment still open, and "Recommendation: comments only" on one with only nits. The first is submitted as GitHub's `REQUEST_CHANGES` (a `COMMENT` review on a PR the bot opened itself), the others as `COMMENT`. Under the recommendation the review says what the pull request does, gives numbered steps for trying it by hand when its description has none, and lists the parts of the description shape in [docs/pr-descriptions.md](../docs/pr-descriptions.md) it lacks (the `User impact:` line, the problem, reproduction, fix and test plan sections). Below those it carries a collapsed "How this review reached its recommendation" section that says, in plain sentences, how many files and rules were read, which passes ran, and how many flagged problems the second reader kept. `checks` is a list of plain-language sentences, one per thing `guidance` looks for, that the section prints under "This project's own criteria"; without it the section names that pass but lists nothing. `linkRules: true` has the validator name the rule a finding from the bug or project-guidance pass breaks, as a link appended to the comment (the link is checked against the docs, and it does not change how findings are deduplicated). `proposeBestPractices: true` lets the validator describe one rule the project lacks, which an automatic run files as an issue titled `Best practice: ...` in the issue repository (one per PR, three per run, skipped when a rule or an earlier issue already says it); it never edits the docs. `labelPrs` labels each pull request once, the first time a review reads it, with the review it needs: `rules` are labels the changed paths decide (`any` or `only` a list of globs, and `exclude`), `judged` maps a label a path cannot decide to the definition one subagent applies; nothing is applied until every label named exists in the PR repository. See [pr-labels.md](../docs/pr-labels.md). Omit the key and the review runs the rules and the bug pass only, with an empty review body. |

**`schedules.sh`** holds this project's cron jobs — see [Schedules](#schedules).

Expand Down
Loading
Loading