Skip to content

Check brave-core pull requests against Chromium's review rules - #211

Closed
netzenbot wants to merge 1 commit into
masterfrom
review-prs-chromium-rules
Closed

netzenbot wants to merge 1 commit into
masterfrom
review-prs-chromium-rules

Conversation

@netzenbot

Copy link
Copy Markdown
Collaborator

User impact: once the rules are synced, a brave-core pull request is also checked against Chromium's own review rules, and a finding links to the rule's file on chromium.googlesource.com. No other project changes.

The problem

Chromium publishes the 838 rules its AI reviewer applies, under agents/knowledge_base/rules/. netzenbot reads none of them, so a brave-core change that breaks one (a new Unretained in a posted task, a WebUI component with no Mocha test) is caught only if our own best-practices docs say the same.

Reproduce

  1. In the bot directory, with project.profile set to brave-core, run python3 scripts/sync-upstream-rules.py.
  2. Run /review-prs #<number> open on a brave-core pull request whose diff adds base::BindOnce(&Foo::Bar, base::Unretained(this)) to a posted task.
  3. Open manifest.json in the work directory and read the pull request's subagent_prompts.

Before: every rules entry is one of our docs (coding-standards.md_0 and so on). Expected: one more entry, chromium-rules-1.md_0, carrying the Chromium rules that diff meets.

I have not run these steps against a live pull request. I ran the same selection and prompt build on the real synced rules with a synthetic diff adding that line: one pool, two rules, prefer-weakptr-over-unretained-for-posted-tasks among them, a 10,313 character prompt. tests/test_upstream_rules.py covers it and fails on the parent commit, where scripts/lib/upstream_rules.py does not exist.

The fix

Chromium's tree is its own index: each directory's COMMON_METADATA lists groups of rules with a path_expression over the changed file and optional content_patterns over the lines the change adds. scripts/sync-upstream-rules.py downloads the tree, and prepare-review.py reads that metadata, with no model, to decide which rules a diff meets. Rules meeting the same files share a pool, chunked like one of our docs, so each chunk is an ordinary detect subagent. A finding must carry the link of a synced rule; a link that names none is dropped.

Only brave-core turns it on, with triggeredOnly: a rule is checked only when the diff adds what it is about. blocking rules are reported as medium and the rest as low, so none stops an approval until someone raises it in severityMap. A profile without the key, or one not yet synced, reviews as before.

A rule reaches the model like this, trimmed from the real run:

<a id="prefer-weakptr-over-unretained-for-posted-tasks"></a>

## Prefer weakptr over unretained for posted tasks

*Chromium rule `general/prefer-weakptr-over-unretained-for-posted-tasks` — rubric: safety, severity: should_fix, report as `low`.*

*rule_link: `https://chromium.googlesource.com/chromium/src/+/main/agents/knowledge_base/rules/general/cpp_safety_and_correctness/prefer-weakptr-over-unretained-for-posted-tasks.md#prefer-weakptr-over-unretained-for-posted-tasks`*

and the profile key it needs:

"upstreamRules": {
  "dir": ".ignore/upstream-rules/chromium",
  "source": "https://chromium.googlesource.com/chromium/src/+archive/main/agents/knowledge_base/rules.tar.gz",
  "linkBase": "https://chromium.googlesource.com/chromium/src/+/main/agents/knowledge_base/rules",
  "minSeverity": "should_fix",
  "triggeredOnly": true,
  "stripPrefixes": ["chromium_src/"]
}
Cost, and what is not covered

On the 60 most recent brave-core commits (October 2026), checking every group a file path alone meets would have been a median of 86 rules and about five more detect subagents per change. triggeredOnly is about half of one. That is why it ships on; turning it off is a one-line change once the findings show they are worth the tokens.

  • The rules come from Chromium main. Brave's Chromium may lack APIs a rule mentions, and the local src/agents has no knowledge_base yet.
  • Only chromium_src/ is translated to the path a rule is written against. browser/ is not chrome/browser/, so the Chrome-specific rules under chrome/ do not fire on our code. The general/ rules are not tied to a directory.
  • The archive is downloaded over https only, and only *.md and COMMON_METADATA are kept. A download that does not parse into rules leaves the ones already synced.
  • Not in this PR: converting our docs to Chromium's format, a semgrep or ast-grep prefilter, and checking proposeBestPractices against upstream headings.

Test plan

  • make lint — passed
  • .venv/bin/python -m pytest tests/test_upstream_rules.py — 52 passed
  • make test on the rebased branch — 1727 passed, 5 failed: the same four that fail on this machine without these changes (test_the_reviewer_is_asked_again_after_the_push and three in TestReviewRequestQueue, which depend on the local reviewer account), and TestStartDelay, which passes alone and failed while the scan ran alongside
  • make check-reviewdog — no findings; both runners also printed "could not run cleanly" because the landrun sandbox is not installed here, so they ran unsandboxed
  • python3 scripts/sync-upstream-rules.py --archive against the real 838-rule tarball, run twice — 838 rules in 145 groups, none in no group, the second run replaced the first
  • python3 scripts/sync-upstream-rules.py run on the deployment (until then brave-core reviews log a warning and run as before)
  • A /review-prs run against a real brave-core pull request shows a chromium-rules-* entry
  • CI passes cleanly

Chromium keeps what its AI reviewer knows as one markdown file per rule plus
COMMON_METADATA files that say when each rule applies, by path and by what the
change adds. review.upstreamRules in a profile points /review-prs at a synced
copy, and prepare-review.py picks the rules a diff meets from that metadata
without a model. brave-core turns it on, restricted to rules a diff triggers by
content; no other profile sets the key.
@bbondy

bbondy commented Nov 13, 2026

Copy link
Copy Markdown
Member

Claudio is handling this differently.

@bbondy bbondy closed this Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants