PR Test Guard is a lightweight PR test-quality checker. It looks for review signals that are easy to miss when a pull request has passing tests or acceptable line coverage but the tests may still be incomplete, weak, mismatched, or over-isolated.
The project does not try to prove that a PR is correct. Its job is to make test-related review risk easier to inspect.
The practical unit is a PR change and the test evidence around it.
For a changed behavior, PR Test Guard may inspect:
- changed production files and lines;
- added, modified, removed, skipped, or weakened tests;
- assertion shapes;
- deterministic related-test candidates from imports, direct calls, test names, and mock targets;
- changed-line coverage when available;
- per-test line contexts when explicitly supplied;
- explicit mock or patch boundaries;
- CI/test-run evidence;
- optional change-intent text from an issue, PR body, task, or structured fixture.
Change intent can improve mapping, but it is context rather than a requirement for every lightweight rule.
Rules should prefer signals that are:
- Local — tied to a concrete PR diff, test, assertion, line, or CI artifact.
- Inspectable — a reviewer can see why the signal fired.
- Deterministic by default — the default path should not require an LLM.
- Cheap enough for CI — lightweight checks should not require an expensive benchmark or hosted service.
- Advisory when uncertain — heuristics should surface risk without pretending to know the final merge decision.
Detection and policy are separate. The checker identifies review signals; repository configuration decides whether a selected rule is hidden, advisory, or error-level for CI.
Repository-specific test path classification is analysis input rather than a confidence policy. Configured test include/exclude globs determine which files participate in test-oriented rules, related-test discovery, and output counts; they do not make any resulting heuristic finding more authoritative.
The cheapest checks compare production-code changes with test changes. Useful signals include:
- production code changed but no test file changed;
- tests were deleted, skipped, or narrowed;
- assertions became visibly weaker;
- a behavioral fix has no obvious nearby test update.
These are review signals, not proof that testing is missing. Existing tests may already cover a change.
Changed-line coverage answers whether changed executable code ran. It is valuable, but it should remain one input rather than the final verdict.
A covered branch can still be backed by a weak assertion. A passing test can still exercise the wrong behavior.
Context-aware coverage adds a separate dynamic layer. PR Test Guard reads JSON
produced by coverage json --show-contexts after a pytest/pytest-cov run with
test contexts enabled. It maps executable changed lines to exact pytest IDs and
preserves setup, run, and teardown phases. Parameterized IDs are retained
as distinct runtime contexts and may also be grouped under their canonical test
node for comparison with static evidence.
The tool does not install a runtime tracer or modify the target test framework. Absent input leaves the existing rules unchanged. Unknown context formats, ambiguous paths, missing related-test contexts, and incomplete artifacts are recorded as unresolved or inconclusive rather than treated as proof that a test missed a change.
The current Python prototype extracts assertion structure and can flag obvious weak patterns. This is intentionally conservative. An assertion that looks weak syntactically can still be meaningful in a richer test context, so this class of signal should be advisory by default.
The direct checker records candidate tests that have deterministic relationships to changed Python symbols. The current context layer recognizes exact import relationships, direct calls to changed symbols, supported mock targets, and test name tokens when another deterministic relationship already exists.
Repositories can also declare source-to-test path mappings for stable
relationships that are not visible from Python syntax, such as integration
tests reached through a public entrypoint. A mapping adds every test function in
a matched tracked test file as a candidate and records both the matched changed
source path and any changed symbols from that path. Configured mappings are
additive, respect test-path exclusions, and remain inspectable through the
configured_path_mapping reason.
This is context for review output, not a test-selection oracle. A related test candidate may still assert the wrong behavior, and no related candidate does not prove that coverage is missing. A configured mapping also does not prove that a test ran or suppress the separate PTG001 signal when no test file changed.
Mocks are neutral. The current structural detector raises a candidate when an explicit patch target overlaps changed code that appears central to the tested path, or when a changed test mocks an internal dependency called on a changed production line without a clear interaction or owner-outcome assertion.
A mock candidate should tell a reviewer where to look; it should not automatically fail a PR. A dependency mock that constrains the changed owner's interaction contract, return value, or exception behavior is treated differently from a mock that simply replaces behavior and then makes a weak existence assertion.
The constrained dependency check is deliberately narrow. It looks for inspectable test evidence such as assert_called_once_with, assert_called_with, call_args / call_count assertions, non-weak assertions over the changed owner result, or pytest.raises around the owner call. It does not infer full business intent or decide that a mock is inherently appropriate.
The checker records one evidence unit for each changed Python function or
method. A unit distinguishes broad related-test candidates from tests that
directly call the changed symbol, tests that replace it with a mock, and direct
calls whose values reach meaningful assertions. This prevents a test change for
one function from hiding a separate changed function with no direct evidence.
Direct evidence propagates through resolvable calls between changed module
functions, imported functions, and self/cls methods so internal helpers can
inherit evidence from a tested public entrypoint. Dynamic dispatch is not
guessed.
Each change unit keeps evidence layers separate:
candidate -> static -> dynamic -> constraint -> counterfactual
A related candidate is not an observed executor. Static direct/call-path evidence is not dynamic execution. Dynamic execution does not establish an assertion constraint, and an assertion-shaped constraint does not establish counterfactual sensitivity. Reports expose the labels and underlying test IDs without assigning a 0–100 confidence score.
Assertion flow is deliberately intraprocedural. It follows a direct changed symbol call through local assignments, attributes, and subscripts. It recognizes comparisons and negated conditions, while treating existence checks, self-comparisons, and unobserved results as insufficient. Fixtures, helper-call summaries, dynamic dispatch, and business-intent correctness remain unknown.
The direct checker can execute a small set of deterministic behavior weakenings against changed Python lines. This path is opt-in through --deep, requires an explicit test command, and runs in an isolated Git worktree. Surviving probes can strengthen a test-quality warning.
This is a bounded PR-scoped signal, not a full mutation-testing campaign or repository-wide mutation score. A generated probe is only a candidate. If the baseline test command fails, PTG006 is skipped. If the configured tests kill the probe, no warning is emitted. Unsupported shapes such as response constructors, symbolic HTTP status constants, and unstable multi-line rewrites remain quiet by design.
Deep mode also supports function-level rollback probes. When the base and PR versions contain the same function signature and deterministic direct-call test evidence exists, the checker restores the base implementation while retaining the PR's tests. A passing rollback produces PTG009. Collection, import, or syntax failures are inconclusive; an ordinary failing test kills the rollback.
The direct checker uses ten stable rule ids for the current Python/pytest scope:
PTG001— production code changed with no test-file change;PTG002— changed Python line uncovered in a supplied coverage XML;PTG003— possible weak assertion added in a changed test;PTG004— suspicious test deletion, skip/xfail, or assertion removal;PTG005— a mock directly replaces a changed Python symbol, or a changed test mocks an unconstrained internal dependency called on a changed production line;PTG006— bounded targeted probe survives the configured tests.PTG007— changed function or method has no deterministic direct or static call-path test evidence;PTG008— direct changed-symbol result does not reach a meaningful assertion;PTG009— compatible base-version function still passes the PR tests.PTG010— all observed deterministic related tests miss every changed executable line in the changed symbol.
PTG010 is advisory by default and intentionally requires all of the following: an explicitly supplied valid context artifact, mapped executable changed lines, deterministic direct/static related-test evidence, parseable runtime contexts for every such related test, and zero changed-line overlap. Without those preconditions the result is inconclusive. This produces a progressive evidence story: PTG007 has no credible relationship; PTG010 has a relationship but no runtime overlap; PTG008 has execution but no meaningful constraint; PTG009 has an apparent constraint but does not distinguish the prior behavior.
Each result includes a rule id, severity, file/line where available, a short message, and evidence text. Severity defaults to warning; .pr-test-guard.yml or --fail-on can promote selected rules to error or suppress them with off. The direct checker also emits related-test candidates as JSON context and a short text/GitHub summary. The older fixture runner retains its research-prototype labels only so existing regression fixtures remain stable during the transition.
The core product should remain callable from a CLI. GitHub Actions is the first automated integration target, not the only runtime.
The intended shape is:
PR Test Guard core
-> local CLI / arbitrary CI
-> GitHub Action wrapper
CI integration should be advisory by default. Repositories can opt into stricter enforcement for selected rules with configuration while keeping heuristic detection inspectable and reversible.
Version 0.8.0 provides a repository-native check command, strict standalone configuration validation, bounded and stably grouped GitHub annotations, reusable Action outputs, configurable test-path recognition and rule policy, JSON report artifacts, dogfooding review-draft helpers, deterministic related-test context, additive source-to-test path mappings, changed-function evidence, opt-in per-test dynamic coverage evidence, bounded assertion-result flow, AST-scoped targeted probes, function rollback probes, and PTG005 constrained dependency-mock suppression for the current Python/pytest scope. Annotation limits affect only delivery noise: the job summary and JSON report retain the complete policy-filtered result. The checker is intentionally conservative: it does not infer full PR correctness, automatically discover every project's test command, instrument repositories automatically, or treat heuristic signals as merge-blocking failures unless the repository explicitly configures that policy.
The immediate engineering goal is real-PR dogfooding and false-positive reduction. Controlled fixtures remain regression tests for the tool rather than a public benchmark.