AI diff auditor

A CLI tool that scans a git diff for patterns that are typical of "almost right, but not quite" AI-generated code — the single biggest frustration developers report when working with AI coding assistants (cited by 66% of developers in the 2025 Stack Overflow Developer Survey).

Instead of reviewing an entire AI-assisted commit line by line, run this against the diff and get a prioritized list of exactly what to double check.

Why this exists

AI coding assistants are very good at producing code that looks correct. They are less good at:

• Not silently swallowing errors (except: / except Exception: with no re-raise)
• Remembering to update tests when they change logic
• Avoiding leftover debug statements and TODOs in "finished" code
• Avoiding accidentally hardcoded secrets when stubbing out config
• Avoiding eval/exec shortcuts that introduce security risk

This tool checks a diff for exactly these patterns, so review time goes to the lines that actually need a human's judgment.

Usage

pip install -r requirements.txt

# Audit a specific commit range
python -m ai_diff_auditor.cli --git-range HEAD~1..HEAD

# Audit a saved diff file
python -m ai_diff_auditor.cli --diff-file changes.diff

# Audit whatever is piped in
git diff | python -m ai_diff_auditor.cli

Example output:

[HIGH] bare-except - app.py:4 - Bare or overly broad except clause added - errors may be silently swallowed
[HIGH] hardcoded-secret - app.py:7 - Possible hardcoded credential/secret introduced
[MEDIUM] missing-test-update - app.py - Logic changed in these files but no test file was updated in this diff - common with AI-generated changes
[LOW] todo-comment - app.py:9 - New TODO/FIXME comment added - may indicate incomplete logic

The process exits with a non-zero status if any high-severity finding is present, so it can gate a CI pipeline or a pre-commit hook.

Running with Docker

docker build -t ai-diff-auditor .
docker run --rm -v $(pwd):/repo -w /repo ai-diff-auditor --git-range HEAD~1..HEAD

Running tests

python -m unittest discover -s tests -v

Architecture

git diff / diff file / stdin
        |
        v
   parse_diff()          -> per-file added/removed lines with line numbers
        |
        v
   per-file rules   ----> bare-except, todo-comment, eval-exec,
        |                 debug-print, hardcoded-secret
        v
   diff-level rules ----> missing-test-update
        |
        v
   CLI output (sorted by severity, non-zero exit on high severity)

Each rule is a small, independent function that takes parsed diff data and returns a list of findings. Adding a new check means writing one function and adding it to PER_FILE_RULES or DIFF_LEVEL_RULES in rules.py — no changes needed elsewhere.

Limitations

This is a heuristic, regex/pattern-based auditor, not a semantic analyzer — it does not parse the full AST or understand data flow across files. It will miss subtler logic bugs and can produce occasional false positives (e.g. a deliberate broad except with a comment explaining why). It's meant to narrow attention during review, not replace it.

Possible extensions

• AST-based checks instead of regex, for fewer false positives
• A GitHub Action that comments findings directly on a pull request
• Configurable severity thresholds and an ignore list for accepted patterns