Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
89 changes: 26 additions & 63 deletions .github/AUTOMATED_REVIEW.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Automated PR Review + Auto-Fix System
# Automated PR Review System

**Status**: Active
**Last Updated**: 2025-11-24
**Workflow**: `.github/workflows/pr-review-auto-fix.yml`
**Last Updated**: 2026-03-03
**Workflow**: `.github/workflows/pr-review.yml`

## Overview

Expand All @@ -11,35 +11,34 @@ Every pull request in the agentready repository receives an automated code revie
1. **Reviews PRs automatically** - Multi-agent review on PR open/update
2. **Maps findings to AgentReady attributes** - Links issues to the 25 attributes
3. **Calculates score impact** - Shows how fixing issues improves certification
4. **Auto-fixes critical issues** - Resolves blockers/criticals (confidence ≥90) automatically
5. **Commits fixes to PR branch** - Human developer sees clean PR

## Security

The workflow uses `pull_request` trigger (not `pull_request_target`) to prevent prompt injection attacks. See [RHOAIENG-51622](https://issues.redhat.com/browse/RHOAIENG-51622) for details.

**Important**: Fork PRs do not receive automated reviews because they don't have access to repository secrets. This is intentional for security.

### For External Contributors

If you're contributing from a fork:
- Push your branch to the main repository instead (e.g., `username/feature-name`)
- Or request manual review from a maintainer

## Workflow

```
PR Opened/Updated
PR Opened/Updated (from main repo branch)
┌───────────────────────────────────────┐
│ Job 1: Code Review │
│ PR Review │
│ (pull_request trigger) │
│ │
│ - Minimize old review comments │
│ - Run /review-agentready command │
│ - Generate .review-results.json │
│ - Post review comment with findings │
└───────────────────────────────────────┘
↓ (if confidence ≥90 issues found)
┌───────────────────────────────────────┐
│ Job 2: Auto-Fix Criticals │
│ │
│ - Download review results │
│ - Fix each critical issue atomically │
│ - Run linters + tests after each fix │
│ - Commit with conventional commits │
│ - Update review comment with status │
│ - Push fixes to PR branch │
└───────────────────────────────────────┘
Developer continues work on clean PR
Developer receives review on their PR
```

## Review Output Format
Expand Down Expand Up @@ -120,10 +119,10 @@ The workflow is enabled by default. To disable:

```bash
# Rename to disable
mv .github/workflows/pr-review-auto-fix.yml .github/workflows/pr-review-auto-fix.yml.disabled
mv .github/workflows/pr-review.yml .github/workflows/pr-review.yml.disabled

# Re-enable later
mv .github/workflows/pr-review-auto-fix.yml.disabled .github/workflows/pr-review-auto-fix.yml
mv .github/workflows/pr-review.yml.disabled .github/workflows/pr-review.yml
```

## Testing
Expand Down Expand Up @@ -157,9 +156,8 @@ mv .github/workflows/pr-review-auto-fix.yml.disabled .github/workflows/pr-review
```

5. **Observe workflow**:
- Check Actions tab: `.github/workflows/pr-review-auto-fix.yml`
- Review job should post comment with findings
- Auto-fix job should commit fixes to PR branch
- Check Actions tab for `PR Review`
- Review comment should be posted on the PR

6. **Verify fixes**:
```bash
Expand All @@ -174,24 +172,7 @@ mv .github/workflows/pr-review-auto-fix.yml.disabled .github/workflows/pr-review

## Customization

### Adjust Confidence Threshold

Edit `.github/workflows/pr-review-auto-fix.yml`:

```yaml
# Change from 90 to 95 for more conservative auto-fixing
if: needs.review.outputs.has_criticals == 'true' # confidence ≥90
# to
if: needs.review.outputs.has_criticals == 'true' # confidence ≥95
```

Also update `.claude/commands/review-agentready.md`:

```markdown
**Critical Issue Criteria** (confidence ≥95): # Changed from 90
```

### Add Custom Focus Areas
### Adjust Review Focus Areas

Edit `.claude/commands/review-agentready.md` under "AgentReady-Specific Focus Areas":

Expand Down Expand Up @@ -220,28 +201,10 @@ class ReviewFormatter:
**Symptom**: Workflow runs but no comment appears on PR

**Solutions**:
1. Check GitHub Actions logs for errors
1. Check GitHub Actions logs for `PR Review`
2. Verify `ANTHROPIC_API_KEY` is set correctly
3. Ensure `pull-requests: write` permission is granted
4. Check if PR is from a fork (may need `pull_request_target`)

### Auto-Fix Not Running

**Symptom**: Review posts but auto-fix job doesn't run

**Solutions**:
1. Verify review found issues with confidence ≥90
2. Check `.review-results.json` artifact was uploaded
3. Review `needs.review.outputs.has_criticals` value in logs

### Fixes Causing Test Failures

**Symptom**: Auto-fix commits but tests fail

**Solutions**:
1. Check the auto-fix logic in `.github/claude-bot-prompt.md`
2. Verify linters run before tests: `black . && isort . && pytest`
3. Consider lowering confidence threshold (fixes might be too aggressive)
4. **Fork PRs**: Reviews only run on PRs from branches in the main repo, not forks

### Rate Limiting

Expand Down
Original file line number Diff line number Diff line change
@@ -1,18 +1,33 @@
name: PR Review + Auto-Fix Criticals
name: PR Review

# Automated code review for pull requests.
#
# Security: Uses pull_request trigger (not pull_request_target) to prevent
# prompt injection attacks. Fork PRs will not receive automated reviews
# since they don't have access to repository secrets.
#
# For fork contributors: Push your branch to the main repo instead of using
# a fork, or request manual review.
#
# Fixes: GitHub #324 (wrong PR context)
# Security: Prevents prompt injection attacks from fork PRs

on:
pull_request_target:
pull_request:
types: [opened, synchronize, reopened]

permissions:
contents: write
contents: read
pull-requests: write
issues: write

jobs:
review:
name: Code Review
runs-on: ubuntu-latest
# Skip reviews on fork PRs (no secrets access anyway)
if: github.event.pull_request.head.repo.full_name == github.repository

steps:
- name: Minimize old Claude review comments
uses: actions/github-script@v8
Expand All @@ -25,7 +40,7 @@ jobs:
});

for (const comment of comments.data) {
if (comment.body.includes('🤖 AgentReady Code Review') &&
if (comment.body.includes('AgentReady Code Review') &&
comment.user.login === 'github-actions[bot]') {
await github.rest.issues.updateComment({
owner: context.repo.owner,
Expand All @@ -38,17 +53,13 @@ jobs:

- name: Checkout repository
uses: actions/checkout@v4
with:
# Use base branch for security - don't execute fork's .claude/commands/
ref: ${{ github.event.pull_request.base.sha }}

- name: Run AgentReady Code Review
uses: anthropics/claude-code-action@v1
with:
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
github_token: ${{ secrets.GITHUB_TOKEN }}
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
allowed_non_write_users: '*' # Enable reviews on fork PRs
settings: |
{
"permissions": {
Expand All @@ -61,12 +72,12 @@ jobs:
}
}
prompt: |
Run the /review-agentready command on this pull request.
Run the /review-agentready command on pull request #${{ github.event.pull_request.number }}.

Provide a comprehensive code review focusing on:
- AgentReady attribute compliance
- Security issues
- Code quality
- Best practices

Post the review as a comment on this PR.
Post the review as a comment on PR #${{ github.event.pull_request.number }}.
Loading