Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions 0024-codeowners-app-matrix/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
*.pem
__pycache__/
.env
results.json
105 changes: 105 additions & 0 deletions 0024-codeowners-app-matrix/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
---
title: "24. CODEOWNERS + GitHub App approval matrix"
status: Concluded
topics:
- github-apps
- security
- branch-protection
---

# 24. CODEOWNERS + GitHub App approval matrix

Date: 2026-07-13

Related: [Auto-merge ADR PR](https://github.com/fullsend-ai/fullsend/pull/2791),
[CODEOWNERS blank-owner PR](https://github.com/fullsend-ai/fullsend/pull/2790)

## Hypotheses

| ID | Statement | Result |
|----|-----------|--------|
| H1 | A GitHub App with `contents: read` can submit an APPROVE review, but it does NOT count toward the "require approvals" branch protection rule. | Confirmed |
| H2 | A GitHub App with `contents: write` can submit an APPROVE review, and it DOES count toward "require approvals". | Confirmed |
| H3 | A blank-owner CODEOWNERS entry removes the code-owner review requirement for PRs that only touch that file. | Confirmed |
| H4 | A PR touching both a blank-owner file and an owned file still requires code-owner approval for the owned file. | Confirmed |
| H5 | When H2 and H3 are both true, a `contents: write` app is the sole approver on a blank-owner-only PR, and the PR is mergeable. | Confirmed |
| H6 | When a PR touches both blank-owner and owned files, even a `contents: write` app cannot satisfy the code-owner requirement for the owned files. | Confirmed |

## Approach

### Test infrastructure

- **Org:** `appdumpster`
- **Repo:** `appdumpster/codeowners-lab`
- **Team:** `@appdumpster/owners` (with the human operator as a member)
- **Apps:**
- `appdumpster-read-bot` — `contents: read`, `pull_requests: write`
- `appdumpster-write-bot` — `contents: write`, `pull_requests: write`

### Repo layout

```
CODEOWNERS:
* @appdumpster/owners
go.mod
go.sum

Files: main.go, go.mod, go.sum
```

### Branch protection on `main`

- Require 1 approval
- Require review from code owners
- No status checks (keep it simple)

### Test matrix

| PR | Branch | Files touched | Approved by | Tests |
|----|--------|--------------|-------------|-------|
| 1 | `test/read-owned` | `main.go` | read-bot | H1 |
| 2 | `test/write-owned` | `main.go` | write-bot | H2, H4 partial |
| 3 | `test/read-blank` | `go.mod` | read-bot | H1, H3 |
| 4 | `test/write-blank` | `go.mod` | write-bot | H2, H3, H5 |
| 5 | `test/read-mixed` | `go.mod` + `main.go` | read-bot | H1, H4 |
| 6 | `test/write-mixed` | `go.mod` + `main.go` | write-bot | H2, H4, H6 |

### Measurement

For each PR, after the bot approves, query:
- `GET /repos/{owner}/{repo}/pulls/{number}/reviews` — confirm the review exists
- `GET /repos/{owner}/{repo}/pulls/{number}/merge` — can it merge? (405 = no, 204 = yes)
- `PUT /repos/{owner}/{repo}/pulls/{number}/merge` — attempt the merge, record status code and error message

## Results

Ran on 2026-07-13 against `appdumpster/codeowners-lab` with two throwaway GitHub Apps.

| PR | Test | `mergeable_state` | Merge attempt | HTTP | Error message |
|----|------|--------------------|---------------|------|---------------|
| #1 | read-bot + owned file | `blocked` | BLOCKED | 403 | Resource not accessible by integration |
| #2 | write-bot + owned file | `blocked` | BLOCKED | 405 | Waiting on code owner review from appdumpster/owners. |
| #3 | read-bot + blank-owner file | `blocked` | BLOCKED | 403 | Resource not accessible by integration |
| #4 | write-bot + blank-owner file | `clean` | **MERGED** | 200 | Pull Request successfully merged |
| #5 | read-bot + mixed files | `unknown` | BLOCKED | 403 | Resource not accessible by integration |
| #6 | write-bot + mixed files | `unknown` | BLOCKED | 405 | Waiting on code owner review from appdumpster/owners. |

### Key observations

1. **`contents: read` vs `contents: write` produces different failure modes.** The read-bot gets 403 "Resource not accessible by integration" — a permission error, not a branch protection error. The write-bot gets 405 with a specific branch protection message. This means the read-bot's approval is invisible to branch protection; it can't even attempt the merge.

2. **Blank-owner CODEOWNERS entries work as documented.** PR #4 (write-bot approving a PR that only touches `go.mod`) had `mergeable_state: clean` and merged successfully. The blank-owner entry removed the code-owner review gate.

3. **Blank-owner doesn't leak to owned files.** PRs #5 and #6 (mixed: `go.mod` + `main.go`) were both blocked. The write-bot's approval satisfied the general approval requirement but not the code-owner requirement for `main.go`.

4. **The only mergeable combination is `contents: write` app + blank-owner-only files.** This is exactly the scenario the [auto-merge ADR](https://github.com/fullsend-ai/fullsend/pull/2791) targets.

## Conclusion

All six hypotheses confirmed. the [auto-merge ADR](https://github.com/fullsend-ai/fullsend/pull/2791)'s design is sound:

- A `contents: write` merge app can approve and merge PRs that only touch blank-owner CODEOWNERS files (like `go.mod`/`go.sum`).
- The same app cannot bypass code-owner review for files with actual owners.
- A `contents: read` review app's approvals are ignored by branch protection entirely.

The combination of blank-owner CODEOWNERS entries + a write-capable merge app is the minimum viable path to bot-driven auto-merge for dependency updates without compromising code-owner review for application code.
100 changes: 100 additions & 0 deletions 0024-codeowners-app-matrix/create_prs.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
#!/usr/bin/env bash
set -euo pipefail

OWNER="appdumpster"
REPO="codeowners-lab"
BASE="main"

MAIN_GO_CONTENT=$(printf 'package main\n\nfunc main() {\n\t// modified\n}\n' | base64 -w0)
GO_MOD_CONTENT=$(printf 'module github.com/appdumpster/codeowners-lab\n\ngo 1.23\n' | base64 -w0)

PR_BODY="Automated test PR for CODEOWNERS experiment."

echo "=== Fetching SHA of main branch"
MAIN_SHA=$(gh api "repos/${OWNER}/${REPO}/git/ref/heads/${BASE}" --jq '.object.sha')
echo "=== main SHA: ${MAIN_SHA}"

create_pr() {
local branch="$1"
local title="$2"
local file1="$3"
local content1="$4"
local file2="${5:-}"
local content2="${6:-}"

echo "=== Creating PR: ${title}"

# Check if branch already exists
if gh api "repos/${OWNER}/${REPO}/git/ref/heads/${branch}" > /dev/null 2>&1; then
echo "=== Branch ${branch} already exists, skipping"
return
fi

# Create branch ref
echo "=== Creating branch ${branch}"
gh api --method POST "repos/${OWNER}/${REPO}/git/refs" \
--field "ref=refs/heads/${branch}" \
--field "sha=${MAIN_SHA}" > /dev/null

# Get current file SHA and commit change for file1
echo "=== Modifying ${file1}"
local file1_sha
file1_sha=$(gh api "repos/${OWNER}/${REPO}/contents/${file1}?ref=${branch}" --jq '.sha')
gh api --method PUT "repos/${OWNER}/${REPO}/contents/${file1}" \
--field "message=test: modify ${file1}" \
--field "content=${content1}" \
--field "sha=${file1_sha}" \
--field "branch=${branch}" > /dev/null

# If a second file is specified, do the same
if [[ -n "${file2}" ]]; then
echo "=== Modifying ${file2}"
local file2_sha
file2_sha=$(gh api "repos/${OWNER}/${REPO}/contents/${file2}?ref=${branch}" --jq '.sha')
gh api --method PUT "repos/${OWNER}/${REPO}/contents/${file2}" \
--field "message=test: modify ${file2}" \
--field "content=${content2}" \
--field "sha=${file2_sha}" \
--field "branch=${branch}" > /dev/null
fi

# Create PR
echo "=== Opening PR"
gh api --method POST "repos/${OWNER}/${REPO}/pulls" \
--field "title=${title}" \
--field "body=${PR_BODY}" \
--field "head=${branch}" \
--field "base=${BASE}" > /dev/null

echo "=== Done: ${title}"
}

create_pr "test/read-owned" \
"H1: read-bot approves owned file (main.go)" \
"main.go" "${MAIN_GO_CONTENT}"

create_pr "test/write-owned" \
"H2: write-bot approves owned file (main.go)" \
"main.go" "${MAIN_GO_CONTENT}"

create_pr "test/read-blank" \
"H1+H3: read-bot approves blank-owner file (go.mod)" \
"go.mod" "${GO_MOD_CONTENT}"

create_pr "test/write-blank" \
"H2+H3+H5: write-bot approves blank-owner file (go.mod)" \
"go.mod" "${GO_MOD_CONTENT}"

create_pr "test/read-mixed" \
"H1+H4: read-bot approves mixed files (go.mod + main.go)" \
"go.mod" "${GO_MOD_CONTENT}" \
"main.go" "${MAIN_GO_CONTENT}"

create_pr "test/write-mixed" \
"H2+H4+H6: write-bot approves mixed files (go.mod + main.go)" \
"go.mod" "${GO_MOD_CONTENT}" \
"main.go" "${MAIN_GO_CONTENT}"

echo ""
echo "=== Open PRs:"
gh api "repos/${OWNER}/${REPO}/pulls?state=open" --jq '.[] | "#\(.number) \(.title)"'
3 changes: 3 additions & 0 deletions 0024-codeowners-app-matrix/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
PyJWT>=2.8,<3
cryptography>=42,<45
requests>=2.31,<3
Loading
Loading