Skip to content

feat(triage): add not-planned action to close out-of-scope issues #54

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rh-hemartin wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat(triage): add not-planned action to close out-of-scope issues #54

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions agents/triage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -175,6 +175,29 @@ When you identify a question, attempt to answer it using the repository context
}
```

### Action: `not-planned`

The issue is out-of-scope, invalid, spam, or should otherwise not be worked on. This action closes the issue with reason `not planned`.

@waynesun09 waynesun09 Jul 10, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[medium] missing-guardrail

not-planned is the only destructive, hard-to-undo triage outcome (it closes a reporter's issue), yet it's the only action without a HARD CONSTRAINT rule integrating it into the decision framework — sufficient, insufficient, prerequisites, and question all have explicit anti-bypass rules (lines 58, 149, 151, 153). Two gaps: (1) nothing biases the agent toward caution when out-of-scope status is ambiguous — the "technical direction counter to documented architectural decisions" criterion is subjective and easy to over-apply to ambitious-but-legitimate feature requests; (2) nothing settles priority against sufficient/triaged for a clearly-written but out-of-scope request, or against insufficient for an out-of-scope issue that is also vague.

Suggested fix: Add a HARD CONSTRAINT, e.g.: "When uncertain whether an issue is truly out of scope, prefer insufficient (ask) or sufficient/triaged (let a human decide) over not-planned. Only use not-planned when out-of-scope/invalid/spam status is unambiguous, and cite the specific scope doc or ADR in the comment when closing on scope grounds."


Use this action for:
- Issues explicitly out of project scope (e.g., feature requests incompatible with project goals)
- Invalid reports (e.g., user error, misconfiguration, or misunderstanding)
- Spam or low-quality submissions
- Issues that would introduce technical direction counter to documented architectural decisions

**Do NOT use this action for:**
- Issues that lack information (use `insufficient` instead)
- Issues that are duplicates of existing ones (use `duplicate` instead)
- Issues blocked on other work (use `prerequisites` instead)

```json
{
"action": "not-planned",
"reasoning": "Brief explanation of why this issue is being closed as not planned",
"comment": "A professional comment explaining why the issue is out of scope or invalid. Be respectful — the reporter may not have understood project boundaries. Link to relevant documentation or related discussions when applicable."
}
```

### Action: `insufficient`

Information is missing that would change the triage outcome. Ask ONE focused, specific clarifying question.
Expand Down
3 changes: 2 additions & 1 deletion docs/triage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ outcome and the post-script applies the corresponding label.
| `triaged` | The issue is fully specified but is a feature or other category that requires human prioritization before coding. |
| `duplicate` | The issue duplicates an existing one. The agent identified the original and the post-script closes the issue. |
| `blocked` | The issue depends on another issue or external condition. The agent identified the blocker. |
| `not-planned` | The issue is out of scope, invalid, or spam. The post-script closes the issue with reason "not planned". |

The `issue-labels` skill may also apply contextual labels (e.g., `area/api`,
`kind/bug`) but these are informational — they do not control agent behavior.
Expand Down Expand Up @@ -79,7 +80,7 @@ invent labels or apply labels not listed here.
## Control labels (never recommend these)

These are managed by the triage pipeline. Never include them in `label_actions`:
`needs-info`, `ready-to-code`, `duplicate`, `blocked`, `triaged`.
`needs-info`, `ready-to-code`, `duplicate`, `blocked`, `triaged`, `not-planned`.

## Area labels

Expand Down
2 changes: 1 addition & 1 deletion schemas/triage-result.schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
"properties": {
"action": {
"type": "string",
"enum": ["insufficient", "duplicate", "sufficient", "prerequisites", "question"]
"enum": ["insufficient", "duplicate", "sufficient", "prerequisites", "question", "not-planned"]
},
"reasoning": {
"type": "string",
Expand Down
25 changes: 25 additions & 0 deletions scripts/post-triage-test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -279,6 +279,31 @@ run_test "question-missing-comment-fails" \
"" \
"true"

run_test "not-planned-posts-comment" \
'{"action":"not-planned","reasoning":"out of scope","comment":"This request is out of scope for the project goals. See docs/scope.md for more details."}' \
"gh issue comment 42 --repo test-org/test-repo --body-file -"

@waynesun09 waynesun09 Jul 10, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[medium] eval-coverage

These tests thoroughly cover the post-script mechanics, but no eval/triage/cases/ fixture covers the agent's decision to choose not-planned. Every other consequential outcome has an eval case guarding decision quality (001-bug-url-encoding, 002-needs-info-vague-crash, 003-feature-request, 004-duplicate-issue), and not-planned is the highest-blast-radius action this agent can take — auto-closing a reporter's issue on subjective criteria — with no regression guard against the model over-applying it. The only other closing action (duplicate) is far more objective, since it must identify a specific existing issue.

Suggested fix: Add an eval case for a clearly out-of-scope issue expecting not-planned (with labels.forbidden mirroring 004-duplicate-issue), and ideally a negative case — an ambitious-but-legitimate feature request that must NOT be closed as not-planned.


run_test "not-planned-applies-label" \
'{"action":"not-planned","reasoning":"out of scope","comment":"This request is out of scope for the project goals."}' \
"gh api repos/test-org/test-repo/issues/42/labels -f labels[]=not-planned --silent"

run_test "not-planned-removes-blocked-label" \
'{"action":"not-planned","reasoning":"out of scope","comment":"This request is out of scope."}' \
"gh api repos/test-org/test-repo/issues/42/labels/blocked -X DELETE --silent"

run_test "not-planned-removes-needs-info-label" \
'{"action":"not-planned","reasoning":"out of scope","comment":"This request is out of scope."}' \
"gh api repos/test-org/test-repo/issues/42/labels/needs-info -X DELETE --silent"

run_test "not-planned-closes-issue" \
'{"action":"not-planned","reasoning":"out of scope","comment":"This request is out of scope for the project goals."}' \
"gh issue close 42 --repo test-org/test-repo --reason not planned"

run_test "not-planned-missing-comment-fails" \
'{"action":"not-planned","reasoning":"out of scope"}' \
"" \
"true"

run_test_stdout "question-control-label-refused" \
'{"action":"question","reasoning":"issue is asking a question","comment":"Answer here.","label_actions":{"reason":"Tried to set question label.","actions":[{"action":"add","label":"question"}]}}' \
"::warning::Refused to add control label 'question' -- control labels are managed by the triage pipeline"
Expand Down
18 changes: 16 additions & 2 deletions scripts/post-triage.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ remove_label() {
# add or remove these via label_actions. This list covers labels that the
# pipeline itself applies (pre-triage.sh resets the first five; the action
# handlers apply blocked/triaged/feature).
CONTROL_LABELS=("needs-info" "ready-to-code" "duplicate" "feature" "blocked" "triaged" "question")
CONTROL_LABELS=("needs-info" "ready-to-code" "duplicate" "feature" "blocked" "triaged" "question" "not-planned")
Comment thread
qodo-code-review[bot] marked this conversation as resolved.

is_control_label() {
local label="$1"
Expand Down Expand Up @@ -341,6 +341,16 @@ ${FAILED_CREATES}"
add_label "question"
;;

not-planned)
if [[ -z "${COMMENT}" ]]; then
echo "ERROR: action is 'not-planned' but no comment provided" >&2
exit 1
fi
remove_label "blocked"
remove_label "needs-info"
add_label "not-planned"
;;
Comment thread
qodo-code-review[bot] marked this conversation as resolved.

Comment on lines +344 to +353

@qodo-code-review qodo-code-review Bot Jul 8, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

2. scripts/ and agents/ changed 📜 Skill insight § Compliance

This PR modifies protected governance/infrastructure paths (agents/ and scripts/), which require
explicit human review and must not be auto-approved. A compliance finding is required whenever
protected paths are changed.

*)
echo "ERROR: unknown action '${ACTION}' — this may be a newer action that post-triage.sh does not handle yet" >&2
exit 1
Expand Down Expand Up @@ -433,10 +443,14 @@ else
printf '%s' "${COMMENT}" | gh issue comment "${ISSUE_NUMBER}" --repo "${REPO}" --body-file -
fi

# --- Post-action: close duplicate issues ---
# --- Post-action: close issues ---

if [[ "${ACTION}" == "duplicate" ]]; then
gh issue close "${ISSUE_NUMBER}" --repo "${REPO}" --reason "duplicate"
fi

if [[ "${ACTION}" == "not-planned" ]]; then
gh issue close "${ISSUE_NUMBER}" --repo "${REPO}" --reason "not planned"
fi

echo "Post-triage complete."
4 changes: 2 additions & 2 deletions scripts/pre-triage.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,13 +27,13 @@ ISSUE_NUMBER=$(basename "${GITHUB_ISSUE_URL}")

echo "Resetting triage labels on ${REPO}#${ISSUE_NUMBER}"

for label in needs-info ready-to-code duplicate feature question; do
for label in needs-info ready-to-code duplicate feature question not-planned; do
gh api "repos/${REPO}/issues/${ISSUE_NUMBER}/labels/${label}" -X DELETE --silent 2>/dev/null || true
done

# Verify no triage labels remain — the pipeline depends on mutual exclusivity.
REMAINING=$(gh api "repos/${REPO}/issues/${ISSUE_NUMBER}/labels" \
--jq '[.[] | select(.name == "needs-info" or .name == "ready-to-code" or .name == "duplicate" or .name == "feature" or .name == "question") | .name] | join(", ")' 2>/dev/null || echo "VERIFY_FAILED")
--jq '[.[] | select(.name == "needs-info" or .name == "ready-to-code" or .name == "duplicate" or .name == "feature" or .name == "question" or .name == "not-planned") | .name] | join(", ")' 2>/dev/null || echo "VERIFY_FAILED")

if [[ "${REMAINING}" == "VERIFY_FAILED" ]]; then
echo "ERROR: cannot verify label state — API call failed"
Expand Down
4 changes: 4 additions & 0 deletions scripts/validate-output-schema-test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,10 @@ run_test "valid-question" \
'{"action":"question","reasoning":"this is a support question","comment":"Based on the docs, Python 4 is not supported. Would you like to open a feature request?"}' \
"true"

run_test "valid-not-planned" \
'{"action":"not-planned","reasoning":"out of scope","comment":"This is out of scope."}' \
"true"

run_test "valid-prerequisites-existing" \
'{"action":"prerequisites","reasoning":"upstream dependency","prerequisites":{"existing":[{"url":"https://github.com/org/repo/issues/99"}],"create":[]},"comment":"Blocked on upstream."}' \
"true"
Expand Down
Loading