feat(triage): add effort-based gating before auto-promoting to coder#36
feat(triage): add effort-based gating before auto-promoting to coder#36rh-hemartin wants to merge 2 commits into
Conversation
|
🤖 Review · |
PR Summary by QodoAdd effort scoring and auto-promotion gating to triage pipeline
AI Description
Diagram
High-Level Assessment
Files changed (4)
|
Code Review by Qodo
Context used✅ Compliance rules (platform):
55 rules 1. Protected scripts/ files modified
|
![qodo-code-review[bot]](https://avatars.githubusercontent.com/in/484649?s=60&v=4){kind=small}
| # ready-to-code, which triggers the code agent. Feature work and anything | ||
| # else receives the triaged label and waits for human prioritization | ||
| # (per #561, only feature issues should require human review before coding). | ||
| # | ||
| # Effort-based gating (#2207): high-effort bug/docs/performance issues | ||
| # (effort >= 2.0) get triaged instead of ready-to-code to route them through | ||
| # human review before dispatching to the code agent. | ||
| CATEGORY=$(jq -r '.triage_summary.category // "unknown"' "${RESULT_FILE}") | ||
| echo "Category: ${CATEGORY}" | ||
| EFFORT=$(jq -r '.triage_summary.effort // "null"' "${RESULT_FILE}") | ||
| echo "Category: ${CATEGORY}, Effort: ${EFFORT}" | ||
| case "${CATEGORY}" in | ||
| bug|documentation|performance) | ||
| echo "Deferring ready-to-code label (${CATEGORY}) until after label_actions..." | ||
| DEFERRED_LABEL="ready-to-code" | ||
| # Effort threshold: >= 2.0 is substantial work, route to human review. | ||
| # Missing effort → conservatively route to human review (triaged). | ||
| if [[ "${EFFORT}" == "null" ]]; then | ||
| echo "Missing effort estimate — applying triaged label for human review..." | ||
| remove_label "ready-to-code" | ||
| add_label "triaged" | ||
| elif awk -v effort="${EFFORT}" 'BEGIN { exit (effort >= 2.0 ? 0 : 1) }'; then | ||
| echo "High effort (${EFFORT}) — applying triaged label for human review..." | ||
| remove_label "ready-to-code" | ||
| add_label "triaged" | ||
| else | ||
| echo "Low effort (${EFFORT}) — deferring ready-to-code label until after label_actions..." | ||
| DEFERRED_LABEL="ready-to-code" | ||
| fi |
There was a problem hiding this comment.
1. Protected scripts/ files modified 📜 Skill insight § Compliance
This PR modifies files under protected governance/infrastructure paths (e.g., scripts/ and agents/), which require explicit human review and must not be auto-approved. Protected-path changes increase governance and supply-chain risk if merged without manual oversight.
| # ready-to-code, which triggers the code agent. Feature work and anything | ||
| # else receives the triaged label and waits for human prioritization | ||
| # (per #561, only feature issues should require human review before coding). | ||
| # | ||
| # Effort-based gating (#2207): high-effort bug/docs/performance issues | ||
| # (effort >= 2.0) get triaged instead of ready-to-code to route them through | ||
| # human review before dispatching to the code agent. | ||
| CATEGORY=$(jq -r '.triage_summary.category // "unknown"' "${RESULT_FILE}") | ||
| echo "Category: ${CATEGORY}" | ||
| EFFORT=$(jq -r '.triage_summary.effort // "null"' "${RESULT_FILE}") | ||
| echo "Category: ${CATEGORY}, Effort: ${EFFORT}" | ||
| case "${CATEGORY}" in | ||
| bug|documentation|performance) | ||
| echo "Deferring ready-to-code label (${CATEGORY}) until after label_actions..." | ||
| DEFERRED_LABEL="ready-to-code" | ||
| # Effort threshold: >= 2.0 is substantial work, route to human review. | ||
| # Missing effort → conservatively route to human review (triaged). | ||
| if [[ "${EFFORT}" == "null" ]]; then | ||
| echo "Missing effort estimate — applying triaged label for human review..." | ||
| remove_label "ready-to-code" | ||
| add_label "triaged" | ||
| elif awk -v effort="${EFFORT}" 'BEGIN { exit (effort >= 2.0 ? 0 : 1) }'; then | ||
| echo "High effort (${EFFORT}) — applying triaged label for human review..." | ||
| remove_label "ready-to-code" | ||
| add_label "triaged" | ||
| else | ||
| echo "Low effort (${EFFORT}) — deferring ready-to-code label until after label_actions..." | ||
| DEFERRED_LABEL="ready-to-code" | ||
| fi |
There was a problem hiding this comment.
2. Docs contradict effort gating 📜 Skill insight ⚙ Maintainability
The triage documentation still states that ready-to-code applies to bug/documentation/performance issues as a category-level rule, but the post-triage logic now routes high-effort (and missing-effort) items to triaged. This mismatch can mislead maintainers and auditors about when auto-promotion to coder actually occurs.
Agent Prompt
## Issue description
`docs/triage.md` describes `ready-to-code`/`triaged` semantics that no longer match the updated post-triage behavior (effort-based gating for bug/docs/performance).
## Issue Context
The post-triage script now applies `triaged` (human review) for bug/documentation/performance items when `effort >= 2.0` or when `effort` is missing, instead of always applying `ready-to-code` for these categories.
## Fix Focus Areas
- docs/triage.md[32-44]
- scripts/post-triage.sh[262-288]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
| EFFORT=$(jq -r '.triage_summary.effort // "null"' "${RESULT_FILE}") | ||
| echo "Category: ${CATEGORY}, Effort: ${EFFORT}" | ||
| case "${CATEGORY}" in | ||
| bug|documentation|performance) | ||
| echo "Deferring ready-to-code label (${CATEGORY}) until after label_actions..." | ||
| DEFERRED_LABEL="ready-to-code" | ||
| # Effort threshold: >= 2.0 is substantial work, route to human review. | ||
| # Missing effort → conservatively route to human review (triaged). | ||
| if [[ "${EFFORT}" == "null" ]]; then | ||
| echo "Missing effort estimate — applying triaged label for human review..." | ||
| remove_label "ready-to-code" | ||
| add_label "triaged" | ||
| elif awk -v effort="${EFFORT}" 'BEGIN { exit (effort >= 2.0 ? 0 : 1) }'; then | ||
| echo "High effort (${EFFORT}) — applying triaged label for human review..." | ||
| remove_label "ready-to-code" | ||
| add_label "triaged" | ||
| else | ||
| echo "Low effort (${EFFORT}) — deferring ready-to-code label until after label_actions..." | ||
| DEFERRED_LABEL="ready-to-code" | ||
| fi |
There was a problem hiding this comment.
3. Missing effort blocks promotion 🐞 Bug ≡ Correctness
scripts/post-triage.sh routes bug/documentation/performance issues to triaged when triage_summary.effort is missing, which can suppress auto-promotion whenever the agent omits the field. Because effort is optional in schemas/triage-result.schema.json, these results still validate, making this behavior easy to trigger unintentionally.
Agent Prompt
## Issue description
The post-triage gating logic treats missing `triage_summary.effort` as a reason to apply the `triaged` label for bug/documentation/performance categories, but the triage result schema does not require `effort`. This means valid (schema-passing) agent outputs can unexpectedly disable auto-promotion.
## Issue Context
- Triage prompt guidance says to estimate effort for bug/docs/performance and to bias toward auto-promotion (round down), but the implementation routes *missing* effort to human review.
- The harness validates against `schemas/triage-result.schema.json` before running `post-triage.sh`, so schema constraints are the right place to prevent missing effort from reaching the gating logic (or the script should default missing effort to a low-effort value).
## Fix Focus Areas
- scripts/post-triage.sh[270-288]
- schemas/triage-result.schema.json[121-144]
- agents/triage.md[247-263]
## What to change
Choose one consistent policy and implement it end-to-end:
1) **Require effort for bug/docs/performance when action=sufficient**
- Update `schemas/triage-result.schema.json` with an `if/then` that, when `action == "sufficient"` and `triage_summary.category` is in `{bug, documentation, performance}`, requires `triage_summary.effort`.
- In `post-triage.sh`, replace the “missing effort => triaged” branch with a hard error (since it should be schema-impossible) or keep a defensive error in case the script is run without schema validation.
OR
2) **Default missing effort to low effort** (to preserve “bias toward auto-promotion”)
- In `post-triage.sh`, treat missing effort as low effort (e.g., default to `1.0` and defer `ready-to-code`) and log a warning.
- Optionally keep schema optional, but consider adding an agent-side requirement if you want stronger guarantees.
Also ensure the written guidance in `agents/triage.md` matches the chosen behavior (especially the handling of missing effort).
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
|
🤖 Finished Review · ✅ Success · Started 6:35 AM UTC · Completed 6:44 AM UTC |
ReviewVerdict: approve · 0 medium, 5 low findings The new head (8408ffc) is a force-push from b2b78ae that addresses all three prior medium findings:
The effort-gating logic is correctly implemented: Findings1. No test for effort just below threshold · low · test-adequacyFile: The test suite covers effort=0.25, 1.0 (below threshold) and effort=2.0, 2.5, 3.0 (at/above threshold), but no test verifies effort=1.5 or 1.99 routes to Remediation: Add a test case with effort=1.5 or 1.99 asserting 2. Test fixtures for non-applicable categories include effort values · low · test-integrityFile: The Remediation: Remove effort from feature/other fixtures, or add a comment explaining they verify effort is ignored for non-applicable categories. 3. Schema validation tests don't cover conditional effort requirement · low · test-adequacyFile: The schema validation test suite has one Remediation: Add schema validation test cases: a bug category without effort that fails, and a feature category without effort that passes. 4. No linked issue for non-trivial feature change · low · authorizationFile: The PR references #2207 in a code comment but no issue URL is linked in the PR metadata. The PR body states it was "Migrated from fullsend-ai/fullsend feat/triage-effort-analysis", providing implicit authorization. 5. Triage effort estimation overlaps with prioritize agent's RICE effort · low · architectural-coherenceFile: The triage agent now estimates effort (0.25–3 scale) matching the prioritize agent's RICE effort dimension. Triage effort gates auto-promotion (< 2.0 → Previous runReviewVerdict: comment · 3 medium, 5 low findings The new head (b2b78ae) is a force-push from the previously-reviewed 8e5ff3f. The effort-gating code across all four files is functionally identical to the prior review. All eight prior findings remain unaddressed. The effort-gating logic is correctly implemented: Findings1. GHA workflow command injection via unsanitized EFFORT interpolation · medium · securityFile: Two interpolation points expose unsanitized
Note: This is defense-in-depth — schema validation ( Remediation: Sanitize EFFORT before all SAFE_EFFORT="${EFFORT//::/}"
SAFE_EFFORT="${SAFE_EFFORT//%0A/}"
SAFE_EFFORT="${SAFE_EFFORT//%0a/}"
SAFE_EFFORT="${SAFE_EFFORT//%0D/}"
SAFE_EFFORT="${SAFE_EFFORT//%0d/}"2. Documentation now contradicts actual behavior —
|
fullsend-ai-review
Bot
added
the
requires-manual-review
waynesun09
left a comment
waynesun09
left a comment
There was a problem hiding this comment.
Multi-agent review squad (5 agents: Claude, Gemini, Codex) findings — Medium and above, deduplicated and verified against the head branch. Two additional Medium findings below couldn't be attached inline since they target files/state outside this PR's diff:
[MEDIUM] docs/triage.md control-label documentation is now stale — docs/triage.md:40-41 (not touched by this PR) still describes ready-to-code as "low-risk (bug, documentation, performance)" and triaged as "feature or other category," with no mention of the new effort threshold. docs/code.md:41 has the same staleness. Already flagged independently by the qodo-code-review bot on this PR ("Docs contradict effort gating").
[MEDIUM] Branch is 24 commits behind main; CI test suite has never run against it — git compare main...feat/triage-effort-analysis shows the branch diverged and is missing main's 88800b2 (adds a label-contradiction guard to the same sufficient) case block this PR modifies) plus the Makefile/script-test.yml CI infra added after this branch was cut. No script-test check has run on this PR — only dispatch/dispatch and DCO. Suggest rebasing onto current main before merge and confirming the effort-gating logic composes cleanly with the label-contradiction guard.
Four inline comments above cover the remaining High/Medium findings (stale triaged label bug, effort-scale mismatch with prioritize.md, optional-but-load-bearing effort schema field, and the round-down bias prompt-injection surface).
| add_label "triaged" | ||
| else | ||
| echo "Low effort (${EFFORT}) — deferring ready-to-code label until after label_actions..." | ||
| DEFERRED_LABEL="ready-to-code" |
There was a problem hiding this comment.
[HIGH] Stale triaged label not removed when re-triage transitions to low effort
The high-effort/missing/non-numeric branches all call remove_label "ready-to-code" before add_label "triaged". This low-effort else branch only sets DEFERRED_LABEL="ready-to-code" — it never calls remove_label "triaged".
docs/triage.md confirms triage re-runs automatically when an issue is edited, and scripts/pre-triage.sh's label reset list (needs-info ready-to-code duplicate feature question) does not include triaged. So this sequence is reachable: issue triaged high-effort → gets triaged → issue edited/re-triaged → LLM now estimates low effort → issue ends up with both triaged and ready-to-code simultaneously, violating the mutual-exclusivity invariant that pre-triage.sh explicitly documents.
Suggested fix: add remove_label "triaged" in this branch (or at the top of the sufficient handler, symmetric with the existing remove_label "blocked" / remove_label "needs-info" calls).
| | 2.0 | Substantial: touches multiple files/layers, architectural consideration, thorough test coverage | | ||
| | 3.0 | Large: multi-file refactor, performance work spanning subsystems, complex integration | | ||
|
|
||
| Higher effort issues (effort >= 2.0) get routed to human review instead of auto-promoting to the coder — similar to how feature issues already require human prioritization. When in doubt between two scores, round down (bias toward auto-promotion). Only estimate effort for bug/docs/performance categories; feature issues already gate on human review. |
There was a problem hiding this comment.
[HIGH] Effort scale doesn't actually match the prioritize agent's RICE effort scale it claims to mirror
agents/prioritize.md's Effort dimension defines 6 tiers (0.25, 0.5, 1, 1.5, 2, 3); this new table has only 5 (0.25, 0.5, 1.0, 2.0, 3.0), omitting 1.5 entirely — which sits directly below the 2.0 gating threshold, the most consequential place to have ambiguity. Descriptions at equivalent levels also diverge (e.g., prioritize's 1.0 = "touches a few files" vs this table's 1.0 = "multi-function change, local refactor").
This isn't cosmetic — it undermines the stated goal of a shared scale between the two agents and creates scoring drift for the same issue evaluated by both (relevant since a triaged issue can later be scored again by the prioritize agent for backlog ranking).
Suggested fix: copy the exact prioritize.md table into triage.md, or explicitly document why triage intentionally uses a coarser scale.
| "recommended_fix": { "type": "string", "minLength": 1 }, | ||
| "proposed_test_case": { "type": "string", "minLength": 1 } | ||
| "proposed_test_case": { "type": "string", "minLength": 1 }, | ||
| "effort": { |
There was a problem hiding this comment.
[MEDIUM] effort is optional while the post-script treats it as load-bearing — silent-failure risk
effort isn't in triage_summary's required array, and no conditional (if/then) requires it for bug/docs/performance categories the way other conditionals in this schema do elsewhere. Since the harness's validation_loop only validates against this schema, a model that silently stops emitting effort would never fail validation — every bug/docs/performance issue would quietly start routing to triaged in post-triage.sh with no alarm raised anywhere.
(Also flagged by the qodo-code-review bot on this PR as "Missing effort blocks promotion".)
Suggested fix: add a conditional schema requirement for effort on bug/docs/performance categories when action: sufficient, or default missing effort to a low value in the script to match the prompt's "bias toward auto-promotion" guidance.
| | 2.0 | Substantial: touches multiple files/layers, architectural consideration, thorough test coverage | | ||
| | 3.0 | Large: multi-file refactor, performance work spanning subsystems, complex integration | | ||
|
|
||
| Higher effort issues (effort >= 2.0) get routed to human review instead of auto-promoting to the coder — similar to how feature issues already require human prioritization. When in doubt between two scores, round down (bias toward auto-promotion). Only estimate effort for bug/docs/performance categories; feature issues already gate on human review. |
There was a problem hiding this comment.
[MEDIUM] Round-down bias instruction is an exploitable prompt-injection surface
"When in doubt between two scores, round down (bias toward auto-promotion)" gives an issue reporter a structural incentive: framing a complex change as trivial ("just a one-line fix") nudges the LLM toward a score that bypasses the human-review gate. This is an easier attack than manipulating category alone, since it only requires making the work sound simple rather than misrepresenting its nature.
It's also inconsistent with the schema's own conservative default — missing effort routes to human review (a round-up bias) — which reinforces that this is a real inconsistency rather than a stylistic nit.
Suggested fix: remove the round-down bias, or invert it to bias toward human review when uncertain, matching the missing-effort fallback behavior.
ralphbean
force-pushed
the
feat/triage-effort-analysis
branch
from
c51787d to
24b4069
Compare
|
🤖 Finished Review · ✅ Success · Started 6:26 PM UTC · Completed 6:35 PM UTC |
fullsend-ai-review
Bot
added
requires-manual-review
ralphbean
force-pushed
the
feat/triage-effort-analysis
branch
from
24b4069 to
2856f9d
Compare
|
🤖 Finished Review · ✅ Success · Started 6:38 PM UTC · Completed 6:51 PM UTC |
fullsend-ai-review
Bot
added
requires-manual-review
ralphbean
force-pushed
the
feat/triage-effort-analysis
branch
from
2856f9d to
60d8500
Compare
|
🤖 Finished Review · ✅ Success · Started 7:13 PM UTC · Completed 7:28 PM UTC |
fullsend-ai-review
Bot
added
requires-manual-review
|
🤖 Review · |
|
🤖 Review · |
|
🤖 Finished Review · ✅ Success · Started 8:02 PM UTC · Completed 8:11 PM UTC |
fullsend-ai-review
Bot
added
requires-manual-review
ralphbean
force-pushed
the
feat/triage-effort-analysis
branch
from
213c3b9 to
39c1f2a
Compare
|
🤖 Review · |
ralphbean
force-pushed
the
feat/triage-effort-analysis
branch
from
39c1f2a to
8e5ff3f
Compare
|
🤖 Finished Review · ✅ Success · Started 8:58 PM UTC · Completed 9:07 PM UTC |
fullsend-ai-review
Bot
added
requires-manual-review
waynesun09
left a comment
waynesun09
left a comment
There was a problem hiding this comment.
One additional Medium finding from this review round (deduplicated against prior comments — the four High/Medium findings already posted on 2026-07-08 remain open and unaddressed by the latest commit, which only fixed the non-numeric-effort fallback, added tests, and corrected a schema description).
| elif awk -v effort="${EFFORT}" 'BEGIN { exit (effort >= 2.0 ? 0 : 1) }'; then | ||
| echo "High effort (${EFFORT}) — applying triaged label for human review..." | ||
| remove_label "ready-to-code" | ||
| add_label "triaged" |
There was a problem hiding this comment.
[MEDIUM] High-effort triaged is applied immediately, bypassing the deferral mechanism built for exactly this ordering problem
ready-to-code is deferred via DEFERRED_LABEL and applied after label_actions specifically to stop its webhook event from being superseded by later label mutations in the same run (per #1752, documented above at the DEFERRED_LABEL declaration and in the final if [[ -n "${DEFERRED_LABEL}" ]] block). The three new/modified branches here (missing effort, non-numeric effort, high effort) all call add_label "triaged" immediately instead of deferring it — so if label_actions adds more labels afterward in the same run, the triaged labeled-event fires first and is exposed to the same supersession risk #1752 was written to eliminate for ready-to-code.
This mirrors the pre-existing feature branch's behavior (also immediate, not deferred), so it's not a new pattern in this codebase — but this PR is the first place bug/documentation/performance issues can land on triaged too, and it's worth an explicit decision rather than inheriting the immediate-apply pattern by default.
Suggested fix: if any downstream automation reacts to the triaged label event the same way it reacts to ready-to-code (e.g., a human-review dashboard or notification), defer it the same way (DEFERRED_LABEL="triaged"). Otherwise, add a one-line comment here noting why triaged doesn't need the same ordering guarantee, so a future reader doesn't have to re-derive it.
Add effort estimation to triage agent (0.25-3 scale matching RICE). Bug/docs/performance issues with effort >= 2.0 get triaged instead of ready-to-code, routing high-complexity work through human review before dispatch. Migrated from fullsend-ai/fullsend feat/triage-effort-analysis. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Hector Martinez <hemartin@redhat.com>
rh-hemartin
force-pushed
the
feat/triage-effort-analysis
branch
from
8e5ff3f to
b2b78ae
Compare
|
🤖 Finished Review · ✅ Success · Started 8:08 AM UTC · Completed 8:17 AM UTC |
fullsend-ai-review
Bot
added
requires-manual-review
- Align effort scale with prioritize agent (6 tiers: 0.25–3) - Bias toward human review (round up) instead of auto-promotion - Sanitize effort value in GHA log output to prevent command injection - Use DEFERRED_LABEL for triaged label to prevent webhook supersession - Remove stale triaged label on re-triage of sufficient issues - Require effort field in schema for bug/docs/performance categories - Update docs to reflect effort-based ready-to-code vs triaged routing Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Hector Martinez <hemartin@redhat.com>
rh-hemartin
force-pushed
the
feat/triage-effort-analysis
branch
from
b2b78ae to
8408ffc
Compare
|
🤖 Finished Review · ✅ Success · Started 8:44 AM UTC · Completed 8:59 AM UTC |
fullsend-ai-review
Bot
added
requires-manual-review
waynesun09
left a comment
waynesun09
left a comment
There was a problem hiding this comment.
Multi-agent review squad (4 agents: Claude, Gemini, Codex) — new findings only, deduplicated against the 8 existing review threads.
[HIGH] Two label-category test fixtures missing effort field silently exercise wrong routing path (scripts/post-triage-test.sh:477-484)
The label-category-contradiction-stripped (category: documentation) and label-category-consistent-passes (category: bug) fixtures were not updated with the effort field. After this PR, missing effort for bug/docs/performance routes to triaged instead of ready-to-code. Both tests still pass because their assertions check label_actions side-effects (warning messages, area labels) rather than the routing label — but they now silently exercise the missing-effort fallback instead of the normal low-effort path.
Suggested fix: add "effort":1.0 to both fixtures, consistent with every other sufficient/bug fixture updated in this PR.
Three additional MEDIUM findings posted inline (schema then clause fragility, no negative schema validation test, effort=2.0 boundary test gap).
| } | ||
| } | ||
| } | ||
| }, |
There was a problem hiding this comment.
[MEDIUM] Schema then clause relies on sibling allOf entry — fragile coupling
This then clause uses properties.triage_summary.required: ["effort"] but doesn't itself declare required: ["triage_summary"] at the root. It works today because allOf[2] already requires triage_summary when action=sufficient. But if someone reorders or removes that entry, this conditional silently stops enforcing anything.
Suggested fix — add defensive required at the then root:
"then": {
"required": ["triage_summary"],
"properties": {
"triage_summary": {
"required": ["effort"]
}
}
}| run_test "valid-sufficient" \ | ||
| '{"action":"sufficient","reasoning":"clear","clarity_scores":{"symptom":0.9,"cause":0.8,"reproduction":0.9,"impact":0.7,"overall":0.85},"triage_summary":{"title":"Bug","severity":"high","category":"bug","problem":"crash","root_cause_hypothesis":"null ptr","reproduction_steps":["step 1"],"impact":"all users","recommended_fix":"fix ptr","proposed_test_case":"test_fix"},"comment":"Triage complete."}' \ | ||
| '{"action":"sufficient","reasoning":"clear","clarity_scores":{"symptom":0.9,"cause":0.8,"reproduction":0.9,"impact":0.7,"overall":0.85},"triage_summary":{"title":"Bug","severity":"high","category":"bug","problem":"crash","root_cause_hypothesis":"null ptr","reproduction_steps":["step 1"],"impact":"all users","recommended_fix":"fix ptr","proposed_test_case":"test_fix","effort":1.0},"comment":"Triage complete."}' \ | ||
| "true" |
There was a problem hiding this comment.
[MEDIUM] No negative test for conditional effort requirement
The schema now conditionally requires effort for bug/docs/performance via the new allOf if/then rule. This test was updated to include effort:1.0 so it passes — but nothing verifies that omitting effort for these categories actually fails validation. If the conditional rule were broken or removed, no test would catch it.
Suggested fix — add two tests:
# Negative: sufficient bug without effort → should fail schema validation
run_test "sufficient-bug-missing-effort-rejected" \
'{"action":"sufficient","reasoning":"clear","clarity_scores":{"symptom":0.9,"cause":0.8,"reproduction":0.9,"impact":0.7,"overall":0.85},"triage_summary":{"title":"Bug","severity":"high","category":"bug","problem":"crash","root_cause_hypothesis":"null ptr","reproduction_steps":["step 1"],"impact":"all users","recommended_fix":"fix ptr","proposed_test_case":"test_fix"},"comment":"Triage complete."}' \
"false"
# Positive: sufficient feature without effort → should pass
run_test "sufficient-feature-no-effort-accepted" \
'{"action":"sufficient","reasoning":"clear","clarity_scores":{"symptom":0.9,"cause":0.8,"reproduction":0.9,"impact":0.7,"overall":0.85},"triage_summary":{"title":"Feature","severity":"medium","category":"feature","problem":"missing","root_cause_hypothesis":"not built","reproduction_steps":["step 1"],"impact":"users","recommended_fix":"build it","proposed_test_case":"test_it"},"comment":"Feature."}' \
"true"|
|
||
| run_test "sufficient-bug-high-effort-gets-triaged" \ | ||
| '{"action":"sufficient","reasoning":"all clear","clarity_scores":{"symptom":0.9,"cause":0.85,"reproduction":0.9,"impact":0.8,"overall":0.87},"triage_summary":{"title":"Refactor auth middleware","severity":"high","category":"bug","problem":"Auth middleware incorrectly handles edge cases","root_cause_hypothesis":"Architectural issue","reproduction_steps":["step 1"],"environment":"Linux","impact":"All users","recommended_fix":"Refactor auth middleware","proposed_test_case":"test_auth_edge_cases","effort":2.5},"comment":"## Triage Summary\n\nSubstantial refactor needed."}' \ | ||
| "gh api repos/test-org/test-repo/issues/42/labels -f labels[]=triaged --silent" |
There was a problem hiding this comment.
[MEDIUM] Missing effort=2.0 boundary test for bug category
The test suite covers effort=0.25, 1.0, 2.5, and 3.0 for bugs, and effort=2.0 for documentation (line 189), but doesn't test the exact boundary value of effort=2.0 for bug. Since the threshold comparison is effort >= 2.0, a bug-specific boundary test would strengthen coverage.
Suggested fix — add after this test:
run_test "sufficient-bug-boundary-effort-gets-triaged" \
'{"action":"sufficient","reasoning":"all clear","clarity_scores":{"symptom":0.9,"cause":0.85,"reproduction":0.9,"impact":0.8,"overall":0.87},"triage_summary":{"title":"Fix complex auth bug","severity":"high","category":"bug","problem":"Auth edge cases","root_cause_hypothesis":"Architectural issue","reproduction_steps":["step 1"],"environment":"Linux","impact":"All users","recommended_fix":"Refactor auth","proposed_test_case":"test_auth","effort":2.0},"comment":"## Triage Summary\n\nBoundary effort."}' \
"gh api repos/test-org/test-repo/issues/42/labels -f labels[]=triaged --silent"2 participants
Summary
effortfieldMigrated from fullsend-ai/fullsend
feat/triage-effort-analysis.Test plan
bash scripts/post-triage-test.sh— all cases pass including new effort-gating scenarioseffortfield with correct bounds🤖 Generated with Claude Code