Fix 2026-07-06 evaluation findings: protocol count, gate semantics, runner hardening, trigger narrowing, CI#10
Merged
Merged
Conversation
The protocol said "Run 10 independent finder angles" while defining 11 angle sections and an 11-value angle enum, leaving the parallel fan-out instruction ambiguous. The intro also named Claude Code's /code-review command in a protocol consumed on harnesses without that command. Both edits are factual alignment with no semantic change; recorded in evaluation.md's new Protocol revisions section. The verbatim-import header now discloses local revisions, and the sha256 pin plus wording assertions in the integration test track the edited file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SYxaVSy1xS1PFzAAesybUJ
Scenario 10 expected triage output to include child issue drafts, which SKILL.md forbids outright (decomposition moved to decomposing-issues and drafting child bodies during triage is a listed Red Flag). The stale expectation would grade correct behavior as a regression. The scenario now expects a Decomposition Handoff and forbids drafted child bodies. The When To Use exclusion routed PR/diff review to requesting-code-review, which is the pre-review checklist for one's own work; review requests belong to superpowers:code-review. The test assertion pinning the old routing asserted the opposite direction and is updated to match. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SYxaVSy1xS1PFzAAesybUJ
…, clean Four operational gaps in scripts/run-skill-evidence.sh: - Sessions ran unbounded; a hung CLI blocked forever. They now run under with_timeout (SKILL_EVIDENCE_TIMEOUT, default 1800s) using timeout(1) when present, else the base-system perl alarm+exec. - Transcripts were written directly at their final path, so an interrupted session left a plausible-looking half transcript in the evidence tree. They are now built in a same-directory temp file and mv'd into place on completion, with an EXIT trap for the temp file. - Rerunning the same skill/slug/harness/phase on the same day silently truncated an existing (possibly committed) transcript. The run subcommand now refuses to overwrite, matching the append-only evidence policy. - Credential copies seeded into the scratch root persisted forever. Scratch dirs are now chmod 700 and a clean subcommand removes the scratch root; also added a command -v presence check for the harness CLI. Verified: shellcheck clean; overwrite guard and clean exercised directly; real claude red preflight passed end-to-end under the timeout wrapper (PREFLIGHT OK); with_timeout kills a 30s hang in 2s (rc=124). The plan document embedding the original runner is marked as a dated historical snapshot rather than kept in sync. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SYxaVSy1xS1PFzAAesybUJ
The fork's only divergence in this file was a deleted trailing space — zero value, and it kept a tuned upstream skill on the sync conflict surface for nothing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SYxaVSy1xS1PFzAAesybUJ
Only code-review has committed green transcripts (8 green plus a no-skill control red, verified by directory count); the four issue-workflow skills have RED baselines only, with green results existing solely as paraphrased evaluation rows. The evidence tree's own standard is transcript-or-it-didn't-happen, so the gap must be stated where the standard is stated. Backfill is tracked as follow-up work. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SYxaVSy1xS1PFzAAesybUJ
Blocking semantics (P0-P2 block, P3 non-blocking, human-only acceptance, rerun-after-fix) were previously stated only in the two workflow skills that route here, while code-review itself — where a gate consumer actually lands — defined none of it. The new section is declared the single authoritative definition. It also closes two gaps the evaluation record exposed: priority labels drift between runs on the same finding (evaluation.md documents P1/P2 swaps), so the latest run's labels are declared authoritative; and the fix-and-rerun loop had no termination condition, so a circuit breaker now stops the loop after two consecutive cycles with an undiminished blocking count and escalates with the findings list. Red Flags gains the matching entry, and the integration test pins the new section. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SYxaVSy1xS1PFzAAesybUJ
…ding
The two-step exact-draft gate had no escape hatch: even an explicit
session-level grant from the human ('for this session, apply labels
without asking') was overridden by the blanket-approval clause, forcing
per-mutation confirmation in owner-operator automation. A standing
pre-authorization now lifts per-mutation confirmation for the mutation
categories the human names in the session. The injection surface does
not widen: repository files still cannot grant it, it is never inferred
from task phrasing, each applied mutation is still shown, and anything
outside the named categories falls back to two-step approval.
The gate text had also drifted across copies: triaging lacked the
two-step exact-draft semantics entirely. Triaging gains it, the new
pre-authorization block is byte-identical in all four issue-workflow
skills, and the decomposing test now asserts that cross-file identity
so future drift fails a test instead of rotting silently. Mutation
Preview semantics extend to record applied mutations under a
pre-authorization. Two pressure scenarios cover the grant and the
repo-file-claims-authorization refusal; live eval runs for them are
tracked as follow-up.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SYxaVSy1xS1PFzAAesybUJ
The description listed bare 'bug report, feature request, support request' as triggers — words that need no tracker context — so a pasted stack trace or a feature idea competed with systematic-debugging and brainstorming and could pull a one-hop task into the three-hop triage pipeline. The trigger list now requires issue-tracker context (issue, issue URL/number, pasted issue report, Triage Result) and explicitly excludes tracker-free bug/feature/support asks. pressure-scenarios gains the previously missing negative cases (a local stack trace and a bare feature idea, both marked must-not-trigger), and the test pins the exclusion plus the negative scenarios. Live eval runs are tracked as follow-up. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SYxaVSy1xS1PFzAAesybUJ
The fork carries ~900 lines of structural tests — including the review-protocol sha256 pin, the evidence-link integrity loop, and the cross-file gate-block identity check — that only ran when someone remembered to run them. They now run on every push and PR. Upstream ships no .github/workflows, so the file adds zero sync conflict surface. Shellcheck runs at warning level to match the current clean baseline (one pre-existing SC2016 info note is deliberate literal-backtick usage). Tool dependencies (shasum, zip, unzip, python3) are all present in the ubuntu-latest image. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SYxaVSy1xS1PFzAAesybUJ
…ario, narrow CI triggers Independent branch review returned no blocking findings and four P3s; three are applied here: - reconciling's main output template hard-coded 'No GitHub mutation was performed' on the very path where a standing pre-authorization can now legitimately perform one, contradicting the gate prose and producing a false audit line; the template fields now carry both states. The Blocked template keeps the hard-coded line — blocked reconciliation performs no mutation by construction. - the gate's key guard 'never inferred from task phrasing' had no negative pressure scenario anywhere; decomposing (the gate's authoritative home) gains one where vague 'handle it as you see fit' phrasing must not be treated as pre-authorization. - CI push triggers narrow to main/dev so PR branches run once and tag pushes not at all. The fourth P3 (unpinned rolling shellcheck on ubuntu-latest) is accepted, not fixed: pinning adds upgrade chores that outweigh fixing a rare new warning; the baseline version is noted in the workflow. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SYxaVSy1xS1PFzAAesybUJ
GitHub issues are disabled on this fork, so the nine follow-ups from the 2026-07-06 evaluation (GREEN transcript backfill, control-baseline discrimination, skill slimming items, upstream PR tracking, live eval runs) land in a committed tracker instead of only in a session summary. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SYxaVSy1xS1PFzAAesybUJ
First CI run on ubuntu failed in the packaging test's tar.gz leg: scripts/package-codex-plugin.sh (upstream code, fork only added TZ=UTC) uses bsdtar-only flags (--uid/--gid/--uname/--gname) that GNU tar rejects, so the script has always been macOS-only and ubuntu merely exposed it. The job now runs on macos-latest, matching the script's real support surface; the GNU tar fact is recorded in followups.md next to the upstream packaging-PR tracking entry rather than patched locally while upstream obra#1901/obra#1910 are in flight on the same file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SYxaVSy1xS1PFzAAesybUJ
macos-latest does not preship shellcheck (the first macos run passed both test steps and died at shellcheck with command not found); install it via brew when absent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SYxaVSy1xS1PFzAAesybUJ
This was referenced Jul 6, 2026
liqiongyu
added a commit
that referenced
this pull request
Jul 8, 2026
Back-merge main into dev — restore dev-first (dev was stuck at #10 / v6.1.1)
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Who is submitting this PR? (required)
claude-fable-5), subagents on Claude Opus 4.8What problem are you trying to solve?
A four-agent evaluation of this fork (skill content, code-review integration, evidence/infrastructure, upstream survey) on 2026-07-06 confirmed a set of concrete defects. Each item below was verified against the tree, not speculated:
review-protocol.mdsaid "Run 10 independent finder angles" while defining 11 angle sections and an 11-value enum — the parallel fan-out instruction was ambiguous. Its intro also named Claude Code's/code-review maxcommand in a protocol consumed on harnesses without that command.triaging-issuespressure scenario 10 expected triage output to include child issue drafts, which SKILL.md forbids and lists as a Red Flag — the stale expectation would grade correct behavior as a regression. The When To Use exclusion also routed PR/diff review torequesting-code-review(the pre-review checklist for one's own work) instead ofcode-review.scripts/run-skill-evidence.shhad four operational gaps: unbounded sessions (a hung CLI blocked forever), non-atomic transcript writes (interruption left plausible half-transcripts in the evidence tree), silent same-day overwrite of committed transcripts, and credential copies seeded into~/.cachewith no cleanup path.code-review, never incode-reviewitself; the fix-and-rerun loop had no termination condition even though the evaluation record shows priority labels drift between runs (P1/P2 swaps on identical bugs).working-from-issuestriggered on bare "bug report, feature request, support request", competing withsystematic-debugging/brainstormingand pulling tracker-free one-hop tasks into the three-hop triage pipeline; its pressure scenarios had zero must-not-trigger negatives.writing-skills/SKILL.mddiverged from upstream by a single deleted trailing space (pure conflict surface); the evidence README did not disclose that GREEN transcripts exist only forcode-review.What does this PR change?
Eleven commits: fixes the protocol count/wording (with sha256 pin update and a disclosure that the import is no longer verbatim), repairs the triaging scenario contradiction and PR-review routing, hardens the evidence runner (timeout, atomic writes, overwrite guard,
cleansubcommand, chmod 700), adds an authoritative Gate Semantics section (circuit breaker, latest-run label authority), adds a standing pre-authorization escape hatch with byte-identical wording across all four issue skills plus a cross-file identity test and three new pressure scenarios, narrows theworking-from-issuestrigger with two negative scenarios, restoreswriting-skillsto upstream bytes, discloses GREEN coverage in the evidence README, adds fork-side CI, and lands a committed follow-up tracker (issues are disabled on this fork).Is this change appropriate for the core library?
This is a fork-internal PR against
heyi-co/superpowersdev, not an upstream contribution. Nothing here is intended for upstream: it repairs fork-owned skills and infrastructure, plus one byte-exact restoration toward upstream. The upstream-facing pieces (packaging fixes tracked as obra#1901/obra#1910) are deliberately left untouched.Existing PRs
Searched fork PRs (#1–#9, open and closed): #8 landed the evidence/runner/code-review restyle this PR builds on; no prior PR addresses these defects. Upstream PRs are intentionally not in scope; the two upstream PRs that overlap fork-modified files (obra#1901/obra#1910 packaging, obra#1931/obra#1932/obra#1934 skill refactors) are recorded in
docs/superpowers/followups.mdas sync-tracking items.Verification
shellcheck -S warningclean over all fork scripts and tests; workflow YAML parses.cleanremoves the scratch root in both branches, a realclaudered preflight passed end-to-end under the timeout wrapper (PREFLIGHT OK), scratch dirs come updrwx------, andwith_timeoutkills a 30s hang in 2s (rc=124; perl fallback verified at rc=142).skills/writing-skills/SKILL.mdverified byte-identical toupstream/main; the four standing pre-authorization blocks verified byte-identical by hash; the evidence README claim verified against a per-skill green/red file count.Environment table
🤖 Generated with Claude Code
https://claude.ai/code/session_01SYxaVSy1xS1PFzAAesybUJ