Skip to content

Fix 2026-07-06 evaluation findings: protocol count, gate semantics, runner hardening, trigger narrowing, CI#10

Merged
liqiongyu merged 13 commits into
devfrom
fix/eval-findings-2026-07-06
Jul 6, 2026
Merged

Fix 2026-07-06 evaluation findings: protocol count, gate semantics, runner hardening, trigger narrowing, CI#10
liqiongyu merged 13 commits into
devfrom
fix/eval-findings-2026-07-06

Conversation

@liqiongyu

Copy link
Copy Markdown

Who is submitting this PR? (required)

Field Value
Your model + version Claude Fable 5 (claude-fable-5), subagents on Claude Opus 4.8
Harness + version Claude Code CLI 2.1.201
All plugins installed superpowers (this fork, working tree), commit-commands, claude-md-management, code-review, pr-review-toolkit, frontend-design
Human partner who reviewed this diff liqiongyu — approved the scope and each fix item after a four-agent fork evaluation; final diff review happens on this PR before merge (merge is human-only in this fork)

What problem are you trying to solve?

A four-agent evaluation of this fork (skill content, code-review integration, evidence/infrastructure, upstream survey) on 2026-07-06 confirmed a set of concrete defects. Each item below was verified against the tree, not speculated:

  1. review-protocol.md said "Run 10 independent finder angles" while defining 11 angle sections and an 11-value enum — the parallel fan-out instruction was ambiguous. Its intro also named Claude Code's /code-review max command in a protocol consumed on harnesses without that command.
  2. triaging-issues pressure scenario 10 expected triage output to include child issue drafts, which SKILL.md forbids and lists as a Red Flag — the stale expectation would grade correct behavior as a regression. The When To Use exclusion also routed PR/diff review to requesting-code-review (the pre-review checklist for one's own work) instead of code-review.
  3. scripts/run-skill-evidence.sh had four operational gaps: unbounded sessions (a hung CLI blocked forever), non-atomic transcript writes (interruption left plausible half-transcripts in the evidence tree), silent same-day overwrite of committed transcripts, and credential copies seeded into ~/.cache with no cleanup path.
  4. Gate semantics (P0-P2 blocking, rerun) were stated only in the two workflow skills routing to code-review, never in code-review itself; the fix-and-rerun loop had no termination condition even though the evaluation record shows priority labels drift between runs (P1/P2 swaps on identical bugs).
  5. The GitHub Mutation Gate had no escape hatch — an explicit session-level grant from the human was still overridden — and its four near-verbatim copies had already drifted (triaging lacked two-step exact-draft semantics entirely).
  6. working-from-issues triggered on bare "bug report, feature request, support request", competing with systematic-debugging/brainstorming and pulling tracker-free one-hop tasks into the three-hop triage pipeline; its pressure scenarios had zero must-not-trigger negatives.
  7. ~900 lines of structural tests (sha256 pin, evidence-link integrity) only ran when someone remembered to run them — no CI.
  8. Misc: writing-skills/SKILL.md diverged from upstream by a single deleted trailing space (pure conflict surface); the evidence README did not disclose that GREEN transcripts exist only for code-review.

What does this PR change?

Eleven commits: fixes the protocol count/wording (with sha256 pin update and a disclosure that the import is no longer verbatim), repairs the triaging scenario contradiction and PR-review routing, hardens the evidence runner (timeout, atomic writes, overwrite guard, clean subcommand, chmod 700), adds an authoritative Gate Semantics section (circuit breaker, latest-run label authority), adds a standing pre-authorization escape hatch with byte-identical wording across all four issue skills plus a cross-file identity test and three new pressure scenarios, narrows the working-from-issues trigger with two negative scenarios, restores writing-skills to upstream bytes, discloses GREEN coverage in the evidence README, adds fork-side CI, and lands a committed follow-up tracker (issues are disabled on this fork).

Is this change appropriate for the core library?

This is a fork-internal PR against heyi-co/superpowers dev, not an upstream contribution. Nothing here is intended for upstream: it repairs fork-owned skills and infrastructure, plus one byte-exact restoration toward upstream. The upstream-facing pieces (packaging fixes tracked as obra#1901/obra#1910) are deliberately left untouched.

Existing PRs

Searched fork PRs (#1#9, open and closed): #8 landed the evidence/runner/code-review restyle this PR builds on; no prior PR addresses these defects. Upstream PRs are intentionally not in scope; the two upstream PRs that overlap fork-modified files (obra#1901/obra#1910 packaging, obra#1931/obra#1932/obra#1934 skill refactors) are recorded in docs/superpowers/followups.md as sync-tracking items.

Verification

  • All six fork-owned test suites pass locally (five skill structural tests + Codex packaging test); shellcheck -S warning clean over all fork scripts and tests; workflow YAML parses.
  • Runner hardening exercised for real: overwrite guard refuses an existing transcript path (exit 1), clean removes the scratch root in both branches, a real claude red preflight passed end-to-end under the timeout wrapper (PREFLIGHT OK), scratch dirs come up drwx------, and with_timeout kills a 30s hang in 2s (rc=124; perl fallback verified at rc=142).
  • skills/writing-skills/SKILL.md verified byte-identical to upstream/main; the four standing pre-authorization blocks verified byte-identical by hash; the evidence README claim verified against a per-skill green/red file count.
  • Independent branch review by a fresh Opus 4.8 subagent (JSON findings contract, adversarial instructions, per-finding command evidence): no P0/P1/P2; four P3 findings — three applied in the final commit (reconciling template/prose mismatch, missing vague-phrasing negative scenario, CI trigger narrowing), one accepted with rationale in the commit message (unpinned rolling shellcheck).
  • Honest limitation: the three behavior-shaping edits (Gate Semantics, standing pre-authorization, trigger narrowing) ship with structural tests and pressure-scenario text but without live eval runs; that gap is recorded as a follow-up entry rather than claimed as verified behavior.

Environment table

Harness Tested Result
Claude Code CLI 2.1.201 (macOS) structural tests, packaging test, real runner preflight session all pass
Codex CLI not run in this session covered by structural/packaging tests only; live eval runs tracked in followups.md

🤖 Generated with Claude Code

https://claude.ai/code/session_01SYxaVSy1xS1PFzAAesybUJ

liqiongyu and others added 13 commits July 6, 2026 18:53
The protocol said "Run 10 independent finder angles" while defining 11
angle sections and an 11-value angle enum, leaving the parallel fan-out
instruction ambiguous. The intro also named Claude Code's /code-review
command in a protocol consumed on harnesses without that command.

Both edits are factual alignment with no semantic change; recorded in
evaluation.md's new Protocol revisions section. The verbatim-import
header now discloses local revisions, and the sha256 pin plus wording
assertions in the integration test track the edited file.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SYxaVSy1xS1PFzAAesybUJ
Scenario 10 expected triage output to include child issue drafts, which
SKILL.md forbids outright (decomposition moved to decomposing-issues and
drafting child bodies during triage is a listed Red Flag). The stale
expectation would grade correct behavior as a regression. The scenario
now expects a Decomposition Handoff and forbids drafted child bodies.

The When To Use exclusion routed PR/diff review to
requesting-code-review, which is the pre-review checklist for one's own
work; review requests belong to superpowers:code-review. The test
assertion pinning the old routing asserted the opposite direction and
is updated to match.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SYxaVSy1xS1PFzAAesybUJ
…, clean

Four operational gaps in scripts/run-skill-evidence.sh:

- Sessions ran unbounded; a hung CLI blocked forever. They now run
  under with_timeout (SKILL_EVIDENCE_TIMEOUT, default 1800s) using
  timeout(1) when present, else the base-system perl alarm+exec.
- Transcripts were written directly at their final path, so an
  interrupted session left a plausible-looking half transcript in the
  evidence tree. They are now built in a same-directory temp file and
  mv'd into place on completion, with an EXIT trap for the temp file.
- Rerunning the same skill/slug/harness/phase on the same day silently
  truncated an existing (possibly committed) transcript. The run
  subcommand now refuses to overwrite, matching the append-only
  evidence policy.
- Credential copies seeded into the scratch root persisted forever.
  Scratch dirs are now chmod 700 and a clean subcommand removes the
  scratch root; also added a command -v presence check for the harness
  CLI.

Verified: shellcheck clean; overwrite guard and clean exercised
directly; real claude red preflight passed end-to-end under the
timeout wrapper (PREFLIGHT OK); with_timeout kills a 30s hang in 2s
(rc=124). The plan document embedding the original runner is marked as
a dated historical snapshot rather than kept in sync.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SYxaVSy1xS1PFzAAesybUJ
The fork's only divergence in this file was a deleted trailing space —
zero value, and it kept a tuned upstream skill on the sync conflict
surface for nothing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SYxaVSy1xS1PFzAAesybUJ
Only code-review has committed green transcripts (8 green plus a
no-skill control red, verified by directory count); the four
issue-workflow skills have RED baselines only, with green results
existing solely as paraphrased evaluation rows. The evidence tree's own
standard is transcript-or-it-didn't-happen, so the gap must be stated
where the standard is stated. Backfill is tracked as follow-up work.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SYxaVSy1xS1PFzAAesybUJ
Blocking semantics (P0-P2 block, P3 non-blocking, human-only
acceptance, rerun-after-fix) were previously stated only in the two
workflow skills that route here, while code-review itself — where a
gate consumer actually lands — defined none of it. The new section is
declared the single authoritative definition.

It also closes two gaps the evaluation record exposed: priority labels
drift between runs on the same finding (evaluation.md documents P1/P2
swaps), so the latest run's labels are declared authoritative; and the
fix-and-rerun loop had no termination condition, so a circuit breaker
now stops the loop after two consecutive cycles with an undiminished
blocking count and escalates with the findings list. Red Flags gains
the matching entry, and the integration test pins the new section.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SYxaVSy1xS1PFzAAesybUJ
…ding

The two-step exact-draft gate had no escape hatch: even an explicit
session-level grant from the human ('for this session, apply labels
without asking') was overridden by the blanket-approval clause, forcing
per-mutation confirmation in owner-operator automation. A standing
pre-authorization now lifts per-mutation confirmation for the mutation
categories the human names in the session. The injection surface does
not widen: repository files still cannot grant it, it is never inferred
from task phrasing, each applied mutation is still shown, and anything
outside the named categories falls back to two-step approval.

The gate text had also drifted across copies: triaging lacked the
two-step exact-draft semantics entirely. Triaging gains it, the new
pre-authorization block is byte-identical in all four issue-workflow
skills, and the decomposing test now asserts that cross-file identity
so future drift fails a test instead of rotting silently. Mutation
Preview semantics extend to record applied mutations under a
pre-authorization. Two pressure scenarios cover the grant and the
repo-file-claims-authorization refusal; live eval runs for them are
tracked as follow-up.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SYxaVSy1xS1PFzAAesybUJ
The description listed bare 'bug report, feature request, support
request' as triggers — words that need no tracker context — so a pasted
stack trace or a feature idea competed with systematic-debugging and
brainstorming and could pull a one-hop task into the three-hop triage
pipeline. The trigger list now requires issue-tracker context (issue,
issue URL/number, pasted issue report, Triage Result) and explicitly
excludes tracker-free bug/feature/support asks.

pressure-scenarios gains the previously missing negative cases (a local
stack trace and a bare feature idea, both marked must-not-trigger), and
the test pins the exclusion plus the negative scenarios. Live eval runs
are tracked as follow-up.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SYxaVSy1xS1PFzAAesybUJ
The fork carries ~900 lines of structural tests — including the
review-protocol sha256 pin, the evidence-link integrity loop, and the
cross-file gate-block identity check — that only ran when someone
remembered to run them. They now run on every push and PR.

Upstream ships no .github/workflows, so the file adds zero sync
conflict surface. Shellcheck runs at warning level to match the
current clean baseline (one pre-existing SC2016 info note is
deliberate literal-backtick usage). Tool dependencies (shasum, zip,
unzip, python3) are all present in the ubuntu-latest image.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SYxaVSy1xS1PFzAAesybUJ
…ario, narrow CI triggers

Independent branch review returned no blocking findings and four P3s;
three are applied here:

- reconciling's main output template hard-coded 'No GitHub mutation was
  performed' on the very path where a standing pre-authorization can now
  legitimately perform one, contradicting the gate prose and producing a
  false audit line; the template fields now carry both states. The
  Blocked template keeps the hard-coded line — blocked reconciliation
  performs no mutation by construction.
- the gate's key guard 'never inferred from task phrasing' had no
  negative pressure scenario anywhere; decomposing (the gate's
  authoritative home) gains one where vague 'handle it as you see fit'
  phrasing must not be treated as pre-authorization.
- CI push triggers narrow to main/dev so PR branches run once and tag
  pushes not at all.

The fourth P3 (unpinned rolling shellcheck on ubuntu-latest) is
accepted, not fixed: pinning adds upgrade chores that outweigh fixing a
rare new warning; the baseline version is noted in the workflow.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SYxaVSy1xS1PFzAAesybUJ
GitHub issues are disabled on this fork, so the nine follow-ups from
the 2026-07-06 evaluation (GREEN transcript backfill, control-baseline
discrimination, skill slimming items, upstream PR tracking, live eval
runs) land in a committed tracker instead of only in a session summary.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SYxaVSy1xS1PFzAAesybUJ
First CI run on ubuntu failed in the packaging test's tar.gz leg:
scripts/package-codex-plugin.sh (upstream code, fork only added TZ=UTC)
uses bsdtar-only flags (--uid/--gid/--uname/--gname) that GNU tar
rejects, so the script has always been macOS-only and ubuntu merely
exposed it. The job now runs on macos-latest, matching the script's
real support surface; the GNU tar fact is recorded in followups.md
next to the upstream packaging-PR tracking entry rather than patched
locally while upstream obra#1901/obra#1910 are in flight on the same file.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SYxaVSy1xS1PFzAAesybUJ
macos-latest does not preship shellcheck (the first macos run passed
both test steps and died at shellcheck with command not found); install
it via brew when absent.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SYxaVSy1xS1PFzAAesybUJ
@liqiongyu liqiongyu merged commit a9ce782 into dev Jul 6, 2026
1 check passed
@liqiongyu liqiongyu deleted the fix/eval-findings-2026-07-06 branch July 6, 2026 13:14
liqiongyu added a commit that referenced this pull request Jul 8, 2026
Back-merge main into dev — restore dev-first (dev was stuck at #10 / v6.1.1)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant