Skip to content

chore(workflows): wire blind grading, and the two corpus convergence sweeps - #1031

Merged
modelmirror merged 1 commit into
stagingfrom
feat/blind-grading-workflow
Aug 4, 2026
Merged

chore(workflows): wire blind grading, and the two corpus convergence sweeps#1031
modelmirror merged 1 commit into
stagingfrom
feat/blind-grading-workflow

Conversation

@modelmirror

Copy link
Copy Markdown
Collaborator

⚠️ Maintainer merge — the permission surface. Three steps across two workflows, all inert until their code is on main. Reported ready, not self-merged.

What

run-evaluate.yml — the blinding bracket (the maintainer-lane half of #1029, whose code merged and is currently inert):

  • Stage the blinded candidates, after the event definition is materialized and before any agent step.
  • Un-alias the blinded evaluations, between the retrieval capture and the process-version stamp.

Both edges of that second position matter, which the comment now says: after the capture, because un-aliasing also resolves aliases inside this cell's retrieval_log.json — run earlier and alias strings ship in the log; before the stamp, because stamp-cell joins each evaluation to its prediction on predictor_id, and under an alias that join misses and drops claim_scores and base_rate_salience_version silently rather than failing.

run-seed.yml — two convergence sweeps beside the scope reconcile they mirror: the application-baseline relabel and the merits-judgment backfill. Both belong in the lane rather than a dev checkout because the corpus pointer is a git file the writers commit to main and corpus-push is last-writer-wins — a stale local copy republishes over concurrent walks, and the corpus-write concurrency group is what serializes them.

Review — one blocker, four recommendations, all fixed

  • blocker · docs said "two maintenance sweeps"; there are now four. Both docs/pipeline.md and docs/data-pipeline.md updated to enumerate all four in order and to state that each converges rather than running once.
  • timeout-minutes on both new sweeps (8 / 10). The job cap is 55 min with the walk taking 2400 s, so the 02:31 window now runs four mutate + full-blob-push + rebase-push sweeps plus the trailing verdict in the remaining ~12 min — and the merits backfill's first non-converged pass reads the content store once per merits-bound grant, an external loop with no internal cap. Overrunning cancels the job, and cancellation is the one outcome continue-on-error does not absorb: the verdict is lost and the guard escalates a healthy run. A timed-out sweep is non-fatal and converges next window.
  • The relabel now reads a freshly fetched tree. Its orphan guard checks committed ledger artifacts in data/, and this checkout is only as fresh as the walk's last checkpoint rebase — up to a window stale. A collect PR merging artifacts for an application docket in that gap would be invisible, and the rename would orphan exactly what the guard exists to prevent. git fetch origin + reset --hard FETCH_HEAD before --apply.
  • Dropped a redundant step-level split-mode env block — the job already sets both variables for every step with identical expressions, and the comment implied a step-level declaration that did not exist, inviting someone to delete the job-level one.
  • "One-time migration" restated as a standing convergence sweep in the CLI docstring and docs/cli.md, now that it is wired to a daily gate.
  • Sweep commit messages made generic (seed: converge corpus (…)): because continue-on-error lets the job proceed, a failed sweep's partial mutation is published by the next sweep's push — correct state, but it was being committed under the wrong name.

Reviewer's verified answers to the questions I raised: the sequential push/pointer-commit sequences do compose (no clobber is reachable — the blob is gitignored, so a rebase cannot revert it under a committed pointer, and the helper's one non-retryable path cannot arise from this job's own prior commit); and a mid-sweep failure is safe to re-enter (each rename is individually atomic, the sweeps are idempotent, and a blob-pushed-but-pointer-uncommitted orphan is litter the next window redoes).

Nits kept deliberately: no success() guard on the un-alias step — gating it on the staging step would let a skipped un-aliasing pass quietly, which is the failure the design exists to prevent.

Promotion constraint

This PR and #1029 must reach main in the same batch. The prompt already points the grader at record/blinded/<alias>/; without these steps nothing creates it. Recorded in the promotion PR (#1019) as well as here.

Linters at the pinned versions: zizmor@1.26.1 --persona=regular → "No findings to report"; actionlint 1.7.12 → exit 0 (run with a shellcheck shim — a bare run silently skips the run:-block linting CI performs). Gate: lint / types (173) / test (2020 passed).

…sweeps

Three steps, all maintainer-lane, all inert until their code is on main.

run-evaluate gains the blinding bracket: stage each predictor's latest
prediction under an opaque alias before the agent runs, un-alias before the
stamp. The second position is load-bearing — stamp-cell joins an evaluation to
its prediction on predictor_id, and under an alias that join misses and drops
claim_scores and base_rate_salience_version silently rather than failing.

run-seed gains two convergence sweeps beside the scope reconcile it mirrors:
the application-baseline relabel and the merits-judgment backfill. Both belong
in the lane rather than a dev checkout because the corpus pointer is a git file
the writers commit to main and corpus-push is last-writer-wins, so a stale
local copy republishes over concurrent walks; the corpus-write concurrency
group is what serializes them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@modelmirror
modelmirror merged commit c46024c into staging Aug 4, 2026
7 checks passed
@modelmirror
modelmirror deleted the feat/blind-grading-workflow branch August 4, 2026 18:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant