chore(workflows): wire blind grading, and the two corpus convergence sweeps - #1031
Merged
Conversation
…sweeps Three steps, all maintainer-lane, all inert until their code is on main. run-evaluate gains the blinding bracket: stage each predictor's latest prediction under an opaque alias before the agent runs, un-alias before the stamp. The second position is load-bearing — stamp-cell joins an evaluation to its prediction on predictor_id, and under an alias that join misses and drops claim_scores and base_rate_salience_version silently rather than failing. run-seed gains two convergence sweeps beside the scope reconcile it mirrors: the application-baseline relabel and the merits-judgment backfill. Both belong in the lane rather than a dev checkout because the corpus pointer is a git file the writers commit to main and corpus-push is last-writer-wins, so a stale local copy republishes over concurrent walks; the corpus-write concurrency group is what serializes them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
run-evaluate.yml— the blinding bracket (the maintainer-lane half of #1029, whose code merged and is currently inert):Both edges of that second position matter, which the comment now says: after the capture, because un-aliasing also resolves aliases inside this cell's
retrieval_log.json— run earlier and alias strings ship in the log; before the stamp, becausestamp-celljoins each evaluation to its prediction onpredictor_id, and under an alias that join misses and dropsclaim_scoresandbase_rate_salience_versionsilently rather than failing.run-seed.yml— two convergence sweeps beside the scope reconcile they mirror: the application-baseline relabel and the merits-judgment backfill. Both belong in the lane rather than a dev checkout because the corpus pointer is a git file the writers commit tomainandcorpus-pushis last-writer-wins — a stale local copy republishes over concurrent walks, and thecorpus-writeconcurrency group is what serializes them.Review — one blocker, four recommendations, all fixed
docs/pipeline.mdanddocs/data-pipeline.mdupdated to enumerate all four in order and to state that each converges rather than running once.timeout-minuteson both new sweeps (8 / 10). The job cap is 55 min with the walk taking 2400 s, so the 02:31 window now runs four mutate + full-blob-push + rebase-push sweeps plus the trailing verdict in the remaining ~12 min — and the merits backfill's first non-converged pass reads the content store once per merits-bound grant, an external loop with no internal cap. Overrunning cancels the job, and cancellation is the one outcomecontinue-on-errordoes not absorb: the verdict is lost and the guard escalates a healthy run. A timed-out sweep is non-fatal and converges next window.data/, and this checkout is only as fresh as the walk's last checkpoint rebase — up to a window stale. A collect PR merging artifacts for an application docket in that gap would be invisible, and the rename would orphan exactly what the guard exists to prevent.git fetch origin+reset --hard FETCH_HEADbefore--apply.docs/cli.md, now that it is wired to a daily gate.seed: converge corpus (…)): becausecontinue-on-errorlets the job proceed, a failed sweep's partial mutation is published by the next sweep's push — correct state, but it was being committed under the wrong name.Reviewer's verified answers to the questions I raised: the sequential push/pointer-commit sequences do compose (no clobber is reachable — the blob is gitignored, so a rebase cannot revert it under a committed pointer, and the helper's one non-retryable path cannot arise from this job's own prior commit); and a mid-sweep failure is safe to re-enter (each rename is individually atomic, the sweeps are idempotent, and a blob-pushed-but-pointer-uncommitted orphan is litter the next window redoes).
Nits kept deliberately: no
success()guard on the un-alias step — gating it on the staging step would let a skipped un-aliasing pass quietly, which is the failure the design exists to prevent.Promotion constraint
This PR and #1029 must reach
mainin the same batch. The prompt already points the grader atrecord/blinded/<alias>/; without these steps nothing creates it. Recorded in the promotion PR (#1019) as well as here.Linters at the pinned versions:
zizmor@1.26.1 --persona=regular→ "No findings to report";actionlint 1.7.12→ exit 0 (run with a shellcheck shim — a bare run silently skips therun:-block linting CI performs). Gate: lint / types (173) / test (2020 passed).