The skill holdout can pick a skill the turn never matched, and the slot produces nothing

[macanderson]

The skill holdout picks a skill before knowing whether this turn's prompt matched it. When the pick names a skill the turn did not match, nothing is withheld and the slot produces no trial. The schedule still counts it, so the next skill holdout is another artifact_holdout_rate × HOLDOUT_ARMS turns away.

This is the remaining half of #5487. That issue fixed which skills the budget is spent on — #6459 narrows the pick to the skills whose control arm is short. It did not fix how often the budget produces anything, because relevance was out of reach for the reason below.

Why the pick cannot simply read the shortlist

SessionMemory::apply_holdout's doc carries the constraint. Every path that injects skills runs its own scoring pass: recall scopes its domains by the paths a turn touched, and the selection seam scopes them by the prompt alone, so the two shortlists differ. If each site picked from the shortlist in front of it, one site could withhold skill A while another injected it, and the ledger would then describe a turn that never happened.

So the pick has to be a function of state every site agrees on. Today that is the loaded catalog, narrowed by the trial ledger. Neither knows what this prompt matched.

The shape of the fix

trials::held_for already solves this for the memory and rule arms: the first pass with a non-empty population settles the pick, caches it on the turn, and every later pass reads it back. The skill arm predates that and never adopted it — crates/stella-cli/src/memory/trials.rs's module header records the same bug being fixed for the other two kinds.

Porting that to skills means the pick can read a matched population, because the cache — not the population — is what makes the sites agree. The care is in the ordering: the pick must be settled by the first site to score, and the withhold must then apply at every later site including ones whose own shortlist would not have contained that skill.

Anchors

• crates/stella-cli/src/memory.rs — SessionMemory::apply_holdout, and its doc's "loaded catalog" argument
• crates/stella-cli/src/memory/trials.rs — held_for, TurnContextTrials::held, and the "One pick per turn" section of the module header
• crates/stella-cli/src/memory/learning/skill_lifecycle.rs — the_holdout_picks_the_skill_whose_control_arm_is_short is the nearest existing witness

Which pillar

Reliability. A skill that is never measured is never promoted from the queue under the shipped require_measured_lift, and never retired when it stops helping — the lifecycle silently does not close.

Verify

A witness in skill_lifecycle.rs: a workspace with two skills where the holdout's ordinal names one the prompt does not match, asserting the turn still records a control trial for a skill it does match. It fails today, because the unmatched pick withholds nothing.

Definition of done

• Implementation complete
• Scoped tests added/updated and passing
• Full CI green
• Code comments and docs updated
• Residue filed as new issues (triage label only)

[macanderson]

Triage (2026-09-08 sweep): P2, area:cli, use-model:pro, pain:accuracy, pain:resolve-rate — a holdout slot that withholds nothing burns the schedule without producing a trial, so a skill can go unmeasured and therefore never promote under require_measured_lift nor retire when it stops helping. use-model:pro because the care is in the ordering — the pick must be settled by the first site to score and then hold at sites whose own shortlist would not contain that skill — and getting that wrong makes the ledger describe a turn that never happened.