[handoff] WS-B Stage A: token-budget admission scheduler

[penta2himajin]

Snapshot

• Branch: claude/token-budget-scheduler
• Last commit: 82b585a @ 2026-07-25
• Working tree: clean
• Last session: 2026-07-25

Status

ready-for-review (correctness phase + GO-bar bench measurement both complete; PR #140 open; default stays OFF pending owner decision)

Next action

Owner: review PR #140 and decide (a) merge as-is (flag default OFF, safe no-op unless opted in), (b) request the budget=1024 tuning experiment noted in notes/21 before merging, or (c) flip QWISP_TOKEN_BUDGET_SCHED default ON in a follow-up once satisfied with the p90 tradeoff documented below.

Verification

• Correctness: xcodebuild build -scheme qwisp BUILD SUCCEEDED, RAWTESTS 92/92, COMPTEST 83/83 (4 new locked tokenbudget_* self-checks), BENCHBATCHTEST PASS — all independently re-run outside the authoring agents.
• GO-bar bench (real model, scripts/bench_lane_budget_ab.sh 24000): steady-decode-lane inter-token-latency while a 24K-token prompt admits concurrently. OFF: p99/max = 93,668ms (one unbounded stall = the other request's full prefill, scales with its size). ON (budget 2048): p99/max = 13,516ms (~7x reduction, bound no longer scales with admitted-prompt size — the motivating failure mode is fixed). Caveat: p90 got worse (17ms → 9,983ms, OFF concentrates pain into 1-2 samples vs ON spreading smaller stalls across ~25%) — full analysis in notes/21's new "Bench verification results" section.

Context pointers

• PR: WS-B Stage A: token-budget admission scheduler (correctness phase) #140 (correctness + bench results both posted as PR comments).
• Spec + results: notes/21-ws-b-stage-a-token-budget-spec.md.
• Bench tool (new, kept as regression tool): scripts/bench_lane_budget_ab.sh + tools/lane_budget_probe.mjs.
• Diff: swift/Sources/QwispCore/ContinuousBatch.swift (AdmitProgress/admitStep/tokenBudget/loopBudgeted/tokenBudgetSelfCheck), swift/Sources/QwispCore/LaneServe.swift (LaneBatchSlots.Prefill state hoist + admitStep override + LaneBackend flag wiring).

Decisions made

• Correctness (locked COMPTEST) and the perf GO-bar are separate gates, same as WS-A's phase-1/phase-2 split — both now land in this PR, flag still default OFF regardless of merge.
• Independent Fable review (askfable) found one real bug (pool-accounting over-charge on prefix-cache-restore completions) — fixed (9edbdb2) and re-verified green. Two lower-risk observations (fake/real budget-consumption semantics differ at non-chunk-multiple budgets; prefix-cache-aware admission: parallel sub-agent fan-out should share the prefill, not re-pay it (measured: batch path loses 2.6x to serialize for lack of this) #121 fan-out restore ordering under budgeted mode is an emergent, not designed, property) recorded as PR comments, not acted on.
• The spec's original GO-bar framing ("~1 chunk-worth of decode latency") was corrected post-measurement: the real bound is "≤1 budget-worth of prefill compute at the admitting request's current depth" (chunk cost itself grows with depth per notes/19 §8) — see notes/21.

Failed approaches

• First bench attempt used a 24K-token prompt that, combined with the default QWISP_LANE_CTX=16384 cap, clamped maxTokens to 0 — the "big" admit became a silent no-op in BOTH the OFF and ON passes (indistinguishable ~210ms noise, not a real measurement). Fixed by explicitly raising QWISP_LANE_CTX=32768 for the bench server only (not the shipped default — that's Stage B's job).
• Second attempt still measured a no-op: a units bug in the probe's filler-prompt generator repeated a 14-word sentence words times instead of words/14 times, producing a ~295K-token prompt (12x past even the raised cap). Fixed by dividing by the sentence's own word count.
• Only after both fixed did the OFF pass show the real, expected 93.7s stall — always sanity-check a "no interleaving benefit measured" result against the actual server log (prompt= token count, gen=0) before trusting it as a negative result.

[penta2himajin]

Session log — 2026-07-25: WS-B kicked off per owner instruction (Next Action (a) from the WS-A handoff). Wrote notes/21-ws-b-stage-a-token-budget-spec.md — full contract for Stage A (token-budget admission scheduler on ContinuousScheduler/LaneBatchSlots), grounded directly in the current code (ContinuousBatch.swift, LaneServe.swift) and notes/19 §3/§8 measurement. Opened this handoff issue and pushed branch claude/token-budget-scheduler. Nothing implemented yet — next session should start from a devloop recon phase reading the admit-to-completion loop before locking the COMPTEST self-checks the spec proposes.

[penta2himajin]

Session log — 2026-07-25 (cont.): devloop recon → 1-round TDD loop (locked RED tests → Sonnet implement → adversarial review, all PASS first round) landed the correctness phase. Independently re-verified outside the agent: build SUCCEEDED, RAWTESTS 92/92, COMPTEST 83/83 (4 new tokenbudget_* cases), BENCHBATCHTEST PASS; read the full diff by hand (scheduler budget accounting, forward-progress guarantee, #121 relocation fidelity) and found no bugs. Committed (4fa19b4), pushed, opened PR #140 — not closing this issue yet, the bench GO-bar measurement (notes/21) is still owed before QWISP_TOKEN_BUDGET_SCHED can go default-ON.

[penta2himajin]

Session log — 2026-07-25 (cont. 2): ran the deferred GO-bar bench (scripts/bench_lane_budget_ab.sh, new tool, real model). Two false starts (QWISP_LANE_CTX cap clamping maxTokens to 0; then a 14x filler-prompt sizing bug) both produced misleadingly-flat OFF/ON comparisons that I caught by checking the server's own request log before trusting a null result. Real measurement: OFF p99/max=93.7s (unbounded single stall), ON p99/max=13.5s (~7x reduction, bound no longer scales with admitted-prompt size) — motivating failure mode fixed. Corrected notes/21's GO-bar framing post-measurement (chunk cost grows with depth, so the bound is 'one budget-worth of prefill at current depth', not a flat decode-token time) and documented the p90 tradeoff honestly rather than only touting the max/p99 win. Also ran an independent Fable review (askfable) of the correctness-phase diff — found and fixed one real bug (pool-accounting over-charge on prefix-cache-restore completions, 9edbdb2), two lower-risk items recorded as PR comments only. PR #140 now has both the correctness verification and the bench results posted; flag stays default OFF, next decision is the owner's.

[penta2himajin]

Session log — 2026-07-25 (final): owner decided default-ON after reviewing the GO-bar measurement. Flipped QWISP_TOKEN_BUDGET_SCHED default 0→1 (00ae72c), rebuilt, re-verified green (RAWTESTS 92/92, COMPTEST 83/83, BENCHBATCHTEST PASS) plus a live smoke test with no env override set. PR #140 merged to main (33c8fba, squash). WS-B Stage A is complete: spec (notes/21), correctness phase, GO-bar bench, default-ON, all landed. Closing — durable record lives in notes/21, not this issue.