prefix-cache-aware admission: parallel sub-agent fan-out should share the prefill, not re-pay it (measured: batch path loses 2.6x to serialize for lack of this)

[penta2himajin]

Motivation (measured, PR #120 harness)

Real OpenCode parallel fan-out = N sub-agent streams arriving simultaneously with a shared 12.8K-char system prefix and same-shape prompts, all greedy. Replayed against both server paths (64GB resident, real captured trace):

path	identity vs strict	total wall
serialize (default)	6/6 byte-identical	72.3s
QWISP_BATCH=4	3/6 diverge	190.6s (2.6x slower)

The batch path's loss is not decode speed — it is admission: ContinuousBatch.admit() runs a standalone full prefill per request (~9K tokens each, no prefix cache, no SuffixSpec), so N concurrent same-prefix requests pay N full prefills. The serialize path wins because #117's RAM-tier prefix cache absorbs the shared prefix (repeat TTFT ~0.2-0.4s). Agentic traffic is prefill-dominated with heavy shared prefixes — any admission that ignores that loses.

Design directions (in increasing ambition)

1. Serialize-path admission ordering (cheap, likely small): when multiple requests are queued on the AsyncLock, grant the lock to the request that best extends the current arena path / RAM store (instead of FIFO). Maximizes warm restores when conversations interleave. May be a no-op in practice — FIFO already arrives prefix-grouped in the captured trace; measure before building.
2. Batch-path admission via the prefix stores (the real fix if batch is revived): admit() consults PrefixRAMStore / in-path slots — restore the longest shared-prefix state into the slot, prefill only the delta. Removes the N× re-prefill penalty that caused the 2.6x loss. Requires the batch model's cache format to accept the Seedless blob (persistentStateData) or its own store — nontrivial: the batch path is MLX QwispModel, a different cache layout.
3. Auto-route (product-level): detect concurrency + shared prefix → stay on serialize (cache-hot, lossless); route to batch only when the workload is decode-dominated with cold-unique prompts (the only regime where batch's uniform admission wins). Could be as simple as: batch only admits requests whose best prefix hit is below a threshold.

Note the strict ordering of value: (3) prevents the regression today with ~no code, (2) is what spec-in-batch would ALSO need (prefix reuse is a prerequisite for batch to beat serialize at all on this traffic), (1) is measure-first.

Acceptance gate

The #120 harness is the gate: replay diff on the captured OpenCode trace. A variant must (a) beat serialize's 72.3s total wall on the fan-out trace, or (b) for routing-only work, never lose to it; identity vs the strict serial reference reported alongside (6/6 for lossless paths; divergence count quantified otherwise).

Relations

• test(tools): OpenCode parallel-agent reproduction — capture proxy + replay harness, first measured verdict #120 (reproduction harness + the measurement motivating this)
• prefix cache: RAM-tier keyed store — interleaved conversations thrash the single-path cache (measured: reuse 4-52% on switches, TTFT up to 7 min) #117 / PR feat(server): RAM-tier prefix cache — keep interleaved conversations warm (#117) #118 (RAM-tier prefix cache — the mechanism that makes serialize win; option 2 reuses its blobs)
• [Draft] 複数ストリーム throughput: continuous batching で dispatch床(GPU 65% idle)を埋める（per-stream correct, 32GB+向き） #6 (continuous batching, closed — its 1.78x was measured on decode-dominated synthetic waves, not agentic shared-prefix traffic; this issue is the missing admission half)

🤖 Generated with Claude Code

[penta2himajin]

Verdict: guard shipped; option 2 parked (measured — it cannot flip the outcome on the target traffic)

Arithmetic on the #120 trace: total batch prefill ≈ 40K tokens, of which only ~13K is shared-prefix dedup headroom. Even a PERFECT prefix-aware admission leaves ~27K unique tokens ⇒ ~70-90s on the MLX path — still ≥ serialize's measured 72.3s, and the MLX-f16 divergence (3/6 streams) is untouched. Batch's winning regime (decode-dominated, cold-unique concurrent prompts) does not occur in harness workloads. Per-request routing between both backends was also examined and rejected: double weight residency.

Shipped (option 3 minimal, PR #125): the real remaining defect was silent regression — QWISP_BATCH + OpenCode = 2.6x slower + divergent with no signal. Now: a startup note under QWISP_BATCH, plus a one-time runtime NOTE when consecutive admitted prompts share a ≥1024-token prefix twice (the agentic-harness signature).

Parked (option 2, design recorded in the issue body): prefix-aware batch admission — revive only when a genuine multi-user cold-prompt workload exists. Acceptance gate remains the #120 replay harness (beat 72.3s on the fan-out trace, identity reported alongside). Option 1 (lock ordering) skipped: the captured trace shows FIFO arrival is already prefix-grouped — nothing to reorder.