streaming/bolt: concurrent requests to amortise expert I/O (Step 0 — not a port of resident lane mode)

[penta2himajin]

Framing: this is not "port lane mode to streaming"

Lane batching (QWISP_LANES) is resident-only today. The obvious reading — lift the tier guard and reuse the resident path — is the wrong shape, because the source of the win is different on a streaming tier.

• On resident, lanes buy decode batching. They save no I/O, because there is none.
• On streaming, expert weights come from flash and I/O is the binding constraint. B concurrent requests route to a union of experts per layer, not B disjoint sets, so the overlapping fraction is loaded once and amortised across B tokens.

So this issue is about amortising expert I/O across concurrent requests, and it should be evaluated on I/O per token, not on decode batching. The resident implementation is a reference for the scheduler and admission surfaces, not for the economics.

Filing it as a Step-0 issue (the shape used by #143 and #154): establish the numbers that decide it before designing anything. The expected outcome is genuinely open — unlike #154 this is not expected to close.

What blocks it today

• LaneBackend.init (swift/Sources/QwispCore/LaneServe.swift) hard-guards DeviceCalibration.defaultC() >= 256, throwing "requires a resident-tier machine (≥32GB)".
• That guard is env-overridable: defaultC() returns min(QWISP_CACHE_C, 256) when set, so QWISP_CACHE_C=256 passes it on any machine. The guard is a tier heuristic, not a capability check.
• The real blocker is the next line: store.residentAll(). SeedlessLaneBatch also states its contract as "All must be resident-mode with the same layer stack". There is no streaming-arena lane path at all.

Why the win could be larger here than on resident

Per layer, one token routes to top-8 experts. B lanes need the union of B×8, minus overlap. Memory expert-reuse-drafting-research recorded ~38% natural expert overlap — measured in a drafting context, not for concurrent independent requests, so it must be re-measured (that is Step 0's main job). If something like it holds:

B	union/layer	expert I/O per token	relative
1	8	8	1.00x
2	~13	6.5	~1.23x
4	~23	5.8	~1.38x

This axis does not exist on resident. It is also strongest exactly where the product is weakest: product-ram-axis-device-floor records slow-NAND MacBooks at 3.7–5.6 tok/s, IO-bound, the only tier where the device floor is genuinely bad.

Machinery that already exists

ExpertArena already has a union-overflow guard for batched work: it detects when one layer's routed distinct-expert count exceeds the arena capacity C, and computes an exact safe prefix. Built to make M>1 speculative verify strict-lossless on streaming (memory streaming-lossless-by-luck-fixed). So "multiple rows whose per-layer expert union may exceed C" is a solved problem in this codebase — for one shape of it.

The new hard problem: overflow handling differs

• Speculation: union > C → shrink the accepted row prefix. Safe; costs only speedup.
• Lanes: you cannot shrink a lane. Union > C for a layer means that layer must be split into multiple passes, which is new scheduling, not a reuse of the guard.

This matters because overflow may be the common case, not the exception: memory footprint-vs-budget records per-layer footprint of 103 experts against C=64 at 8GB — i.e. already overflowing at B=1 in that configuration. If B multiplies the union while C is fixed by RAM, the split path becomes the hot path and its cost dominates the amortisation win.

Counter-pressure: KV arena RAM

Each lane owns its KV arena (~20KB/token, LaneBatchSlots.kvBytesPerToken). On a 32GB-or-less machine, B=4 with 8K prompts is ~640MB of KV on top of the streaming arena and the resident share of weights. The I/O saved may be paid back in RAM, and RAM is what defines this tier. Any GO must show the net.

Step 0 — measure before designing

The key enabler: lane mode is bit-exact, so each lane's expert selections are identical to what that request selects running alone. The union is therefore computable offline from B independent single-stream traces — no concurrent execution, no engine change, no new code path.

1. Capture per-layer top-8 expert selections for N realistic concurrent-workload prompts (the agentic/code mix in notes/ — the same prompt mix the benches use). Single-stream, one at a time.
2. Offline, for B ∈ {2, 3, 4}: compute per-layer union size over B randomly-paired streams, at matched decode positions. Report the distribution, not just the mean.
3. Compute implied expert I/O per token vs B=1, and the fraction of (layer, step) pairs where the union exceeds C at C ∈ {64, 128, 256}.
4. Estimate the RAM ledger: KV arena × B + streaming arena at that C, against the 8/16/24GB tiers.

Kill criteria — decide before looking:

• If implied I/O per token improves by < 15% at B=4, close: the amortisation does not pay for a new scheduling path in the streaming engine.
• If union exceeds C on > 25% of (layer, step) pairs at the tier's default C, close or restrict to larger-C tiers only: the split path would be the common case, and its cost is not modelled by this arithmetic.
• If the RAM ledger is negative at 16GB, restrict scope to 24GB+ streaming before designing anything.

If Step 0 passes, a design issue follows covering: the per-layer split scheduler, whether bolt's slot_remap (in-place RMW of moeSc.inds, inserted between route and gather) composes with B lanes, and admission policy under a shared arena.

Explicitly out of scope

• Lifting the tier guard as a standalone change. It passes trivially via QWISP_CACHE_C=256 and would only reach residentAll() and thrash. Do not "enable" lanes on streaming as an experiment.
• Any change to the resident lane path. Its economics and its measured behaviour (v0.3.11, notes/21, notes/22) stand.

Refs

• swift/Sources/QwispCore/LaneServe.swift — tier guard, residentAll(), LaneBatchSlots
• swift/Sources/QwispCore/SeedlessLaneBatch.swift — "each with its OWN caches", resident-mode contract
• swift/Sources/QwispCore/ExpertArena.swift — union-overflow guard and exact safe-prefix
• memories: expert-reuse-drafting-research (overlap, different context), footprint-vs-budget (103 vs C=64), product-ram-axis-device-floor (slow-NAND floor), streaming-lossless-by-luck-fixed (why the guard exists), faststreaming-tiers-current (current streaming tok/s by tier)
• [Draft] 複数ストリーム throughput: continuous batching で dispatch床(GPU 65% idle)を埋める（per-stream correct, 32GB+向き） #6 (continuous batching, validated on resident), prefix-cache-aware admission: parallel sub-agent fan-out should share the prefill, not re-pay it (measured: batch path loses 2.6x to serialize for lack of this) #121 (prefix-cache-aware admission)