perf(arena): expert-major weight relayout — 9 preads per expert miss → 1 (streaming tier I/O count, not bytes)

[penta2himajin]

Observation

ExpertArena.loadMany → ExpertSource.preadInto issues one pread per (proj, part): {gate,up,down}_proj × {weight,scales,biases} = 9 preads per missed expert, each landing at a different offset in the safetensors shard (the 256-expert stack is proj-major, so expert e's three projections are ~E strides apart, not adjacent).

Verified at HEAD (2026-07-25): the 9-pread count is correct — three projections × three parts
(ExpertSource.swift:11), loadMany schedules jobs.count × 9 operations (ExpertArena.swift:56), and each
operation is one real pread (ExpertSource.swift:144). For this model one missed expert is 1,769,472 bytes
= three 512 KiB weight slices + six 32 KiB scale/bias slices.

Bytes transferred are already minimal — each read is exactly the expert's slice. What is not minimal is the
request count and seek scatter: 9 requests per miss, at offsets ~E strides apart. On a strict-streaming step
with several misses per layer × 40 layers, that is a large multiple of the theoretical minimum request count.

Do not read this as "9 sequential syscalls". They are already issued concurrently — loadMany dispatches all
misses × 9 operations through DispatchQueue.concurrentPerform (ExpertArena.swift:56). The hypothesis under
test is therefore narrower than "syscall overhead": it is that queue depth and seek scatter cost something the
SSD would not charge for one contiguous read. On Apple NVMe that may well be nothing. This is the main reason
Step 0 exists.

Why this is a distinct lever

To be precise about what this does and does not do:

• It reduces I/O request count and seek scatter. It does not reduce bytes. Byte reduction is notes/18 / bolt: long greedy generations can enter repetition loops (tracking) #47's mixed-precision residency territory.
• It is orthogonal to streaming: I/O–compute overlap in the chunk schedule (resident-first submit; bolt next-layer prefetch from frozen routing) #88 (I/O–compute overlap in the chunk schedule): streaming: I/O–compute overlap in the chunk schedule (resident-first submit; bolt next-layer prefetch from frozen routing) #88 hides latency behind compute, this shortens the latency being hidden. They compose; neither subsumes the other.
• It is worthless on resident tiers. This is a ≤32GB-tier issue, and the community rows it targets are the 16–18GB strict band (1.5–9.8 tok/s, README) and the slow-NAND approximation.

Design sketch

Generate an expert-major sidecar at qwisp pull time: for each layer, one contiguous record per expert containing all 9 tensors' slices for that expert. A miss becomes one pread of the whole record into the arena's slot buffers.

Open questions that should be answered before building:

• Slot buffer layout. The arena's slots are keyed "{proj}.{part}" (ExpertArena.slots). A single pread lands one contiguous blob; either the slot buffers become views into one per-slot allocation with the record's internal layout, or the read is followed by a scatter — and a memcpy scatter gives back part of what the single read won. Prefer the former; confirm it is expressible without touching the gather kernels' expectations about slot buffer strides.
• Disk cost. Naively this duplicates the expert weights (~18GB). Options: convert in place and drop the original expert tensors (breaks any consumer expecting stock safetensors layout — including the oracle), or make the sidecar opt-in for the streaming tier only and accept the duplication on machines that by definition have the least room. Neither is obviously right; this is the decision that gates the whole issue.
• Is the win real on NVMe? Apple SSDs handle queued random reads well, and the 9 reads are already issued in parallel by loadMany (confirmed at HEAD, see above). It is entirely possible that at these sizes the 9-read pattern is already bandwidth-bound and the gain is small. Measure before building.

Cheap Step 0

QWISP_SSD_ACCT=1 already accumulates acctBytes / acctReads / acctNanos in ExpertSource
(ExpertSource.swift:17, :154). Two accounting traps to avoid before trusting any number from it:

• No current code prints these counters. A reporting hook is part of Step 0, not a given.
• acctNanos is NOT batch wall time. It sums the durations of calls that ran concurrently, so it
  overstates elapsed time by roughly the achieved parallelism. The elapsed figure to compare against is
  LayerExpertCache.preadNanos, measured around the whole loadMany (ExpertArena.swift:310).
  Using acctNanos as the speed metric would manufacture a win that isn't there.

Before any layout work:

1. Capture reads, bytes and real pread nanos for a strict-streaming decode at C=64 and C=128, throttle off.
2. Compute achieved GB/s per read size. If it is already near the device's sequential rate, the request count is not the problem and this issue closes not-planned.
3. Repeat under QWISP_SSD_THROTTLE_GBS=1.5 — the slow-NAND regime is where request overhead should dominate most, and if the win does not appear there it will not appear anywhere.

A standalone microbench — same total bytes, 9 scattered preads vs 1 contiguous pread, at the real slice sizes — settles it in an afternoon without touching the engine.

Gates (only if Step 0 says go)

• G-A: sidecar records byte-identical to the tensors they replace, all layers × 256 experts, verified against the stock safetensors slices.
• G-B (lossless): OUT_TOKENS byte-identical sidecar-on vs sidecar-off, strict and bolt, C ∈ {64, 128} — the notes/10 G-B idiom. Flag-off byte-unchanged.
• G-D (speed, driver-decided): bench_batch on/off with missTotal / preadNanos / tok/s at throttle 0 and 1.5, plus a real 16GB run (Call for testers: benchmark qwisp on your Apple Silicon Mac (8/16GB + slow-NAND MacBooks especially wanted) #38) — the throttle approximation has already been shown to understate real memory pressure (README streaming rows).

Refs

#88 (I/O–compute overlap — composes with this, does not replace it) · #47 / notes/18 (byte reduction via mixed-precision residency — the other axis) · #38 (community 16GB rows, the target population) · ExpertSource.preadInto (:11, :144, acct at :17/:154), ExpertArena.loadMany (:56) / ensure, LayerExpertCache.preadNanos (:310) · notes/10 (streaming sync/IO decomposition, G-B idiom)