feat(moe): NVMe disk tier for MoE expert banks - #337

Open

CraigStone-Dev wants to merge 38 commits into

FlashML-org:mainfrom

CraigStone-Dev:feat/moe-disk-tier-standalone

Commits on Sep 2, 2026

feat(moe): disk tier v0 -- NVMe-backed MoE experts (VRAM <- RAM <- NVMe)
CraigStone-Dev
committed
cd5f27f
fix(moe): disk tier H2D must view staging bytes as the dst dtype
CraigStone-Dev
committed
ae5f736
fix(moe): disk tier offsets must be absolute (header + data section base)
CraigStone-Dev
committed
5be9f88
test(moe): disk tier test staging must be per-thread (mirror production)
CraigStone-Dev
committed
0654894
fix(moe): disk tier guard must require --cuda-graph-max-bs 0 (None auto-enables graphs)
CraigStone-Dev
committed
01c0687
fix(moe): madvise needs ctypes argtypes (64-bit addr truncation) + regression test
CraigStone-Dev
committed
d7235fe
fix(moe): release_range must remap MAP_PRIVATE in place (shared /dev/zero ignores DONTNEED); index offsets need 64-bit
CraigStone-Dev
committed
e166a53
fix(moe): ExpertBanks is frozen -- use dataclasses.replace for the disk index
CraigStone-Dev
committed
2b33ae7
chore(moe): disk tier preadv failure diagnostics
CraigStone-Dev
committed
97c7c2c
fix(moe): disk tier staging size must use full row bytes (was dropping the row's leading dim -> 8KB buffer, EFAULT/segfault on 512KB segments)
CraigStone-Dev
committed
4f561cd
fix(moe): staging size ceiling rounding (was truncating sub-page rows to 8KB)
CraigStone-Dev
committed
f27d126
fix(moe): disk tier fetch pool threads must run under inference_mode (server tensors)
CraigStone-Dev
committed
de070c3
fix(moe): global-scale banks store per-expert fp32 scalars -- convert+fill, not byte copy; test matches real checkpoint layout
CraigStone-Dev
committed
97e45dd
fix(moe): sync default stream after disk-tier fetches (H2D copies on pool-thread stream must land before GEMM reads slots)
CraigStone-Dev
committed
baea729
fix(moe): guard disk-tier stream sync on cuda availability (CPU-only tests)
CraigStone-Dev
committed
f10a6e6
chore(moe): disk-tier debug instrumentation (per-layer fetch counts + one-shot slot verify)
CraigStone-Dev
committed
e16c93d
fix(moe): disk-tier verify device mismatch + no-crash guard
CraigStone-Dev
committed
dae3ff2
chore(moe): verify RAM-expert slots after PCIe copy
CraigStone-Dev
committed
3379fdc
chore(moe): verify decode-fetched disk expert slot (one-shot)
CraigStone-Dev
committed
3de0053
chore(moe): verify prefill layer 20 + first decode fetch (any layer)
CraigStone-Dev
committed
ee8b9be
chore(moe): verify all 6 fetched disk experts per layer (confirm staging race)
CraigStone-Dev
committed
191d385
chore(moe): verify ALL layer-0 disk experts (prefill) + 3 decode steps (race hunt)
CraigStone-Dev
committed
162098b
chore(moe): verify RAM-expert HOST row too (separate host vs PCIe-copy corruption)
CraigStone-Dev
committed
df3dca7
chore(moe): verify ALL RAM experts post-copy + identify corrupted-slot byte source
CraigStone-Dev
committed
90726fd
chore(moe): instrument copy_missing/fetch_pending (layer 0) for the all-RAM-mismatch hunt
CraigStone-Dev
committed
a9b93b6
chore(moe): verify decode slot-mapping end-to-end (slot data vs bookkeeping expert)
CraigStone-Dev
committed
5fd7d85
fix(moe): clear stale disk-slot bookkeeping in disk-tier prefill materialize
CraigStone-Dev
committed
541ea78
chore(moe): run disk-tier verify_ram on prefill only
CraigStone-Dev
committed
1c4f427
chore(moe): log per-rank bank row shapes vs disk-index row bytes at DiskTier init
CraigStone-Dev
committed
de5963b
fix(moe): close disk-tier staging-buffer reuse race with a per-thread ring
CraigStone-Dev
committed
76fa506
fix(moe): disk-tier fetch workers must run on the rank's CUDA device
CraigStone-Dev
committed
623ca1d
fix(moe): apply reviewer fixes from PR #337 (MT-z)
CraigStone-Dev
committed
de43469
chore(moe): disk-tier review cleanups (PR #337 items 4-5)
CraigStone-Dev
committed
795659d
test(moe): mincore startup check that the released bank tails stay unbacked
CraigStone-Dev
committed
b602e6d

Commits on Sep 3, 2026

Commits on Sep 4, 2026

feat(moe): build the disk index for every NVFP4 family, not just qwen3_5_moe

authored and
CraigStone-Dev
committed
7915ac1
test(moe): regression test for the [copy-miss] probe's disk-tier gate
pi agent
committed
c3abff6