MoE expert paging: load only the needed expert on demand, page rest from HF cache

[joelteply]

Idea

MoE models have N experts but only 1-2 activate per token. On memory-constrained devices (MacBook Air 8-16GB), we don't need ALL experts resident. Load only the expert needed for the current task, page others from HuggingFace cache or disk.

Architecture

Resident (always loaded):
  - Router/gate network (~100MB) — decides which expert to activate
  - Shared attention layers (~2GB) — used by all experts
  - Active expert FFN (~1-2GB) — the one currently needed

Paged (on disk/HF cache, loaded on demand):
  - Other N-1 expert FFNs (~1-2GB each)
  - Loaded when task domain changes (code → reasoning → creative)
  - LRU eviction when memory full

Memory savings

Qwen3.5-35B-A3B has ~8 experts:

• Full model: ~21GB (Q4_K_M GGUF)
• Router + attention + 1 expert: ~4-6GB
• Fits MacBook Air with room for the system

Connection to existing architecture

This IS genome paging (#382) at a deeper level:

• Genome paging: swap LoRA adapters in/out
• Expert paging: swap MoE expert weights in/out
• Same LRU eviction, same on-demand loading, same HF cache

Implementation questions

1. Can GGUF files be partially loaded? (load specific tensor groups)
2. Does Candle support lazy tensor loading?
3. Can we split a GGUF into per-expert shards at publish time?
4. What's the latency for paging in a 2GB expert from SSD? (~200ms?)
5. Could we pre-predict the next expert needed based on task classification?

Why this matters

• MacBook Air becomes viable for the full 35B MoE model
• Each device only stores experts it uses frequently
• Rare experts live on HF, fetched when needed
• The -code-cont suffix literally means "only the code expert is baked in"

Related

• Genome paging: activateSkill/evictLRU not wired end-to-end #382 (genome paging — same architecture, higher level)
• Evaluate Qwen3.5-35B-A3B as local inference model — Opus reasoning distilled, 3B active #417 (Qwen3.5 evaluation)
• 5090 tower: install Unsloth + verify MoE LoRA training works #430 (MoE training)
• Plasticity compaction — prune unused experts permanently for deployment

[joelteply]

Three-tier storage model — HF as the storage medium:

VRAM (L1):  Active expert + shared attention + LoRA    ~4-6GB, microsecond access
Disk (L2):  Recently used experts + adapters            ~20-50GB budget, millisecond access  
HF   (L3):  All published experts + adapters            Infinite, seconds on broadband

Local disk gets a configurable cache budget (default 50GB). LRU eviction keeps it within budget. Haven't used an expert in a week? Evict to L3 (still on HF). Need it? Fetch in ~30s on broadband.

This means:

• 500GB MacBook doesn't fill up with models
• Each install only stores what it actually uses
• New experts are a continuum pull away, not a manual download
• The -cont suffix models on HF ARE the storage medium
• LoRA adapters, experts, base models — all the same cache hierarchy

HF does the heavy lifting on storage and bandwidth. We do the smart caching and paging.

[joelteply]

Publishing strategy for external users (OpenCode, VSCode, generic):

Continuum trains full MoE models with all experts. Then publishes trimmed variants:

continuum-ai/qwen3.5-3b-a3b-cont          # Full: all experts, Continuum tool system
continuum-ai/qwen3.5-3b-a3b-code-cont     # Trimmed: code expert only, Continuum tools
continuum-ai/qwen3.5-3b-a3b-code          # Trimmed: code expert only, generic (VSCode/OpenCode)
continuum-ai/qwen3.5-3b-a3b-reason        # Trimmed: reasoning expert only, generic
continuum-ai/qwen3.5-3b-a3b-code-reason   # Trimmed: code + reasoning, generic

The system builds these automatically using our naming convention:

• -cont suffix = trained for Continuum's tool system
• No -cont = generic, works in any IDE/tool
• Expert name in suffix = only that expert's weights included
• Multiple experts = those experts only

The pipeline:

1. Academy trains on 5090 (full MoE, all experts)
2. Plasticity compaction prunes unused experts per variant
3. adapter/publish pushes to HF with continuum:* tags
4. External users pull the generic variant for their IDE
5. Continuum users get the -cont variant with tool-system training baked in

Our nomenclature becomes the standard for MoE expert-trimmed models. Nobody else is doing this.