5090 tower: install Unsloth + verify MoE LoRA training works

[joelteply]

Problem

The 5090 tower (32GB VRAM, RTX 5090) has PEFT + transformers + bitsandbytes installed but NOT Unsloth. Unsloth is our preferred training framework (2x faster LoRA, memory efficient). Without it, fine-tuning is slower and uses more VRAM.

Additionally, the Qwen3.5-35B-A3B model is MoE (Mixture of Experts) which has specific LoRA training considerations.

MoE Training Challenges

1. Memory: All 35B params loaded during training (not just 3B active). At FP16 = ~70GB — won't fit in 32GB VRAM without quantization
2. Quantized training: Need QLoRA (4-bit base + LoRA adapters) — PEFT + bitsandbytes support this
3. Target layers: LoRA attaches to shared attention layers (q_proj, k_proj, v_proj, o_proj). Expert FFN layers are usually frozen.
4. Router gradients: The MoE gate/router may or may not need gradient updates during LoRA training
5. Unsloth MoE: Unsloth added Qwen3 MoE support — need to verify with this specific model

Steps

1. Install Unsloth on 5090 tower: pip install unsloth
2. Verify Unsloth loads Qwen3.5-35B-A3B with 4-bit quantization
3. Test LoRA training on a small dataset (10 examples, 1 epoch)
4. Measure VRAM usage during training (target: <30GB with QLoRA)
5. If Unsloth doesn't support MoE: fall back to PEFT directly
6. Wire successful training path into Academy pipeline

Hardware

• RTX 5090: 32GB VRAM, CUDA 12, 170 SMs
• Model: /home/joel/.continuum/models/Qwen3.5-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-Q4_K_M.gguf
• Note: For training, need the Safetensors version (not GGUF). May need to download Jackrong/Qwen3.5-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled in Safetensors format.

Related

• Evaluate Qwen3.5-35B-A3B as local inference model — Opus reasoning distilled, 3B active #417 (model evaluation)
• Academy: no full training session proven end-to-end #377 (full academy session e2e)
• Academy teacher requires cloud API key — train local teacher adapter #374 (local teacher — this model could BE the teacher)
• Ship a LoRA-tuned local model that passes coding challenges via our tool system #344 (ship LoRA-tuned model)

[joelteply]

MoE expert specialization strategy:

MoE is ideal — experts are already specialized domains. Fine-tune PER EXPERT:

• Code expert: coding traces from Academy + sentinel tool usage
• Reasoning expert: chain-of-thought from complex tasks
• General expert: leave as-is (Opus distillation already good)

Publish with domain suffix: -code-cont, -reason-cont, -cont (all)

Plasticity compaction opportunity: prune unused experts for deployment targets.
MacBook Air coding-only use case → drop non-code experts → smaller model, same coding quality.

This is the -code suffix Joel mentioned. MoE makes per-domain adapters natural — each expert IS a domain.

[joelteply]

NOTE: Manual pip installs on the tower are how we got into installation nightmares before. Unsloth was installed ad hoc — already showing torchvision version mismatch warnings.

Training deps need to go through install.sh or a pinned requirements-training.txt. Filed for future: any tower dependency change should update the install script, not be done by hand. The install IS the product (#385).

[joelteply]

Qwen3.5-4B loaded successfully on 5090!

• VRAM: 3.4GB at 4-bit quantization
• LoRA adapter attached (r=16, targeting q/k/v/o projections)
• Leaves 28GB VRAM headroom for training batches

Inference hitting a multimodal processor bug (tries to interpret chat template as image URL). Minor fix needed — skip the vision processor for text-only inputs.

Next: Fix inference, run a small training loop, then scale up to real Academy training data.

[joelteply]

SUCCESS! Qwen3.5-4B VL loaded + LoRA + inference working on 5090!

• VRAM: 3.4GB at 4-bit
• LoRA: attached (r=16, q/k/v/o projections)
• Inference: working via processor text-only mode (skip vision when no images)
• Model naturally emits <think> reasoning tags before answering
• VL capability preserved: same model can see screenshots AND write code

Key fix: use AutoProcessor with text= only, remove pixel_values from inputs for text-only requests. Vision requests pass images normally.

Ready for training. Next: create training dataset from Academy traces and start a real LoRA fine-tuning run.

[joelteply]

Training launched on 5090 — 10 Continuum tool-use examples, 3 epochs, Qwen3.5-4B VL + LoRA.

Note: This was done via raw SSH + Python script. Real training MUST go through jtag commands (genome/academy-session, grid/send for remote dispatch). The manual approach proved the model loads and trains, but production training should use the existing Academy pipeline on the tower via Grid commands. #377 tracks the full e2e Academy session.

[joelteply]

FIRST TRAINING RUN COMPLETE ON 5090!

Results:

• Model: Qwen3.5-4B VL, 4-bit quantization, LoRA r=16
• Dataset: 10 Continuum tool-use examples, 3 epochs
• Loss: 2.278 → 1.491 (learning our tool format)
• Training time: 30 seconds
• VRAM: 3.4GB (28GB headroom)
• Adapter saved to /tmp/continuum-lora-test/final

Post-training inference doesn't produce clean tool XML yet (10 examples isn't enough). Need:

1. Hundreds of training examples from Academy traces
2. More epochs
3. Real evaluation against tool-use benchmarks

But the pipeline is PROVEN: load model → attach LoRA → train → save adapter → inference. The 5090 works. Next: scale up training data.

[joelteply]

Verified this session:

• Unsloth installed on 5090 (v2026.3.11)
• Qwen3.5-4B VL loads at 3.4GB VRAM with 4-bit quantization
• LoRA attaches (r=16, q/k/v/o projections)
• First training run completed: 10 examples, loss 2.278→1.491
• Academy session running on tower generating training data
• 35B MoE doesn't fit 32GB (all experts must be resident) — need expert paging (MoE expert paging: load only the needed expert on demand, page rest from HF cache #433) or surgery (MoE surgery: extract individual experts for targeted training + tiny deployment #439)
• Text-only inference works via AutoProcessor (bypass vision for text inputs)
• Training deps added to install.sh (PR install.sh: auto-install training deps on GPU machines #431)