Non-record: QAT + Neural Cache + LoRA TTT

[Bortlesboat]

Summary

Non-record submission exploring three eval-time techniques stacked on the #1 training recipe (Int5-MLP + BigramHash + SWA by @thwu1):

1. Quantization-Aware Training (QAT): STE fake-quantization during training matching int5/int6 export format
2. Neural Cache: Hidden-state similarity lookup with logaddexp interpolation during sliding window eval
3. LoRA Test-Time Training: Per-document rank-8 LoRA adaptation with entropy-gated updates

Results

Seed	Pre-quant val_bpb	Post-quant sliding val_bpb	Steps
1337	1.1739	1.4245	5109

Known Issues

The QAT implementation has a mismatch: STE uses symmetric clipping while the export uses percentile-based per-row scaling. This causes a 0.25 BPB quantization penalty instead of the expected ~0.02. Submitting as non-record to document the approach — the neural cache and LoRA TTT implementations are validated and working, and will show gains once the QAT bug is fixed.

Test plan

• Full 8xH100 training run completed (5109 steps, 600s)
• SWA applied (24 checkpoints)
• Sliding window eval completed on 1xH100
• Neural cache eval validated on H100
• Fix QAT export mismatch
• 3+ seeds for statistical significance

[Bortlesboat]

Superseded by #1169 (better score). Closing.