Non-record: Cosine LR Schedule — -0.070 BPB improvement + Focal Loss Investigation (corrected)#1380

Open

ranausmanai wants to merge 5 commits intoopenai:mainfrom

ranausmanai:focal-loss-lm-pretraining

Commits on Apr 5, 2026

Non-record: Focal Loss for LM Pretraining — 1.1567 int8 BPB on RTX 4000 Ada

ranausmanai
and
claude
committed
Add full experiment log (55+ experiments, 13 rounds) and 8xH100 run script

ranausmanai
and
claude
committed
Add SOTA stack validation: focal loss gives -0.289 pre-quant BPB on SOTA code

ranausmanai
and
claude
committed
Correct focal loss eval bug, add train_gpt.py, update results

ranausmanai
and
claude
committed
Add novel experiment features: z-loss, token dropout, embedding mixup, gamma annealing, logit penalty, multi-cycle cosine

ranausmanai
and
claude
committed