Record: L-BFGS Causal SLOT — val_bpb 1.0046 (3-seed mean)#5
Closed
Record: L-BFGS Causal SLOT — val_bpb 1.0046 (3-seed mean)#5
Conversation
…11473 (3-seed mean) AR self-generated calibration (no val/train data during quantization). Recreated from PR openai#728 at @valerio-oai's request for clarity. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ptq-xsa-bigramhash3072 Record: AR Self-Gen GPTQ + XSA-all + BigramHash 3072×112 — val_bpb 1.11473 (3-seed mean)
3-seed mean 1.0046 (std 0.0003). Beats merged SOTA (1.1147) by 0.110. Novel: L-BFGS causal SLOT — optimizer (L-BFGS), space (logit), and constraint (causal, context-only positions). Passes flip test (PR openai#1240). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
resouer
force-pushed
the
submission/lbfgs-causal-slot
branch
from
46318b4 to
4ede046
Compare
Owner
Author
|
Closing: resouer/main is out of sync with upstream. Branches are clean and ready — will create upstream PRs on openai/parameter-golf when approved. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
3-seed mean val_bpb: 1.0046 (std 0.0003) | ~15.8 MB | 8xH100 SXM | ~556s SLOT eval
Merged SOTA (PR openai#1019, 3-seed mean): 1.88218 nats. This run: 1.69620 nats. Delta: -0.186 nats. Clears the 0.005-nat threshold.
Results (3-seed)
Changes from Merged SOTA (PR openai#1019)
1. L-BFGS Causal SLOT in Logit Space (Novel)
Standard SLOT optimizes delta using loss from ALL positions including future ones — PR openai#1240 proved 100% causal violation. Our causal SLOT restricts optimization to already-scored context positions only. L-BFGS optimizer in logit space (max_iter=25, history=20, focal loss on last 128 tokens, warm-start, delta clamp +/-5). Delta: -0.087 BPP, ~556s eval.
Nearest PR: openai#1318 (L-BFGS logit SLOT, non-causal). Different: causal constraint on optimization — loss from context positions only.
2. Pre-quant AdamW TTT (6 epochs)
AdamW TTT on full-precision EMA weights before GPTQ. Delta: -0.022 BPP, 110s.
3. Coprime-stride multi-shard data loader
Weighted random shard sampling with coprime stride. Delta: -0.003 BPP.
4. Config batch (QK_GAIN=5.0, WARMDOWN=4000, GPTQ damp=0.005)
Delta: ~-0.003 BPP combined.
Compliance
Satisfies all four NoesisGenesis conditions (Issue openai#677):
p_tdepends only on artifact and prefixx_1...x_{t-1}— causal SLOT uses only already-scored positionsModel weights never modified during eval. Only per-window throwaway delta (1024 floats) is optimized then discarded.
Reproduction
Credits
Base: PR openai#1019 (@abaybektursun). Pre-quant TTT: PR openai#1006. Coprime loader: PR openai#1184 (@icryo). L-BFGS SLOT concept: PR openai#1318. Causal SLOT: our PR openai#1306.