Record: GDN-Hybrid (Gated DeltaNet + SWA) — val_bpb 1.028308 (3-seed cold-cache mean)

[Abhishek8108]

Summary

val_bpb = 1.028308 (3-seed cold-cache mean) | 14.48–14.70 MB | 8×H100 SXM

First non-transformer architecture in the 10-min record track. Beats merged SOTA (PR #1493, 1.0810) by 5.27 centiBPB. No TTT. Fixed predictor.

Architecture

GDN-Hybrid replaces the transformer backbone with Gated DeltaNet (delta-rule linear recurrence) + Sliding Window Attention:

[GDN×5] → [SWA] → [GDN×5] → [SWA_shared]

• 33,862,953 params, 512-dim, SP1024 tokenizer
• GDN: fla.layers.GatedDeltaNet, head_dim=64, use_short_conv=True
• SWA: window=512, 8 heads / 4 KV heads, weight-shared across both layers
• QK-Gain 5.0, BigramHash(3072×112) + trigram embeddings, logit softcap 30.0
• Quantization: full-Hessian int6 GPTQ + zstd-22

3-Seed Results (cold-cache, fresh pods)

Seed	Steps	EMA BPB	Quantized BPB	Artifact (bytes)
42	1857	1.017970	1.027163	15,188,240
1337	1858	1.018624	1.027614	15,417,768
2024	1858	1.020559	1.030148	15,314,099
Mean	—	1.019051	1.028308	—
Std	—	0.001356	0.001610	—

All three seeds run on separate fresh pods. Cold-start signature confirmed (step 1 at ~105s, Triton JIT overhead). t-stat = 56.7 vs SOTA threshold, p ≪ 0.01.

Reproduction

pip install flash-linear-attention zstandard sentencepiece
python3 data/cached_challenge_fineweb.py --variant sp1024

SEED=42 ARCH_MODE=D MAX_WALLCLOCK_SECONDS=590 ITERATIONS=9999 \
  TRAIN_SEQ_LEN=2048 TRAIN_BATCH_TOKENS=786432 \
  QK_GAIN_INIT=5.0 GPTQ_ENABLED=1 VAL_LOSS_EVERY=9999 \
  torchrun --standalone --nproc_per_node=8 \
  records/track_10min_16mb/2026-04-11_GDN_Hybrid_DeltaRule/train_gpt.py

Expected on a cold pod: ~1857–1858 steps, quantized BPB ~1.027–1.030. See README for full details.

Compliance

Fixed predictor (Track A). No TTT, no RLS, no SLOT, no n-gram mixer at eval time. GPTQ calibration uses model-generated synthetic sequences only. Sliding-window eval is strictly causal, single-pass, normalized softmax distribution.

Supplemental Evidence

One warm-cache seed=1337 run (same config, pre-compiled Triton kernels) reached 1.015890 BPB (2247 steps). This is not part of the submitted 3-seed claim. The official claim is based solely on the cold-cache runs above.

[SPThole]

BPB metric bug: double-counted leading-space bytes

The build_sentencepiece_luts function in train_gpt.py has a + 1 in base_bytes that double-counts the space byte represented by ▁:

Submission (line ~275)

if piece.startswith("\u2581"):
    has_space[i] = True
    base_bytes[i] = len(piece[1:].encode("utf-8")) + 1   # ← space byte baked in

Compare with the reference implementation in the repo's train_gpt.py:

Reference (line ~196-199)

if piece.startswith("▁"):
    has_leading_space_np[token_id] = True
    piece = piece[1:]
base_bytes_np[token_id] = len(piece.encode("utf-8"))      # no +1

The eval function then adds the space byte again via the has_leading_space_lut conditional (same in both):

tb = base_bytes_lut[tgt]
tb += (has_leading_space_lut[tgt] & ~is_boundary_token_lut[prev])   # +1 again

So for a token like "▁hello" (representing " hello", 6 bytes): the reference correctly counts 5 + 1 = 6, but this submission counts 6 + 1 = 7. Since ~65% of tokens carry a leading space, this inflates the total byte count by ~14%, which deflates the reported BPB by the same factor (BPB = bits / bytes). The claimed 1.028 BPB would be approximately ~1.18 when computed with the reference byte counting.

[Abhishek8108]

You're right — thank you for the careful read.

The bug is confirmed: base_bytes[i] = len(piece[1:].encode("utf-8")) + 1 double-counts the space byte that the eval already adds via has_leading_space_lut. With ~65% of tokens carrying a leading space, the byte denominator is inflated ~14%, deflating the reported BPB by the same factor. The corrected BPB is ~1.18, not 1.028.

Closing this PR.