Add non-record JEPA byte-level encoder-decoder submission#696

Open

gravelBridge wants to merge 28 commits intoopenai:mainfrom

gravelBridge:main

Commits on Mar 24, 2026

jepa test
gravelBridge
committed
Switch tokenizer config to pure byte export
gravelBridge
committed

Commits on Mar 25, 2026

Make JEPA target loss fully end-to-end
gravelBridge
committed
Restore detached JEPA target loss
gravelBridge
committed
Full-sequence byte decoder with cross-patch causal attention
gravelBridge
committed
Rebalance params: depth-recurrent encoder (5x2) + 6-layer decoder
gravelBridge
committed
Teacher-forced decoder: condition on encoder context, not predicted latents
gravelBridge
committed
Revert "Teacher-forced decoder: condition on encoder context, not predicted latents"
gravelBridge
committed
Rebalance loss: CE weight 3.0, pred weight 0.5
gravelBridge
committed
Scale up model to fill 16MB budget: dim 480, 8 decoder layers
gravelBridge
committed
Add sliding window evaluation for final val_bpb
gravelBridge
committed
Add LeakyReLU(0.5)², EMA+SWA, and test-time training
gravelBridge
committed
Apply SWA weights during warmdown and use pure CE for TTT
gravelBridge
committed
Remove SWA, use EMA-only weight averaging
gravelBridge
committed
INT6 optimal-clip quantization, late QAT, LZMA compression
gravelBridge
committed
Reduce default train_seq_len from 4095 to 2047
gravelBridge
committed
Fix train_batch_tokens alignment for seq_len 2047
gravelBridge
committed
Fit within 16MB budget: LZMA preset 9, INT6 for all categories
gravelBridge
committed
Fix TTT divergence: freeze encoder, disable QAT during eval
gravelBridge
committed
Speed up eval: stride 256, TTT epochs 1
gravelBridge
committed
Reduce decoder layers from 8 to 7 to fit 16MB budget
gravelBridge
committed
Remove standalone sliding window eval, use TTT epochs 2
gravelBridge
committed
Adopt SOTA optimizer hyperparameters
gravelBridge
committed
prepare non-record submission
gravelBridge
committed
Merge branch 'openai:main' into main
gravelBridge
authored
Revert non-records files to upstream versions for clean merge
gravelBridge
committed
Merge branch 'openai:main' into main
gravelBridge
authored
Merge branch 'openai:main' into main
gravelBridge
authored

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add non-record JEPA byte-level encoder-decoder submission#696

Add non-record JEPA byte-level encoder-decoder submission#696
gravelBridge wants to merge 28 commits intoopenai:mainfrom
gravelBridge:main

Commits on Mar 24, 2026

jepa test

Switch tokenizer config to pure byte export

Commits on Mar 25, 2026

Make JEPA target loss fully end-to-end

Restore detached JEPA target loss

Full-sequence byte decoder with cross-patch causal attention

Rebalance params: depth-recurrent encoder (5x2) + 6-layer decoder

Teacher-forced decoder: condition on encoder context, not predicted latents

Revert "Teacher-forced decoder: condition on encoder context, not predicted latents"

Rebalance loss: CE weight 3.0, pred weight 0.5

Scale up model to fill 16MB budget: dim 480, 8 decoder layers

Add sliding window evaluation for final val_bpb

Add LeakyReLU(0.5)², EMA+SWA, and test-time training

Apply SWA weights during warmdown and use pure CE for TTT

Remove SWA, use EMA-only weight averaging

INT6 optimal-clip quantization, late QAT, LZMA compression

Reduce default train_seq_len from 4095 to 2047

Fix train_batch_tokens alignment for seq_len 2047

Fit within 16MB budget: LZMA preset 9, INT6 for all categories

Fix TTT divergence: freeze encoder, disable QAT during eval

Speed up eval: stride 256, TTT epochs 1

Reduce decoder layers from 8 to 7 to fit 16MB budget

Remove standalone sliding window eval, use TTT epochs 2

Adopt SOTA optimizer hyperparameters

prepare non-record submission

Merge branch 'openai:main' into main

Revert non-records files to upstream versions for clean merge

Merge branch 'openai:main' into main

Merge branch 'openai:main' into main