CHANGELOG.md

Changelog

All notable changes to OdyssNet will be documented in this file.

The format is based on Keep a Changelog.

[3.6.1] — 2026-09-09

Changed

• The two experiment docstrings point at docs/LIBRARY.md instead of repeating it. experiment_diffusion.py carried seven benchmark tables that docs/LIBRARY.md already held verbatim, and experiment_llm.py carried four more; together they were 589 lines of module docstring in front of the code they describe, which is the case AGENTS.md rules out. The diffusion tables were duplicates and are gone; the LLM ones had no home, so the tokenizer comparison, the --sweep gap and --sweep coldstart results, the --compile speedups and the reference run moved into a new Language Modeling section in docs/LIBRARY.md before the docstring was cut. Every figure is the one that was already published — nothing re-measured, nothing dropped. The docstrings keep the mechanism, the usage lines and the constraints a reader needs before touching the file. Verified with 370 pytest and both --mode smoke runs.

• Version and history references out of the code and the READMEs. "New in 3.0" / "New in 3.1.1" bullets, "the 2.x architecture" as a name for the no-attention arm, and inline "see CHANGELOG 2.6.4" pointers stated the project's history in files meant to describe it as it is. The claims and their numbers stay; only the dates leave. Same for a handful of comments that explained a line by narrating the bug it replaced.

[3.6.0] — 2026-09-09

Changed

• The diffusion example walks the rectified-flow straight path by default, --interpolant rf. Everything downstream of the schedule — q_sample, the targets, the parameterisation conversions, all five samplers, the cadence embedding — reads it through alpha_bar and sigma, so an interpolant is a table here and nothing else. x_u = (1-u) x_0 + u eps divided by its own norm is exactly sqrt(ab) x_0 + sqrt(1-ab) eps at ab = (1-u)²/((1-u)² + u²), with sigma = u/(1-u); both identities are checked in --mode smoke, and _step_euler turned out to already be rectified-flow Euler written in sigma coordinates.

  That settles the parameterisation question a flow interpolant usually raises. The rank ceiling documented since 3.2 belongs to what the network is asked to output, not to the path it walks, so a velocity target would carry epsilon at full rank and hit the same ceiling --predict eps does. x_0 stays and the flow lives in the schedule, where it costs nothing.

  MNIST cannot answer which path is better: all four arms sit at 96–98% at K=4, and the cosine arm's own Frechet distance moved 17.1 to 10.5 between seeds, wider than the gaps under test. CIFAR-10 separates them — 768 neurons, 10 minutes per arm at equal wall clock, two seeds, averaged over K=8 to 64 since K=4 is the degenerate end of every curve:

  arm             frechet 42/123   fidelity 42/123
  cosine             7.48 / 8.25      48.3 / 48.3
  rf                 7.22 / 6.13      48.0 / 49.1
  logitnorm          8.03 / 7.30      39.2 / 37.9
  rf_logitnorm       5.01 / 7.70      45.5 / 45.8
  

  rf gives the same conditioning fidelity as cosine with a better Frechet distance at both seeds. --interpolant cosine restores the old schedule, and --sweep flowmatch is the grid.

Added

• --t-density logit_normal, off by default. It draws --k-range's interior stops from a logistic rather than uniformly, concentrating them where the image is decided rather than spreading them over an axis whose ends are nearly settled (Esser et al. 2024). It does what it claims — measured in the smoke test, the middle half of the schedule holds 52% of uniform stops and 73% of these — and it costs nine to eleven points of conditioning fidelity at both seeds, which is why it is off. The likely reason is a constraint this file has and that paper does not: here the training grid is the sampling grid, so thinning the noisy end leaves the sampler walking through timesteps training barely visited. Untested. rf_logitnorm has the best Frechet distance in the table on one seed and does not reproduce it on the other, which is the pattern --cadence and --echo-cadence both showed.

• --echo-cadence re-measured on CIFAR-10, where it loses. 3.5.0 shipped it off on the grounds that the seeds disagreed; MNIST turned out not to separate the arms at all, the same way it could not separate the interpolants. On CIFAR-10 the E-axis span — the one axis the flag exists to improve — is 7.2 and 4.0 against a plain drawn E's 3.8 and 2.4, behind at both seeds, and 9.6 points of conditioning fidelity worse at one of them. The flag stays for the mechanism it demonstrates and the wording changes from "no separation" to "measured and lost".

Fixed

• A checkpoint older than a config field was scored under today's default for it. adopt_saved_arch only adopted fields the saved payload actually carried, so when --interpolant changed default the base checkpoint — written before the field existed — sampled at 89.2% instead of its measured 91.2%, silently, since nothing about the state dict disagrees. Adopted fields now fall back to what the default was when the checkpoint could have been written, and the smoke test strips a field from a payload to prove it.

[3.5.0] — 2026-09-09

Changed

• The diffusion example draws the thinking depth per batch as well, --e-range 2,6 by default. 3.4.0 made the number of denoising steps something the caller chooses; the depth spent between them was the other value baked into the weights. K decides which timesteps are visited and E decides how long the core thinks between them, so E can be drawn without touching the training distribution at all — nothing in the batch has to know which depth it will be run at, which is why the draw lives in train_step rather than trajectory_batch.

  MNIST, echo 4, 6 minutes per arm at equal wall clock, two seeds. --mode flex gains an echo axis (--flex-e) and reports the two spans separately, since flexibility in K and in E are separate claims; the numbers below are the span of conditioning fidelity across nine step counts at E=4 and across six echo depths at K=16, lower being flatter:

  arm            span 42       span 123
                 K axis / E    K axis / E
  fixed          17.6 / 10.2   16.6 /  6.2
  rand_k         10.8 /  8.0    7.2 /  5.2
  rand_e          9.2 /  5.8    3.6 /  2.6
  rand_ke         5.2 /  3.0    6.0 /  3.2
  ecad           20.8 / 16.6   18.0 / 10.4
  rand_e_ecad     8.8 /  2.8    6.4 /  3.2
  

  A drawn E beats the fixed control on both axes at both seeds and leaves the Frechet distance flat, which is the case --k-range was made the default on. It also flattens the K axis while drawing only E, so the two are not independent knobs — rand_ke is no better than either alone and its Frechet distance is worse everywhere (15.9–17.8 against 9.1–11.5).

  --sweep flexe is the grid; --e-range off restores the fixed depth. Sweep arms now pin both ranges off wherever the axis under test is the other one, --sweep depth included, since a drawn K or E would average away the split those arms exist to compare. Existing checkpoints are unaffected and base still samples 91.2% / 9.341 at K=16.

Added

• --echo-cadence, off by default, which names a hole in the mechanism it cannot yet be shown to fix. A frame is injected once and repeated for every echo step of its run — byte for byte, eleven of eleven transitions identical under torch.equal. So the core can count the steps it has taken and never the ones it has left: the last step of an E=2 run and the second step of an E=6 run are the same input to the same state, and a drawn E asks for the answer at a moment it has no way to see coming.

  The flag widens the frame axis to K*E, so forward resolves ratio = 1 and each echo step carries its own vector: a sinusoidal embedding of the steps remaining and the fraction elapsed. The remaining count is left unnormalised, because "two steps left" has to mean the same thing whatever E is for a depth outside the training range to be readable. Measured on the mechanism rather than argued — run the same frame for two steps, built once for E=2 and once for E=6: without the signal the hidden states are bit-identical, with it they differ by 2.3e-1.

  It is off because the measurement does not support turning it on. Alone it is worse than the fixed control at both seeds, since at a fixed E the remaining-step signal is the same constant sequence every batch and pins the model to that depth harder rather than freeing it — the same shape of result --cadence gave in 3.4.0, more sharply. Paired with a drawn E it leads the E axis at one seed (2.8 against 5.8) and trails at the other (3.2 against 2.6), which is not a separation. It also costs: K*E entries make the frame tensor E times larger, and the arm reached 18% fewer gradient steps in the same wall clock. Widening the input fixes a tensor shape, so it lives in ARCH_FIELDS.

  The general form of this gap is a library one, not an example one: OdyssNet is a network whose depth is time, and nothing in its API lets a model be told how much time it has. That is worth fixing where every example can reach it rather than here.

Fixed

• The diffusion example's --carry independent arm stepped the optimizer mid-trajectory. The memoryless control issues K separate calls and told the trainer to accumulate over cfg.frames instead of that K. Once --k-range became the default in 3.4.0, K moved from 12 to 20 while cfg.frames stayed 16, so the accumulation counter drifted: the step landed part-way through a trajectory and the reported loss was scaled by K/16. Only the control arm was affected — the default trajectory arm makes one call — but the memory sweep's independent row was measured under it and needs re-measuring.

[3.4.0] — 2026-09-04

Changed

• The diffusion example's step count is now a dial the caller turns, and --k-range 12,20 is the default that makes it one. Every other diffusion tool lets you choose the number of denoising steps at sampling time; this one could not, because the training grid is the sampling grid and the weights were fitted to a single cadence. On a fixed-grid checkpoint, conditioning fidelity falls monotonically as K rises — 97.2% at K=6 down to 68.8% at K=64 on MNIST, 77.6% to 32.6% at K=32 on CIFAR-10 — while the Frechet distance is best near the trained grid. The step count was part of the architecture rather than a parameter.

  --k-range LO,HI draws K per batch over a random monotone grid, so the weights meet many cadences instead of one. --mode flex scores a checkpoint across step counts and --sweep flexk trains the arms against a fixed-grid control; --k-range off restores the old behaviour. MNIST, 6 minutes per arm at equal wall clock, echo 2, 500 samples at each of nine step counts, fidelity at K=4/16/64 and the span across all nine, on two seeds:

  arm            K=4   K=16   K=64   span 42   span 123
  fixed         98.4   88.6   55.8      42.8       45.4
  rand_k        97.4   92.2   82.6      14.8       10.4
  cadence       99.2   93.0   68.0      31.2       41.6
  rand_cadence  99.8   94.8   83.4      16.4       19.6
  

  The span is the result: randomising K cuts it three to four times on both seeds and holds the Frechet distance flat from K=12 to K=64, where the fixed arm's climbs from 9.3 to 15.7. It costs one to three points at K=4. Worth noting where the arms were trained — K is drawn from 12 to 20 while the flexibility reaches K=4 and K=64 either side of that range, so what is learned is that cadence is a quantity to read rather than the set of step counts seen. A narrow range is enough, which is why the default is narrow.

  The val MSE column ranks fixed first and disagrees with all of this. It is scored on the fixed --frames grid, that arm's own training distribution and one cadence out of many for the others — the same grid-choice trap that inverted the memory sweep in 3.3.1. The sample columns decide.

  Existing checkpoints are unaffected: they carry k_range in their config, --mode sample and --mode eval do not use it, and base still samples 91.2% at K=16.

Added

• --cadence, off by default, which did not survive its second seed. It widens each frame with the log-sigma stride it is about to take and the fraction of the walk behind it, on the reasoning that a frame cannot infer its own stride until it has seen two of them — by which point the first prediction is made. It does lift conditioning fidelity at every step count, 93.0% against 89.0% at K=16 on the first seed. But its apparent flexibility gain, a span of 31.2 against the fixed arm's 42.8, came back at 41.6 against 45.4 on the second seed, which is no gain at all; and combined with --k-range it makes flexibility consistently worse (16.4 and 19.6 against 14.8 and 10.4) along with the Frechet distance everywhere. Telling the model the stride is evidently not what taught it to read the stride, and the two signals appear to interfere. The flag stays because the fidelity lift is real and reproduced; it is off because the rest is not. Widening the input fixes a tensor shape, so it lives in ARCH_FIELDS.

[3.3.1] — 2026-09-04

Added

• Three more samplers in the diffusion example, and the stop placement as its own axis. --sampler gains euler, euler_a and dpmpp_2m beside ddim and ddpm, and --sigma-schedule selects uniform or karras independently, so "DPM++ 2M Karras" is the two flags together the way the tools that popularised it keep them apart. Every sampler calls the model exactly once per denoising step, which is the constraint this architecture imposes rather than a simplification: a multistage solver evaluates the denoiser twice inside one step, and here a second call would advance the recurrent state a second time. Multistep solvers, whose history is their own previous output, compose with it cleanly.

  --mode bench scores a trained checkpoint across every sampler and placement over several seeds without training anything, which --mode sweep cannot do without re-training an arm per sampler for a choice made after the weights exist.

  Measured on MNIST base, 500 samples, mean of seeds 42/123/54321, cfg 3.0: euler reproduces ddim to three decimals (9.522 Frechet either way), which is the arithmetic checking itself — at eta=0 both integrate the same probability-flow ODE in different coordinates. Two findings against expectation. The second-order dpmpp_2m is behind the first-order solvers, by 1.6 points of fidelity on MNIST and 2.5 on CIFAR-10, because its correction assumes the denoiser is a pure function of (x_t, t) and this one carries its own history — the solver's memory is a second one built on an assumption the architecture breaks. And karras loses by 20 to 50 points on both datasets, because it moves the stops onto timesteps a model trained on its own sampling grid has never seen. Both axes are kept and documented rather than hidden: the first is a real result about stateful denoisers, and the second should behave normally on a model trained across the full schedule.

  The default is unchanged at ddim/uniform, so every published figure stands.

Changed

• The diffusion example's default guidance scale is 3.0, measured up from 2.0. Scored with --mode sample over 500 images on the MNIST base checkpoint, 3.0 leads 2.0 on both sample columns at all three seeds tried — 86.0% to 91.2% at seed 42, 87.4% to 93.0% at 123, 84.8% to 91.8% at 54321, with the Frechet distance improving alongside. Past 3.0 the two columns separate: 5.0 reaches 94.0% fidelity while the Frechet distance turns back to 10.9, which is guidance buying class-purity with variety, so the default stops where both still agree. --mode eval --tag base consequently reads 91.2% rather than 86.0%, batch sensitivity 90.8–92.8%, and carried-versus-wiped 91.4% against 53.6% — a wider gap than before, because guidance amplifies the memoryless arm's error too. The sweep tables are unchanged and still say they were measured at guidance 2.0. --eta stays at 0.0: it helped CIFAR at cfg 5 and hurt it at cfg 3, and hurt MNIST at two seeds of three.

Fixed

• Validator scored every arm on a shared-epsilon grid while the default trains on iid noise, so the val column measured the wrong thing. The grid was built with one epsilon expanded across all K frames regardless of --traj-noise. That is traj_noise_shared's own training distribution and a foreign one for every other arm, and it inflated the reported loss of any iid run: the cifar checkpoint reads 0.0773 on the old grid and 0.0631 on the fixed one, against a training loss of 0.0629. The apparent val-above-train gap on every iid run was this mismatch, not overfitting. The grid is now always iid, which is the distribution that keeps the column comparable across arms — a shared-epsilon trajectory determines x_0 from any two frames, so scoring on it credits an inversion as though it were a denoiser.

  This reverses one claim. traj_noise_shared did not hold the best held-out loss; on the fixed grid it scores 0.1762 against trajectory's 0.0967, which is the inversion shortcut failing exactly as the theory predicted. Every val figure in the four sweep tables, README.md, README_TR.md and docs/LIBRARY.md has been re-measured from the saved checkpoints. The --sweep depth val column reverses order as well: coarser denoising grids make each timestep a harder prediction, so k32_e2 leads on loss while k4_e16 still leads on conditioning fidelity at 95.0%.

  Conditioning fidelity and Frechet are unaffected — they come from measure_samples(), which never calls Validator — so RANK_KEY orderings, the 86.0% headline, the carried-versus-wiped result and the choice of iid as the default all stand as measured. --traj-noise iid remains the default on the sample columns, which are the ones that decide.

• The example read extra_data as a nested dict, but save_checkpoint merges it into the top level. adopt_saved_arch and the resume path both looked up payload["extra_data"]["cfg"], which was always empty. Loading any checkpoint whose architecture differed from the CLI defaults therefore failed outright — --mode sample --tag <an attention run> died with Unexpected key(s) in state_dict: "attn.q_proj.weight" — and --resume always restarted the step counter at zero with best_val at infinity. Both now read the top level.

• --lr was silently dropped on --resume. lr is a parameter-group key, and torch.optim.Optimizer.load_state_dict replaces the group dicts wholesale, so the checkpoint's rate overwrote whatever the command line asked for; ChaosGrad.load_state_dict only backfills keys that are missing, and this one never is. A resume with --lr 1e-2 went on printing ChaosGrad's adaptive estimate and stepping at it. run_session now passes cfg.lr to load_checkpoint, whose lr parameter already existed for exactly this. Zero-config resumes pass None and keep the carried estimate.

• --frames, --echo and --timesteps were ignored on --resume. adopt_saved_arch restored every field in ARCH_FIELDS, and those three sat in it despite fixing no tensor shape — they set which timesteps the grid visits and how many echo steps run between them. A resume asking for --frames 64 --echo 8 silently trained at the checkpoint's 16x4. They move to a new GRID_FIELDS, which the checkpoint supplies only when the command line is quiet: name one and it is honoured, leave it unset and it follows the checkpoint rather than falling back to the dataclass default. Everything left in ARCH_FIELDS does fix a shape and still always comes from the checkpoint — including attn_window, which fixes none but decides the context the weights were trained against.

[3.3.0] — 2026-08-28

Added

• Image diffusion on the chaos core — examples/advanced/experiment_diffusion.py. A class-conditional denoising diffusion model whose denoiser is one OdyssNet, with no UNet and no VAE. The mapping is native rather than bolted on: with pulse_mode=False and a (B, K, F) input run for K*E steps, forward resolves ratio = E by itself, so one denoising timestep is one injected frame, the E echo steps between frames are the depth a UNet would spend on layers, and the hidden state, the attention cache and the plastic trace all cross every frame boundary. The whole reverse trajectory is a single differentiable forward pass, which makes train_batch(..., full_sequence=True) against the trainer's default MSELoss the entire training call.

  vocab_size=[F_in, P] with vocab_mode='continuous' makes the model's own proj and output_decoder the encoder and decoder, and conditioning is fixed-basis — a sinusoidal clock and a class one-hot with a null slot for classifier-free guidance — so every learned parameter is inside OdyssNet. DDPM and DDIM samplers, classifier-free guidance, four ablation grids, a --mode smoke self-test and checkpointing through the library's own functions.

  The example ships a scoring instrument as well as a model: a small fixed convnet gives conditioning fidelity and a Frechet distance in its feature space. That distance is not FID — FID is defined against InceptionV3 — so it is named for what it is and compared only between arms of the same sweep. Its parameters are excluded from every count.

• --predict x0 is the default, because on this architecture the output rank decides what can be denoised at all. The answer is read off n_out neurons, so whatever the network emits is a rank-n_out view of a P-dimensional image. Epsilon is white noise — isotropic, full rank, incompressible — so a rank-192 view keeps 192/784 of its variance and pins the achievable MSE at 0.755 however long training runs. A 573k-parameter run measured 0.767: saturated, not undertrained, and no amount of training or width-at-fixed-n_out would have moved it. Natural images are low rank, and the same 192 directions carry all but 3.4% of MNIST's variance.

  Measured at 700 steps, each target against its own do-nothing predictor, with the per-frame loss at the noisy and clean ends of the schedule:

  target	val MSE / trivial	pure noise	nearly clean
  x_0	13.3%	0.237	0.096
  v	59.7%	0.236	0.998
  eps	80.5%	0.787	0.896

  v is x_0-like at high t and epsilon-like at low t, so it inherits the problem over half the range — visible above as a loss that is fine at pure noise and at the bound when nearly clean. --sweep size carries epsilon arms at four widths, and they behave as the rank argument requires: always above the bound, monotone in n_out, closing on it with training, and sampling at chance throughout.

n_out	bound 1 - n_out/P	measured, 1200 steps
96	0.878	0.900
144	0.816	0.852
192	0.755	0.806 (0.767 by 8.5k steps)
288	0.633	0.710

• The trajectory memory was measured against a matched memoryless control, and it holds. independent runs the same frames with the same targets and the same gradient budget, issued as K separate calls so the denoiser starts each frame with nothing — which is what a UNet sampler does. 600 steps per arm, MNIST, seed 42, guidance 2.0, RTX 3060 Ti:

  arm	val MSE	conditioning fidelity	Frechet	params
  4 attention heads	0.1575	71.8%	38.97	901,184
  trajectory (default)	0.1550	78.4%	40.82	573,376
  attention + plasticity	0.1518	73.2%	47.69	902,720
  plasticity, temporal	0.1564	70.4%	65.79	574,912
  shared-epsilon trajectory	0.1351	54.2%	76.85	573,376
  independent (memoryless)	0.2010	39.4%	83.10	573,376

  Carrying the trajectory doubles conditioning fidelity and halves the Frechet distance at an identical parameter count.

  The same claim falls out of a single checkpoint with no second training run. --mode eval samples one model twice, carried and wiped between denoising steps — identical weights, identical guidance, one line of difference at inference:

  same checkpoint, sampling only	conditioning fidelity	Frechet
  trajectory carried	85.6%	11.0
  wiped each step	58.6%	22.7

  That run also probes sampling-batch sensitivity, since the plastic buffer is a batch mean and a batch generated together would share one memory. With plasticity off — the default — there is nothing to share, and fidelity holds between 84.6% and 87.4% from batch 10 to batch 100.

• --traj-noise iid is the default, on weaker evidence than the theory suggests. Deriving all K frames from one epsilon is the path a perfect DDIM sampler walks, which is why it looks like the right training distribution, and it is a shortcut: two frames of such a trajectory determine x_0 by linear algebra, so the model can learn an inversion instead of a denoiser, and an inversion cannot follow it into sampling. The held-out loss behaves exactly as that predicts — the shared arm wins it at every budget measured. The sample penalty did not replicate: 54.2% conditioning fidelity against trajectory's 78.4% at seed 42, but 83.4% against 83.6% at seed 54321, both at equal wall clock. So iid is the default because it has never been worse, not because the gap is settled, and --sweep memory reports loss and Frechet separately so an arm that trades one for the other stays visible.

• Temporal depth beats denoising resolution at fixed compute. --sweep depth holds K*E = 64 and varies the split, which is a question only this architecture can ask — on a UNet the denoising step count and the depth spent inside one step are different resources. MNIST, seed 54321, 3 min per arm, 573,376 parameters throughout:

  arm	frames	echo	val MSE	conditioning fidelity	Frechet
  k32_e2	32	2	0.1104	73.0%	13.06
  k16_e4	16	4	0.1058	83.6%	15.97
  k8_e8	8	8	0.0986	88.0%	13.96
  k4_e16	4	16	0.1001	95.0%	15.66

  Fidelity climbs monotonically with echo depth while Frechet stays flat in no order, so what improves is the conditioning rather than the sample distribution collapsing. The deepest arm also samples in four denoising steps instead of thirty-two. The default stays 16 x 4: this is one seed at equal wall clock, with step counts spread 11% in the deepest arm's favour, and the ranked table crowns k32_e2 because RANK_KEY is Frechet and Frechet is the column that does not separate here.

• The x_0 width curve, for the per-parameter question. --sweep size, same budget and seed: 221,152 params / 75.6% fidelity, 380,880 / 83.0%, 573,376 / 85.8%, 1,056,672 / 87.6%. Returns are positive and shallow — 4.8x the parameters buys twelve points. n_out scales with the width in this grid, so the curve mixes capacity with output rank; the epsilon arms alongside it move rank alone and stay at chance whatever the width.

• Attention is available on this task and is not the default. It leads on Frechet by 4.5% while behind on conditioning fidelity and on held-out loss for 57% more parameters; on one seed at 500 samples that is not a separation, and per parameter it is a loss. Plasticity is behind on every column. Both are also slow here: at equal wall clock rather than equal gradients the plastic arms reach roughly 6% of the plain arm's step count and attention roughly 28%, because the retained trace grows with the step count, the batch and the neuron count together.

[3.2.0] — 2026-08-24

Changed

• BREAKING: the plastic trace is never assembled as a matrix, and hebb_res='synapse' is gone with it. Every Hebbian write is an outer product and the recurrence only ever asks the trace for h @ L, so the trace is now kept as the (B, N) writes it is made of and contracted on demand — both that product and the row norms its RMSNorm needs. The persistent buffer is materialized once per call, already averaged over the batch, so checkpoints, neurogenesis and transplants see exactly what they saw before. Memory is steps x B x N instead of steps x B x N².

  At 1024 neurons, batch 8 and 96 echo steps the plastic path holds 109 MB where a matrix-held trace held 15.5 GB, or 3.4 GB with gradient_checkpointing, and the step is faster than either. The shape that started this — 1024 neurons, batch 128, 96 steps, hebb_type='both' — asked for 649 GB and now asks for 1.8 GB.

  Per-element decay is what a streamed trace cannot carry: r^t has to scale rows for (h * r^t) @ C to remain a matrix product. hebb_res is therefore 'global' or 'neuron', and the per-synapse resolution is removed. It cost 2N² parameters and 3.1.0's own A/B found it did not earn them — 1,044 parameters to draw level with attention-only's 634. Checkpoints holding t_hebb_factor/s_hebb_factor of shape (N, N) will not load; retrain at 'neuron'.

  3.1.2's lever goes with it: gradient_checkpointing was made to reach the plastic trace because the trace was a matrix, and it no longer is. The flag still covers the step, which is all it now has to cover.

  The form was verified against the matrix one before that one was removed: in float64 the forward pass agreed to 4e-16 and every gradient to 1e-14 — the gain, both factor logits and both decay logits included — across temporal/spatial/both x global/neuron x 1, 2 and 5 steps, under pulse and continuous injection, from a cold and from a populated buffer.

• The novelty gate damps a write by the presynaptic row it lands on, not by the individual synapse. 1 / (1 + rms(W_eff[j,:])) in place of 1 / (1 + |W_eff|). An elementwise gate is the one term in the update that does not factor through an outer product, so the two changes are one change. It is also the better gate: on the record task's shape — sequential patch classification, plasticity on and attention off, where 3.1.0's plasticity A/B was run — it leads on both seeds at 10 epochs.

  seed	per synapse	per row
  123	85.15% / 1.0616	86.22% / 1.0469
  42	86.15% / 1.0364	87.02% / 1.0313

  It is carried as running sums updated once per write rather than read back out of the trace, which costs O(B * N) and adds nothing to the graph, because the gate is detached.

• convergence_hive_mind and convergence_skill_transfer reproduce unchanged. Every hop at 1.000, the pooling contract at 2.98e-08 against a memory scale of 0.270, a bee built after the foraging at 1.000, the hebb_type=None ablation at chance.

Fixed

• Plasticity no longer breaks under AMP on the second call. einsum is on autocast's half list whatever its inputs are, so the trace's constants were being built in half inside an AMP forward pass — which took down the LLM shape (vocab projection, attention, plasticity, batches through the trainer) as soon as the persistent buffer was no longer zero. The constants and the buffer write-back now carry the float32 guard the step already had.

Internal

• Three properties keep the streamed trace affordable, and a refactor should keep all three. The history lives in a preallocated buffer and every contraction runs over all of it behind a mask, so every step has the same shapes and torch.compile can trace them; the buffer is written functionally rather than in place, which is what autograd and Inductor both require of a tensor a graph has read. The read is recomputed in the backward pass rather than held — sound because the history is immutable once written, and the mask must count retired writes rather than the decay position, or the then-live write is counted twice on replay. And only the newest write carries gradient, the decay logit seeing one live factor, which is the same truncation a matrix-held trace applied to its own history.

[3.1.2] — 2026-08-24

Fixed

• gradient_checkpointing now covers the plastic trace, which is the only thing in the graph big enough to matter. The Hebbian update lived outside the checkpointed region: the flag recomputed the step and kept every (B, N, N) intermediate the trace produced, so on a plastic run it bought nothing at all — 6.16 units of B x N² per step with it on against 6.17 with it off. Reading the trace, taking the step and writing the trace back are now one checkpointed region (_step_and_learn), so what survives a step boundary is the trace that has to cross it and the rest is recomputed.

  Measured at 256 neurons, batch 8, 32 steps, in multiples of B x N² retained per step:

  	default	gradient_checkpointing=True
  hebb_type='temporal'	6.3	1.2
  hebb_type='both'	8.3	2.4

  The recompute costs ~70% more time on the plastic step (512 neurons, batch 8, 96 steps: 260 ms against 450 ms), which is the trade to make when memory is what binds. Off the flag nothing changed: outputs, every gradient — hebb_norm.weight, the factor and decay logits included — and the persisted trace buffers are bit-identical to 3.1.1 across temporal/spatial/both x global/neuron/synapse, with and without attention. The retained-per-step ratio is the same at every neuron count, batch and hebb_res; it drifts a little below the table on long rollouts (6.1 and 1.1 at 128 steps). torch.compile behaves as it did before, which includes a standing limitation now worth stating where flags are chosen rather than only here: Dynamo does not trace a checkpointed region, so --compile and --grad-ckpt are alternatives, not a pair.

  This was never a leak: allocation was flat across optimizer steps. It is the price of the 3.1.0 repair — the correlation carries gradient, so the whole per-example trace is an activation, and there are steps x B of them. What was wrong is that the one lever the library offers against it did not reach it.

Added

• A plasticity advisory in experiment_llm.py, alongside the vocabulary and attention ones: what the trace will cost, printed before it costs it, and what batch fits with --grad-ckpt and without. Plasticity is the one term in this model that scales with the batch and the step count, and raising the batch for throughput is the natural move that makes it unaffordable — --batch 128 --hebb both on the default 1024-neuron core, 96 echo steps, needs 649 GB for the trace, or 124 GB with --grad-ckpt.

[3.1.1] — 2026-08-23

Added

• A hive mind on the pooled plastic trace — examples/advanced/convergence_hive_mind.py. Eight bodies run the same core and never touch: separate hidden states, separate attention caches, separate forward passes. What is between them is the persistent Hebbian buffer, which holds the batch mean of the live per-example traces and is handed back to every row at the start of the next call. That pooling exists so checkpoints, neurogenesis and transplants see one (N, N) matrix; run a colony as the batch and the same two lines are a shared memory — linear, superposable, gradient-free, and living on the shared core rather than beside it.

  Each bee is shown one edge of a ring over 8 symbols, drawn fresh every episode so the answer cannot sit in the weights, and nothing else. Every private carrier is then wiped — hidden state zeroed, attention cache reset — so after the wipe no body holds anything of its own, and one query symbol is put to each. One echo step walks one edge, which makes the hop count a reading of temporal depth:

  320 queries per column	hop 0	hop 1	hop 2	hop 3
  together, one memory	1.000	1.000	1.000	1.000
  apart, one bee alone	1.000	0.297	0.134	0.094
  together, memory blank	1.000	0.147	0.141	0.112

  Hop 1 is an edge another body observed; hops 2 and 3 compose two and three edges held by different bodies, so the colony is not only recalling. Chance is 0.125, and the apart column is the same weights on the same inputs with the bees run one at a time instead of together — 0.297 at hop 1 rather than 0.125 because one query in eight happens to be the bee's own edge. 21,280 parameters, 1,000 optimizer steps, final loss 0.0009.

  The controls close the remaining doors. Move one bee's edge and a different bee's answer follows it (1.000), while that same bee run alone answers bit-identically (1.000) — separate forward passes have no channel to leak through, so the isolation needs no argument. Install another colony's memory and the answers follow the installed ring (1.000), not the ring that was asked about (0.147): what carries is the content of the trace, not its presence. Running the colony as one batch and running every bee separately then averaging the memories afterwards agree to 3.0e-08 against a memory scale of 0.317, so the batch axis is doing no secret work — the pooling is an average of independent bodies, and it could as well happen over a network. A body built after the foraging and never run before answers at 1.000 on the colony's memory: the knowledge is not in any bee.

  Only the reading is trained. The study pass runs under torch.no_grad() in every epoch, so the write is the architecture's own plasticity and never receives a gradient; what training shapes is a query that resonates with whatever the fixed rule wrote. Two protocol properties are load-bearing and were measured, not chosen: the study pass is two steps, because a cold start makes the first state a near-one-hot of the injected symbol and the temporal correlation written on the next step therefore lands on that symbol's row; and every evaluation call passes current_state explicitly, because forward re-runs reset_state on a batch-size change and that would zero the very memory the call exists to read.

  A six-step write measures the cost of getting that wrong: single-edge recall survives it (0.994) and the composition does not (hop 2: 0.503, hop 3: 0.194). Two and three steps both hold.

  The channel is the temporal path, measured rather than assumed — same protocol, 1,000 steps on CPU:

  	hop 1	hop 2	hop 3
  hebb_type='temporal' (shipped), seeds 42, 7 and 123	1.000	1.000	1.000
  hebb_type='temporal', attention off	1.000	1.000	1.000
  hebb_type='both', seed 42 / seed 7	1.000	1.000	1.000 / 0.200
  hebb_type='spatial'	0.169	0.134	0.122
  hebb_type=None	0.125	—	—

  An edge is directed and h_prev ⊗ h_t is too, which is why the temporal path carries the colony. h_t ⊗ h_t binds a state to itself, holds no direction, and alone never leaves chance; carried alongside the temporal path it does not add to it — with 'both' the deepest hop came out 1.000 at one seed and 0.200 at another, where temporal alone repeated 1.000 at three, and the loss curve stops bouncing (final loss 0.0009 against 0.4339). Attention is not a channel on this task at all: the cache is per body and is wiped with the state before the query, and the colony measures identically with it off. That leaves exactly one thing between the bodies, which is what the apart and blank rows already said from the other side.

  How deep the composition goes. Trained to seven hops instead of three, the same colony holds 1.000 through hop 4 and then falls away — 0.884, 0.775, 0.228. One echo step is one edge, so that is a reading of how far the pooled memory can be walked, and it is the same number the architecture's own thesis predicts should be there: depth is temporal.

[3.1.0] — 2026-08-23

Fixed

• Attention could not be torch.compiled, and compiling is the largest speedup this architecture has. Attention runs inside torch.amp.autocast(enabled=False) so its cache holds one dtype and the softmax accumulates in float32. Under torch.compile with AMP, Inductor miscompiles that region — its dtype model reports float for a buffer it emits as float16, and o_proj's matmul dies on expected mat1 and mat2 to have the same dtype, but got: struct c10::Half != float. A source-level .float() does not repair it; the same wrong model folds it away. attend and write now carry @torch._dynamo.disable. Running attention in the ambient precision instead, so the whole step traces, was implemented and rejected: on embedded MNIST with four heads it trailed the float32 path by a stable ~0.017 of loss from the fourth epoch on (0.8220 against 0.8043 at epoch 8, same seed and initial weights), and it was not faster where it counts (17.6 ms/batch fully traced in half against 17.4 with the graph break, plasticity and four heads on). What the break does cost: with attention as the only addition to a bare core there is little left to fuse, and compiling buys close to nothing (48.5 ms/batch against 45.4 eager).

• output_scale received exactly zero gradient on every classification-shaped task. OdyssNetTrainer._extract_outputs read the raw final_state on the last-step path while the sequence path read all_states, and only all_states carries the output scaling. Both paths now read all_states.

• Hebbian plasticity was near-inert, and each reason was a defect rather than a property of the idea. Four changes, no new parameters to set:

  1. A normalized, zero-initialized gain (hebb_norm, +N parameters). The strength factor multiplied twice — once accumulating into the trace, once applying it — making the effective gain sigmoid(logit)², roughly 32x below the measured optimum, with nothing able to move it. The trace is now RMS-normalized and scaled by a learnable gain: the factor decides which synapses are plastic, the gain decides how much. The gain starts at zero and construction draws no RNG, so a plastic model and a plain one at the same seed share a core and agree at step zero — hebb_type is a one-variable ablation.
  2. A differentiable correlation. Two stop-gradients sat on this path: one hiding the correlation's dependence on the state, one truncating the trace's own history. Only the second costs memory and only the first costs capability, so the first is gone. On episodic recall that alone recovers three quarters of the gap for the same bytes (0.8321 → 0.9550).
  3. A novelty gate, 1 / (1 + |W_eff|): co-activation across an already-strong synapse says the weight fired the neuron, not that the pattern is new. Parameter-free and detached, so it opens no second-order path through W.
  4. A per-example trace. The live trace is (B, N, N). Shared across a batch the associations average away and a batched forward computes a different function than the same examples run singly (9.5e-3 against 2.98e-7), so training never exercised the mechanism inference uses. The persistent buffer stays (N, N) and holds the batch mean, leaving checkpoints, neurogenesis padding and weight transplants untouched. Memory now grows as steps x B x N².

  The claim is that plasticity works, not that it wins. The gain trains off zero to 3-4% of W's scale where the old trace perturbed W by ~0.1% of its RMS and sat exactly on the no-plasticity control. Every controlled comparison since:

  • Sequential patch classification, attention off, one seed: ahead — 1.0088 / 87.99% at 480 parameters against 1.0548 / 87.41% at 430, epoch 31.
  • The same task with four attention heads held fixed, two seeds: behind, consistently. Epoch 10, seed 42: 0.9642 / 88.92% best against attention-only's 0.9442 / 89.43%; seed 123: 0.9588 / 89.71% against 0.9356 / 90.22%. Both seeds agree in sign and magnitude (+0.02 of loss, -0.5 points for +50 parameters). hebb_res='synapse' does not rescue it: 1,044 parameters to draw level with attention-only's 634.
  • Single-injection classification (embedded MNIST), one seed: slightly behind — 0.8311 / 92.36% against 0.8237 / 92.89% at epoch 8.

  Plasticity earns its place where step T extends what step T-1 built and acts as overfit noise where each step handles an independent chunk. Attention occupies the same role on sequential classification and fills it better, so the two are alternatives more than complements.

• Plasticity was slow for a reason unrelated to plasticity. Profiled on the record configuration (10 neurons, batch 32, 16 steps): 5,845 kernel launches per step for 20 ms of GPU work, 39% of CPU time inside cudaLaunchKernel. Temporal and spatial now run on one stacked path axis, so hebb_type='both' costs one set of kernels instead of two (4,853 launches, 42.0 → 32.8 ms eager), and with the compile bug fixed the whole step fuses: plasticity adds 132% to the step eager and 26% compiled (baseline 13.8 → 6.4 ms, 'both' 32.1 → 8.1 ms, 'both' with four heads 62.7 → 17.3 ms). The 1/N correlation scale now rides in the _offdiag buffer, and W's rows and the memory vector are normalized in one call.

Changed

• convergence_mnist_record, convergence_mnist_tiny and convergence_mnist_reverse_record run temporal attention in place of Hebbian plasticity, on seed 123. convergence_skill_transfer runs both mechanisms. Three choices behind those configurations:

  • Head geometry is sized to the core. Attention's projections scale with the neuron count, so the four heads that cost 204 parameters on record's 10-neuron core would cost 8,288 on tiny's 59 and take that example from 3,717 parameters to 12,005 — tripling the budget of a script whose whole question is what a fixed small budget can do. Tiny gets one head of width 4 (+952) instead.
  • All three write to the cache every thinking step. On reverse-record this is required rather than preferred: the input is a single scalar spread over 21 thinking steps, so the token ratio is 21 and the default attn_write='token' would leave exactly one cache entry, written at the last step, with nothing to attend to before it. On record, at one patch per step, 'step' and 'token' are the same thing.
  • Attention is what carries an early input to a later one in all three — the role plasticity held before, and measured better at it above.

• Measured results after the change, 100 epochs each:

  Example	Parameters	Result
  convergence_mnist_record	634 (430 core + 204 attention)	89.89% final, 90.91% peak (epoch 68)
  convergence_mnist_tiny	4,669 (3,717 core + 952 attention)	96.05%, reached at epoch 23 and held to the digit
  convergence_mnist_reverse_record	728	final loss 0.6522
  convergence_skill_transfer	51,796 (6.7% transplanted)	multiply: transplanted 0.016486 against scratch 0.028157

  README figures follow these. The 480-parameter and 484-parameter headlines they replace were measured on configurations the scripts no longer run.

• Decorative convergence plots removed from both READMEs, keeping only the reverse-record sample grid, which shows output rather than a curve. Charts of numbers that move need re-rendering on every change; the numbers themselves are in the tables.

• CONTRIBUTING's set_seed(42) rule now permits a documented deviation, which the three examples above use.

Added

• --compile in experiment_llm.py. Compiles the bound forward rather than wrapping the module, so state, KV caches, plastic buffers and checkpointing keep reaching the real object. Warmup costs a minute or two, longer with --hebb and attention on.
• TestHebbianRepair covers the four properties above: plasticity is an exact no-op at init, the gain receives gradient, a batched forward equals the same examples run singly, the two mechanisms stay distinct across the shared path axis, and every hebb_type × hebb_res combination trains.
• A test pinning hebb_norm.weight to a weight-decay-free optimizer family. It reaches modulation by falling through the classifier rather than by name; a family with decay would pull a zero-initialized gain back to zero and switch plasticity off silently.

[3.0.0] — 2026-08-22

Added

• Temporal attention: the state queries its own past, on the only axis this architecture has. A transformer stacks attention between layers; OdyssNet has none, so attn_heads=4 attaches it along time instead — at every thinking step the state issues one query over a cache of the states written before it, and the result is added to the same pre-activation signal the recurrence, the memory feedback and the input injection feed, so the step's activation and RMSNorm bound it exactly like everything else. Off by default (attn_heads=None builds no module, no parameters, no cache and no per-step cost, and leaves the state dict byte-identical to a 2.x model's). Switching it on changes nothing, on purpose. o_proj is zero-initialized and the module is constructed after the core is initialized so it draws no RNG the core would otherwise have consumed: two models built at the same seed, one with attention and one without, have the same W and produce the same output until training moves the attention weights (test_attention_is_a_no_op_at_init, test_core_weights_identical_across_the_switch). An ablation of attention therefore has one variable in it, and no known-good initialization story — resonant, edge of chaos — has to be re-validated to try it. The query length is always 1, because the core cannot be unrolled in parallel: step t's input is step t-1's output. A forward pass is a sequence of single-query attentions — a transformer's decode phase, never its prefill — so the score matrix is (B, H, 1, L), one row per key. There is nothing there for a fused flash kernel to tile away; F.scaled_dot_product_attention solves a problem this shape does not have. What is expensive is re-materializing K and V, so the carried history and the current call's writes are attended as separate segments and joined at the scores, under a single softmax that is exactly what one softmax over the concatenated keys would give (test_split_segments_match_one_softmax).
• The branch is divided by the square root of its own width, which turned out to be the difference between attention helping and attention destroying the network. o_proj starts at zero and every optimizer here is Adam-family, so its magnitude after k steps is set by the step size rather than by the gradient — |o_proj| ~ k·lr whatever the width — and the contribution it produces grows as sqrt(heads·head_dim) unless something stops it. Measured on multi-query associative recall at 128 neurons (2 pairs, 400 steps, chance 6.2%) with the division absent: no attention 54.6%, attention with o_proj frozen at zero 54.1% (confirming the module is inert as designed), 1 head × 16 → 54.6%, 4 heads × 64 → 5.9%. The wide branch had reached the same |o_proj| as the narrow one and, through four times the width, drowned the recurrence it was meant to assist; the failure was invisible to every knob that sounds relevant (attn_window=1 collapsed too, so it was never about the history). With the division in place every width lands on the baseline — 54.1% to 56.5%, 8 heads × 64 included. Same reasoning as GPT-2's 1/sqrt(2·n_layers) residual-branch initialization, with steps in place of layers.
• A KV cache in two representations, switched automatically, tested to agree. Under torch.no_grad() the cache is a preallocated (B, H_kv, window, D) ring written in place through a narrowed view — allocation-free decoding, which is the point of a KV cache. Under autograd, where in-place writes are illegal, the carry from previous calls is one frozen tensor (saved once for backward however many steps read it) plus a differentiable list of this call's writes. Both evict oldest-first at attn_window, by slicing rather than copying, and produce the same numbers at every window size tested (test_ring_and_segmented_paths_agree). Feeding tokens one at a time matches feeding them together to float tolerance (test_incremental_decode_matches_one_shot), so generation and scoring are the same computation. Two properties make the ring possible and are worth stating because they are what a transformer does not have here: every cached entry is strictly in the past (the write for a step happens after its read), so no causal mask is ever needed; and softmax over keys is permutation-invariant, so a wrapped ring never has to be reordered. Position lives inside each key instead — RoPE is applied at write time against the entry's absolute index, computed on demand in float64 rather than from a table, because a stateful stream reaches positions where float32 has already lost the phase of the fastest-rotating dimension.
• Grouped-query attention, with attn_kv_heads=1 (multi-query) as the default. In a step-sequential model the cache, not the projection matrices, is the term that decides whether a batch fits, and sharing one KV head across the query heads divides it by attn_heads. attn_kv_heads=attn_heads gives classic multi-head. The grouped case needs no key repetition: the query's group axis plays the role a query-length axis would, and broadcasting over it is what sharing a KV head means.
• attn_write and attn_read — the two knobs that actually set the cost. attn_write='token' (default) records one entry per input token, so extra thinking steps do not multiply the cache's length; 'step' records every step, letting the model attend to its own intermediate reasoning at (gap+1)x the cache. attn_read='step' (default) queries every step, 'token' only on the step a token arrives — leaving the echo steps to run on state alone, exactly as pulse_mode does with the input itself. The query, not the entry, is what think_gap multiplies, which makes attn_read the cheaper end of the trade whenever the gap is above 0 (~1.3x throughput at think_gap=1) and a no-op at 0.
• model.reset_rows(mask): zeroes the hidden state and the attention history of selected batch rows, leaving the rest of the batch running. Staggered cold starts need both — a row whose state is zeroed while its KV history survives is neither cold nor warm, and the cold start being simulated would not be one. Zeroing the cache rows is sufficient and is why the ring needs no per-row bookkeeping: an all-zero key row spreads its softmax uniformly over all-zero values, so its attention output is exactly zero, which is what an empty cache gives it (test_reset_row_attends_to_nothing). The LLM harness now uses it in place of a masked fill on model.state.
• attention parameter family in ChaosGrad, checked before projections (which q_proj/k_proj/v_proj/o_proj would otherwise match on the substring alone) and carrying the same weight decay as other connective structure. The QK-norm gains inside the same module go to modulation instead, where nothing decays them — they are not connective structure and their zero is not a neutral point.
• A fifth weight_init slot for the attention projections: ['quiet', 'resonant', 'quiet', 'zero', 'quiet']. Appended rather than inserted, and shorter lists are still right-padded from the defaults, so every four-entry list written against 2.x means exactly what it meant. The slot covers query/key/value only — the output projection is always zero, since that is the property the whole "free to switch on" story rests on.
• Neurogenesis grows the attention projections along their neuron-facing axis, with the same asymmetry the core uses: what the new neurons emit (the q/k/v input columns) starts as small noise so gradients can reach them, what they receive (the o_proj output rows) starts at zero so existing dynamics are undisturbed. Optimizer state is padded top-left like every other parameter, and the cache is dropped outright — every entry in it was written by projections that no longer have that shape.
• experiment_llm.py exposes all of it on the command line — --attn-heads, --attn-kv-heads, --attn-head-dim, --attn-window, --attn-write, --attn-read, --attn-rope/--no-attn-rope, --attn-qk-norm/--no-attn-qk-norm, --attn-dropout — validated before the corpus is tokenized and CUDA is initialized, so a typo costs a second rather than a tokenization pass. Shape-bearing fields (heads, kv_heads, head_dim, qk_norm) are adopted on --resume; the rest change no tensor and stay yours to re-choose. attention_advisory() prints the head geometry, the parameter count and the keys and values the backward pass will keep before training starts, because that term is quadratic in the writes inside one truncated-BPTT window and is otherwise discovered as an OOM. A new --sweep attn preset measures the whole thing against an off arm that is the 2.x architecture exactly, and --mode smoke gained three attention variants plus a second checkpoint round-trip, since attention adds six tensors that a resume must not silently drop.

Measured

• Attention is launch-bound, not compute-bound, and that is the headline cost. Profiled at 1024 neurons / batch 128 / chunk 48 / think_gap=1 on an RTX 3060 Ti: 24,884 kernel launches per training step, 605 ms of the 1.59 s CPU total inside cudaLaunchKernel against 96 ms of actual GPU work per step. One head costs the same as four (26,286 vs 27,175 tok/s), which is that regime's signature rather than a rounding error. Two fixes followed from the profile and are in the shipped code: attention runs with autocast disabled — a query of length 1 gains nothing from fp16, while the implicit casts around it cost more than the math, and the cache then has one dtype in training and inference instead of two — and the segments are joined at the scores instead of being attended separately and merged by their log-sum-exp afterwards. Together they took batch-128 attention from 18,376 to 27,175 tok/s (+48%) and cut peak memory from 0.83 to 0.76 GB.
• Attention learns more per token and costs more per second; both halves are measured. --sweep attn on TinyStories (1024 neurons, 2.6M params, vocab 2048, think_gap=1, batch 128, RTX 3060 Ti) at equal tokens — 600 steps, 3.69M tokens, same seed, off being the 2.x architecture exactly: mqa4 18.57 ppl, read_token 19.75, mha4 21.40, off 22.19. Every attention arm wins, the default four multi-query heads by 16%. At equal wall-clock — 2 min/arm — the ranking inverts because the arms no longer see the same data: off 14.55 ppl on 10.91M tokens, read_token 17.75 on 4.19M, mqa4 20.11 on 3.16M, no_rope 21.27, window32 23.47, mha4 27.38, write_step 32.16. Which of the two comparisons applies depends on whether a run is token-limited or time-limited; on a 2.6M-parameter core on a consumer GPU the clock binds, and the per-token advantage is the half that scales with hardware and core width. 0.0% collapsed cold starts in every arm, attention on or off.
• Where it did not help: synthetic associative recall. The obvious hypothesis for what content addressing should buy is key/value recall, and on a multi-query recall probe (128 neurons, 16 keys, 16 values, one query at the end, chance 6.2%) attention does not deliver it at this scale. 4 pairs / 1500 steps: no attention 31.6%, multi-query 30.0%, multi-head 30.3%. 8 pairs / 2500 steps: no attention 17.0%, multi-query 11.6%, multi-head 10.9%. That probe is what caught the width defect above, so it is a sensitive instrument rather than a broken one; it simply does not show the win. No example script ships claiming otherwise. The measured advantage is the per-token one on language modeling, and it stays a per-token advantage until the hardware makes the per-step overhead disappear.
• The remaining overhead is per step, so batch amortizes it. Off vs on (4 heads, multi-query), same configuration: batch 64 — 46,160 vs 13,265 tok/s (3.48x); batch 128 — 91,803 vs 26,530 (3.46x); batch 256 — 162,014 vs 52,838 (3.07x); batch 512 — 280,870 vs 98,236 (2.86x). With --attn-read token the batch-512 figure is 125,388 tok/s (2.24x). Measured on a settled GPU: the same benchmark run immediately after 40 minutes of continuous load reported 55-166k tok/s on the off rows, so those absolutes are this card's thermal state as much as this code's — the ratios survived it, the throughput did not.

Changed

• Version 3.0.0 across odyssnet/__init__.py, pyproject.toml and CITATION.cff. TemporalAttention is exported from the package root for expert use; nothing in the 2.x API changed shape or default, and all 306 pre-existing tests pass untouched (55 new ones cover attention).

[2.6.9] — 2026-08-10

Added

• --resume-best: recover from a diverged run without editing files by hand. --resume reads <tag>_latest.pth, which is rewritten at every evaluation and again on interrupt, so a loss explosion overwrites it within one eval interval — and resuming then reloads the diverged model. <tag>_best.pth is written only inside the val_loss < best_val branch, so it structurally cannot hold a diverged model; the recovery was always available but only as copy _best.pth _latest.pth at a shell prompt, which is exactly the step someone loses a training run by not knowing. The flag implies --resume and rewinds the step counter, per-row stream positions and optimizer state to that checkpoint, with the next evaluation repairing _latest. Architecture adoption reads the file the run will actually load, not _latest, or the rescue would rebuild from the wrong shapes and fail the strict load it exists to survive. It refuses to start when the tag has a _latest but no _best: falling back to a fresh model there would overwrite the one surviving checkpoint at the first evaluation, which is the destruction the flag was invoked to avoid. With neither file present there is nothing to protect, so it starts fresh like --resume does. Verified end to end on a scratch tag: two checkpoints carrying distinct marker weights confirm --resume-best loads _best and plain --resume still loads _latest, plus both branches of the missing-_best case.

[2.6.8] — 2026-08-10

Added

• --grad-persistence exposes the trainer's "ghost gradients": a fraction of the previous step's gradient is added to the next one. The mechanics are worth knowing before using it — the carry is injected after AMP unscale and before clipping, then re-captured from the clipped result, so it is a bounded geometric series rather than something that can run away, and it stacks on top of ChaosGrad's own Adam-style momentum rather than replacing it. That makes it a second-order knob: measured over six identical steps the weights differ by ~9e-07, because D-adaptation renormalizes the step size and absorbs much of the change. Range is capped at the trainer's documented 0.0-0.9; at 1.0 the carry stops decaying, which is a divergence rather than a setting. One inherited behaviour is called out in --help because it is surprising and would not be discovered otherwise: --resume cannot switch ghosting off. While the flag is left at 0, OdyssNetTrainer.load_state_dict restores the checkpoint's stored fraction and re-populates its accumulated ghost buffer, so a run the user believes has ghosting disabled resumes with both the setting and a warm carry. Verified directly — resuming a 0.5 checkpoint with --grad-persistence 0 came back at 0.5 with 9 ghost tensors reinstated. Library behaviour is left unchanged; the flag documents it, and the escape is a positive value or a new --tag.

[2.6.7] — 2026-08-10

Added

• --injection {pulse,continuous} exposes pulse_mode, which the LLM harness had been fixing at pulse without ever saying so. It decides whether a token's embedding is injected only on the step it arrives — leaving the --think-gap echo steps to run on recurrent state alone — or is held across every step. Verified to be a no-op at --think-gap 0 and to diverge increasingly above it, which makes it a knob about what the thinking steps actually think on, and therefore closer to this architecture's central claim than most of the flags that were already exposed. It is in ARCH_FIELDS so eval/gen reproduce a checkpoint's setting, and deliberately not in RESUME_FIELDS: like think_gap it is a forward-pass argument that cannot change the state dict, so pinning it on resume would remove a knob rather than prevent an error.
• --activation ENC,CORE,MEM and --weight-init ENC,CORE,MEM,GATE, both previously reachable only by editing Cfg. The core's initialization strategy sets the initial spectral behaviour the whole architecture is built around, and the arch sweep never varied either. Both are validated against the model's accepted names — nine activations, thirteen init strategies — with arity checked, before the corpus is tokenized. Note none means identity for --activation but no gate at all for --gates; the help text says so, since the same word doing two things across neighbouring flags is exactly the kind of thing that costs an afternoon.
• --val-batch sets how many shards the validation split is cut into, which is also the number of independent cold starts each score averages. It changes which corpus positions get scored, and only some positions fall into the absorbing state documented under 2.6.2's known issues — so raise it to probe cold-start robustness and keep it fixed when comparing runs.

[2.6.6] — 2026-08-10

Added

• --gates on the command line. The three gate activations — input/output scaling, the core signal, and memory feedback — were reachable only from inside a sweep preset, so the arch sweep's best arm could be measured but not trained. none,sigmoid,identity beat the default on both held-out loss and tokens consumed (2.6204 against 2.6495 at 25% fewer tokens), which made it the one configuration a user would most want and the one they could not select. none disables a gate outright and creates no parameter for it, so the flag changes the model's parameter set rather than only its behaviour; the values are validated against OdyssNet's accepted activations before the corpus is tokenized, so a typo costs a second instead of a tokenization pass and a CUDA init.

[2.6.5] — 2026-08-10

Added

• --tokenizer byte: one id per byte, 256 ids, nothing to train and nothing to pin. Chosen for a different reason than the other options rather than as a cheaper one. It is the only vocabulary under which a character is reliably its own token, which is what character-level work requires — a subword merge hides the units the model has to manipulate, so a model that has only ever seen 100 as a single id has learned nothing about the three digits inside it. It also makes the vocabulary table negligible: 0.20M parameters against the 1.05M core at 1024 neurons, where the default 2048-token BPE costs 1.57M and is the larger half of the model, leaving the chaos core as most of what is being measured. The price is sequence length — 1.0 byte/token against the trained BPE's 3.6 on TinyStories, so the same text is ~3.6x more timesteps and this architecture is latency-bound on sequential steps; it is a deliberate choice for tasks where the token boundary matters, not a general default. Decoding tolerates a window that begins partway through a multi-byte character, which is normal when scoring fixed-size token windows and must not raise.

[2.6.4] — 2026-08-10

Added

• --tokenizer in the LLM example: a ready-made tiktoken encoding, or the corpus-trained BPE. --tokenizer {gpt2,r50k_base,p50k_base,cl100k_base,o200k_base} skips the vocabulary-training pass entirely — nothing to train, nothing to pin, no tokenizers dependency, and a 46 MB corpus tokenizes in about a second. --tokenizer bpe (the default) is unchanged. Both backends sit behind one small Tokenizer interface (encode / encode_batch / decode / get_vocab_size / bos_id), so the cache builder, validator and generator never touch a backend directly. The tiktoken path uses encode_ordinary, not encode: the latter raises on text containing a special-token string such as <|endoftext|>, and a corpus is text rather than a prompt template.
• The default stays a 2048-token in-domain BPE, and the reason is now measured rather than asserted. On a held-out 906 KB slice of tinystories.txt, bytes per token: trained BPE 1024 → 3.079, 2048 → 3.601, 4096 → 3.908; tiktoken gpt2 (50,257) → 3.914, cl100k_base (100,277) → 4.146, o200k_base (200,019) → 4.190. A 4096-token corpus BPE therefore matches GPT-2's 50k vocabulary to within 0.2% at one twelfth the ids, because an in-domain vocabulary spends every merge on text the model will actually see. The industrial answer is a large vocabulary (Llama 3 128k, Qwen ~151k, Gemma 256k, and even Mistral/Phi-3/SmolLM at 32k–49k) but it does not transfer: those models are ≥2048 wide, so the table amortizes, whereas at this example's defaults cl100k_base buys 6% more compression for 24× the embedding parameters — 2.6M parameters becomes 78M, of which 98% is lookup table, and the batch × chunk × vocab logits tensor alone is 4.9 GB in fp32 at the documented batch 256 / chunk 48. The tiktoken path exists for the cases where that trade is the right one (a standard vocabulary to compare against, a larger core, or simply not wanting to wait for a training pass); --help carries the table.
• vocab_advisory() prints the parameter split (embedding table vs chaos core) at the top of every non-quiet run, warns when the logits tensor passes 0.5 GB with a concrete --batch suggestion, and warns when the vocabulary table exceeds 90% of the parameters. Advisory, never a hard limit — a large table is the correct call on a large core, and it is the silent version of this trade that costs an afternoon.

Fixed

• --resume destroyed the run it was asked to continue whenever an architecture flag was omitted. adopt_saved_arch was only wired into eval/gen, so a resumed run rebuilt the model from CLI defaults: --mode train --tag v4k --vocab-size 4096 followed by --mode train --tag v4k --resume reverted to vocab 2048, loaded the wrong token cache, failed load_checkpoint(strict=True) on the shape mismatch, and hit run_session's except Exception: starting fresh — which then trained a randomly initialized model and saved it over the trained checkpoint at the first validation, since --resume deliberately bypasses guard_overwrite. Reproduced end to end, including the clobber. Resume now adopts the checkpoint's architecture before the corpus is loaded, exactly as eval/gen do (which also closes the same latent hole for --neurons, --n-in and --n-out, predating this release). It adopts RESUME_FIELDS — the subset that decides what build() allocates or which corpus is read — rather than all of ARCH_FIELDS: pinning think_gap, activation and weight_init too would have quietly removed a knob, and continuing a run at a different temporal depth is a legitimate experiment here (--resume --cold-start-every 0 is already documented as a mid-run change). The split is drawn on what can actually change the state dict, checked rather than assumed: all nine accepted activations are parameter-free (_build_activation returns only stock nn modules, so the state dict is identical for each), weight_init is dead once weights load, and think_gap is a forward-pass argument — while gates does add or remove core_gate, input_gate, output_gate and memory_gate, and is therefore adopted. --resume --think-gap 3 is verified to keep gap 3 while still adopting the checkpoint's shape fields. eval/gen continue to adopt everything, since there the goal is to reproduce how the checkpoint was scored.
• A failed checkpoint load now stops instead of continuing. "Starting fresh" was never a graceful degradation on this path — the only way to reach it is a checkpoint that exists and will not load, and continuing past it overwrites that checkpoint with an untrained model. Loading the weights is now fatal on failure (SystemExit naming the error, offering --tag <name>_v2 or an explicit --overwrite; verified to exit 1 and leave the file md5-identical). The guard is deliberately narrow: restoring the trainer's state runs outside it and only warns, because that payload is the piece most likely to gain or lose keys between versions — 2.6.3 fixed exactly that for the optimizer — and a failure there costs counters rather than weights, with nothing trained yet. Verified with a checkpoint whose trainer_state_dict was made unreadable: the run warns, keeps the loaded weights, step count and best_val, and continues.
• --mode eval / --mode gen loaded the corpus before adopting the checkpoint's architecture. adopt_saved_arch ran after load_corpus, so a checkpoint trained at a non-default --vocab-size was scored against whatever corpus the command line happened to name — the tokenizer and vocabulary size decide which token cache is read. Pre-existing, and --tokenizer would have widened it from "a forgotten flag" to "the wrong tokenizer decodes the output into fluent-looking garbage, silently". The adoption now happens before the corpus is touched; --mode gen --tag <t> with no flags at all correctly rebuilds a tiktoken checkpoint's tokenizer from the file.
• A missing tokenizer file made an existing token cache unusable, and the example crashed on it. data/token_cache/ outlives examples/advanced/ckpt/, so cleaning the checkpoint directory left caches whose pinned BPE was gone; load_corpus hit its cache branch, called get_tokenizer with nothing to train on, and raised FileNotFoundError — --mode smoke at the defaults was dead on any machine in that state. The cache is now invalidated and rebuilt alongside a fresh tokenizer, with a message saying so. Retraining the tokenizer next to the old cache was tried first and is wrong: BPE training is deterministic given identical input (two runs in one process produce identical vocabularies), but nothing records what an existing cache's tokenizer was trained on, and measured against the cache in data/, retraining today's 2048 vocabulary re-encodes the validation half to 414,805 tokens where the cache holds 414,778 — close enough to look fine, different enough that every id is suspect. Rebuilding tinystories.txt at vocab 2048 under a freshly trained tokenizer yields 12,301,738 train / 414,805 val tokens against the previous cache's 12,301,242 / 414,778; the 2.6.2 reference run's exact figures are defined against the older pairing and are not reproducible without it.
• The cache stamp now includes the tokenizer, so switching backends cannot read the previous one's ids. The trained-BPE stamp keeps its historical <corpus>.v<size>.val<chars> form byte-for-byte — existing caches, and the numbers measured against them, stay valid — while tiktoken caches are named by encoding, which cannot collide with v<digits>.
• --vocab-size now defaults to a sentinel rather than to 2048, so passing it alongside a tiktoken encoding says the flag is being ignored instead of quietly appearing to do something. Cfg.tokenizer is part of ARCH_FIELDS and round-trips through checkpoints.
• generate() no longer falls back to a hardcoded token id 0 when a prompt encodes to nothing; it asks the tokenizer (bos_id), because id 0 is <s> in the trained BPE and an ordinary byte token in a tiktoken encoding.

[2.6.3] — 2026-08-09

Fixed

• ChaosGrad's spike brake released itself while the loss was still climbing. The brake_ceiling relaxed geometrically toward 1.0 on every report_loss call, conditioned on nothing — observed on an LLM run (batch 1024, lr=None): the brake fired at step ~18,650, and by ~18,850 the run had its full step size back while the loss was still rising past 5.5; it then diverged to 100% collapsed cold starts. A slow climb is exactly the shape the release logic couldn't see: each step stayed under the 1.2× ratio test against the post-spike reseeded EWMA, so no new spike fired, and 200 quiet calls undid the brake mid-divergence. The pre-spike EWMA is now kept per-group as a recovery reference (brake_ref, round-trips through state_dict, carried through neurogenesis next to brake_ceiling): the fast release only runs once the loss EWMA is back inside the reference band (brake_ratio-wide, floored at the loss stream's own σ so an asymptotically-approached reference can't hold the brake hostage), and while the loss remains elevated the ceiling merely crawls (0.9995/call, ~1400 calls to undo one hit) so a genuine regime shift — a loss that legitimately settles higher — eventually frees the step size instead of being pinned forever. Repeated spikes keep the lowest reference seen: a staircase divergence must recover to the original healthy level, not the previous stair. Replayed against the divergence-shaped stream (calm → spike → 200-call slow climb → recovery): old ceiling back at 1.0 within 100 climb calls, new ceiling held at ~0.52 through the climb and fully healed within 100 calls of actual recovery. Fixed-rate mode (lr=<float>) remains untouched by the brake, as documented. Scope, stated precisely: this removes the release-mid-climb mechanism that preceded the observed divergence — it has not yet been shown end-to-end that the held step is small enough to stop that particular run (an lr=None re-run at the failing config is the missing measurement), and the silent absorbing-state entry under Known issues (2.6.2) has healthy training loss throughout, so it remains out of any loss-watching brake's reach by construction.

• Loading an older checkpoint into a newer ChaosGrad silently dropped newly-added per-group keys. torch.optim.Optimizer.load_state_dict replaces the group dicts wholesale with the saved ones, so any key added to defaults after a checkpoint was written vanished on load and resurfaced as a KeyError mid-training — the exact gap that shipped once before with the brake-config fields (2.6.1). load_state_dict is now overridden to backfill missing keys from defaults; rms0 (computed at construction, deliberately not in defaults) is preserved from the pre-load groups so the traction anchor keeps pointing at the initial weight scale.

• OdyssNetTrainer's anomaly spike detector zeroed its variance estimate after every spike — the same degenerate-σ bug fixed in ChaosGrad's brake in 2.6.1 (a trivially-satisfied 3σ test for the next ~20 calls, spamming the anomaly_hook). Now retains the pre-spike variance, mirroring report_loss.

• Switching a fixed-rate ChaosGrad history to adaptive mode crashed with a KeyError. Fixed-rate mode never creates the estimator state s/p0, so a checkpoint trained under lr=<float> whose groups were later set back to lr=None blew up on the first adaptive step. The estimator state is now lazily initialized at the switch, with the distance reference point at the current (warm) weights.

• experiment_llm.py printed lr auto on resumed runs that were actually in fixed-rate mode: load_checkpoint(lr=None) means "don't touch", so a fixed-rate checkpoint stayed fixed while the banner claimed auto — and since "auto" was also the CLI default, there was no way to say "actually return this checkpoint to the estimator" at all. --lr now defaults to keep (fresh runs use auto; --resume keeps the checkpoint's stored mode), an explicit --lr auto moves a fixed-rate checkpoint back to the online estimate (possible now that the fixed→adaptive switch above no longer crashes), and the banner reports the optimizer's actual mode, noting when the checkpoint's mode differs from the CLI's.

• experiment_llm.py could not tokenize a corpus larger than roughly a tenth of available RAM. The cache builder read the whole file into a string, split it, re-joined it into blocks, and accumulated token ids in a Python list — boxed ints at ~36 bytes each. Measured on data/wikisent2.txt (934 MB): the old path was killed at 46.4 GB resident and still climbing; the streaming replacement finishes in 84 s at 0.81 GB peak (417.5M tokens). The file is now read line-by-line in binary from a byte range, encoded in batches, and written straight through to a flat uint16 file that is memory-mapped for training, so peak cost is one encode batch regardless of corpus size. Reading a byte range instead of a decoded string moves three boundary concerns into the caller's hands; the streaming rewrite got each of them wrong before it got them right, so they are recorded here as guidance rather than as discoveries. Line endings: binary reads preserve the CR of a CRLF file and a byte-level BPE makes every one its own token — one wasted token per line, ~5% of tinystories.txt, and a silent shift of every metric against a text-mode tokenization; newlines are normalized at the single decode site. Split alignment: searching raw bytes for b"\n\n" never matches CRLF (b"\r\n\r\n"), and three of the four corpora in data/ are CRLF or have no blank lines, so a paragraph-aligned split lands mid-line and potentially mid-UTF-8; aligning on a bare b"\n" is correct under both conventions, whereas a preference-ordered separator list matched at the end of the search window on wikisent2.txt and produced an empty validation set. Budget: val_chars is capped at a fifth of the corpus so a small file plus the default budget cannot leave an empty training half.

• The token cache would have silently corrupted any vocabulary above 65535 ids. Ids were written as uint16 unconditionally, and numpy does not warn on an out-of-range cast — a tiktoken-scale vocabulary (o200k_base is ~200k) would have wrapped around into plausible-looking garbage with no error anywhere in the pipeline. The element width is now chosen from the vocabulary size, recorded in the cache filename (a flat token file has no header, so reading uint32 bytes as uint16 is not detectable after the fact), and the ids actually produced are range-checked per block — special tokens are not always inside the nominal vocab size, so the declared size alone is not trustworthy.

Changed

• Token cache format is a flat .bin (memory-mapped, uint16 or uint32) rather than .npy. Pre-existing .npy caches are still read and take priority, so numbers already measured against one stay valid; delete the .npy pair to re-tokenize under the new scheme. On tinystories.txt a fresh cache yields 12,301,242 train tokens against the old 12,294,433 (0.06%), and the reference checkpoint scores val loss 2.0791 (ppl 8.00) on it versus 2.0996 (ppl 8.16) on the original cache. Those two numbers are not comparable — same weights, different held-out split — and the published reference stays defined by the original cache.

[2.6.2] — 2026-08-09

Added

• cold_start_every (default 32) in the LLM example: during training each batch row's hidden state is independently zeroed with probability 1/cold_start_every, giving an exponential spread of context lengths and keeping cold and warm contexts mixed in every batch. Without it the carried state is never reset, so gradient descent puts no pressure on cold-start behaviour at all. Measured at 1500 steps / batch 128 / same seed: cold_start_every=0 collapsed 15.6% of held-out shards (pooled val loss 39.55, ppl 1e13) while its training loss read a healthy 2.689; cold_start_every=32 collapsed 0.0% (pooled val loss 2.68, ppl 14.60) at training loss 2.706. The failure is removed at no measurable cost in fit quality.
• Reference run for the rewritten LLM example (--mode train --minutes 25 --batch 256, 2,625,280 params, RTX 3060 Ti): held-out loss 2.2009, ppl 9.03, bits/byte 0.8764, median cold start 2.3154, 0.0% collapsed cold starts, 226.15M tokens at 150,761 tok/s. Not a README-advertised metric — a reproducible baseline for the harness, quoted with the exact command that produces it.
• Cold-start collapse is now a measured quantity. examples/advanced/experiment_llm.py reports val_p50 (median per-shard loss — what the model does on a typical cold start) and val_coldfail (fraction of shards that collapsed) alongside pooled loss/PPL/bits-per-byte. The pooled mean alone is unreadable when this failure mode is present: a handful of collapsed shards at ~240 nats swamp it and the run looks like a global divergence when most shards are healthy.

Changed

• examples/advanced/experiment_llm.py rewritten. Was an unconditional infinite training loop with a blocking input() prompt on architecture mismatch, HuggingFace streaming (network-dependent, non-reproducible ordering, MAX_START_SKIP randomness), exp(label_smoothed_loss) reported as "perplexity", and best-checkpoint selection on training loss over a streaming corpus. Now --mode {smoke,sweep,train,eval,gen} over a local uint16 memmap token cache, with a held-out split, unsmoothed-CE perplexity, and compute-matched sweep presets (gap, arch, batch, coldstart, optim, window). TBPTT chunk 5 → 48 tokens: at chunk 5 a 512-token batch spent 103 full ChaosGrad steps over the 4.2M-element core, so the optimizer cost more than the model did.
• The LLM example is now zero-config (lr=None, ChaosGrad's estimator) instead of the pinned 1e-4 it carried since 2.6.0; --lr <float> still selects fixed-rate mode. The reference run above was measured under the automatic estimate. CLAUDE.md's convention list, which named the LLM example among those keeping an explicit rate, is updated to match. Note the estimator pins at its traction cap (~3.9e-3) within ~300 steps and stays there for the whole run — the optim sweep preset exists to compare that against pinned rates, but has not been run at a meaningful budget, so no claim is made that auto beats fixed here.
• experiment_llm.py validation runs in fp32 with TF32 disabled — not because precision was ever implicated (AMP on/off and TF32 on/off were measured to agree to four decimals) but because eval throughput is irrelevant and a metric used to rank sweep arms should not move with kernel choice. What did make the score batch-size dependent — same weights scoring 2.6 at eval batch 8 and 39 at batch 32 — was shard layout: eval batch size determines how the val split is cut and therefore which corpus positions each shard starts from, and only some starting positions fall into the absorbing state described under Known issues. val_coldfail now reports that directly instead of letting it distort the pooled mean.

Fixed

• OdyssNet.forward allocated a throwaway CPU tensor per timestep (torch.tensor(t)) to pass the step index into checkpoint.checkpoint. With use_reentrant=False plain ints are accepted; on an LLM-scale sequence that was one allocation per tokens x thinking_steps iteration. Behaviour-identical.
• OdyssNet.forward scaled the full (B, T, N) activity tensor by output_scale_vec before slicing the output neurons out of it, in vocab-projection mode. Now slices first and scales the slice — mathematically identical (output_scale_vec is 1.0 outside output_pos), but on long sequences that tensor is the largest allocation in the forward pass and only len(output_ids) of its columns are ever read.

Known issues

• The chaos core can develop a degenerate absorbing state that training loss cannot see. Measured on a 1024-neuron LM: after ~1500 steps with ChaosGrad's estimate pinned at its traction cap (3.93e-3), certain cold starts drive the hidden state into a regime producing ~240 nats/token which it never leaves, while training loss continued improving smoothly (2.67 nats). Per-row and single-row-batch replays confirm the rows are fully independent and the effect is reproducible per starting position, so this is dynamics, not numerics. Two mechanisms hide it: stateful TBPTT never re-enters the bad basin once training is on a benign trajectory, and ChaosGrad's spike brake watches the training loss stream, which stays smooth throughout — a blind spot in the brake's coverage worth noting for the next optimizer pass. cold_start_every is the example-level mitigation and is sufficient in every configuration measured so far, but it treats the symptom; the underlying dynamics are unchanged.
• The LLM example's --sweep gap verdict (gap=1 best at equal wall-clock, gap=3/gap=5 far behind) is a wall-clock result at a 1.3 min/arm budget, where the deeper arms only consumed 1.5-2.3M tokens against gap=1's 5.68M. It does not establish that deeper thinking is worse per token — that needs a token-matched run, which the harness supports but which has not been done.

[2.6.1] — 2026-08-07

Fixed

• ChaosGrad's loss-spike brake permanently scarred the estimator on ordinary noise. Diagnosed via instrumented probes on convergence_mnist_record.py: the brake was firing every ~150-350 batches on nothing worse than normal per-batch classification variance (confirmed unrelated to Hebbian plasticity — fires more often with hebb_type=None), and each trigger permanently multiplied d_numerator/d_max by brake_factor with no way back. By epoch 25 of a 3k-subset probe, effective_lr had collapsed 500x and training had silently frozen — matching the ~83-87% accuracy plateau seen in full 100-epoch runs. Replaced the permanent mutation with a transient brake_ceiling multiplier applied only at the point of the actual parameter update, leaving the estimator's own bookkeeping untouched so it keeps learning the true scale while suppressed. The ceiling relaxes geometrically back toward 1.0 every report_loss call rather than on a fixed window: an isolated spike heals in tens of steps, a fast cascade (genuine divergence — the delayed-adder case this brake exists for) still compounds down. Also fixed a secondary bug where the post-spike variance reseed collapsed to 0.0, making the sigma test degenerate for ~20 calls after every fire.
• The traction limit's trust_ratio anchor selection was a hard cutoff sitting exactly on the library's own init scale. _trust_cap excluded groups with rms0 < 1e-3 or 0.9 ≤ rms0 ≤ 1.1 outright — but micro_quiet_warm (one of OdyssNet's own bundled init strategies) initializes at std=1e-3, the exact value of the floor. Measured on convergence_mnist_record.py (which uses micro_quiet_warm): the same seed produced a traction cap of 0.000254 on CPU vs 0.004266 on CUDA — a 16.8x difference from nothing but which side of the 1e-3 cutoff a group's RNG-drawn rms0 happened to land on. In practice this washed out on our examples (both land below the floor on the CUDA runs the READMEs are measured on), but it meant the cap a library user got was silently discontinuous in a region their own init could easily sit in. Replaced the hard include/exclude with _anchor_weight (a smooth ramp over the same two boundaries) plus _trust_cap blending each group toward the smallest already-fully-trusted group's own rms0 (not a fixed constant — an earlier version that blended toward a fixed reference passed every test but silently moved the cap on record.py itself, from 0.006 to 0.0025, by letting an excluded near-zero group's fallback undercut a real, smaller anchor elsewhere in the same model; caught by re-measuring the actual example configs before merging, not by the test suite). Same cap on every example already validated (confirmed by direct measurement on record.py's and the adder's actual param groups), continuous everywhere else. If literally every group is excluded (e.g. a lone all-zero-initialized parameter with no other family to anchor against), the cap is disabled rather than pinned to an arbitrary value — same as before.
• Neurogenesis.expand() rebuilt ChaosGrad via from_model without forwarding brake_factor (or the newly-exposed brake_sigma/brake_ratio/brake_ema_alpha) — a user who customized the brake would silently get the defaults back the moment their network grew. Now forwarded from the pre-expansion param group.
• brake_sigma/brake_ratio/brake_ema_alpha were first exposed as plain instance attributes, which torch.optim.Optimizer.state_dict() doesn't serialize (only param_groups and state are) — the same class of bug as the Neurogenesis one above, one mechanism over: a save/load round trip would have silently reverted a customized brake back to defaults. Moved into defaults so they ride the param groups like brake_factor already does.
• examples/advanced/convergence_skill_transfer.py's "Claim check" required the transplanted model's average loss across the whole run to beat scratch's — but the transplanted model's epoch-0 loss is inflated by the 93%-freshly-initialized new region, dragging the average up regardless of how well it ultimately converges. The check printed "No clear transfer win" on every run even when transplanted beat scratch on final loss and time-to-threshold by a wide margin. Now judges the claim by final loss and first-epoch-below-threshold instead.

Changed

• brake_sigma, brake_ratio, and brake_ema_alpha are now constructor parameters (previously hidden class constants a library user could not reach at all), defaults unchanged (3.0, 1.2, 0.05). The EWMA memory (brake_ema_alpha) in particular was tuned against OdyssNet's own ~16-32 batch-size examples; a user training at a very different batch size sees a differently-scaled loss-noise floor and can now retune it instead of silently inheriting ours.
• The brake's loss-EWMA/variance now use a bias-correction-style warmup (alpha = max(brake_ema_alpha, 1/(brake_step+1)), the same technique already used for Adam's own bias correction elsewhere in this file): the first ~20 calls after construction (or after a checkpoint reload) converge the estimate from every sample seen so far rather than committing to the steady-state window immediately. Verified this doesn't introduce spurious early spikes by logging (loss, ema, std, is_spike) for the delayed adder's first 50 report_loss calls — one legitimate-looking fire, no cluster near the start.
• convergence_skill_transfer.py: retuned add_epochs 500→250, mul_epochs 1500→500 — the shorter add-phase avoids overfitting the small model before transplant, giving a consistent, clear transfer win instead of a partial one.
• Re-validated numbers across README.md/README_TR.md under the fixed brake: MNIST 98.62%→98.71%, MNIST Revive 98.54%→98.70%, MNIST Tiny 95.15%→95.58%, MNIST Scaled 97.38%→98.01%, MNIST (8k) Embed 94.08%→93.71% (within run-to-run noise), Skill Transfer speedup 3.0x→3.6x. Sine Wave/Latch/Stopwatch log excerpts refreshed to match current runs.
• convergence_mnist_record.py: confirmed at full scale — previously froze around epoch 15-20 (~83-87% plateau, the exact symptom the brake fix targets), now trains cleanly through all 100 epochs, landing at 87.98% (peak 88.46%, epoch 86) zero-config. The README's 90.14% "WORLD RECORD" banner predates this optimizer entirely (different scheduler/preset pipeline, since removed) and is left as-is with an added status note — not a regression target, since the script itself changed too much for a like-for-like comparison. LR set to None (zero-config).
• Re-validated after the trust-cap and brake-warmup changes above: delayed adder (2000 epochs, no divergence, final loss ~0.00017), convergence_mnist_record.py 25-epoch diagnostic probe (d_max flat at 0.060141, effective_lr steady at 0.005958 with brake recovering after each dip — unchanged from before this fix), XOR seeds 42/123 (both solve zero-config), MNIST-3k/6-epoch probe (89.20%, byte-identical to the pre-change run), full pytest tests/ (297/297).

Known issues

• convergence_sine_wave.py shows a late-training instability under the fixed brake: loss explodes from ~0.001 to 0.01-0.06 starting around epoch 7900 of a 10000-epoch run. Not yet root-caused. EPOCHS reduced 10000→6800 as a stopgap (avoids the window entirely) rather than a fix; worth the same instrumentation approach used to diagnose the record.py brake issue.

[2.6.0] — 2026-08-06

Added

• ChaosGrad — OdyssNet's bespoke zero-config optimizer, rebuilt from first principles and now the default. Combines Adam-style per-synapse preconditioning with online distance adaptation (D-adaptation class estimator): no learning rate is required. Exported as odyssnet.ChaosGrad.
• Architecture-aware family policy: parameters are auto-classified into chaos_core, memory_feedback, projections, plasticity, and modulation families. Weight decay applies only to connective structure; Hebbian logits and gates are never decayed. The chaos core's zero-diagonal constraint is enforced inside the optimizer step.
• Anchored traction limit (trust_ratio): the applied step scale is capped at a fraction of the network's initial weight scale, shielding tiny chaotic networks from distance-estimator overshoot (stock Prodigy could not solve the 9-parameter XOR example; ChaosGrad solves it zero-config with the step scale settling at the previously hand-tuned value).
• Loss-spike brake (brake_factor): on a statistical loss spike the distance estimate is scaled down and re-grows only if the landscape supports it, fixing late-training divergence on sharpening temporal tasks (delayed adder). The trainer feeds the loss stream automatically; custom loops can call optimizer.report_loss(loss).
• trainer.get_diagnostics() now includes ChaosGrad health metrics under the optimizer key, and current_lr reports the live step-scale estimate.

Changed

• OdyssNetTrainer default optimizer is now ChaosGrad (lr=None → automatic estimation). Passing an explicit lr selects ChaosGrad's fixed-rate mode (AdamW-equivalent updates under the family policy) instead of AdamW.
• All convergence examples now run zero-config (no lr argument). Precision-record examples (embed/record/reverse-record/LLM) keep their tuned rates via fixed-rate mode.
• Neurogenesis.expand() migrates ChaosGrad with fresh family grouping while preserving the step-scale estimate.
• convergence_mnist_embed.py moved to zero-config (lr removed) — matches its previously tuned fixed-rate result under ChaosGrad.
• convergence_adder.py epochs raised 500 → 2000 to ride out the mid-training loss spike before the loss-spike brake settles the estimate back down.
• Re-validated and refreshed the advertised numbers in README.md/README_TR.md for MNIST, MNIST Revive, MNIST Tiny/Scaled, MNIST (8k) Embed, Sine Wave, Adder, Latch, Stopwatch, Detective, and Skill Transfer under ChaosGrad v2.6 — most improved (e.g. MNIST Tiny 90.2% → 95.15%, MNIST Revive 97.8% → 98.54%). MNIST (Record) and MNIST Reverse (Generation) numbers are intentionally left untouched pending further optimizer work.

Fixed

• Neurogenesis.expand() silently copied shape-tracking optimizer state tensors (exp_avg, exp_avg_sq, ...) without resizing them, crashing the first optimizer step after expansion.
• Examples with emoji output crashed on legacy Windows code pages (cp1254); affected scripts now reconfigure stdout to UTF-8.

Removed

• prodigyopt dependency (Prodigy is superseded by ChaosGrad).

[2.5.0] — 2026-04-30

Added

• Spatial Hebbian Plasticity: Introduced a co-activation learning mechanism (classic Hebbian) alongside the existing STDP-style learning.
• hebb_mode functionality (hebb_type): hebb_type is now repurposed to act as the mechanism toggle: None (disabled), "temporal", "spatial", or "both".
• hebb_res: Controls the structural resolution ("global", "neuron", "synapse"). Defaults to "neuron".

Changed

• BREAKING: Replaced single-path Hebbian parameters with path-specific prefixes (t_hebb_factor and s_hebb_factor, etc.). Existing checkpoints utilizing hebb_factor will need to be loaded with strict=False and re-trained, or manually patched, as we have prioritized a clean architecture over legacy support.

[2.4.0] — 2026-04-10

Added

• Prodigy optimizer is now the default when lr=None (the new default). Prodigy auto-calibrates the learning rate continuously — no manual LR tuning required. Requires the prodigyopt package (pip install prodigyopt), now listed as a core dependency.

Changed

• OdyssNetTrainer default lr changed from 1e-4 to None. Passing lr=None (default) activates Prodigy; passing an explicit float (e.g. lr=1e-4) still selects AdamW with weight_decay=0.01.

[2.3.1] — 2026-04-09

Added

• Added ODYSSNET_DISABLE_PLOT environment variable support to TrainingHistory.plot() to bypass interactive plotting during automated runs.
• Updated examples/test_all.py to automatically set ODYSSNET_DISABLE_PLOT=1 before spawning sub-processes.

Fixed

• Fixed bug in save_checkpoint where os.makedirs crashes if a bare filename is provided (e.g. "model.pt") due to an empty directory string.
• Fixed 5/6/7-space indentations across codebase to comply with strict 4-space PEP 8 guidelines.
• Cleaned up several unused imports (torch.nn, Dataset, math) in advanced examples.

Changed

• Default learning rate in OdyssNetTrainer changed to 1e-4 (previously 1e-3).
• Centralized repetitive output-extraction and autocast resolution logic in OdyssNetTrainer into private helper methods (_extract_outputs, _get_autocast_ctx), standardizing logic.
• Optimized optimizer state transferring logic in neurogenesis.expand(), collapsing multiple loops.

[2.3.0] — 2026-04-06

Removed

• Removed ChaosGrad optimizer — replaced with standard AdamW as default.
• Removed bitsandbytes dependency and all NO_BNB environment variable usage.
• Removed trigger_plateau_escape() from trainer (was ChaosGrad-specific).
• Renamed micro_quiet_8bit init strategy to micro_quiet_warm.

Changed

• Default optimizer is now torch.optim.AdamW(lr=1e-3, weight_decay=0.01).
• Diagonal zeroing of chaos core W matrix is now enforced by the trainer.
• get_diagnostics() simplified — removed ChaosGrad-specific metrics.

[2.2.0] - 2026-04-06

Added

• ChaosGrad v2.2 "The Learning Teacher": Zero-hyperparameter optimizer with Analytic Hypergradient Descent. All meta-parameters (LR, momentum, weight decay, centralization) are autonomously adapted per-parameter at each step.
• Heterogeneous Synaptic Plasticity (hebb_type): Three resolution levels (global, neuron, synapse) for online Hebbian learning with fully differentiable logit parameters.
• Parametric Gating: Configurable per-branch gates (encoder_decoder, core, memory) with identity and sigmoid modes.
• Label Smoothing: Integrated into trainer for classification tasks.
• Debug Mode (debug=True): NaN/Inf diagnosis with per-operation forward-pass checks and automatic detect_anomaly.
• Enhanced Diagnostics: Both ChaosGrad.get_diagnostics() and OdyssNetTrainer.get_diagnostics() now support a debug parameter.
  • ChaosGrad debug mode includes per-parameter statistics (min/max/std) for learning rate, beta, alpha, decay, per-group breakdowns, and step count statistics.
  • Trainer debug mode includes gradient persistence tracking, anomaly detection state, loss tracking buffer info, AMP scaler state, and gradient statistics (norms/means).
• Training history plotting (plot_history): Utility to visualize loss, learning rate, and custom metrics over training.
• pyproject.toml for standard Python packaging (pip install -e .).
• CONTRIBUTING.md with example standards, initialization protocols, and contributor checklist.
• LICENSE file (MIT).
• CHANGELOG.md (this file).

Changed

• Removed legacy ChaosScheduler — ChaosGrad now handles scheduling at granular synaptic level.
• Renamed PoC/ to examples/, PoC/experiments/ to examples/advanced/ for open-source clarity.
• Removed ChaosGradConfig — ChaosGrad requires only a genesis lr.
• Removed sys.path.append hacks from all example scripts (use pip install -e . instead).