T1+T2 — Consensus, liveness & anti-abuse: pinned corpus, an eval server that can't wedge, and the freeze cheat closed

[vex0209-bt]

Summary

Stacked on #7 (review that first — this diffs against it). Six commits: the duel becomes reproducible, the machinery that runs it stops wedging, and the cheats that our particular metric invites are closed.

dbab2cc	Two honest validators could reach different verdicts on the same challenger, and neither would ever find out.
8a61b1d	The eval server could hold the subnet's only GPU lock forever — and the failure looked exactly like "we're busy".
389b9aa	A duel was awaited inline, so a tick took hours, during which the validator set no weights and looked dead to the chain.
0bbb102	Every duel re-downloaded the king's ~30-70 GB of weights.
a2be2cd	A model that just holds the conditioning frame could take the crown. Copying the king, and monopolizing the GPU, were both free.
0bf25ee	A wrong-architecture model cost hours of GPU to discover.

---

1. Make the duel a pure function of the chain (dbab2cc)

Input	Before	After
Which clips	live list_objects → permutation(len(keys))	pinned, digest-verified corpus manifest → select by index
Where in the video	ffmpeg scene detection, at duel time	clip_start baked into the manifest, offline
Ground truth	x264 re-encode → JPEG → PIL	raw RGB, one ffmpeg call, truth_sha256 verified
Prompt / frames / fps / resolution	eval box's own env vars + dataclass defaults	chain.toml [gen], sent explicitly, echoed
Metric / delta / alpha / bootstrap	validator's own env vars	chain.toml [duel], sent explicitly, echoed
A clip that fails to download	silently skipped — a duel could run on 3 clips and be recorded as normal	typed error, duel aborts

The clip set depended on the corpus size, so uploading one video reshuffled the entire exam — and only for validators that had seen the upload. The window inside each video was scene-detected at duel time, and scene-cut timestamps vary across ffmpeg builds, so two validators could carve a different five seconds out of the same video. And the duel parameters came from per-box env vars, while the prompt, resolution, fps and negative prompt weren't sent at all — the eval box supplied its own.

The corpus manifest is built once, offline, listing every clip with its window already chosen and its truth already hashed. At duel time nothing is detected, listed, or skipped. Pre-filtering offline is sound because the reasons a video is unusable — too short, no single-shot window — are seed-independent properties of the video; the seed only ever picked among viable candidates.

The spec has no defaults and forbids extras. A field with a default is a field a validator can silently forget, and forgetting one produces a plausible verdict that quietly disagrees with everyone else's. The eval server is now a pure executor of the request — it reads nothing about how to duel from its own environment — and echoes the spec back, which the validator verifies before crowning.

Determinism, honestly. Torch flags buy run-to-run reproducibility on one box, not bit-exactness across GPU architectures for a 14B bf16 model — and no flag will. So this removes every other source of noise (metric_device = "cpu": scoring in fp32 costs minutes against hours of generation), leaving generation as the only fuzzy step, and makes the residual noise visible: per-clip distances and frame digests, so validators diff at three levels — frames (bit-exact?), distances (close?), verdict (same crown?).

Also fixed: frame order broke at 100 frames (frame_%02d.jpg + sorted() puts frame_100 before frame_11 — and the pinned config is 81 frames); the truth was lossy twice (x264 then JPEG); paired_bootstrap_verdict carried a wall-clock timestamp, so two validators that agreed perfectly still produced different verdict bytes; load_video_pipeline silently fell back to a different pipeline, so the "pinned" pipeline was not pinned; and 31-bit seed truncation threw away half the bits blake2b produced.

2. Stop the eval server wedging the subnet (8a61b1d)

It holds the only GPU behind one global lock, and a duel takes hours. Every way that lock can leak halts the subnet — and it all looks like "busy".

• The lock leaked whenever Thread.start() raised. It was released in the worker's finally, which never runs if the thread never starts. The box then answered 409 to every future duel, forever, while looking perfectly healthy.
• A duel could end with no terminal event, which was indistinguishable from a busy server — so a broken duel read as "try again later", permanently.
• The SSE log was a queue.Queue: a late subscriber blocked forever, and two subscribers split the stream. Now an append-only log with a cursor, so late subscribers replay from the start.
• The watchdog watches progress, not wall-clock. A slow-but-alive 70 GB download is never killed; a dead socket is.
• Bind safety: POST /eval makes the box download and execute an arbitrary model. On 0.0.0.0 with no auth that is RCE with a REST API — now loopback by default, and a public bind without a token refuses to start.

⚠️ The bug this commit matters most for

A validator with its own broken corpus would have quarantined every honest miner on the subnet. The attempt ledger quarantines an artifact once its attempts are exhausted — whatever the failure class. A validator whose corpus didn't match the manifest, or whose eval box ran a stale chain.toml, would fail every duel transiently, four times each, and then permanently lock out every miner, for a mistake that was entirely its own.

Hence ErrorClass.LOCAL: faults that are ours cost the challenger nothing and stop the validator instead. It is not a blanket amnesty — a genuinely broken model is still PERMANENT and still quarantined.

3. Bound the validator tick (389b9aa)

A duel was awaited inline — hours in which the validator set no weights (the chain concludes it is dead), published no dashboard, and couldn't report what it was doing. Now: dispatch → persist an in-flight slot → return in seconds. Weights and the dashboard run every tick, including mid-duel.

Restart-safe: the slot is persisted, so a validator that restarts mid-duel re-attaches. Before, the eval box spent hours on a duel nobody would ever read, and 409'd everyone else the whole time. A verdict against a deposed king is discarded (and the challenger is not marked seen, so it gets a fair duel against the king it actually has to beat). LEOMA_EVAL_TIMEOUT is deleted — the validator no longer guesses how long a video duel takes.

4. Stop re-downloading the king (0bbb102)

materialize_model's cache check looked for a root config.json and a top-level *.safetensors — a transformers layout. A diffusers snapshot has neither (it has model_index.json, and the weights live in transformer/, vae/, text_encoder/). So the check never hit, and every duel rmtreed the cache and pulled the king's ~30-70 GB again — for the same king, against every challenger in the queue.

A completion marker replaces it, which also fixes the latent bug that an interrupted download looked exactly like a valid cache. Plus LRU eviction with a keep-set (run inline before a download — the only moment disk pressure matters), a warm king pipeline, and release_pipeline() — grep empty_cache across the repo previously returned zero hits.

Same root cause, second bug: sha256_safetensors globbed non-recursively, matched zero files on a diffusers layout, and returned the sha256 of the empty string — a constant, for every model on earth. As the copy-detector it would have flagged every model as identical to every other one.

5. The cheats our metric invites (a2be2cd)

The freeze cheat. Leoma scores closeness to the real continuation, which creates a cheat the reference subnet structurally cannot have: a model that just holds the conditioning frame scores well on any clip that barely moves, having learned nothing. The 1-frame version died in #7; a full-length freeze is still a legal generation, and it can genuinely beat a mediocre king.

The fix makes the cheat a third duelist: freeze_frames is a GenerateFn, so it runs through the same run_duel loop, the same metric, the same clips and the same paired_bootstrap_verdict. One statistical primitive, two opponents — a challenger must beat the king and the cheat. The margin is scale-free (a fraction of the cheat's own mean distance), so it survives any metric recalibration; LPIPS, MSE and flow live on wildly different scales and an absolute margin would silently mean something different on each. Launch setting is LCB-only (0.0), because we have not yet measured on real hardware how much headroom a good model has — avg_freeze_distance is published so it can be measured rather than invented.

A king that fails the gate raises a loud alarm but is not auto-deposed: auto-dethroning on a gate failure would let an attacker shift the corpus static and hand the crown to a marginal challenger.

Copy-of-king, two layers: a different hotkey re-committing the king's exact digest was handed a full multi-hour duel (it could never win — a copy ties exactly — it was simply free to repeat). Rejected pre-dispatch now. And at duel time, bit-identical generations on every clip are caught for free, because the duel is deterministic — that catches a copy repackaged under a new digest, by what it does rather than what it claims to be.

Rate limit. The seen-set (hotkey|digest) is an idempotency gate, not a cost gate: a hotkey can change one byte, re-upload, and buy a brand-new free multi-hour duel. Now a cooldown, a per-reign cap, and a reign refresh so a durable king never becomes a permanent lockout. Strikes are only for gate rejections, never for losing — a miner whose honest model isn't good enough has done nothing wrong.

6. The architecture lock (0bf25ee)

A wrong-architecture model cost a full dispatch: tens of GB downloaded, two 14B pipelines loaded, the eval lock held — and only then the discovery that it was never loadable. The prescreen runs the lock on a config-only fetch (~200 KB) on the validator, before dispatch: seconds, not hours. It finally consumes three hooks dead since they were written (materialize_model(config_only=True), whose docstring literally says "use for the validator's per-challenger arch/lock validation", plus EXTRA_LOCK_KEYS and ARCH_BASE_REPO).

The lock is nested and per-component, because a diffusers snapshot has no root config.json. It validates by diffing against the base repo's own configs rather than numbers hand-copied into chain.toml — hand-transcribed shapes rot the day the pinned base is bumped, and the lock would start rejecting the very architecture it exists to enforce. Fail-open on infrastructure, fail-closed on architecture: if the prescreen itself can't run, that's our problem and must not be charged to the miner.

---

Tests

441 passing, up from 249. The decode tests run the real ffmpeg — stubbing it would test nothing about what actually went wrong.

• test_frame_order_survives_past_100_frames — decodes a 120-frame ramp, asserts monotonic luma
• test_the_clip_set_does_not_depend_on_the_corpus_SIZE
• test_a_box_that_decodes_video_differently_refuses_to_duel
• test_a_spec_missing_one_field_is_rejected_not_defaulted
• test_released_when_the_worker_thread_cannot_even_start — the permanent-409 lock leak
• test_a_SLOW_but_progressing_duel_is_NOT_killed — the watchdog's critical half
• test_a_broken_corpus_does_NOT_quarantine_a_single_miner
• test_the_slot_survives_a_restart_and_is_settled_later
• test_a_complete_diffusers_snapshot_is_reused — the 70 GB re-download
• test_a_freeze_cheater_that_BEAT_the_king_is_still_rejected
• test_bit_identical_generations_are_caught_at_duel_time_too
• test_a_wrong_arch_model_never_reaches_the_GPU

Verified by building a wheel and importing it from a clean venv outside the source tree.

⚠️ Operator action

leoma corpus build-manifest --corpus-id leoma-corpus-v1
leoma corpus publish-manifest manifest.json    # prints the digest
# paste into chain.toml [corpus].manifest_digest
leoma corpus verify --sample 4                 # run on EVERY eval box before it duels

Until [corpus].manifest_digest (and [seed].seed_digest, from #7) are pinned, validators refuse to duel and burn to UID 0. Deliberate: an unpinned corpus is not reproducible. It is a runtime refusal, not a crash — a validator that crash-loops can't burn, can't publish a dashboard, and can't tell you why it's unhappy.

The corpus must be ≥ 20× n_clips (640 clips at the pinned 32), or a miner can simply memorize the exam.

Known risk, not papered over

delta_threshold is still uncalibrated against cross-GPU noise. This work makes divergence detectable (frame digests, per-clip distances, verdict_digest), not impossible. The measurement — same model as both king and challenger across the fleet, take the |Δdistance| tail, set delta above it — needs the GPU box, and remains the subnet's largest open consensus risk. Bigger than any bug fixed here.