Harden the king-of-the-hill subnet: build, state integrity, liveness, weight correctness, and the short-generation exploit

[vex0209-bt]

Summary

The first hardening tranche on top of the king-of-the-hill re-architecture (#6): make the package actually build, then fix the correctness bugs that make the subnet untrustworthy. Plus the dashboard publisher, which had not been merged.

Every fix here is Leoma-native — shaped by what Leoma actually is (slow multi-GB diffusers duels, a live corpus, a reference-distance metric), not transplanted from the reference subnet. The organizing principle throughout: the chain is the queue. Leoma has no persistent work queue — the challenger list is re-derived every tick from get_all_revealed_commitments — so the answer to failure is never a durable queue, it's a decision function over a stateless work list.

1. It did not build, and it was not tested (21e27ca)

• Dockerfile copied leoma.py, deleted in 8d5bd4c — the image could not build at all.
• chain.toml was never packaged, yet chain_config open()s it at import, resolving into site-packages/ in a non-editable install ⇒ both services would have crash-looped on startup even once the build was fixed. It now lives at leoma/chain.toml, resolves via importlib.resources, and is declared as package-data — it is consensus-critical, so the pin must travel with the code.
• Compose ran the eval server from an image with no torch. Split into a slim validator image + Dockerfile.eval (CUDA base + the [eval] extra). Both now fail at build time if the package can't import.
• numpy was only in [eval] but is imported at module scope on the validator path.
• CI's path filter watched the deleted leoma.py instead of leoma/**, and there was no test workflow at all. Added one whose three jobs map 1:1 onto exactly these bug classes: pytest; wheel import-safety from outside the source tree; docker build smoke.

2. Silent state destruction (60e8ef6)

JsonBucketStore.get swallowed every exception and returned None, so a transient bucket outage was indistinguishable from an empty bucket: KingState.load returned blank state — re-seeding genesis, re-dueling every past challenger, wiping history — and the next flush overwrote the good bucket state with the blank one.

• get now distinguishes a genuine miss (S3 NoSuchKey) from an error (StoreUnavailable / StoreCorrupt), with bounded retries.
• load refuses to start on a partial or failed read; main() exits for the supervisor unless LEOMA_FORCE_FRESH_STATE=1. A validator that cannot read its own state must not run.
• State is now one canonical, atomically-PUT object (state/state.json, schema v2) instead of five non-atomic keys that could leave the king updated but seen-set and history stale. Legacy keys are mirrored and migrate transparently.
• Every Minio call moved to a worker thread (they were blocking the async loop), and clients get explicit urllib3 pools — minio's default is 300 s × 5 retries, so one hung socket could stall a validator for ~25 minutes.

3. One bad model wedged the whole subnet (32651d1)

process_challengers did return (not continue) on any duel error, so a single challenger whose repo 404s permanently blocked every later challenger — a free griefing vector, and nothing was ever recorded as failed.

• failures.py classifies BUSY / TRANSIENT / PERMANENT, defaulting to TRANSIENT — a bad transient costs four retries, a bad permanent locks a legitimate miner out. (Auth errors are deliberately not permanent: our own token misconfig would otherwise quarantine every miner on the subnet.)
• Retries are an attempt ledger keyed by hotkey|digest with block-based backoff — the chain re-supplies the work item for free next tick, so we only need to supply the memory. Quarantine is artifact-scoped, so a miner who fixes the model gets a clean slate; it is never a ban on a person. Only a hotkey with several quarantined digests is dropped at scan time — which finally wires scan_reveals' long-dead blacklist= hook.
• dispatch_duel now raises EvalBusy vs EvalJobFailed instead of returning None for both, so a broken duel is no longer mistaken for a busy server; a verdictless stream is an explicit transient failure rather than silence.
• stats.failed is incremented for the first time, and quarantined challengers finally produce the error history row the frontend has been typed for since day one.
• The unopposed-crown path is deleted. With no king and no seed_digest, the first reveal was crowned with no duel and no validation. We now burn to UID 0 instead.

4. Weights, and the freeze cheat's root (cd22b2c)

set_weights' return value was discarded. bittensor returns (success, message), and its rate-limit path returns False without submitting an extrinsic — so a no-op advanced last_weight_block as if the weights had landed, blocking retries for ~1 h and misreporting state. The result is now inspected (a rate-limit no-op is not a genuine failure), last_weight_block only advances on real success, failures back off exponentially, and a set is forced at startup. And since the chain is the queue, the chain is also the weight clock (blocks_since_last_update) — which makes "state says the weights landed but they didn't" structurally impossible.

The short-generation exploit. metrics._align truncated both arrays to the shorter one, so a 1-frame generation was scored against a 1-frame truth — and frame 0 of the truth is the conditioning frame the model was handed:

metric	score for a 1-frame generation
flow / clip / temporal	0.0 — a literal perfect score
mse / ssim	≈ 0
lpips (the production default)	≈ 0

Every metric, including the default, handed a near-perfect score to a model that emitted one frame. A generation shorter than the truth is now rejected (longer is still truncated to the truth, which stays authoritative), and the guard sits above the lazy torch/cv2 imports so it is enforced — and unit-testable — with neither installed. Two existing tests asserted the vulnerable behavior and have been inverted.

5. Public dashboard (3d1d832)

Duel history in KingState, a pure build_dashboard(), and dashboard.json published to the validator's own bucket each tick and on crown: king + 5-deep king chain with equal-weight shares, stats, queue, history, and the chain/duel params. No API, no database — the same bucket the state already lives in.

Tests

244 passing, up from 163. Each headline bug has a pinned regression test:

• test_transport_error_raises_not_none, test_load_refuses_partial_state — the blank-state overwrite
• test_one_failing_challenger_does_not_block_the_rest — the head-of-line block
• test_rate_limited_no_op_does_not_advance — the bittensor no-op
• test_one_frame_generation_is_rejected_by_every_metric — parametrized over all six metrics
• test_no_king_no_seed_burns_and_crowns_nobody — the unopposed crown

Verified end-to-end by building a real wheel and importing it from a clean venv outside the source tree.

⚠️ Operator action required before deploy

Pin chain.toml [seed].seed_digest. Now that the unopposed-crown path is gone, a subnet with no king and no seed digest burns 100 % to UID 0 rather than crowning an unevaluated model. That is the correct safe behavior, but it is an emission consequence: the seed must be pinned with or before this change.

Next in the series

Corpus manifest + chain-pinned consensus surface, eval-server resilience (watchdog, SSE replay, in-flight slot), GPU cache/teardown/eviction, the freeze-baseline gate, the diffusers config-lock, and rate limiting.