T1+T2 — Consensus, liveness & anti-abuse: pinned corpus, an eval server that can't wedge, and the freeze cheat closed

README.md

@@ -49,10 +49,16 @@ per-clip seed**; each generation is scored against the **real continuation** wit
 
 The challenger is crowned only if it is **confidently** better: the per-clip advantage
 `king_distance − challenger_distance` is bootstrapped, and the crown passes iff the lower-confidence
-bound `lcb > delta_threshold`. All duel parameters (`LEOMA_DELTA_THRESHOLD`, `LEOMA_ALPHA`,
-`LEOMA_N_BOOTSTRAP`, `LEOMA_DUEL_METRIC`, `LEOMA_DUEL_N_CLIPS`) **must match across validators** for
-consensus. The block hash is unpredictable until mined, so miners cannot overfit to the test set, yet
-every validator reproduces the identical verdict.
+bound `lcb > delta_threshold`.
+
+Every input that can change a verdict — the corpus, the prompt, the frame count, the resolution, the
+metric, the threshold — is pinned in **`chain.toml`**, hashed into a `consensus_digest` that is sent
+with each eval request and **echoed back in the verdict**. A validator running a different config
+cannot quietly disagree with the rest of the subnet: the mismatch is refused at the door. The held-out
+clips come from a **digest-pinned corpus manifest** (not a live bucket listing), with each clip's
+window and ground-truth hash fixed offline, so two validators provably grade the same exam. The block
+hash is unpredictable until mined, so miners cannot overfit to the test set, yet every validator
+reproduces the identical verdict.
 
 ---
 
@@ -94,6 +100,35 @@ leoma servers eval-server   # FastAPI on EVAL_SERVER_PORT (default 9000)
 The validator reaches it over `EVAL_SERVER_URL` (default `http://localhost:9000`, usually an SSH
 tunnel). One duel runs at a time. See `ecosystem.eval.config.js` for a PM2 launcher.
 
+Before a new eval box is allowed to duel, prove it decodes the pinned corpus byte-identically:
+
+```bash
+leoma corpus verify --sample 4
+```
+
+A box whose ffmpeg produces even slightly different pixels measures every distance against different
+ground truth — silently, confidently, and wrongly. This takes a minute and rules that out.
+
+---
+
+## Corpus (subnet operator)
+
+The duel's held-out clips come from a **pinned manifest**, not a live bucket listing. The manifest
+fixes which videos, the window inside each one, and the hash of the decoded ground truth — so every
+validator provably grades the same exam. Building it once, offline, is what removes the whole class of
+"two honest validators disagree" bugs: nothing is scene-detected, listed or skipped at duel time.
+
+```bash
+leoma corpus build-manifest --corpus-id leoma-corpus-v1   # decides windows, hashes truth
+leoma corpus publish-manifest manifest.json               # uploads; prints the digest
+# paste that digest into chain.toml [corpus].manifest_digest, then ship it
+```
+
+**Until `[corpus].manifest_digest` and `[seed].seed_digest` are pinned, validators refuse to duel and
+burn 100% to UID 0.** That is deliberate: an unpinned corpus is not reproducible, and an unevaluated
+first challenger must never be crowned. Rotate the corpus by rebuilding with a new `corpus-id` and
+re-pinning — a version bump, auditable in git.
+
 ---
 
 ## Miner setup

docker-compose.yml

@@ -20,12 +20,7 @@ services:
       WALLET_NAME: ${WALLET_NAME:-default}
       HOTKEY_NAME: ${HOTKEY_NAME:-default}
       EVAL_SERVER_URL: ${EVAL_SERVER_URL:-http://eval-server:9000}
-      # Duel parameters — identical across validators for consensus.
-      LEOMA_DUEL_METRIC: ${LEOMA_DUEL_METRIC:-lpips}
-      LEOMA_DUEL_N_CLIPS: ${LEOMA_DUEL_N_CLIPS:-32}
-      LEOMA_DELTA_THRESHOLD: ${LEOMA_DELTA_THRESHOLD:-0.0025}
-      LEOMA_ALPHA: ${LEOMA_ALPHA:-0.001}
-      LEOMA_N_BOOTSTRAP: ${LEOMA_N_BOOTSTRAP:-10000}
+      # Duel parameters live in chain.toml (the consensus surface), not here.
       # This validator's own state bucket (durable king state).
       OBJECT_STORAGE_BACKEND: ${OBJECT_STORAGE_BACKEND:-r2}
       R2_OWN_BUCKET: ${R2_OWN_BUCKET:-}

ecosystem.config.js

@@ -22,12 +22,7 @@ module.exports = {
         // Where duels are dispatched (SSH-tunnel to the GPU eval box).
         EVAL_SERVER_URL: "http://localhost:9000",
 
-        // Duel parameters — MUST match across validators for consensus.
-        LEOMA_DUEL_METRIC: "lpips",
-        LEOMA_DUEL_N_CLIPS: "32",
-        LEOMA_DELTA_THRESHOLD: "0.0025",
-        LEOMA_ALPHA: "0.001",
-        LEOMA_N_BOOTSTRAP: "10000",
+        // Duel parameters live in chain.toml (the consensus surface), not here.
         LEOMA_KING_CHAIN_SIZE: "5",
         LEOMA_WEIGHT_INTERVAL: "300",
         LEOMA_BURN_UID: "0",

env.validator.example

@@ -13,12 +13,12 @@ NETUID=99
 # Eval server the validator dispatches duels to (SSH-tunnel to the GPU box).
 EVAL_SERVER_URL=http://localhost:9000
 
-# Duel parameters — keep identical across all validators for consensus.
-LEOMA_DUEL_METRIC=lpips
-LEOMA_DUEL_N_CLIPS=32
-LEOMA_DELTA_THRESHOLD=0.0025
-LEOMA_ALPHA=0.001
-LEOMA_N_BOOTSTRAP=10000
+# Duel parameters are NOT set here any more. The metric, clip count, threshold,
+# alpha, bootstrap count and every generation knob are the CONSENSUS SURFACE and
+# live in chain.toml, hashed into a consensus_digest that is sent with every eval
+# request and echoed in every verdict. An env var is per-box, and a per-box exam is
+# not consensus: one validator with a stray LEOMA_DELTA_THRESHOLD used to be enough
+# to silently disagree with the rest of the subnet forever.
 LEOMA_KING_CHAIN_SIZE=5
 LEOMA_WEIGHT_INTERVAL=300
 LEOMA_BURN_UID=0

leoma/app/validator/dashboard.py

@@ -63,6 +63,21 @@ def build_dashboard(
 
     king = _king_entry(state.king, uid_map) if state.king else {}
 
+    # The duel currently on the GPU. The dashboard used to go dark for the entire
+    # length of a duel — hours in which the most interesting thing in the subnet was
+    # happening and the site showed nothing at all.
+    live = None
+    if state.inflight:
+        slot = dict(state.inflight)
+        live = {
+            "eval_id": slot.get("eval_id"),
+            "hotkey": slot.get("hotkey"),
+            "uid": uid_map.get(slot.get("hotkey", "")),
+            "model_repo": slot.get("model_repo"),
+            "model_digest": slot.get("model_digest"),
+            "dispatched_block": slot.get("dispatched_block"),
+        }
+
     return {
         "updated_at": updated_at,
         "chain": chain_meta,
@@ -72,6 +87,11 @@ def build_dashboard(
         "stats": dict(state.stats),
         "queue": list(queue or []),
         "history": list(state.history),
+        "live": live,
+        # Why the validator is not crowning anyone, if it isn't: an unpinned corpus, a
+        # missing seed digest, a stale eval box. Without this the operator sees a
+        # subnet burning 100% to UID 0 and no reason anywhere.
+        "degraded": state.degraded,
     }
 
 

leoma/app/validator/failures.py

@@ -5,22 +5,35 @@
 queue with ``requeue_front``/``retry_count``; we need a *decision function* over a
 stateless work list. This module is that decision.
 
-Three classes, and the distinction matters:
+Four classes, and the distinctions all matter:
 
 * ``BUSY``      — a property of the SERVER (the eval box is running someone else's
                   duel). Not a failure; consumes no attempt. The caller must
                   ``break`` (continuing would just 409 N more times).
-* ``TRANSIENT`` — a property of the ENVIRONMENT (network, disk, GPU, our corpus).
-                  Retry with block-based backoff; quarantine only after the attempt
-                  budget is exhausted.
+* ``TRANSIENT`` — a property of the ENVIRONMENT (network, disk, GPU). Retry with
+                  block-based backoff; quarantine only after the attempt budget is
+                  exhausted.
 * ``PERMANENT`` — a property of the ARTIFACT (the repo 404s, the weights won't
                   load, the arch is wrong). ``repo@digest`` is immutable, so this
                   can never succeed; quarantine it.
+* ``LOCAL``     — a property of **THIS VALIDATOR** (our corpus doesn't match the
+                  pinned manifest, our eval box runs a stale chain.toml). Costs the
+                  challenger **nothing**: no attempt, no backoff, no quarantine. The
+                  caller must ``break`` — every challenger would hit the identical
+                  wall, including the king.
 
 **Design rule: when in doubt, TRANSIENT.** A misclassified transient costs four
-retries. A misclassified permanent locks a legitimate miner out of an artifact.
-That asymmetry is why auth errors are deliberately *not* permanent: a validator's
-own token misconfiguration would otherwise quarantine every miner on the subnet.
+retries. A misclassified permanent locks a legitimate miner out of an artifact. That
+asymmetry is why auth errors are deliberately *not* permanent: a validator's own
+token misconfiguration would otherwise quarantine every miner on the subnet.
+
+**LOCAL exists because TRANSIENT is not actually harmless.** The attempt ledger
+quarantines an artifact once its attempts are ``exhausted``, whatever the class. So a
+validator whose *own* corpus was broken would fail every duel transiently, four times
+each, and then quarantine **every miner on the subnet** — permanently locking out
+honest models because of its own misconfiguration. Faults that are ours must therefore
+not touch the challenger's ledger at all. They are a reason to stop dueling, not a
+reason to blame whoever happened to be next in the queue.
 """
 from __future__ import annotations
 
@@ -32,6 +45,7 @@ class ErrorClass(str, Enum):
     BUSY = "busy"
     TRANSIENT = "transient"
     PERMANENT = "permanent"
+    LOCAL = "local"
 
 
 @dataclass(frozen=True)
@@ -48,6 +62,26 @@ def is_permanent(self) -> bool:
     def is_transient(self) -> bool:
         return self.kind is ErrorClass.TRANSIENT
 
+    @property
+    def is_local(self) -> bool:
+        return self.kind is ErrorClass.LOCAL
+
+
+# OUR fault, not the challenger's. These would fail identically for every model on
+# the subnet — including the reigning king — so they can never be evidence about a
+# particular challenger. Checked FIRST, before anything else can claim them.
+_LOCAL_SIGNS: tuple[tuple[str, str], ...] = (
+    ("corpus_integrity", "corpus_integrity"),
+    ("consensus_config", "consensus_config"),
+    ("consensus_mismatch", "consensus_mismatch"),
+    ("consensus_echo_mismatch", "consensus_echo_mismatch"),
+    ("code_mismatch", "code_mismatch"),
+    ("does not match the manifest", "corpus_integrity"),
+    ("decoded ground truth does not match", "corpus_integrity"),
+    ("corpus manifest digest mismatch", "corpus_integrity"),
+    ("manifest_digest is not pinned", "consensus_config"),
+    ("different consensus surface", "consensus_mismatch"),
+)
 
 # Substring -> (class, reason). Order matters: the first match wins, so the
 # PERMANENT artifact signatures are checked before the generic transient ones.
@@ -136,6 +170,9 @@ def classify_remote(message: str, reason: str = "") -> DuelFailure:
 
     token = reason.strip().lower()
     if token:
+        for _, r in _LOCAL_SIGNS:
+            if token == r:
+                return DuelFailure(ErrorClass.LOCAL, r, message)
         for _, r in _PERMANENT_SIGNS:
             if token == r:
                 return DuelFailure(ErrorClass.PERMANENT, r, message)
@@ -145,6 +182,10 @@ def classify_remote(message: str, reason: str = "") -> DuelFailure:
         if token.startswith("watchdog_stall"):
             return DuelFailure(ErrorClass.TRANSIENT, token, message)
 
+    hit = _match(text, _LOCAL_SIGNS)
+    if hit:
+        return DuelFailure(ErrorClass.LOCAL, hit, message)
+
     hit = _match(text, _PERMANENT_SIGNS)
     if hit:
         return DuelFailure(ErrorClass.PERMANENT, hit, message)
@@ -165,6 +206,23 @@ def classify(exc: BaseException) -> DuelFailure:
     if isinstance(exc, EvalJobFailed):
         return classify_remote(exc.detail, exc.reason)
 
+    # The typed eval errors say who is at fault directly — no substring guessing.
+    from leoma.eval.errors import (
+        ChallengerFault,
+        ConsensusConfigError,
+        CorpusIntegrityError,
+        DuelCancelled,
+    )
+
+    if isinstance(exc, (CorpusIntegrityError, ConsensusConfigError)):
+        return DuelFailure(ErrorClass.LOCAL, exc.reason, str(exc))
+    if isinstance(exc, DuelCancelled):
+        # Checked BEFORE ChallengerFault would be, and deliberately transient: a
+        # watchdog stall or an operator's DELETE says nothing about the model.
+        return DuelFailure(ErrorClass.TRANSIENT, exc.reason, str(exc))
+    if isinstance(exc, ChallengerFault):
+        return DuelFailure(ErrorClass.PERMANENT, exc.reason, str(exc))
+
     type_name = type(exc).__name__.lower()
     message = str(exc)
     text = f"{type_name} {message}".lower()
@@ -173,6 +231,10 @@ def classify(exc: BaseException) -> DuelFailure:
         reason = _match(text, _TRANSIENT_SIGNS) or "eval_unreachable"
         return DuelFailure(ErrorClass.TRANSIENT, reason, message)
 
+    hit = _match(text, _LOCAL_SIGNS)
+    if hit:
+        return DuelFailure(ErrorClass.LOCAL, hit, message)
+
     hit = _match(text, _PERMANENT_SIGNS)
     if hit:
         return DuelFailure(ErrorClass.PERMANENT, hit, message)