55queue with ``requeue_front``/``retry_count``; we need a *decision function* over a
66stateless work list. This module is that decision.
77
8- Three classes, and the distinction matters :
8+ Four classes, and the distinctions all matter :
99
1010* ``BUSY`` — a property of the SERVER (the eval box is running someone else's
1111 duel). Not a failure; consumes no attempt. The caller must
1212 ``break`` (continuing would just 409 N more times).
13- * ``TRANSIENT`` — a property of the ENVIRONMENT (network, disk, GPU, our corpus).
14- Retry with block-based backoff; quarantine only after the attempt
15- budget is exhausted.
13+ * ``TRANSIENT`` — a property of the ENVIRONMENT (network, disk, GPU). Retry with
14+ block-based backoff; quarantine only after the attempt budget is
15+ exhausted.
1616* ``PERMANENT`` — a property of the ARTIFACT (the repo 404s, the weights won't
1717 load, the arch is wrong). ``repo@digest`` is immutable, so this
1818 can never succeed; quarantine it.
19+ * ``LOCAL`` — a property of **THIS VALIDATOR** (our corpus doesn't match the
20+ pinned manifest, our eval box runs a stale chain.toml). Costs the
21+ challenger **nothing**: no attempt, no backoff, no quarantine. The
22+ caller must ``break`` — every challenger would hit the identical
23+ wall, including the king.
1924
2025**Design rule: when in doubt, TRANSIENT.** A misclassified transient costs four
21- retries. A misclassified permanent locks a legitimate miner out of an artifact.
22- That asymmetry is why auth errors are deliberately *not* permanent: a validator's
23- own token misconfiguration would otherwise quarantine every miner on the subnet.
26+ retries. A misclassified permanent locks a legitimate miner out of an artifact. That
27+ asymmetry is why auth errors are deliberately *not* permanent: a validator's own
28+ token misconfiguration would otherwise quarantine every miner on the subnet.
29+
30+ **LOCAL exists because TRANSIENT is not actually harmless.** The attempt ledger
31+ quarantines an artifact once its attempts are ``exhausted``, whatever the class. So a
32+ validator whose *own* corpus was broken would fail every duel transiently, four times
33+ each, and then quarantine **every miner on the subnet** — permanently locking out
34+ honest models because of its own misconfiguration. Faults that are ours must therefore
35+ not touch the challenger's ledger at all. They are a reason to stop dueling, not a
36+ reason to blame whoever happened to be next in the queue.
2437"""
2538from __future__ import annotations
2639
@@ -32,6 +45,7 @@ class ErrorClass(str, Enum):
3245 BUSY = "busy"
3346 TRANSIENT = "transient"
3447 PERMANENT = "permanent"
48+ LOCAL = "local"
3549
3650
3751@dataclass (frozen = True )
@@ -48,6 +62,26 @@ def is_permanent(self) -> bool:
4862 def is_transient (self ) -> bool :
4963 return self .kind is ErrorClass .TRANSIENT
5064
65+ @property
66+ def is_local (self ) -> bool :
67+ return self .kind is ErrorClass .LOCAL
68+
69+
70+ # OUR fault, not the challenger's. These would fail identically for every model on
71+ # the subnet — including the reigning king — so they can never be evidence about a
72+ # particular challenger. Checked FIRST, before anything else can claim them.
73+ _LOCAL_SIGNS : tuple [tuple [str , str ], ...] = (
74+ ("corpus_integrity" , "corpus_integrity" ),
75+ ("consensus_config" , "consensus_config" ),
76+ ("consensus_mismatch" , "consensus_mismatch" ),
77+ ("consensus_echo_mismatch" , "consensus_echo_mismatch" ),
78+ ("code_mismatch" , "code_mismatch" ),
79+ ("does not match the manifest" , "corpus_integrity" ),
80+ ("decoded ground truth does not match" , "corpus_integrity" ),
81+ ("corpus manifest digest mismatch" , "corpus_integrity" ),
82+ ("manifest_digest is not pinned" , "consensus_config" ),
83+ ("different consensus surface" , "consensus_mismatch" ),
84+ )
5185
5286# Substring -> (class, reason). Order matters: the first match wins, so the
5387# PERMANENT artifact signatures are checked before the generic transient ones.
@@ -136,6 +170,9 @@ def classify_remote(message: str, reason: str = "") -> DuelFailure:
136170
137171 token = reason .strip ().lower ()
138172 if token :
173+ for _ , r in _LOCAL_SIGNS :
174+ if token == r :
175+ return DuelFailure (ErrorClass .LOCAL , r , message )
139176 for _ , r in _PERMANENT_SIGNS :
140177 if token == r :
141178 return DuelFailure (ErrorClass .PERMANENT , r , message )
@@ -145,6 +182,10 @@ def classify_remote(message: str, reason: str = "") -> DuelFailure:
145182 if token .startswith ("watchdog_stall" ):
146183 return DuelFailure (ErrorClass .TRANSIENT , token , message )
147184
185+ hit = _match (text , _LOCAL_SIGNS )
186+ if hit :
187+ return DuelFailure (ErrorClass .LOCAL , hit , message )
188+
148189 hit = _match (text , _PERMANENT_SIGNS )
149190 if hit :
150191 return DuelFailure (ErrorClass .PERMANENT , hit , message )
@@ -165,6 +206,23 @@ def classify(exc: BaseException) -> DuelFailure:
165206 if isinstance (exc , EvalJobFailed ):
166207 return classify_remote (exc .detail , exc .reason )
167208
209+ # The typed eval errors say who is at fault directly — no substring guessing.
210+ from leoma .eval .errors import (
211+ ChallengerFault ,
212+ ConsensusConfigError ,
213+ CorpusIntegrityError ,
214+ DuelCancelled ,
215+ )
216+
217+ if isinstance (exc , (CorpusIntegrityError , ConsensusConfigError )):
218+ return DuelFailure (ErrorClass .LOCAL , exc .reason , str (exc ))
219+ if isinstance (exc , DuelCancelled ):
220+ # Checked BEFORE ChallengerFault would be, and deliberately transient: a
221+ # watchdog stall or an operator's DELETE says nothing about the model.
222+ return DuelFailure (ErrorClass .TRANSIENT , exc .reason , str (exc ))
223+ if isinstance (exc , ChallengerFault ):
224+ return DuelFailure (ErrorClass .PERMANENT , exc .reason , str (exc ))
225+
168226 type_name = type (exc ).__name__ .lower ()
169227 message = str (exc )
170228 text = f"{ type_name } { message } .lower ()
@@ -173,6 +231,10 @@ def classify(exc: BaseException) -> DuelFailure:
173231 reason = _match (text , _TRANSIENT_SIGNS ) or "eval_unreachable"
174232 return DuelFailure (ErrorClass .TRANSIENT , reason , message )
175233
234+ hit = _match (text , _LOCAL_SIGNS )
235+ if hit :
236+ return DuelFailure (ErrorClass .LOCAL , hit , message )
237+
176238 hit = _match (text , _PERMANENT_SIGNS )
177239 if hit :
178240 return DuelFailure (ErrorClass .PERMANENT , hit , message )
0 commit comments