Feature: Manual/Automated Switch to Burn Mode for Exploit Detection

[joellidin]

Context

An exploit has been identified where miners skip backward steps to achieve artificially high MFU scores while passing weight verification through rebinding techniques.

Example:

• Miner UID 15 using _SKIP_BACKWARD = 4 achieves 86.6-86.8% MFU (exceeding the plausible maximum of 85.0%)
• Lower skip values (e.g., 3 or below) would pass verification while still being exploitative
• Code uses rebinding tricks to pass weight verification despite incomplete training

Request

Add capability for authorized operators to trigger "burn mode" for the subnet when exploits are detected. This would:

1. Emergency Response: Allow immediate switching to 100% burn rate when exploits are confirmed
2. Targeted Action: Option to burn specific bad actors (UIDs) while preserving honest miner emissions
3. Transparency: Ensure mechanism is not arbitrary - based on concrete exploit evidence

References

• Submission: https://www.tplr.ai/tournament/v19_commit_7889797_15
• Exploit: Skip backward steps with _SKIP_BACKWARD = N to reduce compute while maintaining plausible-looking weight updates

Rationale

Without this capability, exploits can persist until code patches are deployed, allowing bad actors to collect undeserved emissions while honest miners are displaced.

[machine-god-deus]

Machine God picked this up and is drafting a spec now.

[machine-god-deus]

Machine God approved the implementation spec and has begun implementation.

Approved spec

Now I have full context. Here is the implementation spec:

---

Implementation Spec: Burn Mode for Exploit Detection

Overview

Add a runtime-toggleable burn mode that allows authorized operators to redirect emissions away from exploitative miners. When activated, burn mode overrides the burn_rate from hparams.json and optionally blocks specific UIDs from receiving weight. State persists across restarts via the existing validator_state KV store. Operators interact through authenticated API endpoints on the existing FastAPI server.

The existing WeightSetter already splits emissions between burn_uid and a winner. This feature adds a runtime override layer on top of that logic — no new infrastructure, no new services.

Scope

1. BurnMode data model — Pydantic model with enabled, burn_rate_override, blocked_uids, reason, activated_at, activated_by.
2. Persistence — Serialize/deserialize burn mode state through the existing validator_state KV store in database.py.
3. WeightSetter integration — Load burn mode state at the top of set_weights(). When enabled: use overridden burn rate, exclude blocked UIDs from winning.
4. API endpoints — POST /burn-mode/activate, POST /burn-mode/deactivate, GET /burn-mode/status. Protected by the existing HMAC API key auth.
5. Logging — slog-equivalent structured logging (Python logging) for all burn mode state transitions.

Out of Scope

• New authentication mechanisms (wallet-based signatures, role-based access). The existing X-API-Key HMAC auth is sufficient.
• Automatic exploit detection heuristics. This is a manual/operator-driven toggle.
• Changes to hparams.json schema. Burn mode is purely runtime/DB-driven.
• UI/TUI changes for burn mode display.
• Changes to miner code (neurons/miner.py).
• Metagraph cadence overrides. Burn mode respects existing set_weights_interval_blocks.

Files / Components Likely to Change

File	Change
src/crusades/chain/weights.py	Add _load_burn_mode(), modify set_weights() to check burn mode before weight distribution, add blocked-UID exclusion logic
src/crusades/chain/burn_mode.py	New file. BurnMode Pydantic model, serialization helpers, BurnModeManager class encapsulating DB read/write
src/crusades/api/server.py	Add 3 endpoints: activate, deactivate, status. Wire Database dependency for burn mode persistence
src/crusades/storage/database.py	Add get_burn_mode() and set_burn_mode() convenience methods (thin wrappers over get_validator_state/set_validator_state with JSON serialization)
neurons/validator.py	Pass Database reference to API server startup if not already available (verify wiring)

Implementation Plan

Step 1: BurnMode Model (src/crusades/chain/burn_mode.py)

Create a new file with:

from pydantic import BaseModel, Field
from datetime import datetime

class BurnMode(BaseModel):
    enabled: bool = False
    burn_rate_override: float = Field(default=1.0, ge=0.0, le=1.0)
    blocked_uids: list[int] = Field(default_factory=list)
    reason: str = ""
    activated_at: datetime | None = None
    activated_by: str = ""

    @classmethod
    def inactive(cls) -> "BurnMode":
        return cls(enabled=False)

This is a pure data object. ~30 lines.

Step 2: Database Persistence (src/crusades/storage/database.py)

Add two methods to the Database class:

BURN_MODE_KEY = "burn_mode"

async def get_burn_mode(self) -> BurnMode:
    raw = await self.get_validator_state(BURN_MODE_KEY)
    if raw is None:
        return BurnMode.inactive()
    return BurnMode.model_validate_json(raw)

async def set_burn_mode(self, burn_mode: BurnMode) -> None:
    await self.set_validator_state(BURN_MODE_KEY, burn_mode.model_dump_json())

Reuses existing validator_state table. No schema migration. ~15 lines added.

Step 3: WeightSetter Integration (src/crusades/chain/weights.py)

Modify set_weights() (currently lines 64–233):

1. At the top of the method, after metagraph sync, add:

   burn_mode = await self.db.get_burn_mode()

2. If burn_mode.enabled:

   • Override self.burn_rate with burn_mode.burn_rate_override for this cycle.
   • After determining winner_uid from leaderboard, check if winner_uid in burn_mode.blocked_uids. If blocked:
     • Log the block event.
     • Fall through to _set_burn_only_weights() (100% to burn_uid), OR find next non-blocked winner from leaderboard.
     • For simplicity in v1: if winner is blocked, send 100% to burn_uid via _set_burn_only_weights().
   • Log that burn mode is active with reason.

3. If burn_mode.burn_rate_override == 1.0 and no blocked UIDs, this is functionally equivalent to the existing _set_burn_only_weights() path — minimal code change.

Key constraint: burn_mode state takes precedence over self.burn_rate (from hparams.json). This is checked every cycle since set_weights() already hits DB for _load_previous_winner().

~30 lines of logic added to existing method.

Step 4: API Endpoints (src/crusades/api/server.py)

Add three endpoints. All are protected by the existing verify_api_key dependency (global on the app). The API is currently read-only GET; these are the first POST endpoints, which is appropriate since burn mode is an operator action.

POST /burn-mode/activate

Request body:

{
  "burn_rate_override": 1.0,
  "blocked_uids": [15],
  "reason": "UID 15 using _SKIP_BACKWARD exploit"
}

• Validates input via Pydantic model.
• Sets activated_at = datetime.utcnow().
• Persists via db.set_burn_mode().
• Returns the stored BurnMode state.
• Logs activation at WARNING level.

POST /burn-mode/deactivate

• Sets BurnMode.inactive() in DB.
• Returns confirmation.
• Logs deactivation at WARNING level.

GET /burn-mode/status

• Returns current BurnMode state from DB.
• No side effects.

The Database instance is already available on app.state (wired during create_app or accessible via validator). If not currently on app.state, add it during app initialization — the validator already holds a Database reference.

~60 lines added to server.py.

Step 5: Wiring Verification

Confirm in neurons/validator.py that:

1. The Database instance is accessible to the API server. The validator creates self.db at line 132; the API server is started (if present) with access to the same DB path. If the API server creates its own Database instance from DATABASE_PATH env, burn mode persistence works automatically since it's the same SQLite file.
2. No additional wiring needed if API server already instantiates its own Database.

Test and Verification Plan

Unit Tests (tests/test_burn_mode.py — new file)

1. BurnMode model tests: Serialization round-trip, validation constraints (burn_rate_override bounds), inactive() factory.
2. Database persistence tests: set_burn_mode() then get_burn_mode() returns same state. get_burn_mode() with no prior state returns inactive().
3. WeightSetter burn mode tests: Mock db.get_burn_mode() to return active burn mode. Assert weight distribution uses override rate. Assert blocked UID triggers burn-only weights.

Integration Tests (tests/test_burn_mode_api.py — new file)

1. API endpoint tests using FastAPI TestClient:
   • Activate with valid payload → 200, state persisted.
   • Activate without API key → 401.
   • Deactivate → 200, state reset.
   • Status reflects current state.
   • Activate with invalid burn_rate_override (e.g., 1.5) → 422.

Manual Verification

# Lint
ruff check src/crusades/chain/burn_mode.py src/crusades/api/server.py
# Type check
mypy src/crusades/chain/burn_mode.py
# Tests
pytest tests/test_burn_mode.py tests/test_burn_mode_api.py -v

Verification Contract

The default contract (docker compose -f docker/compose.yml up -d) boots a full validator with GPU requirements. This is misaligned with the task — burn mode is a control-plane feature testable without GPUs or chain access.

Override: Tests should run via pytest tests/test_burn_mode*.py with an in-memory SQLite database and mocked chain/metagraph. No Docker boot required for verification. If the machine-god harness requires Docker, the compose file needs no modification — the validator container will pick up the new code on next image build. The compose file is NOT modified by this task.

Guardrail Exceptions

None. This task does not modify:

• CI/workflow files
• Package scripts or build config
• Lint configuration
• pyproject.toml dependencies (all required packages — pydantic, fastapi, sqlalchemy — are already present)
• Docker configuration

Parallel Work Units

• Unit 1: Data model + persistence — files: src/crusades/chain/burn_mode.py (new), src/crusades/storage/database.py (add 2 methods)
• Unit 2: API endpoints — files: src/crusades/api/server.py (add 3 routes). Depends on Unit 1.
• Unit 3: WeightSetter integration — files: src/crusades/chain/weights.py (modify set_weights()). Depends on Unit 1.
• Unit 4: Tests — files: tests/test_burn_mode.py (new), tests/test_burn_mode_api.py (new). Depends on Units 1–3.
• Shared/coordination: Unit 1 must complete first. Units 2 and 3 can proceed in parallel after Unit 1.

Risks / Open Questions

1. API key not configured in production: If DASHBOARD_API_KEY is unset, burn mode endpoints are unauthenticated. This is an existing API-wide issue, not introduced by this feature. Recommend documenting that API key MUST be set when burn mode is deployed. Consider logging a WARNING on startup if no API key is configured and burn mode endpoints exist.

2. Race between API write and weight-setting read: Operator activates burn mode via API while set_weights() is mid-execution. Low risk — set_weights() reads burn mode at the top of each cycle (~20 min cadence). The next cycle picks up the change. Acceptable latency for an emergency response tool.

3. Blocked UID is the only competitive miner: If all non-blocked miners are below threshold, set_weights() should fall through to burn-only mode (already the existing fallback). No special handling needed.

4. SQLite concurrent access: The API server and validator may access SQLite concurrently. aiosqlite with WAL mode (default for SQLAlchemy async) handles this. Confirm WAL mode is enabled in Database.__init__.

5. No audit trail beyond logs: Burn mode activation/deactivation is logged but not stored historically. For v1 this is acceptable. A future enhancement could store an event log table.

[machine-god-deus]

Machine God completed delivery.

Pull requests:

• Feature: Manual/Automated Switch to Burn Mode for Exploit Detection #78