Document checkpoint and GPU broker contracts

Author: echobt

docs/miner/README.md

@@ -61,11 +61,21 @@ def compute_loss(model, batch, ctx):
 
 def train_step(model, batch, optimizer, ctx):
     ...
+
+def save_checkpoint(model, checkpoint_dir, ctx):
+    ...
+
+def load_checkpoint(model, checkpoint_dir, ctx):
+    ...
 ```
 
 `configure_optimizer` gives full optimizer and LR control. Use it for custom optimizers, parameter groups, schedulers, or learning rates that should not be reduced to evaluator fallback choices. Without that hook, the fallback optimizer may apply safe evaluator defaults/caps, including learning-rate caps.
 
-`train_step` can implement a fully custom update step. Use it when you need a training loop other than the evaluator default. It must return a loss tensor and stay within sandbox and resource limits.
+`train_step` can implement a fully custom update step. Use it when you need a training loop other than the evaluator default. It must return a loss tensor and stay within sandbox and resource limits. PRISM launches 1-8 GPU container runs with single-node torchrun, including `torchrun --standalone --nnodes=1 --nproc-per-node=1` for a 1 GPU run. When PRISM wraps default multi-process training with DDP, a custom `train_step` that bypasses the default loop must be DDP-safe and rank-aware.
+
+Use `save_checkpoint(model, checkpoint_dir, ctx)` and `load_checkpoint(model, checkpoint_dir, ctx)` only for model state inside the evaluator-provided checkpoint workspace. The checkpoint fields on `ctx` are `checkpoint_dir`, `resume_checkpoint_dir`, `checkpoint_api_version`, `attempt`, `is_resume`, `rank`, `local_rank`, `world_size`, `distributed_backend`, `device`, and `checkpoint_metadata`. `save_checkpoint` may return `None`, a checkpoint-dir-relative `str`, or the exact shape `{"path": str, "metadata": dict[str, object]}`. Return `None` only when no checkpoint artifact should be recorded; return a checkpoint-dir-relative `str` or the exact dict shape when PRISM should accept and record a produced checkpoint artifact. PRISM records accepted checkpoint artifacts through manifest paths under the run artifact root. External checkpoint paths and miner-selected resume sources are not supported. The workspace cap is decimal 10G, exactly `10_000_000_000` bytes.
+
+Evaluator resume in v1 is retry-only after eligible infrastructure or eviction failures. It does not resume sandbox failures, miner code failures, scoring failures, or policy failures.
 
 ## Artifact Manifest
 

docs/operators.md

@@ -55,6 +55,8 @@ PRISM_PLATFORM_EVAL_PIDS_LIMIT=512
 PRISM_DOCKER_NETWORK=none
 ```
 
+For broker-backed runs, PRISM sends only the official Platform SDK GPU field `gpu_count`. `gpu_count=None` or an omitted field is CPU-only; a positive integer is a GPU request. Platform owns `gpu_resource_name`, defaulting to `nvidia.com/gpu`, and Kubernetes broker mode maps positive counts to `resources.limits['nvidia.com/gpu']` unless Platform is configured with a different resource name. PRISM GPU device IDs are metadata for observability, not Kubernetes placement semantics. Single-GPU runs use `torchrun --standalone --nnodes=1 --nproc-per-node=1`; PRISM does not claim multi-node support.
+
 ## Review Configuration
 
 Recommended production review settings:

docs/scaling.md

@@ -44,6 +44,20 @@ Official score-eligible GPU runs use a fixed runtime profile. The default policy
 
 Official scoring uses the fixed profile from SQL runtime config or its defaults. Autosplit is allowed only for non-scoring or development paths such as smoke runs. It is not an official scoring shortcut.
 
+### Broker GPU Contract
+
+PRISM passes the actual lease size to the Platform broker as `gpu_count`. `gpu_count=None` or an omitted `gpu_count` means CPU-only execution and must not create a Kubernetes GPU resource limit. A positive integer requests that many GPUs. Invalid values such as `0`, negatives, booleans, strings, or floats fail validation before placement.
+
+Platform owns `gpu_resource_name`; PRISM does not pass it. In Kubernetes broker mode, Platform maps a positive `gpu_count` to `resources.limits['nvidia.com/gpu']` by default, or to the configured Platform-owned resource name. Prism GPU environment variables, labels, payload metadata, and device IDs are observability and backward-compatibility metadata only. Device IDs are not Kubernetes placement semantics, and this contract is not an arbitrary TPU, AMD, or custom accelerator abstraction.
+
+## Single-Node Torchrun And DDP
+
+PRISM's distributed v1 scope is single-node only. Runs with 1-8 GPUs use single-node torchrun with one process per GPU, including `torchrun --standalone --nnodes=1 --nproc-per-node=1` for a 1 GPU run. Requests above 8 GPUs are rejected. This documents command and environment support, not proof that every submission succeeds on 8 GPUs.
+
+PRISM DDP-wraps default training before running the default loop. Rank 0 writes shared checkpoint and manifest artifacts, including `prism_run_manifest.v1.json`; other ranks participate in training and synchronization without writing those shared artifacts. Custom `train_step` implementations that bypass the default loop must be DDP-safe and rank-aware.
+
+PRISM does not support multi-node distributed training in v1.
+
 ## Bad Scaling Predictors
 
 The following signals are weak predictors of frontier-scale performance when used alone:

docs/security.md

@@ -104,7 +104,15 @@ Container limits include:
 * read-only runtime option
 * optional GPU type and count
 
-Official score-eligible GPU jobs use the fixed official GPU profile from SQL runtime config or defaults, with a maximum of 8 GPUs. Autosplit is for non-scoring or development paths only.
+Official score-eligible GPU jobs use the fixed official GPU profile from SQL runtime config or defaults, with a maximum of 8 GPUs. Autosplit is for non-scoring or development paths only. PRISM sends the actual lease size as `gpu_count`: `None` or omitted means CPU-only, while a positive integer requests GPUs. Platform owns `gpu_resource_name` and maps the count to Kubernetes `resources.limits['nvidia.com/gpu']` by default. PRISM does not pass `gpu_resource_name`; device IDs remain metadata for logs and manifests, not Kubernetes placement semantics. Network isolation depends on the cluster CNI and the NetworkPolicy enforcement configured by Platform and the operator.
+
+## Checkpoint Workspace Security
+
+Checkpoint hooks write only inside an evaluator-owned checkpoint workspace. PRISM rejects absolute paths, `..` traversal, symlinks, and any checkpoint path outside that workspace. The checkpoint workspace cap is decimal 10G, exactly `10_000_000_000` bytes.
+
+Evaluator resume in v1 is retry-only after eligible infrastructure or eviction failures, with validated same submission, code, architecture, and recipe lineage. It is not used for sandbox failures, miner code failures, scoring failures, or policy failures. Miners cannot select arbitrary external checkpoint paths or resume sources, and PRISM does not support object-store or cloud checkpoint upload in this contract.
+
+Distributed execution is single-node only in v1. PRISM launches 1-8 GPU container runs with single-node torchrun, including `torchrun --standalone --nnodes=1 --nproc-per-node=1` for a 1 GPU run. PRISM does not support multi-node distributed training.
 
 ## Scientific Security References
 

docs/submissions.md

@@ -87,6 +87,12 @@ def compute_loss(model, batch, ctx):
 
 def train_step(model, batch, optimizer, ctx):
     ...
+
+def save_checkpoint(model, checkpoint_dir, ctx):
+    ...
+
+def load_checkpoint(model, checkpoint_dir, ctx):
+    ...
 ```
 
 | Hook | Purpose | Attribution |
@@ -96,13 +102,53 @@ def train_step(model, batch, optimizer, ctx):
 | `infer` | Fallback inference path when `inference_logits` is absent. | Training or inference owner |
 | `compute_loss` | Custom loss, auxiliary losses, regularization. | Training owner |
 | `train_step` | Fully custom update step. | Training owner |
+| `save_checkpoint` | Save model state into the evaluator-provided checkpoint workspace. | Training owner |
+| `load_checkpoint` | Load model state from an evaluator-approved retry checkpoint workspace. | Training owner |
 
 Use `configure_optimizer` when you need complete optimizer and LR control, including parameter groups, custom optimizer classes, scheduler setup, clipping wrappers, or learning rates outside evaluator defaults. If `configure_optimizer` is absent, the fallback optimizer may apply safe evaluator defaults/caps, including learning-rate caps, while still reading recipe metadata where allowed.
 
-Use `train_step` when the default `zero_grad`, `loss.backward`, gradient clipping, and `optimizer.step` loop is not enough. `train_step` can implement a fully custom update step, as long as it returns a valid loss tensor and stays within the sandbox and resource limits.
+Use `train_step` when the default `zero_grad`, `loss.backward`, gradient clipping, and `optimizer.step` loop is not enough. `train_step` can implement a fully custom update step, as long as it returns a valid loss tensor and stays within the sandbox and resource limits. PRISM launches 1-8 GPU container runs with single-node torchrun, including `torchrun --standalone --nnodes=1 --nproc-per-node=1` for a 1 GPU run. When PRISM launches multi-process torchrun/DDP, PRISM wraps the model, but custom `train_step` implementations are responsible for DDP-safe and rank-aware behavior if they bypass the default loop. PRISM does not support multi-node distributed training.
 
 If both `inference_logits` and `infer` exist, `inference_logits` takes precedence.
 
+## Checkpoint Hooks
+
+Submissions may define these exact optional hook signatures:
+
+```python
+def save_checkpoint(model, checkpoint_dir, ctx):
+    ...
+
+def load_checkpoint(model, checkpoint_dir, ctx):
+    ...
+```
+
+`checkpoint_dir` is an evaluator-owned directory. Save into that directory or a child path under it. Do not choose an external checkpoint path. `load_checkpoint` is called only when PRISM approves a retry resume source for the same submission, code, architecture, and recipe lineage. v1 resume is retry-only after eligible infrastructure or eviction failures, not sandbox failures, miner code failures, scoring failures, or policy failures. PRISM does not support arbitrary external checkpoint resume.
+
+Checkpoint and distributed fields on `ctx` are:
+
+* `checkpoint_dir`
+* `resume_checkpoint_dir`
+* `checkpoint_api_version`
+* `attempt`
+* `is_resume`
+* `rank`
+* `local_rank`
+* `world_size`
+* `distributed_backend`
+* `device`
+* `checkpoint_metadata`
+
+Accepted `save_checkpoint` return schemas are:
+
+* `None`, when no checkpoint artifact was produced for PRISM to record.
+* A checkpoint-dir-relative `str`, when the main checkpoint file or directory is below `checkpoint_dir` and should be accepted and recorded.
+* The exact shape `{"path": str, "metadata": dict[str, object]}`. `path` is checkpoint-dir-relative, and `metadata` contains JSON-compatible checkpoint metadata for an accepted and recorded checkpoint artifact.
+
+Writing files under `checkpoint_dir` is not enough for PRISM to record a produced checkpoint. Return a checkpoint-dir-relative `str` or the exact dict shape above when the hook creates a checkpoint artifact that should be accepted for manifest recording and retry resume.
+
+Absolute paths, `..` traversal, symlinks, and paths outside `checkpoint_dir` are rejected. PRISM records accepted checkpoint artifacts in `prism_run_manifest.v1.json` using artifact-root-relative manifest paths, not host paths. The checkpoint workspace cap is exactly decimal 10G, `10_000_000_000` bytes.
+
 ## Artifact Manifest
 
 Official and smoke evaluators write `prism_run_manifest.v1.json`. The manifest is the scoring contract for artifacts and metrics, not a free-form log. Submitted metrics are not free-form claims. They must be derived from artifacts, evaluator logs, and manifest fields that validators can check.

tests/test_docs_contract.py

@@ -152,12 +152,147 @@ def test_submission_docs_clarify_custom_training_boundaries() -> None:
         "configure_optimizer` gives full optimizer and LR control",
         "fallback optimizer may apply safe evaluator defaults/caps",
         "train_step` can implement a fully custom update step",
+        "custom `train_step` implementations are responsible for DDP-safe and rank-aware behavior",
         "training_for_arch` submission cannot silently change architecture family",
         "Submitted metrics are not free-form claims",
     ):
         assert expected in combined
 
 
+
+def test_checkpoint_hook_contract_docs_are_exact() -> None:
+    submissions_doc = read_doc("docs/submissions.md")
+    miner_doc = read_doc("docs/miner/README.md")
+    security_doc = read_doc("docs/security.md")
+    combined = f"{submissions_doc}\n{miner_doc}\n{security_doc}"
+
+    for expected in (
+        "save_checkpoint(model, checkpoint_dir, ctx)",
+        "load_checkpoint(model, checkpoint_dir, ctx)",
+        "`checkpoint_dir`",
+        "`resume_checkpoint_dir`",
+        "`checkpoint_api_version`",
+        "`attempt`",
+        "`is_resume`",
+        "`rank`",
+        "`local_rank`",
+        "`world_size`",
+        "`distributed_backend`",
+        "`device`",
+        "`checkpoint_metadata`",
+        "`None`, when no checkpoint artifact was produced for PRISM to record",
+        "A checkpoint-dir-relative `str`",
+        "should be accepted and recorded",
+        'The exact shape `{"path": str, "metadata": dict[str, object]}`',
+        "Writing files under `checkpoint_dir` is not enough for PRISM to record",
+        "Return `None` only when no checkpoint artifact should be recorded",
+        "return a checkpoint-dir-relative `str` or the exact dict shape",
+        "artifact-root-relative manifest paths",
+        "decimal 10G, `10_000_000_000` bytes",
+        "decimal 10G, exactly `10_000_000_000` bytes",
+        "retry-only after eligible infrastructure or eviction failures",
+        "same submission, code, architecture, and recipe lineage",
+    ):
+        assert expected in combined
+
+    forbidden_checkpoint_semantics = (
+        "`None`, when the hook writes its checkpoint files directly under `checkpoint_dir`",
+        "None, when the hook writes its checkpoint files directly under checkpoint_dir",
+    )
+    for phrase in forbidden_checkpoint_semantics:
+        assert phrase not in combined
+
+
+def test_distributed_docs_define_v1_single_node_scope() -> None:
+    submissions_doc = read_doc("docs/submissions.md")
+    miner_doc = read_doc("docs/miner/README.md")
+    scaling_doc = read_doc("docs/scaling.md")
+    security_doc = read_doc("docs/security.md")
+    combined = f"{submissions_doc}\n{miner_doc}\n{scaling_doc}\n{security_doc}"
+
+    for expected in (
+        "single-node only",
+        "Runs with 1-8 GPUs use single-node torchrun",
+        "torchrun --standalone --nnodes=1 --nproc-per-node=1",
+        "Requests above 8 GPUs are rejected",
+        "Rank 0 writes shared checkpoint and manifest artifacts",
+        "PRISM DDP-wraps default training",
+        "custom `train_step` implementations are responsible for DDP-safe and rank-aware behavior",
+        (
+            "Custom `train_step` implementations that bypass the default loop "
+            "must be DDP-safe and rank-aware"
+        ),
+        "does not support multi-node distributed training",
+        "command and environment support, not proof that every submission succeeds on 8 GPUs",
+    ):
+        assert expected in combined
+
+
+def test_public_docs_define_platform_gpu_broker_contract() -> None:
+    scaling_doc = read_doc("docs/scaling.md")
+    security_doc = read_doc("docs/security.md")
+    operators_doc = read_doc("docs/operators.md")
+    combined = f"{scaling_doc}\n{security_doc}\n{operators_doc}"
+
+    for expected in (
+        "`gpu_count=None` or an omitted `gpu_count` means CPU-only",
+        "A positive integer requests that many GPUs",
+        "Platform owns `gpu_resource_name`",
+        "PRISM does not pass `gpu_resource_name`",
+        "resources.limits['nvidia.com/gpu']",
+        "Device IDs are not Kubernetes placement semantics",
+        "device IDs remain metadata",
+        "torchrun --standalone --nnodes=1 --nproc-per-node=1",
+        "does not claim multi-node support",
+        "Network isolation depends on the cluster CNI",
+    ):
+        assert expected in combined
+
+    forbidden_claims = (
+        "device IDs are Kubernetes placement semantics",
+        "multi-node distributed training is supported",
+        "supports arbitrary TPU",
+        "supports AMD accelerator abstraction",
+    )
+    combined_lower = combined.lower()
+    for phrase in forbidden_claims:
+        assert phrase.lower() not in combined_lower
+
+
+def test_checkpoint_docs_do_not_overpromise_unsupported_scope() -> None:
+    public_docs = [
+        "README.md",
+        "docs/submissions.md",
+        "docs/miner/README.md",
+        "docs/scaling.md",
+        "docs/security.md",
+    ]
+    combined = "\n".join(read_doc(path) for path in public_docs)
+    combined_lower = combined.lower()
+
+    for expected in (
+        "PRISM does not support arbitrary external checkpoint resume",
+        "External checkpoint paths and miner-selected resume sources are not supported",
+        "Miners cannot select arbitrary external checkpoint paths or resume sources",
+        "does not support object-store or cloud checkpoint upload",
+        "does not support multi-node distributed training",
+    ):
+        assert expected in combined
+
+    forbidden_phrases = (
+        ".omo",
+        "Prometheus",
+        "Metis",
+        "workflow artifacts",
+        "arbitrary external checkpoint resume is supported",
+        "miner-selected external checkpoint paths are supported",
+        "object-store checkpoint upload is supported",
+        "cloud checkpoint upload is supported",
+        "multi-node distributed training is supported",
+    )
+    for phrase in forbidden_phrases:
+        assert phrase.lower() not in combined_lower
+
 def test_security_doc_clarifies_metric_review_limits() -> None:
     security_doc = read_doc("docs/security.md")