[Allocator 3/5 — after #71] Qualify physical engine lifecycles with real inference

README.md

@@ -622,6 +622,22 @@ grid --local allocator status --grid allocator-control
 grid --local allocator mode automatic --grid allocator-control
 ```
 
+Before granting a new physical runtime production lifecycle authority, qualify it on that machine
+with actual inference. The command writes an owner-only evidence report and always attempts to stop
+what it warmed; it never deletes a pre-existing artifact:
+
+```bash
+grid --local allocator qualify ollama <model> --artifact-sha256 <digest>
+grid --local allocator qualify comfyui comfyui:image_generation \
+  --endpoint http://127.0.0.1:8188
+grid --local allocator qualify vllm <served-model> \
+  --artifact-source hf://owner/repo@<commit> \
+  --artifact-sha256 <snapshot-identity> --artifact-size-mb <bound>
+```
+
+See [physical runtime qualification](docs/allocator-runtime-qualification.md) for canary cleanup,
+failure interpretation, and the evidence captured on Forge.
+
 Allocator enrollment verifies the managed llama.cpp runtime and installs Grid's version- and
 SHA-256-pinned build when it is absent. It also waits briefly for a just-started provider identity
 to become visible at the relay, so the two fresh-node commands above are safe to run back to back.

cli/allocator_qualification.py

@@ -0,0 +1,161 @@
+"""Physical runtime lifecycle qualification command."""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import time
+from pathlib import Path
+from typing import Any
+
+import httpx
+
+from shared import paths
+from shared.allocator.orchestrator import ComfyUIBackend, OllamaBackend, VllmBackend
+from shared.allocator.qualification import qualify_runtime, save_report
+from shared.media.media_handler import MediaHandler
+
+
+def cmd_allocator_qualify(args: argparse.Namespace) -> int:
+    if not 1 <= args.port <= 65_535:
+        raise SystemExit("--port must be in [1, 65535]")
+    if args.timeout <= 0 or args.artifact_size_mb < 0:
+        raise SystemExit("--timeout must be positive and --artifact-size-mb must be non-negative")
+    if args.tensor_parallel_size < 1 or args.max_tokens < 1:
+        raise SystemExit("tensor parallel size and max tokens must be positive")
+    if not 0.0 < args.gpu_memory_utilization <= 1.0:
+        raise SystemExit("--gpu-memory-utilization must be in (0, 1]")
+    if args.max_model_len < 0:
+        raise SystemExit("--max-model-len must be non-negative")
+    if args.image_size < 64 or args.steps < 1:
+        raise SystemExit("ComfyUI image size must be at least 64 and steps must be positive")
+    backend = _backend(args)
+    report = qualify_runtime(
+        runtime=args.runtime,
+        backend=backend,
+        model_id=args.model,
+        port=args.port,
+        inference=_inference(args),
+        artifact_source=args.artifact_source,
+        artifact_sha256=args.artifact_sha256,
+        artifact_size_mb=args.artifact_size_mb,
+        cleanup_artifact=args.cleanup_artifact,
+    )
+    report_path = _report_path(args, report.started_at)
+    save_report(report_path, report)
+    if args.json:
+        print(json.dumps({**report.to_dict(), "report_path": str(report_path)}, indent=2))
+    else:
+        verdict = "PASS" if report.passed else "FAIL"
+        print(f"Runtime qualification {verdict}: {report.runtime} · {report.model_id}")
+        for item in report.steps:
+            marker = "✓" if item.passed else "✗"
+            print(f"  {marker} {item.name:<20} {item.duration_seconds:7.2f}s  {item.detail}")
+        print(f"  report  {report_path}")
+    return 0 if report.passed else 1
+
+
+def _backend(args: argparse.Namespace) -> Any:
+    endpoint = _endpoint(args)
+    if args.runtime == "ollama":
+        return OllamaBackend(endpoint, timeout=args.timeout)
+    if args.runtime == "comfyui":
+        bundle = args.model.removeprefix("comfyui:")
+        return ComfyUIBackend(endpoint, bundles=(bundle,))
+    cache = (
+        Path(args.cache_dir).expanduser()
+        if args.cache_dir
+        else paths.grid_home() / "allocator-qualification" / "vllm"
+    )
+    return VllmBackend(
+        cache,
+        tensor_parallel_size=args.tensor_parallel_size,
+        gpu_memory_utilization=args.gpu_memory_utilization,
+        max_model_len=args.max_model_len,
+        enforce_eager=args.enforce_eager,
+        use_flashinfer_sampler=(False if args.disable_flashinfer_sampler else None),
+        readiness_timeout=args.timeout,
+    )
+
+
+def _inference(args: argparse.Namespace):
+    endpoint = _endpoint(args)
+    if args.runtime == "ollama":
+
+        def ollama(_handle):
+            with httpx.Client(timeout=args.timeout, trust_env=False) as client:
+                response = client.post(
+                    f"{endpoint}/api/generate",
+                    json={
+                        "model": args.model,
+                        "prompt": args.prompt,
+                        "stream": False,
+                        "keep_alive": -1,
+                    },
+                )
+            response.raise_for_status()
+            text = str(response.json().get("response") or "")
+            return len(text), "Ollama /api/generate"
+
+        return ollama
+    if args.runtime == "vllm":
+
+        def vllm(handle):
+            with httpx.Client(timeout=args.timeout, trust_env=False) as client:
+                response = client.post(
+                    f"http://127.0.0.1:{handle.port}/v1/chat/completions",
+                    json={
+                        "model": args.model,
+                        "messages": [{"role": "user", "content": args.prompt}],
+                        "max_tokens": args.max_tokens,
+                    },
+                )
+            response.raise_for_status()
+            text = str(response.json()["choices"][0]["message"]["content"])
+            return len(text), "vLLM OpenAI-compatible chat"
+
+        return vllm
+
+    def comfyui(_handle):
+        if args.model not in (
+            "comfyui:image_generation",
+            "comfyui:krea2",
+            "comfyui:z_image",
+        ):
+            raise RuntimeError(
+                "physical ComfyUI qualification currently requires an image-generation bundle"
+            )
+        handler = MediaHandler(comfyui_url=endpoint)
+        output_units = 0
+        for line in handler.handle_request(
+            "media/image/generate",
+            {
+                "model": args.model,
+                "prompt": args.prompt,
+                "width": args.image_size,
+                "height": args.image_size,
+                "steps": args.steps,
+            },
+        ):
+            terminal = str(line)
+            if '"error"' in terminal:
+                raise RuntimeError(terminal[:500])
+            output_units += len(terminal)
+        return output_units, "ComfyUI completed workflow output"
+
+    return comfyui
+
+
+def _endpoint(args: argparse.Namespace) -> str:
+    if args.endpoint:
+        return str(args.endpoint).rstrip("/")
+    return "http://127.0.0.1:11434" if args.runtime == "ollama" else "http://127.0.0.1:8188"
+
+
+def _report_path(args: argparse.Namespace, started_at: float) -> Path:
+    if args.report:
+        return Path(args.report).expanduser()
+    scope = hashlib.sha256(f"{args.runtime}\0{args.model}".encode()).hexdigest()[:16]
+    stamp = time.strftime("%Y%m%d-%H%M%S", time.gmtime(started_at))
+    return paths.grid_home() / "allocator-qualification" / "reports" / f"{stamp}-{scope}.json"

cli/parser.py

@@ -30,6 +30,7 @@
     cmd_allocator_tick,
     cmd_allocator_token_write,
 )
+from .allocator_qualification import cmd_allocator_qualify
 from shared.allocator.scenario import SCENARIO_STRATEGIES
 
 from .allocator_scenario import (
@@ -623,6 +624,55 @@ def _add_allocator(sub) -> None:
     tick.add_argument("--json", action="store_true", help="Emit machine-readable JSON.")
     tick.set_defaults(handler=cmd_allocator_tick)
 
+    qualify = allocator_sub.add_parser(
+        "qualify",
+        help="Prove a physical engine's full managed lifecycle with real inference",
+    )
+    qualify.add_argument("runtime", choices=("ollama", "comfyui", "vllm"))
+    qualify.add_argument("model")
+    qualify.add_argument(
+        "--endpoint",
+        default="",
+        help="Native engine base URL (defaults by runtime).",
+    )
+    qualify.add_argument("--artifact-source", default="")
+    qualify.add_argument("--artifact-sha256", default="")
+    qualify.add_argument("--artifact-size-mb", type=int, default=0)
+    qualify.add_argument("--port", type=int, default=28901)
+    qualify.add_argument("--tensor-parallel-size", type=int, default=1)
+    qualify.add_argument(
+        "--gpu-memory-utilization",
+        type=float,
+        default=0.90,
+        help="Fraction of each visible GPU vLLM may reserve (default: 0.90).",
+    )
+    qualify.add_argument(
+        "--max-model-len",
+        type=int,
+        default=0,
+        help="Override vLLM context length; zero keeps the model default.",
+    )
+    qualify.add_argument(
+        "--enforce-eager",
+        action="store_true",
+        help="Disable CUDA graph compilation for a lower-impact shared-node qualification.",
+    )
+    qualify.add_argument(
+        "--disable-flashinfer-sampler",
+        action="store_true",
+        help="Use vLLM's native sampler when FlashInfer JIT is unavailable or incompatible.",
+    )
+    qualify.add_argument("--cache-dir", default=None)
+    qualify.add_argument("--prompt", default="Reply with exactly GRID.")
+    qualify.add_argument("--max-tokens", type=int, default=32)
+    qualify.add_argument("--image-size", type=int, default=256)
+    qualify.add_argument("--steps", type=int, default=1)
+    qualify.add_argument("--timeout", type=float, default=900.0)
+    qualify.add_argument("--cleanup-artifact", action="store_true")
+    qualify.add_argument("--report", default=None)
+    qualify.add_argument("--json", action="store_true")
+    qualify.set_defaults(handler=cmd_allocator_qualify)
+
     token = allocator_sub.add_parser("token", help="Provision the node control capability")
     token_sub = token.add_subparsers(dest="allocator_token_command", required=True)
     token_write = token_sub.add_parser(

docs/allocator-runtime-qualification.md

@@ -0,0 +1,85 @@
+# Physical runtime qualification
+
+Grid does not treat adapter unit tests or an open TCP port as proof that it can manage an engine on
+real hardware. `grid --local allocator qualify` executes the same native lifecycle boundary used by
+the allocator and writes an owner-only JSON report below
+`~/.grid/allocator-qualification/reports/`.
+
+Every successful report proves, in order:
+
+1. Native cached-model inventory and exact artifact identity.
+2. Artifact fetch with a digest and size bound when it was not already cached.
+3. Warm/load through the engine's native API.
+4. Process/model ownership and native readiness.
+5. A real generated response or completed ComfyUI workflow.
+6. The runtime activity probe used during drain.
+7. Native drain/stop or model-memory unload.
+
+Teardown is attempted even after a failed inference. `--cleanup-artifact` removes only an artifact
+fetched by that same qualification run; a model already present at the inventory step is never
+deleted. Without that flag all artifacts remain cached.
+
+## Commands
+
+Ollama uses its registry digest and does not stop the shared daemon:
+
+```bash
+grid --local allocator qualify ollama smollm2:135m \
+  --artifact-sha256 <digest> --timeout 300
+```
+
+ComfyUI qualification requires an installed image-generation bundle. It submits the real bundled
+workflow at 256×256 and one step by default, waits for an output, checks the queue, then calls the
+native `/free` memory-unload API:
+
+```bash
+grid --local allocator qualify comfyui comfyui:image_generation \
+  --endpoint http://127.0.0.1:8188 --timeout 900
+```
+
+vLLM qualification pins a full Hugging Face commit. Its `artifact_sha256` is the allocator's
+deterministic identity for that exact repository snapshot, not a mutable branch name:
+
+```bash
+grid --local allocator qualify vllm Qwen/example \
+  --artifact-source hf://Qwen/example@<40-hex-commit> \
+  --artifact-sha256 <snapshot-identity> \
+  --artifact-size-mb <maximum-download-mb> \
+  --tensor-parallel-size 2 --timeout 1800
+```
+
+On a shared GPU, bound the canary and shorten its compilation window with
+`--gpu-memory-utilization`, `--max-model-len`, and `--enforce-eager`. If the installed vLLM and
+FlashInfer wheels have incompatible JIT toolchains, `--disable-flashinfer-sampler` selects vLLM's
+native sampler. Grid activates sibling vLLM build tools and a wheel-packaged CUDA compiler when the
+host has only an NVIDIA driver; a source-build path still requires Python development headers.
+
+The command downloads into Grid's isolated qualification cache, starts a Grid-owned vLLM child,
+proves it through `/v1/models`, runs an OpenAI-compatible chat completion, and reaps the child.
+
+## Forge evidence, 2026-09-02
+
+Machine A physically passed the full Ollama lifecycle with `smollm2:135m`: exact digest inventory,
+warm, ownership, readiness, a real `/api/generate` response, activity probe, and unload. The warm
+took 0.81 seconds and inference 2.13 seconds after the artifact was cached. The disposable 270 MB
+canary was removed after the report; only the pre-existing model remained.
+
+The same run found a model-specific failure for the pre-existing `gpt-oss:20b`: Ollama returned
+`tensor "blk.0.ffn_down_exps.weight" size overflow` during warm. Grid correctly withheld readiness
+and did not attempt inference or delete the artifact. That is a failed artifact/runtime
+qualification, not evidence that the Ollama lifecycle adapter is healthy for that model.
+
+Machine D physically passed vLLM 0.28 with Qwen2.5-Coder-0.5B-Instruct at an immutable commit. The
+run used 35% GPU utilization, a 2,048-token context, eager mode, and the native sampler. It fetched
+the bounded snapshot, proved process ownership and `/v1/models` readiness, returned `GRID` through
+the OpenAI-compatible chat API, observed zero active requests, stopped the child, and cleaned the
+canary artifact. The run also established that fresh Ubuntu needs Python development headers for
+Triton.
+
+Machine A physically passed ComfyUI 0.34.0 with the installed Z-Image bundle on MPS. The run proved
+inventory and artifact identity, loaded the text encoder, diffusion model, and VAE, completed a real
+256-by-256 image workflow, collected the exact output from ComfyUI history, observed an empty queue,
+and released model memory through `/free`.
+
+A report is not transferable between machines: the engine version, accelerator, driver, and
+artifacts are part of what the physical run is testing.

docs/allocator.md

@@ -889,6 +889,20 @@ grid --remote models <grid>
 grid --remote chat -m <model.gguf> "hello"
 ```
 
+Before enabling a newly installed engine adapter on a physical provider, run its real lifecycle
+qualification locally on that host. It validates inventory, immutable identity, warm, ownership,
+native readiness, real inference, activity, and drain/stop, then writes durable evidence:
+
+```bash
+grid --local allocator qualify ollama <model> --artifact-sha256 <digest>
+grid --local allocator qualify comfyui comfyui:image_generation
+grid --local allocator qualify vllm <model> \
+  --artifact-source hf://owner/repo@<commit> \
+  --artifact-sha256 <snapshot-identity> --artifact-size-mb <bound>
+```
+
+See [physical runtime qualification](allocator-runtime-qualification.md) for exact behavior.
+
 On a remote provider enrolled with `--dedicated`, the node uses Grid's multi-engine lifecycle
 orchestrator. It manages installed `llama.cpp`, Ollama, ComfyUI, and vLLM runtimes through one
 `LOAD → WARM → READY → DRAIN → UNLOAD` contract while the existing provider remains the only relay
@@ -1396,20 +1410,20 @@ The design follows several primary systems results while preserving Grid's alloc
 - Remote allocator nodes require a relay with the authenticated allocator sidecar and enrollment
   bridge enabled. Policy administration remains controller-only; remote providers may enroll only
   their own already-live identity.
-- The first managed process boundary is Grid-owned llama.cpp model runtimes. ComfyUI, external
-  Ollama/vLLM/LM Studio, API, and manually started engines are inventory and routing sources, not
-  processes the allocator may stop. The mixed-framework logical fixture starts and stops its own
-  ComfyUI process as test-fixture setup/cleanup; allocator actions do not masquerade as ComfyUI
-  model lifecycle mutations.
+- Dedicated allocator providers have lifecycle adapters for Grid-owned llama.cpp and vLLM children,
+  plus model-memory control for loopback Ollama and ComfyUI services. External/manual processes
+  remain inventory and routing sources until explicitly enrolled; lifecycle authority never follows
+  from protocol detection alone. LM Studio and generic API engines remain routing-only.
 - The current autonomous.ai NVIDIA engines are vLLM/CUDA even though live discovery labels their
   ownership class `external`. Framework identity and lifecycle ownership are independent: those
   engines participate in routing and placement evidence, but discovery alone does not grant Grid
   permission to start, drain, or stop them. Local auto-discovery publishes the detected runtime;
   when pointing at an engine explicitly, use `grid join --at <url> -m <model> --kind vllm` (or the
   corresponding kind) so runtime-constrained profiles can use the inventory.
-- Managed llama.cpp can autonomously fetch an exact, size-bounded, SHA-256-pinned Hugging Face GGUF.
-  ComfyUI bundles and externally owned vLLM artifacts still require their runtime-specific install
-  paths; the generic action fields are present, but those lifecycle adapters are not yet claimed.
+- Managed llama.cpp and vLLM can autonomously fetch exact, size-bounded immutable artifacts. Ollama
+  verifies the native registry digest and refuses to overwrite a pre-existing tag with another
+  digest. ComfyUI workflow assets still require their runtime-specific installation path and are
+  unloaded from accelerator memory rather than deleted.
 - Capacity is refreshed by the node as stable physical capacity plus dynamic non-Grid reserve.
   Device count and per-device VRAM are preserved, and profiles may fail closed with
   `min_gpu_count` and `min_gpu_memory_mb` constraints. This covers basic tensor-parallel