feat(deployments): reconcile controller and prerequisite DAG (AIRCORE-758) (#315)

* feat(deployments): add reconcile controller and prerequisite DAG (#758)

Introduce DeploymentsController with deployment/volume reconcilers,
prerequisite gating, drift recovery, and orphan cleanup on top of the
755 plugin scaffold. Stacks on PR #280 (AIRCORE-755).

AIRCORE-758

Signed-off-by: Tyler Bray <tbray@nvidia.com>

* fix(deployments): address remaining PR #315 CodeRabbit review items

Re-raise NemoEntityConflictError in deployment reconciler create/drift/save
paths, gate Docker integration tests on BACKEND_CLASSES, document split
list-health signals in README, and add ge=0 validation on DriftRecoveryPolicy.

AIRCORE-758

Signed-off-by: Tyler Bray <tbray@nvidia.com>

* chore(deployments): regenerate plugin OpenAPI spec for drift recovery fields

The lint-web-sdk check failed because plugins/nemo-deployments/openapi/openapi.yaml
was out of sync with the updated DriftRecoveryPolicy and Prerequisite models.

Signed-off-by: Tyler Bray <tbray@nvidia.com>

* fix(deployments): address PR #315 review feedback from benmccown and CodeRabbit

Retry backend delete failures in DELETING state, rename drift/orphan config
fields, rename listing helpers to entity_client, trim README to Controller
section, and apply remaining review nits (docstrings, RuntimeError guards,
substrate wording, test coverage).

AIRCORE-758

Signed-off-by: Tyler Bray <tbray@nvidia.com>

---------

Signed-off-by: Tyler Bray <tbray@nvidia.com>

Author: tylersbray

plugins/nemo-deployments/README.md

@@ -1,11 +1,24 @@
 # NeMo Deployments Plugin
 
-Substrate-agnostic deployment lifecycle for the NeMo Platform: entity schemas,
-CRUD APIs, a `DeploymentBackend` ABC, and an executor registry.
+Backend-agnostic deployment lifecycle for the NeMo Platform: entity schemas,
+CRUD APIs, a `DeploymentBackend` ABC, an executor registry, and a background
+reconcile controller (`DeploymentsController`).
 
-## Tests
+## Controller
 
-```bash
-uv sync
-uv run pytest plugins/nemo-deployments/tests/unit -v
-```
+Register `DeploymentsController` via the `nemo.controllers` entry point. The controller
+paginates non-terminal deployment/volume lists, reconciles volumes before deployments,
+gates deployment create on mounted volumes reaching `BOUND`, and writes status via the
+entity client (including endpoints and status history). Orphan backend resource cleanup
+runs on a configurable interval and is skipped when the deployment list is unhealthy.
+
+The controller exposes `is_healthy`, which is `False` when either the deployment-list or
+volume-list query fails. Internally these are tracked separately so operators can tell
+which list query failed without losing that signal behind a single boolean.
+
+Per-config drift backoff overrides live on `DeploymentConfig.driftRecovery`; unset fields
+fall back to `DeploymentsConfig.controller`.
+
+Prerequisites are currently declared on `DeploymentConfig` and resolve to a single
+deployment per config name in a workspace. Multiple deployments sharing one config is
+unsupported until prerequisites move to the `Deployment` entity (follow-up work).

plugins/nemo-deployments/openapi/openapi.yaml

@@ -14,6 +14,9 @@ dependencies = [
 [project.entry-points."nemo.services"]
 deployments = "nemo_deployments_plugin.service:DeploymentsService"
 
+[project.entry-points."nemo.controllers"]
+deployments = "nemo_deployments_plugin.controller:DeploymentsController"
+
 [build-system]
 requires = ["hatchling"]
 build-backend = "hatchling.build"

plugins/nemo-deployments/pyproject.toml

@@ -8,7 +8,7 @@
 from typing import Any, ClassVar
 
 from nemo_platform_plugin.config import NemoConfig
-from pydantic import BaseModel, Field
+from pydantic import BaseModel, Field, model_validator
 
 
 class ExecutorConfigEntry(BaseModel):
@@ -17,6 +17,30 @@ class ExecutorConfigEntry(BaseModel):
     config: dict[str, Any] = Field(default_factory=dict)
 
 
+class ControllerConfig(BaseModel):
+    """Configuration for the deployments reconcile controller."""
+
+    interval_seconds: int = Field(default=5, gt=0, description="Reconciliation loop interval in seconds.")
+    drift_recovery_max_attempts: int = Field(default=5, ge=0, description="Max drift recovery attempts before FAILED.")
+    drift_recovery_initial_delay_seconds: int = Field(
+        default=5, ge=0, description="Initial delay for drift recovery backoff."
+    )
+    drift_recovery_max_delay_seconds: int = Field(
+        default=300, ge=0, description="Max delay cap for drift recovery backoff."
+    )
+    orphan_cleanup_interval_seconds: int = Field(
+        default=30,
+        ge=0,
+        description="Run orphaned backend resource cleanup after this many seconds (0 disables).",
+    )
+
+    @model_validator(mode="after")
+    def _validate_backoff(self) -> ControllerConfig:
+        if self.drift_recovery_initial_delay_seconds > self.drift_recovery_max_delay_seconds:
+            raise ValueError("drift_recovery_initial_delay_seconds must not exceed drift_recovery_max_delay_seconds")
+        return self
+
+
 class DeploymentsConfig(NemoConfig):
     plugin_name: ClassVar[str] = "deployments"
     plugin_description: ClassVar[str] = "Configuration for the NeMo Platform deployments plugin."
@@ -27,5 +51,6 @@ class DeploymentsConfig(NemoConfig):
     )
     default_executor: str | None = Field(
         default=None,
-        description="Fallback executor when Deployment.executor is unset.",
+        description="Default executor when Deployment.executor is unset.",
     )
+    controller: ControllerConfig = Field(default_factory=ControllerConfig)

plugins/nemo-deployments/src/nemo_deployments_plugin/config.py

@@ -4,7 +4,7 @@
 """Shared constants for the deployments plugin."""
 
 MANAGED_BY_LABEL = "nemo-deployments"
-"""Label value backends use to tag substrate resources for orphan cleanup."""
+"""Label value backends use to tag backend resources for orphan cleanup."""
 
 ENTITY_TYPE_DEPLOYMENT_CONFIG = "deployments_deployment_config"
 ENTITY_TYPE_DEPLOYMENT = "deployments_deployment"