feat: FournosCluster controller with GPU discovery, sentinel locking, and Kueue integration

README.md

@@ -293,6 +293,347 @@ Copied secrets are named `<fjob-name>-<secret-name>` and carry
 `ownerReferences` back to the FournosJob, so Kubernetes garbage-collects
 them automatically when the job is deleted.
 
+## PSAPCluster — Cluster Management
+
+The `PSAPCluster` custom resource provides a single pane of glass for cluster
+state, GPU inventory, and ownership locking.
+
+### Viewing clusters
+
+```bash
+oc get psapclusters -n psap-automation
+```
+
+```
+NAME          OWNER   GPUS      KUBECONFIG    LOCKED   AGE
+athena-fire           8x H200   Valid         false    3h
+psap-mgmt                       Valid         false    3h
+```
+
+### Onboarding a cluster via PSAPCluster
+
+1. Create a kubeconfig Secret in the secrets namespace (default `psap-secrets`):
+
+```bash
+oc create secret generic kubeconfig-<cluster-name> \
+  --from-file=kubeconfig=/path/to/kubeconfig \
+  -n psap-secrets
+```
+
+2. Add a ResourceFlavor and quota for the cluster in
+   `config/kueue-cluster-config.yaml` (see [Onboarding a new cluster](#onboarding-a-new-cluster)).
+
+3. Create a `PSAPCluster` resource:
+
+```yaml
+apiVersion: fournos.dev/v1
+kind: PSAPCluster
+metadata:
+  name: my-cluster
+spec:
+  kubeconfigSecret: kubeconfig-my-cluster
+```
+
+```bash
+oc apply -f my-cluster.yaml -n psap-automation
+```
+
+The controller automatically:
+- Validates the kubeconfig secret
+- Discovers GPUs on the target cluster and updates the global `fournos-queue`
+  ClusterQueue quotas for this cluster's ResourceFlavor
+- Self-heals the lock if the sentinel job is deleted externally
+
+### How cluster locking works
+
+Locking uses the same mechanism as exclusive FournosJobs: **cluster-slot quota**.
+Each cluster has 100 `fournos/cluster-slot` quota in the global `fournos-queue`
+ClusterQueue. An exclusive job requests all 100 slots, blocking any other job
+from being scheduled on that cluster.
+
+When you set `spec.owner` on a PSAPCluster, the controller creates a **sentinel
+FournosJob** — a lightweight job with `lockOnly: true` and `exclusive: true`
+that requests all 100 cluster-slots without running any pipeline. This holds
+the cluster's quota, preventing other jobs from being admitted.
+
+When you clear `spec.owner`, the controller deletes the sentinel job, freeing
+the cluster-slots. Any pending jobs are then eligible for admission by Kueue.
+
+### Locking a cluster
+
+```bash
+# Lock with a 4-hour TTL (auto-expires after 4 hours)
+oc patch psapcluster athena-fire -n psap-automation --type merge \
+  -p '{"spec":{"owner":"userA","ttl":"4h"}}'
+
+# Lock indefinitely (must be manually unlocked)
+oc patch psapcluster athena-fire -n psap-automation --type merge \
+  -p '{"spec":{"owner":"userA"}}'
+
+# Unlock (deletes the sentinel, pending jobs proceed)
+oc patch psapcluster athena-fire -n psap-automation --type merge \
+  -p '{"spec":{"owner":""}}'
+```
+
+#### TTL format
+
+| Format | Example | Duration |
+|--------|---------|----------|
+| `Nm`   | `30m`   | 30 minutes |
+| `Nh`   | `4h`    | 4 hours |
+| `Nd`   | `2d`    | 2 days |
+
+If `ttl` is omitted, the lock does not expire and must be cleared manually.
+
+### Common scenarios
+
+#### Scenario 1: UserA needs the cluster for manual work
+
+UserA is running manual experiments on `athena-fire` and wants to make sure
+no automated jobs interfere.
+
+```bash
+oc patch psapcluster athena-fire -n psap-automation --type merge \
+  -p '{"spec":{"owner":"userA","ttl":"4h"}}'
+```
+
+**What happens:**
+1. The controller creates a sentinel FournosJob (`psapcluster-lock-athena-fire`)
+   that holds all 100 cluster-slots on `athena-fire`.
+2. Any new jobs targeting `athena-fire` queue up with the message:
+   *"Cluster athena-fire is exclusively locked by psapcluster-lock-athena-fire,
+   waiting for it to finish"*
+3. UserA does his manual work.
+4. After 4 hours (or when UserA unlocks early), the sentinel is deleted and
+   queued jobs automatically proceed.
+
+#### Scenario 2: Locking while a job is already running
+
+UserB's benchmark job is mid-run on `athena-fire` when UserA locks the cluster.
+
+```bash
+oc patch psapcluster athena-fire -n psap-automation --type merge \
+  -p '{"spec":{"owner":"userA","ttl":"2h"}}'
+```
+
+**What happens:**
+1. The sentinel FournosJob is created and goes to **Pending** — it cannot be
+   admitted because UserB's job already holds cluster-slots.
+2. UserB's job **continues running uninterrupted** until it completes normally.
+3. Once UserB's job finishes and releases its cluster-slots, Kueue admits the
+   sentinel. UserA now has the lock.
+4. Any jobs submitted after step 1 queue behind the sentinel.
+
+There is no preemption — running jobs always finish. The lock takes effect
+after currently running work completes.
+
+#### Scenario 3: UserA finishes early
+
+UserA locked the cluster for 4 hours but finished after 1 hour.
+
+```bash
+oc patch psapcluster athena-fire -n psap-automation --type merge \
+  -p '{"spec":{"owner":""}}'
+```
+
+**What happens:**
+1. The controller deletes the sentinel FournosJob.
+2. Kueue sees 100 cluster-slots freed on `athena-fire`.
+3. Any pending jobs are immediately eligible for admission — no need to wait
+   for the TTL to expire, no Slack ping at 10:30pm.
+
+#### Scenario 4: UserA forgets to unlock
+
+UserA locked the cluster with `ttl: 4h` and left for the day.
+
+**What happens:**
+1. After 4 hours, the reconciler detects the TTL has expired.
+2. It automatically clears `spec.owner` and deletes the sentinel FournosJob.
+3. Pending jobs proceed as if UserA had unlocked manually.
+
+#### Scenario 5: Submitting a job while a cluster is locked
+
+UserC submits a FournosJob targeting `athena-fire` while UserA has it locked.
+
+```bash
+oc create -f my-job.yaml -n psap-automation
+```
+
+**What happens:**
+1. The job goes through the normal lifecycle: Resolving (Forge) → Pending.
+2. During the Pending phase, Kueue cannot admit it because the sentinel holds
+   all cluster-slots.
+3. The job status shows: *"Cluster athena-fire is exclusively locked by
+   psapcluster-lock-athena-fire, waiting for it to finish"*
+4. When UserA unlocks (or TTL expires), the sentinel is deleted and UserC's
+   job is admitted automatically.
+
+The job's Forge resolution happens normally while it waits — only Kueue
+admission is blocked by the lock.
+
+### PSAPCluster spec fields
+
+| Field | Required | Default | Description |
+|-------|----------|---------|-------------|
+| `kubeconfigSecret` | yes | — | Name of the K8s Secret holding the target cluster kubeconfig |
+| `owner` | no | — | Person or team claiming exclusive use. Setting this creates a sentinel FournosJob that locks the cluster |
+| `ttl` | no | — | Auto-expiry duration (e.g. `4h`, `30m`, `2d`). Lock persists indefinitely if omitted |
+| `gpuDiscoveryInterval` | no | `5m` | How often to probe the target cluster for GPU hardware |
+
+### PSAPCluster status fields
+
+| Field | Description |
+|-------|-------------|
+| `kubeconfigStatus` | `Valid`, `Missing`, `Invalid`, or `Unreachable` |
+| `locked` | Whether the cluster is currently locked |
+| `lockExpiresAt` | When the lock auto-expires (null if no TTL) |
+| `lockJobName` | Name of the sentinel FournosJob holding cluster quota while locked |
+| `gpuSummary` | Human-readable GPU summary (e.g. `8x H200`) |
+| `hardware.gpus` | Array of `{vendor, model, shortName, count, nodeCount}` |
+| `hardware.totalGPUs` | Total GPU count across all types |
+| `hardware.lastDiscovery` | Timestamp of last successful GPU discovery |
+| `hardware.consecutiveFailures` | Number of consecutive discovery failures |
+| `conditions` | `KubeconfigValid`, `GPUDiscovered` |
+
+### Troubleshooting PSAPCluster
+
+**Cluster shows `Unreachable`:**
+The controller failed to connect to the target cluster 5 or more times
+consecutively. The kubeconfig reconciler will automatically reset the status
+to `Valid` once the secret is accessible, and GPU discovery will retry on the
+next cycle. Check:
+- Is the target cluster up? Try `oc --kubeconfig=<path> cluster-info`
+- Is the kubeconfig secret valid? `oc get secret <name> -n psap-secrets -o yaml`
+- Check operator logs: `oc logs deployment/fournos -n psap-automation | grep <cluster-name>`
+
+**Cluster shows `Missing` kubeconfig:**
+The kubeconfig secret referenced by `spec.kubeconfigSecret` does not exist
+in the secrets namespace. Create it with:
+```bash
+oc create secret generic kubeconfig-<name> \
+  --from-file=kubeconfig=/path/to/kubeconfig -n psap-secrets
+```
+
+**Sentinel FournosJob stuck in Pending:**
+This means another job is currently running on the cluster. The sentinel
+queues behind it. Check what's running:
+```bash
+oc get fournosjobs -n psap-automation -l fournos.dev/exclusive-cluster=<cluster-name>
+```
+The lock takes effect once the running job finishes.
+
+**Sentinel FournosJob deleted externally:**
+The self-healing reconciler (runs every 30s) detects the missing sentinel and
+recreates it automatically if `spec.owner` is still set. No manual action needed.
+
+**Lock not expiring:**
+- Verify `ttl` is set: `oc get psapcluster <name> -n psap-automation -o jsonpath='{.spec.ttl}'`
+- Check `lockExpiresAt`: `oc get psapcluster <name> -n psap-automation -o jsonpath='{.status.lockExpiresAt}'`
+- Check operator logs for errors in the TTL reconciler
+
+**GPU count shows 0 or is missing:**
+- The target cluster may not have GPU nodes, or GPU device plugins are not installed
+- Check node labels: `oc get nodes --show-labels | grep gpu` on the target cluster
+- Verify the kubeconfig has permission to list nodes
+
+**Want to see the sentinel job details:**
+```bash
+oc get fournosjob psapcluster-lock-<cluster-name> -n psap-automation -o yaml
+```
+
+### PSAPCluster configuration
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `FOURNOS_PSAPCLUSTER_TIMER_INTERVAL_SEC` | `30` | Reconciliation timer interval |
+| `FOURNOS_GPU_DISCOVERY_DEFAULT_INTERVAL_SEC` | `300` | Default GPU discovery interval |
+| `FOURNOS_GPU_DISCOVERY_TIMEOUT_SEC` | `10` | Timeout for connecting to target clusters |
+| `FOURNOS_CLUSTER_DISCOVERY_INTERVAL_SEC` | `60` | Interval for auto-discovery scan of kubeconfig secrets |
+
+### Testing cluster locking end-to-end
+
+This walkthrough verifies that the sentinel FournosJob mechanism correctly
+blocks and unblocks Kueue workloads on a locked cluster.
+
+**1. Lock the cluster and verify the sentinel:**
+
+```bash
+# Lock
+oc patch psapcluster athena-fire -n psap-automation --type merge \
+  -p '{"spec":{"owner":"mehul"}}'
+
+# Confirm sentinel job was created and admitted
+oc get fournosjobs -n psap-automation | grep psapcluster-lock
+# Expected:
+#   psapcluster-lock-athena-fire   mehul   Admitted   athena-fire   Cluster lock held on athena-fire
+
+# Confirm Kueue workload is admitted, holding all 100 cluster-slots
+oc get workloads -n psap-automation
+# Expected:
+#   psapcluster-lock-athena-fire   fournos-queue   fournos-queue   True
+```
+
+**2. Submit a competing workload and verify it is blocked:**
+
+```bash
+oc create -n psap-automation -f - <<'EOF'
+apiVersion: kueue.x-k8s.io/v1beta2
+kind: Workload
+metadata:
+  name: test-lock-workload
+  labels:
+    kueue.x-k8s.io/queue-name: fournos-queue
+spec:
+  queueName: fournos-queue
+  podSets:
+    - name: launcher
+      count: 1
+      template:
+        spec:
+          containers:
+            - name: placeholder
+              image: registry.k8s.io/pause:3.9
+              resources:
+                requests:
+                  fournos/cluster-slot: "100"
+          nodeSelector:
+            fournos.dev/cluster: athena-fire
+          restartPolicy: Never
+EOF
+
+# Verify test workload stays pending (not admitted)
+oc get workloads -n psap-automation
+# Expected:
+#   psapcluster-lock-athena-fire   fournos-queue   fournos-queue   True
+#   test-lock-workload             fournos-queue                          <-- no ADMITTED
+
+# Describe shows the reason: "insufficient unused quota for fournos/cluster-slot"
+oc describe workload test-lock-workload -n psap-automation
+```
+
+**3. Unlock the cluster and verify the competing workload is admitted:**
+
+```bash
+# Unlock
+oc patch psapcluster athena-fire -n psap-automation --type merge \
+  -p '{"spec":{"owner":""}}'
+
+# After a few seconds, sentinel is deleted and test workload gets admitted
+oc get workloads -n psap-automation
+# Expected:
+#   test-lock-workload   fournos-queue   fournos-queue   True
+
+# Sentinel job should be gone
+oc get fournosjobs -n psap-automation | grep psapcluster-lock
+# Expected: no results
+```
+
+**4. Clean up:**
+
+```bash
+oc delete workload test-lock-workload -n psap-automation
+```
+
 ## Configuration
 
 All settings are read from environment variables with the `FOURNOS_` prefix:

config/forge/samples/job-full.yaml

@@ -2,10 +2,11 @@ apiVersion: fournos.dev/v1
 kind: FournosJob
 metadata:
   generateName: forge-full-sample-
+  namespace: psap-automation
 spec:
   owner: perf-team
   displayName: forge-full-sample
-  cluster: cluster-1
+  cluster: athena-fire
   pipeline: forge-full
   executionEngine:
     forge:

dev/mock-psapcluster.yaml

@@ -0,0 +1,13 @@
+apiVersion: fournos.dev/v1
+kind: PSAPCluster
+metadata:
+  name: psap-mgmt
+spec:
+  kubeconfigSecret: kubeconfig-psap-mgmt
+---
+apiVersion: fournos.dev/v1
+kind: PSAPCluster
+metadata:
+  name: athena-fire
+spec:
+  kubeconfigSecret: kubeconfig-athena-fire

fournos-cluster/Containerfile

@@ -0,0 +1,16 @@
+FROM registry.access.redhat.com/ubi10/python-312-minimal:10.1
+
+WORKDIR /opt/app-root/src
+
+COPY pyproject.toml ./
+RUN mkdir -p fournos_cluster && touch fournos_cluster/__init__.py \
+    && pip install --no-cache-dir --upgrade pip \
+    && pip install --no-cache-dir . \
+    && rm -rf fournos_cluster
+
+COPY fournos_cluster/ fournos_cluster/
+RUN pip install --no-cache-dir --no-deps .
+
+USER 1001
+
+ENTRYPOINT ["python", "-m", "fournos_cluster", "--liveness=http://0.0.0.0:8080/healthz"]