Mount secrets via projected volume into Tekton pods

Fournos_Design_Document.md

@@ -55,7 +55,7 @@ Jobs are submitted as `FournosJob` custom resources ([manifests/crd.yaml](manife
 | `spec.displayName`           | no           | Human-readable job name (defaults to `metadata.name`)                                            |
 | `spec.pipeline`              | no           | Tekton Pipeline name (default: `fournos-full`)                                                   |
 | `spec.priority`              | no           | Kueue WorkloadPriorityClass name                                                                 |
-| `spec.secretRefs`            | no           | Vault-synced K8s Secret names (`vault-<entry>`) to mount into the pipeline. Populated by Forge during the Resolving phase. Each must be a K8s Secret with `fournos.dev/vault-entry=true`. |
+| `spec.secretRefs`            | no           | Vault-synced K8s Secret names (`vault-<entry>`) to mount into the pipeline. Populated by Forge during the Resolving phase. Each must be a K8s Secret with `fournos.dev/vault-entry=true` in `FOURNOS_SECRETS_NAMESPACE`. During the Admitted phase the operator copies them into the operator namespace and mounts them as a projected volume at `/var/run/secrets/fournos/<entry-name>/`. |
 | `spec.exclusive`             | no           | If `true`, locks the target cluster so no other FournosJob can run there. Requires `spec.cluster`. |
 | `spec.shutdown`              | no           | Shutdown action: `Stop` (graceful, runs finally tasks) or `Terminate` (immediate, skips finally tasks). Both wait for the PipelineRun to finish before releasing Kueue quota. |
 
@@ -192,7 +192,7 @@ sequenceDiagram
 1. **on_create**: Operator validates the spec (cluster exists if specified, `exclusive` requires `cluster`). If `spec.shutdown` is set (`Stop` or `Terminate`), immediately sets `phase=Stopped`. Otherwise sets `phase=Resolving`.
 2. **timer (Resolving)**: Launches a Forge resolve K8s Job that patches the FournosJob spec with `hardware` (if not user-provided) and `secretRefs`. Polls the Job for completion. On success, reads the FournosJob spec, validates hardware (GPU type checked against Kueue), validates `secretRefs` against Vault secrets, creates the Kueue Workload (exclusive jobs request all 100 `fournos/cluster-slot` units; normal jobs request 1), and sets `phase=Pending`. Failed resolve Jobs are preserved for debugging.
 3. **timer (Pending)**: Polls the Workload for Kueue admission. On admission, extracts the assigned cluster and sets `phase=Admitted`.
-4. **timer (Admitted)**: Reads `secretRefs` from the FournosJob spec, resolves the kubeconfig Secret, creates the Tekton PipelineRun with `ownerReferences` pointing at the FournosJob, sets `phase=Running`.
+4. **timer (Admitted)**: Reads `secretRefs` from the FournosJob spec, copies each referenced secret from `secrets_namespace` into the operator namespace (per-job name `<fjob-name>-<ref>`, with `ownerReferences` for automatic cleanup), resolves the kubeconfig Secret, creates the Tekton PipelineRun with a projected `vault-secrets` volume mounting all copied secrets at `/var/run/secrets/fournos/<entry-name>/` and `ownerReferences` pointing at the FournosJob, sets `phase=Running`.
 5. **timer (Running)**: Polls the PipelineRun for completion. On success/failure, deletes the Workload and sets `phase=Succeeded` or `phase=Failed`.
 6. **timer (any non-terminal phase, shutdown)**: If `spec.shutdown` is set (`Stop` or `Terminate`) and the job has a PipelineRun (Admitted/Running), the timer cancels the PipelineRun — `Stop` uses Tekton's `CancelledRunFinally` (runs `finally` tasks), `Terminate` uses `Cancelled` (skips `finally` tasks) — and sets `phase=Stopping`. The Workload is **not** deleted yet — it stays alive to hold the cluster slot while the PipelineRun winds down. If no PipelineRun exists (Pending), the Workload is deleted immediately and the job goes straight to `phase=Stopped`.
 7. **timer (Stopping)**: Polls the PipelineRun until it reaches a terminal state (`succeeded` or `failed`). Once complete, deletes the Workload to release Kueue quota and sets `phase=Stopped`.
@@ -359,7 +359,7 @@ fournos/
     execution.py           # reconcile_admitted, reconcile_running
   core/
     constants.py           # Shared label keys, Phase enum, cluster-slot constants
-    clusters.py            # ClusterRegistry (kubeconfig lookup, secretRef resolution)
+    clusters.py            # ClusterRegistry (kubeconfig lookup, secretRef resolution, cross-namespace secret copying)
     resolve.py             # ResolveClient (Forge resolve Job management)
     tekton.py              # TektonClient (PipelineRun CRUD)
     kueue.py               # KueueClient (Workload CRUD, admission checks, cluster-slot requests)
@@ -406,11 +406,11 @@ README.md
 - **Stateless operator** — all job state lives in Kubernetes resources (FournosJob CRs, PipelineRuns, Workloads), not in memory. Crash-safe via `on_resume`.
 - **Timer-based reconciliation** — the operator polls Workload admission and PipelineRun completion via a kopf timer (5s interval), eliminating the need for callback tasks or watch streams on third-party resources
 - **Operator cleans up on completion** — Kueue Workloads are deleted when the PipelineRun reaches a terminal state, releasing quota without relying on external callbacks
-- **ownerReferences for cascade deletion** — Workloads and PipelineRuns carry `ownerReferences` pointing at their FournosJob, so Kubernetes automatically cascade-deletes them when the job is removed
+- **ownerReferences for cascade deletion** — Workloads, PipelineRuns, resolve Jobs, and copied secrets all carry `ownerReferences` pointing at their FournosJob, so Kubernetes automatically cascade-deletes them when the job is removed
 - **Exclusive locking via Kueue semaphore** — each cluster flavor has 100 `fournos/cluster-slot` units. Normal jobs request 1 slot; exclusive jobs request all 100. Kueue enforces mutual exclusion atomically — no operator-level blocking, labels, or in-memory state needed. Hardware-only jobs are automatically steered to clusters with available slots.
 - **Shutdown via spec field** — the `spec.shutdown` enum supports two modes: `Stop` (Tekton `CancelledRunFinally` — runs `finally` cleanup tasks) and `Terminate` (Tekton `Cancelled` — skips `finally` tasks). Both transition to an intermediate `Stopping` phase while the PipelineRun winds down. The Workload (and its quota) is kept alive until the PipelineRun completes, ensuring the cluster slot is not released prematurely. Only then does the operator delete the Workload and set `phase=Stopped`. The enum is extensible for future shutdown strategies. The FournosJob stays around in `Stopped` phase for inspection, unlike deletion which cascades and removes the record.
 - **Mandatory Resolving phase** — every job passes through a `Resolving` phase before entering `Pending`. During this phase, a Forge K8s Job runs to determine hardware requirements (`gpuType`, `gpuCount`) and secret references (`secretRefs`). Forge patches these values directly into the FournosJob spec (hardware only when not already user-provided). The operator validates them after the Job completes. Failed resolve Jobs are preserved for debugging.
-- **Vault-based secret management** — pipeline secrets originate in a HashiCorp Vault and are synced to K8s Secrets on demand via `hacks/sync_vault_secrets.py` into a dedicated secrets namespace (`FOURNOS_SECRETS_NAMESPACE`, default `psap-secrets`). The K8s Secret name uses a `vault-` prefix followed by the Vault entry name (e.g. `vault-my-creds`); entries that are not valid DNS-1123 names are rejected during sync. Each synced Secret carries a `fournos.dev/vault-entry=true` label. Secret references are populated by Forge during the Resolving phase on `spec.secretRefs`. The operator validates them in the secrets namespace before creating the Workload (during Resolving) and reads them from spec when creating the PipelineRun (during Admitted). Missing or non-vault refs fail the job immediately rather than creating a broken PipelineRun.
+- **Vault-based secret management with cross-namespace injection** — pipeline secrets originate in a HashiCorp Vault and are synced to K8s Secrets on demand via `hacks/sync_vault_secrets.py` into a dedicated secrets namespace (`FOURNOS_SECRETS_NAMESPACE`, default `psap-secrets`). The K8s Secret name uses a `vault-` prefix followed by the Vault entry name (e.g. `vault-my-creds`); entries that are not valid DNS-1123 names are rejected during sync. Each synced Secret carries a `fournos.dev/vault-entry=true` label. Secret references are populated by Forge during the Resolving phase on `spec.secretRefs`. The operator validates them in the secrets namespace before creating the Workload (during Resolving). During the Admitted phase, the operator copies each referenced secret from the secrets namespace into the operator namespace with a per-job name (`<fjob-name>-<secret-name>`) and `ownerReferences` to the FournosJob for automatic cleanup. The copies are combined into a single projected volume (`vault-secrets`) mounted at `/var/run/secrets/fournos/<entry-name>/<key>`, with each secret's keys placed under a subdirectory matching its original name. This avoids key collisions across secrets and works with a static `volumeMount` in the Task YAML regardless of how many secrets a job uses. An empty projected volume (no secrets) is always emitted so the static mount never fails. Missing or non-vault refs fail the job during Resolving rather than creating a broken PipelineRun.
 - **Multiple pipelines** — `fournos-full` (prepare → run → cleanup) and `fournos-run-only` (run only), selectable per job
 - **Target clusters need nothing installed** — FORGE runs on the hub cluster inside Tekton Task pods and communicates with targets via remote `oc`/`kubectl` commands through kubeconfig Secrets (stored in the dedicated secrets namespace)
 

README.md

@@ -85,7 +85,7 @@ oc delete FournosJob -n $FOURNOS_NAMESPACE <name>      # cleanup
 | `spec.displayName` | no | Human-readable job name (defaults to `metadata.name`) |
 | `spec.pipeline` | no | Tekton Pipeline name (default: `fournos-full`) |
 | `spec.priority` | no | Kueue WorkloadPriorityClass name |
-| `spec.secretRefs` | no | Vault-synced K8s Secret names (prefixed with `vault-`) to mount into the pipeline. Populated by Forge during the Resolving phase. The operator verifies each name as a K8s Secret with the `fournos.dev/vault-entry=true` label. |
+| `spec.secretRefs` | no | Vault-synced K8s Secret names (prefixed with `vault-`) to mount into the pipeline. Populated by Forge during the Resolving phase. The operator validates each name in `FOURNOS_SECRETS_NAMESPACE`, copies the secrets into the operator namespace, and mounts them as a projected volume at `/var/run/secrets/fournos/<entry-name>/`. |
 | `spec.exclusive` | no | If `true`, locks the target cluster so no other FournosJob can run there. Requires `spec.cluster`. |
 | `spec.shutdown` | no | Shutdown action: `Stop` cancels gracefully (Tekton `CancelledRunFinally` — runs `finally` tasks); `Terminate` cancels immediately (Tekton `Cancelled` — skips `finally` tasks). Both wait for the PipelineRun to finish before releasing Kueue quota. |
 
@@ -269,9 +269,25 @@ make sync-vault-secrets-dry-run      # preview only
 The synced secrets are labelled `fournos.dev/vault-entry=true` and
 `app.kubernetes.io/managed-by=fournos-vault-sync` for easy identification.
 Secret references are populated by Forge during the Resolving phase directly
-on the FournosJob `spec.secretRefs` field. The operator verifies each
+on the FournosJob `spec.secretRefs` field. The operator validates each
 referenced Secret exists in the secrets namespace and carries the vault
-label before proceeding.
+label during the Resolving phase, then copies them into the operator
+namespace during the Admitted phase and mounts them as a projected volume
+into the PipelineRun pods. Each secret's keys are placed under a
+subdirectory matching the original name:
+
+```
+/var/run/secrets/fournos/
+  vault-my-creds/
+    username
+    password
+  vault-other-creds/
+    token
+```
+
+Copied secrets are named `<fjob-name>-<secret-name>` and carry
+`ownerReferences` back to the FournosJob, so Kubernetes garbage-collects
+them automatically when the job is deleted.
 
 ## Configuration
 
@@ -304,7 +320,7 @@ The operator runs as a single-replica Deployment using
 1. **Resolves** job requirements by launching a Forge K8s Job that populates the FournosJob spec with GPU type/count and secret references
 2. **Creates** a Kueue Workload with the resolved GPU resources (owned by the FournosJob via `ownerReferences`)
 3. **Polls** (5 s timer) for Kueue admission and assigned cluster
-4. **Launches** a Tekton PipelineRun with FORGE parameters (owned by the FournosJob via `ownerReferences`)
+4. **Copies** referenced Vault secrets from the secrets namespace into the operator namespace (per-job copies with `ownerReferences` for automatic cleanup) and **launches** a Tekton PipelineRun with FORGE parameters and the secrets mounted as a projected volume at `/var/run/secrets/fournos/` (owned by the FournosJob via `ownerReferences`)
 5. **Watches** the PipelineRun until completion
 6. **Deletes** the Workload to release Kueue quota
 

config/forge/workflows/tasks.yaml

@@ -26,7 +26,7 @@ spec:
       imagePullPolicy: Always
       env:
       - name: KUBECONFIG
-        value: /workspace/kubeconfig/kubeconfig
+        value: /var/run/secrets/fournos-kubeconfig/kubeconfig
       - name: FORGE_CONFIG
         value: "$(params.forge-config)"
       - name: FOURNOS_ENV
@@ -35,7 +35,10 @@ spec:
         value: "$(params.job-step)"
       volumeMounts:
         - name: kubeconfig
-          mountPath: /workspace/kubeconfig
+          mountPath: /var/run/secrets/fournos-kubeconfig
+          readOnly: true
+        - name: vault-secrets
+          mountPath: /var/run/secrets/fournos
           readOnly: true
       script: |
         #!/usr/bin/env bash

config/fournos-validation/workflows/tasks.yaml

@@ -16,10 +16,10 @@ spec:
       image: bitnami/kubectl:latest
       env:
         - name: KUBECONFIG
-          value: /workspace/kubeconfig/kubeconfig
+          value: /var/run/secrets/fournos-kubeconfig/kubeconfig
       volumeMounts:
         - name: kubeconfig
-          mountPath: /workspace/kubeconfig
+          mountPath: /var/run/secrets/fournos-kubeconfig
           readOnly: true
       script: |
         #!/usr/bin/env bash

dev/job-secret-demo.yaml

@@ -0,0 +1,13 @@
+apiVersion: fournos.dev/v1
+kind: FournosJob
+metadata:
+  generateName: mock-secret-demo-
+spec:
+  owner: dev
+  displayName: mock-secret-demo
+  cluster: cluster-1
+  pipeline: fournos-run-only
+  forge:
+    project: secret-demo
+    args:
+      - demo

dev/mock-pipelines/tasks.yaml

@@ -51,6 +51,10 @@ spec:
               name: fournos-mock-config
               key: sleep
               optional: true
+      volumeMounts:
+        - name: vault-secrets
+          mountPath: /var/run/secrets/fournos
+          readOnly: true
       script: |
         #!/bin/sh
         MOCK_SLEEP="${MOCK_SLEEP:-3}"
@@ -61,6 +65,8 @@ spec:
         echo "$(params.env)" | sed 's/^/  /'
         echo "[mock-run] gpu-count=$(params.gpu-count)"
         echo "[mock-run] job=$(params.job-name)"
+        echo "[mock-run] vault-secrets:"
+        find /var/run/secrets/fournos -type f 2>/dev/null | sort | while read f; do echo "  $f ($(wc -c < "$f") bytes)"; done
         echo "[mock-run] simulating workload (${MOCK_SLEEP}s)..."
         sleep "$MOCK_SLEEP"
         echo "[mock-run] done"

dev/mock-resolve/resolve.sh

@@ -1,8 +1,10 @@
 #!/usr/bin/env bash
 # Mock Forge resolve script — patches the FournosJob spec with resolved values.
 #
-# Forge writes hardware (only when not user-provided) and secretRefs
-# directly into the FournosJob spec.
+# In production, Forge determines hardware requirements and secret
+# references by inspecting the project.  This mock sets hard-coded
+# defaults for hardware only when the user hasn't provided them, and
+# always sets secretRefs.
 #
 # Expected env vars (set by the operator):
 #   FOURNOS_JOB_NAME    — FournosJob name to patch
@@ -22,25 +24,15 @@ if [[ -z "${EXISTING_HW}" ]]; then
   kubectl patch fournosjob "${FOURNOS_JOB_NAME}" \
     -n "${FOURNOS_NAMESPACE}" \
     --type=merge \
-    -p '{
-      "spec": {
-        "hardware": {
-          "gpuType": "a100",
-          "gpuCount": 2
-        },
-        "secretRefs": []
-      }
-    }'
+    -p '{"spec":{"hardware":{"gpuType":"a100","gpuCount":2}}}'
 else
-  echo "[mock-resolve] user-provided hardware found (${EXISTING_HW}), patching secretRefs only"
-  kubectl patch fournosjob "${FOURNOS_JOB_NAME}" \
-    -n "${FOURNOS_NAMESPACE}" \
-    --type=merge \
-    -p '{
-      "spec": {
-        "secretRefs": []
-      }
-    }'
+  echo "[mock-resolve] user-provided hardware found (${EXISTING_HW}), keeping"
 fi
 
+echo "[mock-resolve] setting secretRefs"
+kubectl patch fournosjob "${FOURNOS_JOB_NAME}" \
+  -n "${FOURNOS_NAMESPACE}" \
+  --type=merge \
+  -p '{"spec":{"secretRefs":["vault-placeholder"]}}'
+
 echo "[mock-resolve] done"

dev/mock-secrets.yaml

@@ -37,3 +37,16 @@ metadata:
 type: Opaque
 stringData:
   kubeconfig: "mock"
+
+---
+# Mock Vault-synced secret for the secret-demo sample job.
+apiVersion: v1
+kind: Secret
+metadata:
+  name: vault-placeholder
+  labels:
+    fournos.dev/vault-entry: "true"
+    app.kubernetes.io/managed-by: fournos-vault-sync
+type: Opaque
+stringData:
+  placeholder: "mock-secret-value"