Fournos

Fournos (φούρνος) = "oven" in Greek.

Fournos is a Kubernetes operator that schedules benchmark jobs via Kueue and executes them as Tekton PipelineRuns on remote clusters through a pluggable execution engine.

Jobs are submitted as FournosJob custom resources. Every job first passes through a mandatory Resolving phase where a resolve Job (driven by the configured execution engine) populates GPU requirements and secret references directly on the FournosJob spec. The operator then creates a Kueue Workload for quota management, waits for admission, and launches the corresponding Tekton PipelineRun.

Cluster dependencies

The following operators must be installed in the cluster before deploying Fournos:

Red Hat OpenShift Pipelines (1.21)
Red Hat build of Kueue (1.3)
Builds for Red Hat OpenShift Operator (1.7)
Red Hat OpenShift GitOps (1.20)
- only for the GitOps deployment of Fournos

Quick start

python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
pre-commit install

Submitting a job

Create a FournosJob resource. Use generateName for automatic unique naming and displayName for a human-readable label:

apiVersion: fournos.dev/v1
kind: FournosJob
metadata:
  generateName: sample-run-benchamark-
spec:
  owner: perf-team
  displayName: sample-run-benchmark
  cluster: cluster-1
  hardware:
    gpuType: a100
    gpuCount: 2
  pipeline: forge-full
  executionEngine:
    forge:
      project: llmd
      args:
        - cks
      configOverrides:
        batch_size: 64
  env:
    OCPCI_SUITE: regression
    OCPCI_VARIANT: nightly

FOURNOS_WORKLOAD_NAMESPACE=fournos-$USER-dev
oc create -f config/forge/samples/job-full.yaml -n $FOURNOS_WORKLOAD_NAMESPACE     # returns the generated name, e.g. forge-full-sample-x7k2m
oc get FournosJobs -n $FOURNOS_WORKLOAD_NAMESPACE -w            # watch status transitions
oc patch FournosJob <name> -n $FOURNOS_WORKLOAD_NAMESPACE --type merge -p '{"spec":{"shutdown":"Stop"}}'        # graceful stop (runs finally tasks)
oc patch FournosJob <name> -n $FOURNOS_WORKLOAD_NAMESPACE --type merge -p '{"spec":{"shutdown":"Terminate"}}'   # immediate terminate (skips finally tasks)
oc delete FournosJob -n $FOURNOS_WORKLOAD_NAMESPACE <name>      # cleanup

Spec fields

Field	Required	Description
`spec.executionEngine`	yes	Execution engine configuration. The single top-level key is the engine name (e.g. `forge`); its value is opaque engine-specific config passed through as-is.
`spec.env`	no	Environment variables available to the execution engine (read from the FournosJob spec via K8s API)
`spec.cluster`	*	Pin to a specific cluster (Kueue ResourceFlavor). Since `exclusive` defaults to `true`, this also locks the cluster — set `exclusive: false` for shared access.
`spec.hardware.gpuType`	*	Short GPU model name — e.g. `a100`, `h200`. The operator prepends the `FOURNOS_GPU_RESOURCE_PREFIX` (default `fournos/gpu-`) automatically, so do not include the full resource path.
`spec.hardware.gpuCount`	with gpuType	Number of GPUs (minimum 1)
`spec.owner`	no	Team or individual that owns this job
`spec.displayName`	no	Human-readable job name (defaults to `metadata.name`)
`spec.pipeline`	no	Tekton Pipeline name (default: `fournos-full`). The Pipeline must carry a `fournos.dev/resolve-image` annotation with the full image reference for the resolve Job.
`spec.priority`	no	Kueue WorkloadPriorityClass name
`spec.secretRefs`	no	Vault-synced K8s Secret names (prefixed with `vault-`) to mount into the pipeline. Populated by the execution engine during the Resolving phase. The operator validates each name in `FOURNOS_SECRETS_NAMESPACE`, copies the secrets into the operator namespace, and mounts them as a projected volume at `/var/run/secrets/fournos/<entry-name>/`.
`spec.exclusive`	no (default `true`)	If `true`, locks the target cluster so no other FournosJob can run there. Requires `spec.cluster`. Hardware is optional — when omitted the Workload only requests cluster-slot resources for locking.
`spec.clusterless`	no (default `false`)	If `true`, runs without cluster access — no kubeconfig is passed to the execution environment. Cannot be combined with `cluster` or `exclusive: true`. Hardware specifications are optional — if omitted, the job runs on hub cluster resources without Kueue hardware scheduling.
`spec.scheduledStartTime`	no	ISO 8601 UTC timestamp (e.g. `2026-08-18T15:00:00Z`). When set, the job stays in `Scheduled` phase until this time, then proceeds to `Resolving`. Mutually exclusive with `schedule`.
`spec.schedule`	no	Cron expression for recurring execution (e.g. `0 20 * * *`). The job enters `Recurring` phase and the operator creates child FournosJob CRs on each cron tick, labeled with `fournos.dev/recurring-parent`. Mutually exclusive with `scheduledStartTime`.
`spec.shutdown`	no	Shutdown action: `Stop` cancels gracefully (Tekton `CancelledRunFinally` — runs `finally` tasks); `Terminate` cancels immediately (Tekton `Cancelled` — skips `finally` tasks). Both wait for the PipelineRun to finish before releasing Kueue quota.

* spec.hardware is optional for all jobs except those requiring explicit GPU scheduling. Hardware specifications are handled as follows:

Exclusive cluster jobs (exclusive: true + cluster): Hardware may be omitted since the Workload only needs cluster-slot resources for locking.
Non-exclusive jobs (exclusive: false): Hardware is optional — when omitted, the Workload only requests cluster-slot resources for shared access.
Clusterless jobs (clusterless: true): Hardware is optional since jobs run on hub cluster resources without Kueue scheduling. Every job passes through the Resolving phase where the execution engine populates spec.hardware (if not already set) and spec.secretRefs directly on the FournosJob. Since exclusive defaults to true, any job with spec.cluster locks the cluster exclusively — including jobs that also specify spec.hardware. Set exclusive: false for shared access. Jobs without spec.cluster must set exclusive: false.

Status

The operator writes status to .status:

Field	Description
`phase`	[`Scheduled` →] [`Recurring` ↻] `Resolving` → `Pending` → `Admitted` → `Running` → `Succeeded` / `Failed` / `Stopping` → `Stopped`
`lastScheduledTime`	For recurring jobs, the timestamp when the last child job was created
`cluster`	Cluster assigned by Kueue (or `[clusterless]` for clusterless jobs)
`pipelineRun`	Name of the Tekton PipelineRun
`dashboardURL`	Tekton Dashboard link (if configured)
`message`	Error details on failure

Clusterless Jobs

Clusterless jobs (clusterless: true) run entirely on the hub cluster without access to target clusters. They bypass Kueue admission and skip the Pending phase for faster execution.

Examples

Basic clusterless job:

apiVersion: fournos.dev/v1
kind: FournosJob
metadata:
  name: hub-only-job
spec:
  clusterless: true
  exclusive: false  # Required
  executionEngine:
    forge:
      project: "local/benchmark"
      args: ["unit-test"]

Clusterless Job Lifecycle

Resolving → Admitted → Running → Succeeded/Failed
            ↑
        Skips Pending (no Kueue admission)

Benefits:

⚡ Faster startup: No Kueue admission delay
🧹 Resource efficient: No target cluster resource blocking
🎯 Hub-focused: Perfect for CI/validation workflows

Restrictions:

Cannot combine with cluster specification
Must use exclusive: false
Cannot use lockOnly: true
No target cluster kubeconfig available to execution environment

Scheduled and Recurring Jobs

One-time scheduled jobs

Set spec.scheduledStartTime to defer execution to a future time. The job enters Scheduled phase and automatically transitions to Resolving once the time is reached. When scheduledStartTime is omitted or in the past, the job starts immediately.

apiVersion: fournos.dev/v1
kind: FournosJob
metadata:
  generateName: deferred-bench-
spec:
  owner: perf-team
  scheduledStartTime: "2026-08-18T15:00:00Z"
  cluster: cluster-1
  executionEngine:
    forge:
      project: llm-d
      args: [cks]

Lifecycle: Scheduled → Resolving → Pending → Admitted → Running → Succeeded/Failed

Recurring jobs

Set spec.schedule with a cron expression to create a recurring job template. The job enters Recurring phase and the operator creates child FournosJob CRs on each cron tick. The parent stays in Recurring phase indefinitely.

apiVersion: fournos.dev/v1
kind: FournosJob
metadata:
  name: nightly-bench
spec:
  owner: perf-team
  schedule: "0 20 * * *"
  cluster: cluster-1
  executionEngine:
    forge:
      project: llm-d
      args: [cks]

Child jobs are labeled with fournos.dev/recurring-parent for easy querying:

kubectl get fjobs -l fournos.dev/recurring-parent=nightly-bench

To manually trigger a child job outside the cron schedule:

kubectl annotate fjob nightly-bench fournos.dev/trigger-now=true

The operator creates a child immediately and resets the annotation to false.

To stop recurring execution, delete the parent job.

Restrictions:

scheduledStartTime and schedule are mutually exclusive
Invalid cron expressions or timestamps fail the job immediately

Local development

Prerequisites: Podman, kind, and kubectl.

make dev-setup    # creates a kind cluster, installs Tekton + Kueue + CRD, applies mock resources
make dev-run      # starts the operator locally (connects to the kind cluster)

Both targets default to the fournos-local-dev namespace. Override with FOURNOS_WORKLOAD_NAMESPACE=<YOUR_NAMESPACE> make dev-setup dev-run.

In another terminal:

FOURNOS_WORKLOAD_NAMESPACE=fournos-local-dev make test   # run the integration test suite

make dev-teardown # deletes the kind cluster

dev-setup installs real Tekton Pipelines and Kueue controllers into the kind cluster, but substitutes lightweight mock Tasks (echo + sleep) in place of the real execution engine runner. The dev environment uses its own Kueue config (dev/mock-kueue-config.yaml) with four mock clusters and synthetic GPU quotas, plus matching kubeconfig Secrets (kubeconfig-cluster-{1..4}) in the dedicated secrets namespace (psap-secrets).

Before opening a PR

make lint                        # lint (fournos/ + tests/)
make test                        # integration tests (operator must be running)

Deployment

Execution engine on the hub: config/forge/ is the real OpenShift configuration for this repo — ImageStreams, Builds, Tekton Tasks and Pipelines, and sample jobs you apply to a cluster. It is not the same as the lightweight stand-ins under dev/mock-pipelines/, which make dev-setup installs on kind for local testing only.

Deploy the full stack (namespaces, CRD, RBAC, Kueue config, workflows, Deployment). The operator runs in a dedicated controller namespace; execution resources (Tekton, Kueue, FournosJobs) live in the execution namespace:

make deploy \
  FOURNOS_CONTROLLER_NAMESPACE=fournos-controller-$USER \
  FOURNOS_WORKLOAD_NAMESPACE=fournos-$USER-dev \
  FOURNOS_SECRETS_NAMESPACE=psap-secrets

Onboarding a new cluster

Three things are needed to make a target cluster available to Fournos:

Create a kubeconfig Secret in the dedicated secrets namespace:

FOURNOS_SECRETS_NAMESPACE=psap-secrets
CLUSTER_NAME=<name>
oc create secret generic kubeconfig-${CLUSTER_NAME} \
  --from-file=kubeconfig=/path/to/auth/kubeconfig \
  -n $FOURNOS_SECRETS_NAMESPACE

The secret name must match the FOURNOS_KUBECONFIG_SECRET_PATTERN (default kubeconfig-{cluster}). Secrets are stored in the dedicated namespace (FOURNOS_SECRETS_NAMESPACE, default psap-secrets).

Add a ResourceFlavor and quota in config/kueue-config.yaml. Add a new ResourceFlavor with a matching fournos.dev/cluster nodeLabel, and list it under the fournos-queue ClusterQueue with the appropriate GPU/CPU quotas. Then apply:

oc apply -f config/kueue-config.yaml

Verify connectivity by submitting a lightweight validate-only job. Edit cluster (and optionally hardware) in config/fournos-validation/samples/test-connectivity-job.yaml to match the new target, then:

FOURNOS_WORKLOAD_NAMESPACE=fournos-$USER-dev
oc create -f config/fournos-validation/samples/test-connectivity-job.yaml -n $FOURNOS_WORKLOAD_NAMESPACE
oc get fournosjobs -n $FOURNOS_WORKLOAD_NAMESPACE -w        # should reach Succeeded

This runs the fournos-validate-only pipeline, which only checks oc cluster-info against the target — no benchmark workload is launched. If the job reaches Succeeded, the kubeconfig secret and Kueue quota are correctly configured. If it fails, check the operator logs and the PipelineRun status for details.

Deploying the execution engine workflow configuration

Apply the production execution engine assets from config/forge/ (not the kind mocks in dev/mock-pipelines/). Deploy the cluster configuration (Builds + Tekton) to the execution namespace:

oc apply -n $FOURNOS_WORKLOAD_NAMESPACE -f config/forge/images/is_forge.yaml
cat config/forge/images/build_forge-main.yaml \
  | NAMESPACE=$FOURNOS_WORKLOAD_NAMESPACE envsubst '$NAMESPACE' \
  | oc apply -f- -n $FOURNOS_WORKLOAD_NAMESPACE
oc create -n $FOURNOS_WORKLOAD_NAMESPACE -f config/forge/images/buildrun_forge-main.yaml

for wf_file in config/forge/workflows/*.yaml; do
  cat "$wf_file" | NAMESPACE=$FOURNOS_WORKLOAD_NAMESPACE envsubst '$NAMESPACE' | oc apply -f- -n $FOURNOS_WORKLOAD_NAMESPACE
done

Synchronizing secrets from Vault

Pipeline jobs can reference Kubernetes Secrets via spec.secretRefs. These secrets originate in a HashiCorp Vault instance. Because there is no permanent programmatic access to the vault, secrets are synchronized manually on demand — whenever the vault content changes.

The sync script reads vault entries and creates one Opaque Secret per entry in the dedicated secrets namespace (FOURNOS_SECRETS_NAMESPACE, default psap-secrets), using a vault- prefix followed by the vault entry name as the K8s Secret name (e.g. vault entry my-creds becomes Secret vault-my-creds). Entries whose names are not valid DNS-1123 subdomain names are skipped with an error. Individual keys within an entry that are not valid K8s Secret data keys (allowed: alphanumeric, -, _, .) are also skipped. Existing secrets are updated in-place.

# 1. Set the required environment variables
export VAULT_ADDR="https://vault.example.com"   # Vault server URL
export VAULT_TOKEN="s.xxxxx"                     # your short-lived token
export VAULT_SECRET_PATH="path/to/secrets"       # directory path within the KV engine

# 2. Sync all vault entries under the configured path
python hacks/sync_vault_secrets.py -n psap-secrets

# 3. Preview without touching the cluster
python hacks/sync_vault_secrets.py -n psap-secrets --dry-run

Makefile shortcuts (VAULT_ADDR, VAULT_TOKEN, and VAULT_SECRET_PATH must be set):

make sync-vault-secrets              # syncs all entries
make sync-vault-secrets-dry-run      # preview only

The synced secrets are labelled fournos.dev/vault-entry=true and app.kubernetes.io/managed-by=fournos-vault-sync for easy identification. Secret references are populated by the execution engine during the Resolving phase directly on the FournosJob spec.secretRefs field. The operator validates each referenced Secret exists in the secrets namespace and carries the vault label during the Resolving phase, then copies them into the operator namespace during the Admitted phase and mounts them as a projected volume into the PipelineRun pods. Each secret's keys are placed under a subdirectory matching the original name:

/var/run/secrets/fournos/
  vault-my-creds/
    username
    password
  vault-other-creds/
    token

Copied secrets are named <fjob-name>-<secret-name> and carry ownerReferences back to the FournosJob, so Kubernetes garbage-collects them automatically when the job is deleted.

Configuration

All settings are read from environment variables with the FOURNOS_ prefix:

Variable	Default	Description
`FOURNOS_WORKLOAD_NAMESPACE`	required	Namespace for FournosJobs and execution resources
`FOURNOS_SECRETS_NAMESPACE`	`psap-secrets`	Namespace where kubeconfig and vault-synced secrets are stored
`FOURNOS_TEKTON_DASHBOARD_URL`		Tekton Dashboard base URL
`FOURNOS_KUBECONFIG_SECRET_PATTERN`	`kubeconfig-{cluster}`	Pattern for resolving cluster names to Secret names
`FOURNOS_VAULT_SECRET_PATTERN`	`vault-{entry}`	Pattern for naming vault-synced Secrets
`FOURNOS_KUEUE_LOCAL_QUEUE_NAME`	`fournos-queue`	Kueue LocalQueue name
`FOURNOS_GPU_RESOURCE_PREFIX`	`fournos/gpu-`	Resource name prefix for GPU types
`FOURNOS_LOG_LEVEL`	`INFO`	Logging level
`FOURNOS_GC_INTERVAL_SEC`	`300`	Resource GC interval (seconds)
`FOURNOS_RESOLVE_DEADLINE_SEC`	`300`	Deadline for the resolve Job (seconds)
`FOURNOS_RESOLVE_JOB_TEMPLATE`	`config/forge/resolve_job.yaml`	Path (relative to project root) to the Job YAML template for the resolve step. Override with `dev/mock-resolve/resolve_job.yaml` for local dev/CI.
`FOURNOS_ARTIFACT_PVC_SIZE`	`1Gi`	Size of the per-PipelineRun PVC used for shared artifact storage across pipeline tasks

Architecture

FournosJob CR ──→ Operator ──→ Resolve Job (e.g. FORGE, patches FournosJob spec) ──→ Kueue Workload ──→ (admission) ──→ Tekton PipelineRun ──→ Execution Engine (e.g. FORGE) ──→ target cluster

The operator runs as a single-replica Deployment using kopf. On each FournosJob, it:

Resolves job requirements by launching a resolve K8s Job (using the configured execution engine image) that populates the FournosJob spec with GPU type/count and secret references
Creates a Kueue Workload with the resolved GPU resources (owned by the FournosJob via ownerReferences)
Polls (5 s timer) for Kueue admission and assigned cluster
Copies referenced Vault secrets from the secrets namespace into the operator namespace (per-job copies with ownerReferences for automatic cleanup) and launches a Tekton PipelineRun with FJOB_NAME + FOURNOS_WORKLOAD_NAMESPACE (so the execution engine can look up the full FournosJob spec), the secrets mounted as a projected volume at /var/run/secrets/fournos/ (owned by the FournosJob via ownerReferences), and a shared artifacts workspace backed by a volumeClaimTemplate PVC for cross-task artifact storage (managed by Tekton)
Watches the PipelineRun until completion
Deletes the Workload to release Kueue quota

Setting spec.shutdown on a FournosJob triggers cancellation of the PipelineRun and transitions to phase=Stopping. Stop uses Tekton's CancelledRunFinally (runs finally cleanup tasks); Terminate uses Cancelled (skips finally tasks). In both cases the operator keeps the Kueue Workload alive until the PipelineRun finishes, ensuring the cluster slot is not released prematurely. Once done, the Workload is deleted and the job moves to phase=Stopped.

Deleting a FournosJob automatically cascade-deletes its Workload and PipelineRun through Kubernetes owner references.

Target clusters need nothing installed — the execution engine runs on the hub cluster inside Tekton Task pods and communicates with targets via oc/kubectl through kubeconfig Secrets.

For a detailed breakdown of the CRD, scheduling, operator internals, and key design decisions, see the Design Document.

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
config		config
dev		dev
fournos-ui		fournos-ui
fournos		fournos
hacks		hacks
manifests		manifests
tests		tests
.dockerignore		.dockerignore
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
Containerfile		Containerfile
Fournos_Design_Document.md		Fournos_Design_Document.md
LICENSE		LICENSE
Makefile		Makefile
OWNERS		OWNERS
README.md		README.md
pyproject.toml		pyproject.toml

Navigation Menu

Uh oh!

Repository files navigation

Fournos

Cluster dependencies

Quick start

Submitting a job

Spec fields

Status

Clusterless Jobs

Examples

Clusterless Job Lifecycle

Scheduled and Recurring Jobs

One-time scheduled jobs

Recurring jobs

Local development

Before opening a PR

Deployment

Onboarding a new cluster

Deploying the execution engine workflow configuration

Synchronizing secrets from Vault

Configuration

Architecture

About

Releases

Packages

Contributors

Languages

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

Fournos

Cluster dependencies

Quick start

Submitting a job

Spec fields

Status

Clusterless Jobs

Examples

Clusterless Job Lifecycle

Scheduled and Recurring Jobs

One-time scheduled jobs

Recurring jobs

Local development

Before opening a PR

Deployment

Onboarding a new cluster

Deploying the execution engine workflow configuration

Synchronizing secrets from Vault

Configuration

Architecture

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages