diff --git a/.gitignore b/.gitignore index a6e7e83..d01554b 100644 --- a/.gitignore +++ b/.gitignore @@ -13,3 +13,4 @@ build/ secrets/ fournos.log fournos.pid +.vscode/ diff --git a/fournos-ui/.gitignore b/fournos-ui/.gitignore index bb12cb0..43d2f85 100644 --- a/fournos-ui/.gitignore +++ b/fournos-ui/.gitignore @@ -33,4 +33,7 @@ app/mock_data.py kustomize/overlays/*/params.env kustomize/overlays/*/projects.yaml kustomize/overlays/*/kustomization.yaml +kustomize/overlays/*/oauth-cookie-secret.yaml kustomize/overlays/*/rolebinding-*.yaml +projects-local.yaml +cpt.yaml diff --git a/fournos-ui/README.md b/fournos-ui/README.md index 1df9566..0eb1dc9 100644 --- a/fournos-ui/README.md +++ b/fournos-ui/README.md @@ -14,15 +14,17 @@ A web dashboard for managing [Fournos](https://github.com/openshift-psap/fournos ## Architecture ``` -┌─────────────┐ ┌──────────────────┐ ┌────────────┐ -│ Browser │────▶│ FastAPI + HTMX │────▶│ Kubernetes │ -│ │◀────│ (Dashboard) │◀────│ API │ -└─────────────┘ └────────┬─────────┘ └────────────┘ - │ - ┌────────▼─────────┐ - │ PostgreSQL │ - │ (job history) │ - └──────────────────┘ + ┌─── Pod ──────────────────────────────┐ +┌─────────┐ ┌───────┐ │ ┌─────────────┐ ┌───────────────┐ │ ┌────────────┐ +│ Browser │──▶│ Route │──┼▶│ OAuth Proxy │──▶│ FastAPI + HTMX│─┼──▶│ Kubernetes │ +│ │◀──│ (TLS) │◀─┼─│ (:8443) │◀──│ (:8000) │◀┼───│ API │ +└─────────┘ └───────┘ │ └──────┬──────┘ └───────┬───────┘ │ └────────────┘ + │ │ │ │ + │ ▼ ▼ │ + │ OpenShift OAuth ┌────────────┐ │ + │ Server │ PostgreSQL │ │ + │ └────────────┘ │ + └──────────────────────────────────────┘ ``` - **FastAPI** backend with **Jinja2** templates and **HTMX** for dynamic updates. @@ -32,59 +34,161 @@ A web dashboard for managing [Fournos](https://github.com/openshift-psap/fournos ## Prerequisites -- A Kubernetes / OpenShift cluster with the [Fournos Operator](https://github.com/openshift-psap/fournos-operator) installed. +- An OpenShift cluster (4.14+) with the [Fournos Operator](https://github.com/openshift-psap/fournos-operator) installed. +- **cert-manager** operator installed on the cluster (for TLS certificate issuance). - A container registry to push the dashboard image. -- `kubectl` or `oc` CLI configured with cluster access. +- `oc` CLI configured with cluster-admin access (needed for initial setup). ## Getting Started -### 1. Clone and configure the overlay +### 1. Configure the overlay ```bash cd kustomize/overlays/ocp/ +``` + +Create the required config files from the examples: -# Copy example files +```bash cp kustomization.yaml.example kustomization.yaml +cp oauth-cookie-secret.yaml.example oauth-cookie-secret.yaml cp projects.yaml.example projects.yaml -cp params.env.example params.env cp ../../base/postgresql-secret.env.example postgresql-secret.env ``` +These files are gitignored because they contain secrets or cluster-specific values. +For an existing deployment, you can pull values from the cluster instead (see "Pulling config from a live cluster" below). + Edit each file with your values: -- **`kustomization.yaml`** -- Set your dashboard image, PostgreSQL image, target namespace, and storage class. - **`projects.yaml`** -- Define your Forge projects, clusters, and presets. -- **`postgresql-secret.env`** -- Set your database credentials. -- **`params.env`** -- Set your storage class and size. +- **`postgresql-secret.env`** -- Set your database credentials (PGHOST, PGPORT, PGUSER, PGPASSWORD, PGDATABASE). + +Edit `kustomization.yaml` and replace these values: +- Dashboard container image (e.g. `quay.io/your-org/fournos-dashboard:latest`) +- `FOURNOS_NAMESPACE` -- the namespace where FournosJobs run +- `storageClassName` -- your cluster's storage class (`oc get sc` to list) + +Generate the OAuth cookie secret: + +```bash +# Replace the placeholder in oauth-cookie-secret.yaml +COOKIE=$(openssl rand -base64 32) +# macOS: +sed -i '' "s|REPLACE_ME_WITH_OUTPUT_OF_openssl_rand_base64_32|${COOKIE}|" oauth-cookie-secret.yaml +# Linux: +# sed -i "s|REPLACE_ME_WITH_OUTPUT_OF_openssl_rand_base64_32|${COOKIE}|" oauth-cookie-secret.yaml +``` + +Verify the OAuth proxy image matches your cluster version: + +```bash +# Get the correct image for your cluster (digest may differ per OCP version) +oc adm release info --image-for=oauth-proxy +``` + +If the output differs from what's in `patch-deployment-oauth-proxy.yaml`, update the image field in that file. ### 2. Build and push the dashboard image -### 3. Deploy to the cluster +```bash +podman build -t quay.io/your-org/fournos-dashboard:latest . +podman push quay.io/your-org/fournos-dashboard:latest +``` + +### 3. Deploy TLS certificate (one-time) + +The dashboard uses a Let's Encrypt certificate for trusted HTTPS. This requires cert-manager to be installed on the cluster. ```bash -cd kustomize/overlays/ocp/ +# Apply the ClusterIssuer (cluster-scoped, only needed once) +oc apply -f kustomize/overlays/ocp/letsencrypt-clusterissuer.yaml + +# Verify it's ready +oc get clusterissuer letsencrypt-production +``` + +### 4. Deploy to the cluster +```bash # Apply the main stack -oc kustomize . | oc apply -f - +oc apply -k kustomize/overlays/ocp/ # Apply the cross-namespace RoleBinding (grants dashboard access to the jobs namespace) -oc apply -f rolebinding-psap-automation.yaml +# Replace FOURNOS_NAMESPACE with your target namespace (e.g. psap-automation) +oc apply -f - < ``` -Open http://localhost:8000 +You will be redirected to the OpenShift login page. After authenticating with your cluster credentials, you'll land on the dashboard. Any user who can log into the OpenShift cluster can access the UI. + +### Pulling config from a live cluster + +If the dashboard is already deployed and you need to recreate the overlay files: + +```bash +# projects.yaml +oc get configmap fournos-projects -n fournos-dashboard -o jsonpath='{.data.projects\.yaml}' > projects.yaml + +# postgresql-secret.env +oc get secret postgresql-secret -n fournos-dashboard -o go-template='PGHOST={{index .data "PGHOST" | base64decode}} +PGPORT={{index .data "PGPORT" | base64decode}} +PGUSER={{index .data "PGUSER" | base64decode}} +PGPASSWORD={{index .data "PGPASSWORD" | base64decode}} +PGDATABASE={{index .data "PGDATABASE" | base64decode}} +' > postgresql-secret.env + +# Dashboard image +oc get deployment fournos-dashboard -n fournos-dashboard \ + -o jsonpath='{.spec.template.spec.containers[?(@.name=="dashboard")].image}' + +# Storage class +oc get pvc -n fournos-dashboard -o jsonpath='{.items[0].spec.storageClassName}' +``` ## Configuration @@ -101,9 +205,17 @@ All configuration is via environment variables (set in the deployment manifest): | `KUBECONFIG` | Path to kubeconfig (local dev only) | in-cluster config | | `FORGE_GITHUB_REPO` | GitHub `owner/repo` for PR listing | `openshift-psap/forge` | -## Security Considerations +## Security + +Authentication is handled by the **OpenShift OAuth proxy** sidecar container. The proxy intercepts all requests to the Route, redirects unauthenticated users to the OpenShift login page, and only forwards traffic to the FastAPI app after successful authentication. + +- **Who can access:** Any user who can authenticate to the OpenShift cluster. +- **TLS:** The Route uses a Let's Encrypt certificate (auto-renewed by cert-manager). Traffic between the Route and the pod is re-encrypted using a service-ca cert. +- **Local dev bypass:** When developing locally or using `oc port-forward` to port 8000, the OAuth proxy is bypassed entirely (traffic goes directly to FastAPI). + + + -This dashboard is designed as an **internal tool** and does **not** include built-in authentication or authorization. As described above, the tool is accessible when port-forwarding from the cluster where it's running. Future development may include auth. ## Local Development @@ -130,7 +242,17 @@ fournos-ui/ │ └── templates/ # Jinja2 HTML templates ├── kustomize/ │ ├── base/ # Generic K8s manifests -│ └── overlays/ocp/ # Environment-specific overrides +│ └── overlays/ocp/ # OpenShift deployment overlay +│ ├── kustomization.yaml +│ ├── dashboard-route.yaml # Route with cert-manager annotations +│ ├── dashboard-certificate.yaml # Let's Encrypt Certificate CR +│ ├── letsencrypt-clusterissuer.yaml # ACME ClusterIssuer (apply separately) +│ ├── oauth-cookie-secret.yaml # OAuth proxy session secret +│ ├── patch-deployment-oauth-proxy.yaml # Adds OAuth sidecar to Deployment +│ ├── patch-service-oauth.yaml # Adds TLS port to Service +│ ├── patch-serviceaccount-oauth.yaml # Adds OAuth redirect annotation +│ ├── projects.yaml # (user-created) project config +│ └── postgresql-secret.env # (user-created) DB credentials ├── Dockerfile └── requirements.txt ``` diff --git a/fournos-ui/app/k8s_client.py b/fournos-ui/app/k8s_client.py index abfce55..974b124 100644 --- a/fournos-ui/app/k8s_client.py +++ b/fournos-ui/app/k8s_client.py @@ -355,11 +355,22 @@ def list_pods_for_job(job_name: str, namespace: str | None = None) -> list[dict] container_ready = False restarts = 0 + exit_code = None + term_reason = "" + term_message = "" if pod.status.container_statuses: for cs in pod.status.container_statuses: if cs.ready: container_ready = True restarts += cs.restart_count + terminated = ( + cs.state.terminated if cs.state and cs.state.terminated + else (cs.last_state.terminated if cs.last_state and cs.last_state.terminated else None) + ) + if terminated: + exit_code = terminated.exit_code + term_reason = terminated.reason or "" + term_message = terminated.message or "" if pod.metadata.name.startswith("affinity-assistant"): continue @@ -373,6 +384,9 @@ def list_pods_for_job(job_name: str, namespace: str | None = None) -> list[dict] "ready": container_ready, "restarts": restarts, "age_minutes": age_minutes, + "exit_code": exit_code, + "term_reason": term_reason, + "term_message": term_message, "_created": created, }) pods.sort(key=lambda p: p["_created"] or datetime.min.replace(tzinfo=timezone.utc)) diff --git a/fournos-ui/app/main.py b/fournos-ui/app/main.py index 72e51a0..d364228 100644 --- a/fournos-ui/app/main.py +++ b/fournos-ui/app/main.py @@ -10,6 +10,7 @@ from pathlib import Path from typing import Any +import yaml from fastapi import FastAPI, Form, HTTPException, Query, Request from fastapi.responses import HTMLResponse, RedirectResponse, StreamingResponse from fastapi.staticfiles import StaticFiles @@ -501,12 +502,15 @@ async def delete_history_job(job_name: str): @app.get("/api/jobs/{job_name}/logs/{pod_name}") async def stream_logs(job_name: str, pod_name: str): - """Stream live pod logs via SSE (only for running jobs).""" + """Stream or fetch pod logs via SSE.""" job_pods = await asyncio.to_thread(k8s_client.list_pods_for_job, job_name) - pod_names = {p["name"] for p in job_pods} - if pod_name not in pod_names: + pod_map = {p["name"]: p for p in job_pods} + if pod_name not in pod_map: raise HTTPException(status_code=404, detail="Pod not found for this job") + pod = pod_map[pod_name] + is_running = pod.get("phase") in ("Running", "Pending") + async def generate(): stop = asyncio.Event() queue: asyncio.Queue[str | None] = asyncio.Queue(maxsize=64) @@ -514,7 +518,7 @@ async def generate(): def _reader(): try: - for line in k8s_client.read_pod_log(pod_name, follow=True): + for line in k8s_client.read_pod_log(pod_name, follow=is_running): if stop.is_set(): break try: @@ -550,6 +554,7 @@ async def submit_form(request: Request): "submit_job.html", projects=projects, pipelines=list(settings.default_pipelines), + fournos_namespace=settings.fournos_namespace, ) @@ -594,6 +599,347 @@ async def github_open_prs(): raise HTTPException(status_code=502, detail=f"GitHub API error: {exc}") +_RHAIIS_ORCHESTRATION = "projects/rhaiis/orchestration" + + +def _github_fetch_yaml(path: str) -> dict: + """Fetch a single YAML file from the forge GitHub repo and return parsed content.""" + import urllib.request + import json as _json + + url = f"https://api.github.com/repos/{settings.forge_github_repo}/contents/{path}" + req = urllib.request.Request(url, headers={"Accept": "application/vnd.github+json"}) + with urllib.request.urlopen(req, timeout=15) as resp: + meta = _json.loads(resp.read()) + + download_url = meta.get("download_url", "") + if not download_url: + return {} + + raw_req = urllib.request.Request(download_url) + with urllib.request.urlopen(raw_req, timeout=15) as resp: + return yaml.safe_load(resp.read()) or {} + + +def _github_list_yamls(directory: str) -> list[str]: + """List .yaml file paths in a forge GitHub repo directory.""" + import urllib.request + import json as _json + + url = f"https://api.github.com/repos/{settings.forge_github_repo}/contents/{directory}" + req = urllib.request.Request(url, headers={"Accept": "application/vnd.github+json"}) + with urllib.request.urlopen(req, timeout=15) as resp: + items = _json.loads(resp.read()) + + return sorted( + item["path"] for item in items + if isinstance(item, dict) and item.get("name", "").endswith(".yaml") + ) + + +_CATEGORY_KEYS = { + "rhaiis.accelerator": "accelerator", + "rhaiis.engine": "engine", + "rhaiis.cluster_tag": "cluster", + "tests.rhaiis.run_benchmark": "benchmark", + "tests.rhaiis.model_key": "model", + "tests.rhaiis.workload_key": "workload", +} + +_CATEGORY_PLURAL = { + "accelerator": "accelerators", + "engine": "engines", + "cluster": "clusters", + "benchmark": "benchmarks", + "model": "models", + "workload": "workloads", +} + + +def _parse_cpt_models(raw_models) -> list[dict]: + """Normalize __models from a CPT definition into [{name, preset, overrides, tp}, ...]. + + Keys may contain a ``/suffix`` to allow the same preset multiple times + with different settings (e.g. ``llama-70b/tp2``). The part before ``/`` + is the Forge preset name; the full key is used as the display label. + """ + if isinstance(raw_models, dict): + result = [] + for m, ov in raw_models.items(): + preset = m.split("/")[0] + entry: dict[str, Any] = {"name": m, "preset": preset, "overrides": {}} + if isinstance(ov, dict): + entry["tp"] = ov.pop("__tp", None) + entry["overrides"] = ov + result.append(entry) + return result + return [{"name": m, "preset": m, "overrides": {}} for m in raw_models] + + +def _fetch_rhaiis_config_from_github() -> dict: + """Fetch and categorize rhaiis presets from the forge GitHub repo.""" + config_dir = f"{_RHAIIS_ORCHESTRATION}/config.d" + presets_dir = f"{_RHAIIS_ORCHESTRATION}/presets.d" + + categories: dict[str, list[dict]] = { + "quick_presets": [], + "accelerators": [], + "engines": [], + "clusters": [], + "models": [], + "workloads": [], + "benchmarks": [], + } + + model_display_names: dict[str, str] = {} + model_tp_sizes: dict[str, int] = {} + try: + models_data = _github_fetch_yaml(f"{config_dir}/models.yaml") + for key, val in models_data.items(): + if isinstance(val, dict): + model_display_names[key] = val.get("name", key) + tp = ( + val.get("vllm_args", {}).get("tensor-parallel-size") + or val.get("sglang_args", {}).get("tp-size") + or val.get("tensor_parallel") + or val.get("tp_size") + or val.get("tp") + ) + if tp is not None: + try: + model_tp_sizes[key] = int(tp) + except (ValueError, TypeError): + pass + except Exception as exc: + logger.warning("Failed to fetch models.yaml from GitHub: %s", exc) + + cluster_gpu_types: dict[str, str] = {"hera": "h200", "zeus": "h200"} + try: + clusters_data = _github_fetch_yaml(f"{config_dir}/clusters.yaml") + for key, val in clusters_data.items(): + if isinstance(val, dict): + gpu = val.get("gpu_type", val.get("gpuType", val.get("gpu"))) + if gpu: + cluster_gpu_types[key] = str(gpu) + except Exception as exc: + logger.debug("No clusters.yaml in config.d, using defaults: %s", exc) + + engine_images: dict[str, dict[str, str]] = {} + try: + rhaiis_data = _github_fetch_yaml(f"{config_dir}/rhaiis.yaml") + for ename, edata in (rhaiis_data.get("engines") or {}).items(): + if isinstance(edata, dict): + for accel, img in (edata.get("images") or {}).items(): + if isinstance(img, str): + engine_images.setdefault(ename, {})[accel] = img + except Exception as exc: + logger.debug("Failed to fetch rhaiis.yaml for engine defaults: %s", exc) + + workload_profiles: dict[str, dict] = {} + try: + workloads_data = _github_fetch_yaml(f"{config_dir}/workloads.yaml") + for wk, wv in workloads_data.items(): + if isinstance(wv, dict): + workload_profiles[wk] = wv + except Exception as exc: + logger.debug("Failed to fetch workloads.yaml: %s", exc) + + model_key_to_preset: dict[str, str] = {} + workload_key_to_preset: dict[str, str] = {} + cpt_pipelines: list[dict] = [] + + try: + preset_files = _github_list_yamls(presets_dir) + except Exception as exc: + logger.error("Failed to list presets.d from GitHub: %s", exc) + preset_files = [] + + # First pass: categorize simple presets and detect compound ones + compound_presets: list[tuple[str, dict]] = [] + + for file_path in preset_files: + try: + data = _github_fetch_yaml(file_path) + except Exception as exc: + logger.warning("Failed to fetch %s: %s", file_path, exc) + continue + + if data.get("__cpt"): + for key, entry in data.items(): + if key.startswith("__") or not isinstance(entry, dict): + continue + raw_models = entry.get("__models", []) + models_list = _parse_cpt_models(raw_models) + cpt_pipelines.append({ + "key": key, + "description": entry.get("__description", ""), + "engine": entry.get("__engine", ""), + "accelerator": entry.get("__accelerator", ""), + "models": models_list, + "workloads": entry.get("__workloads", []), + "overrides": { + k: v for k, v in entry.items() + if not k.startswith("__") + }, + }) + continue + + for key, overrides in data.items(): + if key.startswith("__"): + continue + if not isinstance(overrides, dict): + continue + + matched_cats = [ + cat for cat_key, cat in _CATEGORY_KEYS.items() + if cat_key in overrides + ] + + if len(matched_cats) >= 2: + compound_presets.append((key, overrides)) + continue + + if "rhaiis.accelerator" in overrides: + categories["accelerators"].append({"key": key, "label": key.upper(), "overrides": dict(overrides)}) + elif "rhaiis.engine" in overrides: + categories["engines"].append({"key": key, "label": key, "overrides": dict(overrides)}) + elif "rhaiis.cluster_tag" in overrides: + cluster_tag = overrides["rhaiis.cluster_tag"] + entry = {"key": key, "label": key.capitalize(), "overrides": dict(overrides)} + gpu = cluster_gpu_types.get(cluster_tag, cluster_gpu_types.get(key)) + if gpu: + entry["gpu_type"] = gpu + categories["clusters"].append(entry) + elif "tests.rhaiis.run_benchmark" in overrides: + categories["benchmarks"].append({"key": key, "label": key.capitalize(), "overrides": dict(overrides)}) + elif "tests.rhaiis.model_key" in overrides: + model_key = overrides["tests.rhaiis.model_key"] + display = model_display_names.get(model_key, key) + entry = {"key": key, "label": display, "overrides": dict(overrides)} + tp = model_tp_sizes.get(model_key) + if tp: + entry["gpu_count"] = tp + categories["models"].append(entry) + model_key_to_preset[model_key] = key + elif "tests.rhaiis.workload_key" in overrides: + wk = overrides["tests.rhaiis.workload_key"] + entry: dict[str, Any] = {"key": key, "label": key, "overrides": dict(overrides)} + profile = workload_profiles.get(wk) + if profile: + entry["profile"] = profile + categories["workloads"].append(entry) + workload_key_to_preset[wk] = key + + _SETTINGS_KEYS = { + "tests.rhaiis.warmup": "warmup", + "rhaiis.profiler.enabled": "profiler", + "tests.rhaiis.slack_notify_always": "slack", + "rhaiis.agent_analysis.enabled": "agent_analysis", + "caliper.postprocess.csv_dashboard.enabled": "csv_dashboard", + "rhaiis.compare_versions.enabled": "compare_versions", + "tests.rhaiis.run_benchmark": "benchmark", + } + + # Second pass: build quick presets with fill mappings + for key, overrides in compound_presets: + fills: dict[str, Any] = {} + if "tests.rhaiis.model_key" in overrides: + mk = overrides["tests.rhaiis.model_key"] + fills["model"] = model_key_to_preset.get(mk, "") + if "tests.rhaiis.workload_key" in overrides: + wk = overrides["tests.rhaiis.workload_key"] + fills["workload"] = workload_key_to_preset.get(wk, "") + if "tests.rhaiis.version" in overrides: + fills["version"] = overrides["tests.rhaiis.version"] + for cfg_key, fill_key in _SETTINGS_KEYS.items(): + if cfg_key in overrides: + fills[fill_key] = bool(overrides[cfg_key]) + + categories["quick_presets"].append({ + "key": key, + "label": key.replace("-", " ").replace("_", " ").title(), + "fills": fills, + "overrides": dict(overrides), + }) + + engine_defaults: dict[str, str] = {} + for ename, accel_versions in engine_images.items(): + for accel, ver in accel_versions.items(): + engine_defaults[f"{accel}_{ename}"] = ver + + accel_keys = {e["key"] for e in categories["accelerators"]} + engine_keys = {e["key"] for e in categories["engines"]} + invalid_combos = [ + {"accelerator": a, "engine": e} + for a in accel_keys for e in engine_keys + if f"{a}_{e}" not in engine_defaults + ] + + categories["engine_defaults"] = engine_defaults + categories["invalid_combos"] = invalid_combos + categories["workload_profiles"] = workload_profiles + + if not cpt_pipelines: + local_dir = Path(__file__).resolve().parent.parent + for local_cpt in sorted(local_dir.glob("cpt*.yaml")): + try: + with open(local_cpt) as f: + cpt_data = yaml.safe_load(f) or {} + if not cpt_data.get("__cpt"): + continue + for key, entry in cpt_data.items(): + if key.startswith("__") or not isinstance(entry, dict): + continue + raw_models = entry.get("__models", []) + models_list = _parse_cpt_models(raw_models) + cpt_pipelines.append({ + "key": key, + "description": entry.get("__description", ""), + "engine": entry.get("__engine", ""), + "accelerator": entry.get("__accelerator", ""), + "models": models_list, + "workloads": entry.get("__workloads", []), + "overrides": {k: v for k, v in entry.items() if not k.startswith("__")}, + }) + logger.info("Loaded CPT pipeline(s) from local %s", local_cpt) + except Exception as exc: + logger.debug("Failed to load local CPT file %s: %s", local_cpt, exc) + + categories["cpt_pipelines"] = cpt_pipelines + + return categories + + +_rhaiis_config_cache: dict | None = None + + +@app.get("/api/rhaiis-config") +async def rhaiis_config(): + """Return categorized rhaiis preset options for the submit form.""" + global _rhaiis_config_cache + if _rhaiis_config_cache is None: + result = await asyncio.to_thread(_fetch_rhaiis_config_from_github) + if result.get("accelerators") and result.get("engines"): + _rhaiis_config_cache = result + else: + logger.warning("rhaiis config fetch returned incomplete data — not caching") + return result + return _rhaiis_config_cache + + +@app.post("/api/rhaiis-config/refresh") +async def rhaiis_config_refresh(): + """Force-refresh the cached rhaiis config from GitHub.""" + global _rhaiis_config_cache + _rhaiis_config_cache = None + result = await asyncio.to_thread(_fetch_rhaiis_config_from_github) + if result.get("accelerators") and result.get("engines"): + _rhaiis_config_cache = result + return {"status": "ok", "accelerators": len(result.get("accelerators", [])), + "engines": len(result.get("engines", [])), + "models": len(result.get("models", []))} + + @app.post("/submit") async def submit_job( request: Request, @@ -606,6 +952,12 @@ async def submit_job( exclusive: str = Form("false"), config_overrides_raw: str = Form(""), pull_sha: str = Form(""), + rhaiis_args: str = Form(""), + rhaiis_version: str = Form(""), + rhaiis_overrides: str = Form(""), + priority: str = Form("manual"), + gpu_type: str = Form(""), + gpu_count: str = Form("1"), ): exclusive_bool = exclusive.lower() in ("true", "on", "1", "yes") @@ -621,36 +973,65 @@ async def submit_job( version_key = _get_version_config_key(project) config_overrides[version_key] = version - args = [preset] if preset else [] + display_name = f"{project} {preset}".strip() + + if project == "rhaiis" and rhaiis_args.strip(): + args = [a.strip() for a in rhaiis_args.split(",") if a.strip()] + display_name = f"rhaiis-{cluster}-{'-'.join(args[:2])}" if args else f"rhaiis-{cluster}" + if rhaiis_version.strip(): + config_overrides["tests.rhaiis.version"] = rhaiis_version.strip() + if rhaiis_overrides.strip(): + import json as _json + try: + rh_ov = _json.loads(rhaiis_overrides) + if isinstance(rh_ov, dict): + config_overrides.update(rh_ov) + except (ValueError, TypeError): + pass + else: + args = [preset] if preset else [] - job_name = sanitize_job_name(f"forge-{project}") + generate_name = f"rhaiis-{cluster}-" if project == "rhaiis" else f"forge-{project}-" pull_sha = pull_sha.strip() env: dict[str, str] = {} if pull_sha: env["PULL_PULL_SHA"] = pull_sha + spec: dict[str, Any] = { + "cluster": cluster, + "displayName": display_name, + "owner": owner or "fournos-dashboard", + "pipeline": pipeline, + "exclusive": exclusive_bool, + "priority": priority, + "executionEngine": { + "forge": { + "project": project, + "args": args, + "configOverrides": config_overrides, + } + }, + } + + try: + gpu_count_int = int(gpu_count) if gpu_count.strip() else 1 + except ValueError: + gpu_count_int = 1 + if gpu_type.strip(): + spec["hardware"] = {"gpuType": gpu_type.strip(), "gpuCount": gpu_count_int} + + if project == "rhaiis": + spec["secretRefs"] = ["psap-forge-dashboard-s3", "psap-forge-notifications"] + body = { "apiVersion": f"{settings.fournos_api_group}/{settings.fournos_api_version}", "kind": "FournosJob", "metadata": { - "name": job_name, + "generateName": generate_name, "namespace": settings.fournos_namespace, }, - "spec": { - "cluster": cluster, - "displayName": f"{project} {preset}".strip(), - "owner": owner or "fournos-dashboard", - "pipeline": pipeline, - "exclusive": exclusive_bool, - "executionEngine": { - "forge": { - "project": project, - "args": args, - "configOverrides": config_overrides, - } - }, - }, + "spec": spec, } if env: @@ -664,10 +1045,11 @@ async def submit_job( "submit_job.html", projects=projects, pipelines=list(settings.default_pipelines), + fournos_namespace=settings.fournos_namespace, error=str(exc), ) - created_name = created.get("metadata", {}).get("name", job_name) + created_name = created.get("metadata", {}).get("name", generate_name) try: async with db.async_session() as session: @@ -690,6 +1072,127 @@ async def submit_job( return RedirectResponse(url=f"/jobs/{created_name}", status_code=303) +@app.post("/api/submit-cpt") +async def submit_cpt(request: Request): + """Submit a CPT pipeline — creates one FournosJob per model.""" + import json as _json + + payload = await request.json() + models: list[str] = payload.get("models", []) + workloads: list[str] = payload.get("workloads", []) + accelerator: str = payload.get("accelerator", "nvidia") + engine: str = payload.get("engine", "vllm") + cluster: str = payload.get("cluster", "hera") + pipeline: str = payload.get("pipeline", "forge-full") + owner: str = payload.get("owner", "fournos-dashboard") + priority: str = payload.get("priority", "manual") + version_label: str = payload.get("version_label", "") + pull_sha: str = payload.get("pull_sha", "") + overrides: dict = payload.get("overrides", {}) + engine_version: str = payload.get("engine_version", "") + + if not models or not workloads: + raise HTTPException(status_code=400, detail="models and workloads are required") + + config = _rhaiis_config_cache or await asyncio.to_thread(_fetch_rhaiis_config_from_github) + model_entries = {m["key"]: m for m in config.get("models", [])} + cluster_entries = {c["key"]: c for c in config.get("clusters", [])} + gpu_type = cluster_entries.get(cluster, {}).get("gpu_type", "h200") + + results = [] + for model_item in models: + if isinstance(model_item, dict): + model_preset = model_item.get("preset", model_item.get("name", "")) + model_label = model_item.get("name", model_preset) + per_model_overrides = model_item.get("overrides", {}) + else: + model_preset = model_item + model_label = model_item + per_model_overrides = {} + + model_entry = model_entries.get(model_preset, {}) + model_preset_overrides = model_entry.get("overrides", {}) + model_key = model_preset_overrides.get("tests.rhaiis.model_key", model_preset) + cpt_tp = model_item.get("tp") if isinstance(model_item, dict) else None + gpu_count = cpt_tp or model_entry.get("gpu_count", 1) + + args = [accelerator, engine, cluster, model_preset] + + job_overrides: dict[str, Any] = {} + job_overrides.update(overrides) + job_overrides.update(per_model_overrides) + job_overrides["tests.rhaiis.workload_keys"] = workloads + if version_label: + job_overrides["tests.rhaiis.version"] = version_label + if engine_version: + job_overrides[f"rhaiis.engines.{engine}.images.{accelerator}"] = engine_version + + display_name = f"rhaiis-cpt-{model_preset}-{cluster}" + generate_name = f"rhaiis-cpt-{cluster}-" + + env: dict[str, str] = {} + if pull_sha.strip(): + env["PULL_PULL_SHA"] = pull_sha.strip() + + spec: dict[str, Any] = { + "cluster": cluster, + "displayName": display_name, + "owner": owner, + "pipeline": pipeline, + "exclusive": False, + "priority": priority, + "hardware": {"gpuType": gpu_type, "gpuCount": gpu_count}, + "secretRefs": ["psap-forge-dashboard-s3", "psap-forge-notifications"], + "executionEngine": { + "forge": { + "project": "rhaiis", + "args": args, + "configOverrides": job_overrides, + } + }, + } + + body = { + "apiVersion": f"{settings.fournos_api_group}/{settings.fournos_api_version}", + "kind": "FournosJob", + "metadata": { + "generateName": generate_name, + "namespace": settings.fournos_namespace, + }, + "spec": spec, + } + + if env: + body["spec"]["env"] = env + + try: + created = await asyncio.to_thread(k8s_client.create_fournos_job, body) + created_name = created.get("metadata", {}).get("name", generate_name) + results.append({"model": model_label, "job_name": created_name, "status": "created"}) + + try: + async with db.async_session() as session: + async with session.begin(): + await db.upsert_job( + session, + name=created_name, + project="rhaiis", + preset=f"cpt-{model_preset}", + cluster=cluster, + pipeline=pipeline, + owner=owner, + status="Pending", + config_overrides=job_overrides, + fjob_spec=body.get("spec", {}), + ) + except Exception as exc: + logger.error("DB upsert failed for CPT job %s: %s", created_name, exc) + except Exception as exc: + results.append({"model": model_label, "error": str(exc), "status": "failed"}) + + return {"status": "ok", "jobs": results, "total": len(results)} + + # --------------------------------------------------------------------------- # Routes: Schedules # --------------------------------------------------------------------------- diff --git a/fournos-ui/app/static/style.css b/fournos-ui/app/static/style.css index 9733a1b..bd566ed 100644 --- a/fournos-ui/app/static/style.css +++ b/fournos-ui/app/static/style.css @@ -1,20 +1,22 @@ :root { - --bg-primary: #0d1117; - --bg-secondary: #161b22; - --bg-tertiary: #21262d; - --border: #30363d; - --text-primary: #e6edf3; - --text-secondary: #8b949e; - --text-muted: #6e7681; - --accent-blue: #58a6ff; - --accent-green: #3fb950; - --accent-red: #f85149; - --accent-yellow: #d29922; - --accent-purple: #bc8cff; - --accent-orange: #f0883e; + --bg-primary: #ffffff; + --bg-secondary: #f6f8fa; + --bg-tertiary: #eaeef2; + --border: #d0d7de; + --text-primary: #1f2328; + --text-secondary: #656d76; + --text-muted: #8b949e; + --accent-blue: #0969da; + --accent-green: #1a7f37; + --accent-red: #cf222e; + --accent-yellow: #9a6700; + --accent-purple: #8250df; + --accent-orange: #bc4c00; + --shadow-sm: 0 1px 3px rgba(31, 35, 40, 0.06); + --shadow-md: 0 3px 12px rgba(31, 35, 40, 0.08); --font-mono: 'SF Mono', 'Cascadia Code', 'Fira Code', monospace; --font-sans: -apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif; - --radius: 6px; + --radius: 8px; } * { margin: 0; padding: 0; box-sizing: border-box; } @@ -37,12 +39,13 @@ a:hover { text-decoration: underline; } /* Header */ header { - background: var(--bg-secondary); + background: var(--bg-primary); border-bottom: 1px solid var(--border); - padding: 12px 0; + padding: 14px 0; position: sticky; top: 0; z-index: 100; + box-shadow: var(--shadow-sm); } header .container { @@ -58,9 +61,9 @@ header .container { } .logo { - font-size: 20px; - font-weight: 600; - letter-spacing: -0.3px; + font-size: 22px; + font-weight: 700; + letter-spacing: -0.4px; color: var(--text-primary); text-decoration: none; } @@ -94,7 +97,7 @@ header .container { } .nav-link.active { - background: rgba(88, 166, 255, 0.1); + background: rgba(9, 105, 218, 0.08); color: var(--accent-blue); } @@ -136,11 +139,11 @@ main { padding: 24px 0; } .filter-bar select, .filter-bar input { - background: var(--bg-secondary); + background: var(--bg-primary); border: 1px solid var(--border); border-radius: var(--radius); color: var(--text-primary); - padding: 6px 12px; + padding: 8px 12px; font-size: 13px; outline: none; } @@ -152,11 +155,12 @@ main { padding: 24px 0; } /* Jobs table wrapper: handles border + radius */ .table-wrap { - background: var(--bg-secondary); + background: var(--bg-primary); border: 1px solid var(--border); border-radius: var(--radius); overflow: visible; position: relative; + box-shadow: var(--shadow-sm); } /* Jobs table */ @@ -164,7 +168,7 @@ main { padding: 24px 0; } width: 100%; border-collapse: separate; border-spacing: 0; - background: var(--bg-secondary); + background: var(--bg-primary); } .jobs-table th { @@ -193,7 +197,7 @@ main { padding: 24px 0; } .jobs-table tbody tr:last-child td:first-child { border-bottom-left-radius: var(--radius); } .jobs-table tbody tr:last-child td:last-child { border-bottom-right-radius: var(--radius); } -.jobs-table tr:hover td { background: rgba(88, 166, 255, 0.04); } +.jobs-table tr:hover td { background: rgba(9, 105, 218, 0.03); } .job-name { font-family: var(--font-mono); @@ -214,33 +218,33 @@ main { padding: 24px 0; } } .phase-running { - background: rgba(210, 153, 34, 0.15); + background: rgba(154, 103, 0, 0.08); color: var(--accent-yellow); - border: 1px solid rgba(210, 153, 34, 0.3); + border: 1px solid rgba(154, 103, 0, 0.2); } .phase-succeeded { - background: rgba(63, 185, 80, 0.15); + background: rgba(26, 127, 55, 0.08); color: var(--accent-green); - border: 1px solid rgba(63, 185, 80, 0.3); + border: 1px solid rgba(26, 127, 55, 0.2); } .phase-failed { - background: rgba(248, 81, 73, 0.15); + background: rgba(207, 34, 46, 0.08); color: var(--accent-red); - border: 1px solid rgba(248, 81, 73, 0.3); + border: 1px solid rgba(207, 34, 46, 0.2); } .phase-stopped { - background: rgba(139, 148, 158, 0.15); + background: rgba(101, 109, 118, 0.08); color: var(--text-secondary); - border: 1px solid rgba(139, 148, 158, 0.3); + border: 1px solid rgba(101, 109, 118, 0.2); } .phase-resolving { - background: rgba(188, 140, 255, 0.15); + background: rgba(130, 80, 223, 0.08); color: var(--accent-purple); - border: 1px solid rgba(188, 140, 255, 0.3); + border: 1px solid rgba(130, 80, 223, 0.2); } .phase-unknown { @@ -291,8 +295,8 @@ main { padding: 24px 0; } .detail-header h2 { font-family: var(--font-mono); - font-size: 18px; - font-weight: 600; + font-size: 22px; + font-weight: 700; } .back-link { @@ -312,20 +316,21 @@ main { padding: 24px 0; } } .card { - background: var(--bg-secondary); + background: var(--bg-primary); border: 1px solid var(--border); border-radius: var(--radius); overflow: hidden; + box-shadow: var(--shadow-sm); } .card-header { - padding: 12px 16px; - font-size: 13px; + padding: 14px 18px; + font-size: 15px; font-weight: 600; - color: var(--text-secondary); - text-transform: uppercase; - letter-spacing: 0.5px; - background: var(--bg-tertiary); + color: var(--text-primary); + text-transform: none; + letter-spacing: 0; + background: var(--bg-secondary); border-bottom: 1px solid var(--border); } @@ -374,9 +379,9 @@ main { padding: 24px 0; } margin-top: 2px; } -.condition-true { background: rgba(63, 185, 80, 0.2); color: var(--accent-green); } -.condition-false { background: rgba(248, 81, 73, 0.2); color: var(--accent-red); } -.condition-unknown { background: rgba(210, 153, 34, 0.2); color: var(--accent-yellow); } +.condition-true { background: rgba(26, 127, 55, 0.1); color: var(--accent-green); } +.condition-false { background: rgba(207, 34, 46, 0.1); color: var(--accent-red); } +.condition-unknown { background: rgba(154, 103, 0, 0.1); color: var(--accent-yellow); } .condition-info { flex: 1; } .condition-type { font-weight: 600; color: var(--text-primary); } @@ -409,7 +414,7 @@ main { padding: 24px 0; } /* Log viewer */ .log-viewer { - background: #010409; + background: #f6f8fa; border: 1px solid var(--border); border-radius: var(--radius); padding: 16px; @@ -420,34 +425,34 @@ main { padding: 24px 0; } overflow-y: auto; white-space: pre-wrap; word-break: break-all; - color: var(--text-secondary); + color: var(--text-primary); margin-top: 20px; } .log-viewer .log-line { display: block; } -.log-viewer .log-info { color: #58a6ff; } -.log-viewer .log-warn { color: #d29922; } -.log-viewer .log-error { color: #f85149; } -.log-viewer .log-task { color: #3fb950; font-weight: 600; } -.log-viewer .log-command { color: #bc8cff; } -.log-viewer .log-separator { color: #30363d; } +.log-viewer .log-info { color: var(--accent-blue); } +.log-viewer .log-warn { color: var(--accent-yellow); } +.log-viewer .log-error { color: var(--accent-red); } +.log-viewer .log-task { color: var(--accent-green); font-weight: 600; } +.log-viewer .log-command { color: var(--accent-purple); } +.log-viewer .log-separator { color: var(--border); } .owner-badge { font-size: 12px; padding: 2px 8px; border-radius: 12px; - background: rgba(88, 166, 255, 0.1); + background: rgba(9, 105, 218, 0.08); color: var(--accent-blue); - border: 1px solid rgba(88, 166, 255, 0.2); + border: 1px solid rgba(9, 105, 218, 0.15); } .project-badge { font-size: 12px; padding: 2px 8px; border-radius: 12px; - background: rgba(188, 140, 255, 0.1); + background: rgba(130, 80, 223, 0.08); color: var(--accent-purple); - border: 1px solid rgba(188, 140, 255, 0.2); + border: 1px solid rgba(130, 80, 223, 0.15); font-family: var(--font-mono); } @@ -464,19 +469,20 @@ main { padding: 24px 0; } /* Pipeline Timeline */ .pipeline-timeline { - background: var(--bg-secondary); + background: var(--bg-primary); border: 1px solid var(--border); border-radius: var(--radius); padding: 20px 24px; margin-bottom: 24px; + box-shadow: var(--shadow-sm); } .pipeline-timeline-header { - font-size: 13px; + font-size: 15px; font-weight: 600; - color: var(--text-secondary); - text-transform: uppercase; - letter-spacing: 0.5px; + color: var(--text-primary); + text-transform: none; + letter-spacing: 0; margin-bottom: 16px; } @@ -512,27 +518,27 @@ main { padding: 24px 0; } } .tl-succeeded { - background: rgba(63, 185, 80, 0.25); + background: rgba(26, 127, 55, 0.1); color: var(--accent-green); - border: 1px solid rgba(63, 185, 80, 0.4); + border: 1px solid rgba(26, 127, 55, 0.25); } .tl-running { - background: rgba(210, 153, 34, 0.25); + background: rgba(154, 103, 0, 0.1); color: var(--accent-yellow); - border: 1px solid rgba(210, 153, 34, 0.4); + border: 1px solid rgba(154, 103, 0, 0.25); animation: tl-pulse 2s infinite; } @keyframes tl-pulse { - 0%, 100% { background: rgba(210, 153, 34, 0.25); } - 50% { background: rgba(210, 153, 34, 0.15); } + 0%, 100% { background: rgba(154, 103, 0, 0.1); } + 50% { background: rgba(154, 103, 0, 0.05); } } .tl-failed { - background: rgba(248, 81, 73, 0.25); + background: rgba(207, 34, 46, 0.1); color: var(--accent-red); - border: 1px solid rgba(248, 81, 73, 0.4); + border: 1px solid rgba(207, 34, 46, 0.25); } .tl-pending { @@ -542,13 +548,13 @@ main { padding: 24px 0; } } .tl-cancelled { - background: rgba(139, 148, 158, 0.15); + background: rgba(101, 109, 118, 0.08); color: var(--text-secondary); - border: 1px solid rgba(139, 148, 158, 0.3); + border: 1px solid rgba(101, 109, 118, 0.2); } .tl-skipped { - background: rgba(139, 148, 158, 0.08); + background: rgba(101, 109, 118, 0.04); color: var(--text-muted); border: 1px dashed var(--border); } @@ -653,29 +659,35 @@ main { padding: 24px 0; } .btn:hover { background: var(--bg-tertiary); text-decoration: none; } .btn-primary { - background: rgba(88, 166, 255, 0.15); - color: var(--accent-blue); - border-color: rgba(88, 166, 255, 0.3); + background: var(--accent-blue); + color: #ffffff; + border-color: var(--accent-blue); } -.btn-primary:hover { background: rgba(88, 166, 255, 0.25); } +.btn-primary:hover { background: #0860ca; border-color: #0860ca; } .btn-secondary { color: var(--text-secondary); } .btn-danger { color: var(--accent-red); - border-color: rgba(248, 81, 73, 0.3); + border-color: rgba(207, 34, 46, 0.3); } -.btn-danger:hover { background: rgba(248, 81, 73, 0.1); } +.btn-danger:hover { background: rgba(207, 34, 46, 0.06); } .btn-sm { padding: 4px 10px; font-size: 12px; } /* Forms */ -.page-header { margin-bottom: 24px; } -.page-header h2 { margin-bottom: 4px; } +.page-header { margin-bottom: 28px; } +.page-header h2 { + font-size: 26px; + font-weight: 700; + letter-spacing: -0.3px; + margin-bottom: 6px; + color: var(--text-primary); +} -.submit-form { max-width: 900px; } +.submit-form { max-width: none; } .form-grid { display: grid; @@ -755,8 +767,8 @@ main { padding: 24px 0; } } .alert-error { - background: rgba(248, 81, 73, 0.1); - border: 1px solid rgba(248, 81, 73, 0.3); + background: rgba(207, 34, 46, 0.06); + border: 1px solid rgba(207, 34, 46, 0.2); color: var(--accent-red); } @@ -791,20 +803,20 @@ main { padding: 24px 0; } } .banner-success { - background: rgba(63, 185, 80, 0.1); - border: 1px solid rgba(63, 185, 80, 0.3); + background: rgba(26, 127, 55, 0.06); + border: 1px solid rgba(26, 127, 55, 0.2); color: var(--accent-green); } .banner-error { - background: rgba(248, 81, 73, 0.1); - border: 1px solid rgba(248, 81, 73, 0.3); + background: rgba(207, 34, 46, 0.06); + border: 1px solid rgba(207, 34, 46, 0.2); color: var(--accent-red); } .banner-stopped { - background: rgba(139, 148, 158, 0.1); - border: 1px solid rgba(139, 148, 158, 0.3); + background: rgba(101, 109, 118, 0.06); + border: 1px solid rgba(101, 109, 118, 0.2); color: var(--text-secondary); } @@ -821,13 +833,14 @@ main { padding: 24px 0; } align-items: center; gap: 6px; padding: 8px 16px; - background: var(--bg-secondary); + background: var(--bg-primary); border: 1px solid var(--border); border-radius: var(--radius); font-size: 13px; font-weight: 500; color: var(--accent-blue); transition: background 0.15s; + box-shadow: var(--shadow-sm); } .result-link:hover { background: var(--bg-tertiary); text-decoration: none; } @@ -850,9 +863,9 @@ main { padding: 24px 0; } font-size: 11px; padding: 2px 8px; border-radius: 12px; - background: rgba(240, 136, 62, 0.1); + background: rgba(188, 76, 0, 0.08); color: var(--accent-orange); - border: 1px solid rgba(240, 136, 62, 0.2); + border: 1px solid rgba(188, 76, 0, 0.15); } /* Current step info (live table) */ @@ -885,9 +898,9 @@ main { padding: 24px 0; } font-size: 11px; padding: 1px 6px; border-radius: 10px; - background: rgba(210, 153, 34, 0.12); + background: rgba(154, 103, 0, 0.08); color: var(--accent-yellow); - border: 1px solid rgba(210, 153, 34, 0.25); + border: 1px solid rgba(154, 103, 0, 0.2); white-space: nowrap; } @@ -923,10 +936,10 @@ main { padding: 24px 0; } top: 100%; z-index: 50; min-width: 160px; - background: var(--bg-secondary); + background: var(--bg-primary); border: 1px solid var(--border); border-radius: var(--radius); - box-shadow: 0 8px 24px rgba(0, 0, 0, 0.4); + box-shadow: var(--shadow-md); padding: 4px 0; flex-direction: column; } @@ -971,7 +984,7 @@ main { padding: 24px 0; } } .actions-menu-item.actions-danger:hover { - background: rgba(248, 81, 73, 0.08); + background: rgba(207, 34, 46, 0.06); } /* Toast notifications */ @@ -984,10 +997,10 @@ main { padding: 24px 0; } border-radius: var(--radius); font-size: 13px; font-weight: 500; - background: var(--bg-secondary); + background: var(--bg-primary); color: var(--text-primary); border: 1px solid var(--border); - box-shadow: 0 8px 24px rgba(0, 0, 0, 0.4); + box-shadow: var(--shadow-md); opacity: 0; transform: translateY(12px); transition: opacity 0.25s, transform 0.25s; @@ -1000,6 +1013,6 @@ main { padding: 24px 0; } } .toast-error { - border-color: rgba(248, 81, 73, 0.3); + border-color: rgba(207, 34, 46, 0.3); color: var(--accent-red); } diff --git a/fournos-ui/app/templates/components/job_detail_dynamic.html b/fournos-ui/app/templates/components/job_detail_dynamic.html index 6bd1ef7..581b482 100644 --- a/fournos-ui/app/templates/components/job_detail_dynamic.html +++ b/fournos-ui/app/templates/components/job_detail_dynamic.html @@ -22,6 +22,9 @@ {% if phase in ('Succeeded', 'Failed', 'Stopped') %}
{% if phase == 'Succeeded' %}Execution succeeded{% elif phase == 'Failed' %}Execution failed{% else %}Execution stopped{% endif %} + {% if phase == 'Failed' and job.status.message %} +
{{ job.status.message }}
+ {% endif %}
{% endif %} @@ -123,7 +126,8 @@ Name Status - Ready + Reason + Exit Code Restarts Age Logs @@ -139,16 +143,28 @@ {{ pod.phase }} - {{ "Yes" if pod.ready else "No" }} + + {% if pod.term_reason %} + + {{ pod.term_reason }} + + {% else %} + - + {% endif %} + + + {% if pod.exit_code is not none %} + {{ pod.exit_code }} + {% else %} + - + {% endif %} + {{ pod.restarts }} {{ pod.age_minutes }}m - {% if phase in ("Running", "Pending", "Admitted") %} View logs - {% else %} - - - {% endif %} {% endfor %} diff --git a/fournos-ui/app/templates/job_detail.html b/fournos-ui/app/templates/job_detail.html index a947e0f..9c48fc8 100644 --- a/fournos-ui/app/templates/job_detail.html +++ b/fournos-ui/app/templates/job_detail.html @@ -33,6 +33,9 @@

{{ job.metadata.name }}

{% if phase == 'Succeeded' %}Execution succeeded{% elif phase == 'Failed' %}Execution failed{% else %}Execution stopped{% endif %} {% if job.get('_duration_seconds') is not none %} after {{ format_duration(job['_duration_seconds']) }}{% endif %} + {% if phase == 'Failed' and job.get('status', {}).get('message') %} +
{{ job.status.message }}
+ {% endif %}
{% endif %} @@ -58,26 +61,26 @@

{{ job.metadata.name }}

@@ -135,12 +138,12 @@

{{ job.metadata.name }}

-{% if not is_history and phase in ("Running", "Pending", "Admitted") %} +{% if not is_history %}
Logs
- Select a pod above to view its logs. + Click "View logs" on a pod above to see its output.
@@ -148,6 +151,7 @@

{{ job.metadata.name }}

{% endblock %} diff --git a/fournos-ui/app/templates/jobs_list.html b/fournos-ui/app/templates/jobs_list.html index 1c1d0a7..c1482c0 100644 --- a/fournos-ui/app/templates/jobs_list.html +++ b/fournos-ui/app/templates/jobs_list.html @@ -140,10 +140,17 @@ Run Again + {% if job.phase in ('Succeeded', 'Failed', 'Stopped') %} + {% elif job.phase in ('Running', 'Pending', 'Resolving', 'Admitted') %} + + {% endif %} @@ -243,6 +250,22 @@ .catch(err => showToast('Error: ' + err.message, true)); } +function stopHistoryJob(jobName, btn) { + if (!confirm('Stop job ' + jobName + '?')) return; + const menu = btn.closest('.actions-menu'); + menu.classList.remove('open'); + fetch('/api/jobs/' + jobName + '/cancel', { method: 'POST' }) + .then(r => r.json()) + .then(data => { + if (data.status === 'ok') { + showToast('Stop requested for ' + jobName); + } else { + showToast('Error: ' + (data.detail || 'Unknown error'), true); + } + }) + .catch(err => showToast('Error: ' + err.message, true)); +} + function deleteHistoryJob(jobName, btn) { if (!confirm('Delete job ' + jobName + ' from history? This cannot be undone.')) return; const menu = btn.closest('.actions-menu'); diff --git a/fournos-ui/app/templates/schedules.html b/fournos-ui/app/templates/schedules.html index 4805b5e..61ed9fc 100644 --- a/fournos-ui/app/templates/schedules.html +++ b/fournos-ui/app/templates/schedules.html @@ -88,7 +88,7 @@

Version Resolver (optional)

Version Resolver (optional) - +