-
Notifications
You must be signed in to change notification settings - Fork 7
Add fournos-ui dashboard #97
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 5 commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
72aa751
Add fournos-ui dashboard
ashtarkb ba9481e
fournos-ui: harden deployment, fix async I/O, and improve RBAC model
ashtarkb 1a375cb
fournos-ui: add PR commit SHA field and conditional version display
ashtarkb 387fd00
fournos-ui: fix SSE reader leak, cluster filter, and filename mismatch
ashtarkb a396a53
fournos-ui: add GitHub PR picker and sort pods by creation time
ashtarkb 58407c3
fournos-ui: offload GitHub fetch to thread to avoid blocking event loop
ashtarkb File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| __pycache__ | ||
| *.pyc | ||
| *.pyo | ||
| .git | ||
| .gitignore | ||
| .env | ||
| .env.* | ||
| *.md | ||
| .venv | ||
| venv | ||
| .mypy_cache | ||
| .pytest_cache | ||
| .ruff_cache | ||
| .DS_Store | ||
|
|
||
| app/mock_data.py | ||
|
|
||
| kustomize/ | ||
| resolvers/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| # Secrets - never commit actual credentials | ||
| kustomize/overlays/*/postgresql-secret.env | ||
| *secret*.env | ||
|
|
||
| # Python | ||
| __pycache__/ | ||
| *.pyc | ||
| *.pyo | ||
| .venv/ | ||
| venv/ | ||
|
|
||
| # Database files | ||
| *.db | ||
| *.sqlite | ||
| *.sqlite3 | ||
|
|
||
| # IDE | ||
| .idea/ | ||
| .vscode/ | ||
|
|
||
| # Environment | ||
| .env | ||
| .env.* | ||
| !.env.example | ||
|
|
||
| # OS | ||
| .DS_Store | ||
|
|
||
| # Dev-only | ||
| app/mock_data.py | ||
|
|
||
| # Local overlay overrides (users create their own from *.example) | ||
| kustomize/overlays/*/params.env | ||
| kustomize/overlays/*/projects.yaml | ||
| kustomize/overlays/*/kustomization.yaml | ||
| kustomize/overlays/*/rolebinding-*.yaml |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| FROM python:3.12-slim AS base | ||
|
|
||
| RUN groupadd --gid 1001 app && \ | ||
| useradd --uid 1001 --gid app --create-home app | ||
|
|
||
| WORKDIR /opt/fournos-dashboard | ||
|
|
||
| COPY requirements.txt . | ||
| RUN pip install --no-cache-dir -r requirements.txt | ||
|
|
||
| COPY app/ app/ | ||
|
|
||
| RUN chown -R app:app /opt/fournos-dashboard | ||
|
|
||
| USER app | ||
|
|
||
| EXPOSE 8000 | ||
|
|
||
| ENTRYPOINT ["uvicorn", "app.main:app", \ | ||
| "--host", "0.0.0.0", \ | ||
| "--port", "8000", \ | ||
| "--workers", "1", \ | ||
| "--log-level", "info"] | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,136 @@ | ||
| # Fournos Dashboard | ||
|
|
||
| A web dashboard for managing [Fournos](https://github.com/openshift-psap/fournos-operator) performance testing jobs on Kubernetes. Submit jobs, monitor live runs, schedule recurring tests, and review historical results -- all from one place. | ||
|
|
||
| ## What It Does | ||
|
|
||
| - **Live job monitoring** -- Watch running FournosJobs with real-time log streaming (SSE) and pipeline progress tracking. | ||
| - **Job submission** -- Submit new FournosJobs with project, preset, cluster, and config override selection. Optionally pick an open Forge PR to test with -- the dashboard fetches open PRs from GitHub and fills in the commit SHA automatically. | ||
| - **GitHub PR integration** -- Lists open pull requests from the [Forge repo](https://github.com/openshift-psap/forge) directly in the submit form. Since Forge is a public repository, no GitHub token is needed. | ||
| - **Scheduling** -- Create Kubernetes CronJobs for recurring test runs, with optional version-resolver scripts that dynamically determine parameters at runtime. | ||
| - **History** -- Browse completed jobs stored in PostgreSQL with status, duration, and direct links to MLflow artifacts. | ||
| - **Schedule tracking** -- See which schedule triggered each job (manual vs. scheduled) and view all runs for a given schedule. | ||
|
|
||
| ## Architecture | ||
|
|
||
| ``` | ||
| ┌─────────────┐ ┌──────────────────┐ ┌────────────┐ | ||
| │ Browser │────▶│ FastAPI + HTMX │────▶│ Kubernetes │ | ||
| │ │◀────│ (Dashboard) │◀────│ API │ | ||
| └─────────────┘ └────────┬─────────┘ └────────────┘ | ||
| │ | ||
| ┌────────▼─────────┐ | ||
| │ PostgreSQL │ | ||
| │ (job history) │ | ||
| └──────────────────┘ | ||
| ``` | ||
|
|
||
| - **FastAPI** backend with **Jinja2** templates and **HTMX** for dynamic updates. | ||
| - **Kubernetes Python client** for watching FournosJob CRs, streaming pod logs, and managing CronJobs. | ||
| - **PostgreSQL** (via SQLAlchemy async + asyncpg) for persisting job metadata and schedule tracking. | ||
| - A background **watcher thread** monitors FournosJob events and archives them to PostgreSQL automatically. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - A Kubernetes / OpenShift cluster with the [Fournos Operator](https://github.com/openshift-psap/fournos-operator) installed. | ||
| - A container registry to push the dashboard image. | ||
| - `kubectl` or `oc` CLI configured with cluster access. | ||
|
|
||
| ## Getting Started | ||
|
|
||
| ### 1. Clone and configure the overlay | ||
|
|
||
| ```bash | ||
| cd kustomize/overlays/ocp/ | ||
|
|
||
| # Copy example files | ||
| cp kustomization.yaml.example kustomization.yaml | ||
| cp projects.yaml.example projects.yaml | ||
| cp params.env.example params.env | ||
| cp ../../base/postgresql-secret.env.example postgresql-secret.env | ||
| ``` | ||
|
|
||
| Edit each file with your values: | ||
| - **`kustomization.yaml`** -- Set your dashboard image, PostgreSQL image, target namespace, and storage class. | ||
| - **`projects.yaml`** -- Define your Forge projects, clusters, and presets. | ||
| - **`postgresql-secret.env`** -- Set your database credentials. | ||
| - **`params.env`** -- Set your storage class and size. | ||
|
|
||
| ### 2. Build and push the dashboard image | ||
|
|
||
| ### 3. Deploy to the cluster | ||
|
|
||
| ```bash | ||
| cd kustomize/overlays/ocp/ | ||
|
|
||
| # Apply the main stack | ||
| oc kustomize . | oc apply -f - | ||
|
|
||
| # Apply the cross-namespace RoleBinding (grants dashboard access to the jobs namespace) | ||
| oc apply -f rolebinding-psap-automation.yaml | ||
| ``` | ||
|
|
||
| This creates: | ||
| - A `fournos-dashboard` namespace | ||
| - PostgreSQL StatefulSet with persistent storage | ||
| - Dashboard Deployment, Service, ServiceAccount | ||
| - ClusterRole for FournosJob/CronJob/Pod access | ||
| - RoleBinding in the target namespace (e.g. `psap-automation`) | ||
| - Projects ConfigMap | ||
|
|
||
| ### 4. Access the dashboard | ||
|
|
||
| ```bash | ||
| oc port-forward -n fournos-dashboard svc/fournos-dashboard 8000:8000 | ||
| ``` | ||
|
|
||
| Open http://localhost:8000 | ||
|
|
||
|
|
||
| ## Configuration | ||
|
|
||
| All configuration is via environment variables (set in the deployment manifest): | ||
|
|
||
| | Variable | Description | Default | | ||
| |---|---|---| | ||
| | `DATABASE_URL` | PostgreSQL connection string (required) | *none -- must be set* | | ||
| | `FOURNOS_NAMESPACE` | Namespace where FournosJobs run | *set via overlay* | | ||
| | `PROJECTS_CONFIG_PATH` | Path to projects YAML | `/etc/fournos-dashboard/projects.yaml` | | ||
| | `K8S_REQUEST_TIMEOUT` | Timeout for K8s API calls (seconds) | `30` | | ||
| | `LOG_LEVEL` | Logging level | `INFO` | | ||
| | `KUBECONFIG` | Path to kubeconfig (local dev only) | in-cluster config | | ||
| | `FORGE_GITHUB_REPO` | GitHub `owner/repo` for PR listing | `openshift-psap/forge` | | ||
|
|
||
| ## Security Considerations | ||
|
|
||
| This dashboard is designed as an **internal tool** and does **not** include built-in authentication or authorization. As described above, the tool is accessible when port-forwarding from the cluster where it's running. Future development may include auth. | ||
|
|
||
| ## Local Development | ||
|
|
||
| ```bash | ||
| pip install -r requirements.txt | ||
|
|
||
| # Set DATABASE_URL and KUBECONFIG, then: | ||
| uvicorn app.main:app --reload --port 8000 | ||
| ``` | ||
|
|
||
| ## Project Structure | ||
|
|
||
| ``` | ||
| fournos-ui/ | ||
| ├── app/ | ||
| │ ├── main.py # FastAPI routes and Jinja2 rendering | ||
| │ ├── config.py # Environment-based settings | ||
| │ ├── db.py # SQLAlchemy models and queries | ||
| │ ├── k8s_client.py # Kubernetes API wrapper (with timeouts) | ||
| │ ├── watcher.py # Background FournosJob event watcher | ||
| │ ├── forge_discovery.py # Project discovery from ConfigMap | ||
| │ ├── models.py # Pydantic/dataclass models | ||
| │ ├── static/ # CSS, HTMX, htmx-sse.js | ||
| │ └── templates/ # Jinja2 HTML templates | ||
| ├── kustomize/ | ||
| │ ├── base/ # Generic K8s manifests | ||
| │ └── overlays/ocp/ # Environment-specific overrides | ||
| ├── Dockerfile | ||
| └── requirements.txt | ||
| ``` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| """Application configuration loaded from environment variables.""" | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import os | ||
| from dataclasses import dataclass, field | ||
|
|
||
|
|
||
| @dataclass(frozen=True) | ||
| class Settings: | ||
| database_url: str = field( | ||
| default_factory=lambda: os.environ["DATABASE_URL"] | ||
| ) | ||
|
|
||
| fournos_namespace: str = field( | ||
| default_factory=lambda: os.environ.get("FOURNOS_NAMESPACE", "fournos-jobs") | ||
| ) | ||
|
|
||
| kubeconfig_path: str | None = field( | ||
| default_factory=lambda: os.environ.get("KUBECONFIG") | ||
| ) | ||
|
|
||
| forge_repo_path: str | None = field( | ||
| default_factory=lambda: os.environ.get("FORGE_REPO_PATH") | ||
| ) | ||
|
|
||
| projects_config_path: str = field( | ||
| default_factory=lambda: os.environ.get("PROJECTS_CONFIG_PATH", "/etc/fournos-dashboard/projects.yaml") | ||
| ) | ||
|
|
||
| fournos_api_group: str = "fournos.dev" | ||
| fournos_api_version: str = "v1" | ||
| fournos_job_plural: str = "fournosjobs" | ||
|
|
||
| tekton_api_group: str = "tekton.dev" | ||
| tekton_api_version: str = "v1" | ||
| tekton_pipelinerun_plural: str = "pipelineruns" | ||
|
|
||
| log_level: str = field( | ||
| default_factory=lambda: os.environ.get("LOG_LEVEL", "INFO") | ||
| ) | ||
|
|
||
| forge_github_repo: str = field( | ||
| default_factory=lambda: os.environ.get("FORGE_GITHUB_REPO", "openshift-psap/forge") | ||
| ) | ||
|
|
||
| k8s_request_timeout_seconds: int = field( | ||
| default_factory=lambda: int(os.environ.get("K8S_REQUEST_TIMEOUT", "30")) | ||
| ) | ||
|
|
||
| jobs_poll_interval_seconds: int = 5 | ||
|
|
||
| default_pipelines: tuple[str, ...] = ( | ||
| "forge-full", | ||
| "forge-prepare-test", | ||
| "forge-test-only", | ||
| "forge-prepare-only", | ||
| "forge-replot", | ||
| ) | ||
|
|
||
|
|
||
| settings = Settings() |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.