BenchFlow

Repeatable LLM inference benchmarks for OpenShift.

Note

This project is experimental and for learning purposes mainly, but the implemented execution paths today are llm-d and RHOAI. Expect some parts to still be highly coupled.

BenchFlow is a packaged control plane for running benchmark scenarios, not a loose collection of scripts. It resolves an experiment into one immutable RunPlan, executes it through Tekton PipelineRuns, captures metrics and artifacts, and pushes the result to MLflow. The current benchmark backends are vllm-project/guidellm and a narrow AIPerf trace replay path with explicit Mooncake and Weka benchmark profiles.

Warning

BenchFlow now locks shared llm-d and RHOAI platform mutations per target cluster by setup key. Same-key runs can share a wave and use spare GPUs in parallel; different-key runs wait until the current admitted wave finishes. Shared platform prerequisites stay installed until a different setup key is requested or you explicitly tear them down.

Quickstart

Install the CLI from the repository root:

pip install -e .

For local development, install the dev extra and enable the pre-commit hooks:

pip install -e '.[dev]'
pre-commit install

Before bootstrapping, create the real secret manifests next to the examples under config/cluster/secrets/. BenchFlow applies every *.yaml file there except *.example.yaml, so cp the examples secrets and remove the .example suffix, and populate them with your credentials.

BenchFlow supports two cluster topologies.

Single cluster

bflow bootstrap --single-cluster
bflow experiment run experiments/smoke/qwen3-06b-llm-d-smoke.yaml

BenchFlow installs Tekton, Kueue, the BenchFlow remote-capacity controller, Grafana, RBAC, GPU prerequisites, and the required PVCs in the same cluster. Kueue admits runs locally against the discovered GPU capacity, and Tekton runs the full workflow there. GPU prerequisites include DCGM exporter metrics and the ServiceMonitor used by BenchFlow system metrics.

Management cluster + remote target cluster(s)

Management cluster:

bflow bootstrap

Target cluster(s):

bflow bootstrap --target-kubeconfig ~/.kube/target-cluster --cluster-name target-cluster

Run from the management cluster:

bflow experiment run experiments/smoke/qwen3-06b-llm-d-smoke.yaml --cluster-name target-cluster

In this mode, the management cluster runs Tekton, Kueue, and the BenchFlow remote-capacity controller. Kueue queues executions by target cluster and admits them only when the target has enough GPU capacity. The target cluster does not need Tekton; BenchFlow launches the runtime work there through plain Kubernetes Jobs using the stored kubeconfig Secret. Target bootstrap also reconciles DCGM exporter metrics unless accelerator prerequisites are explicitly disabled.

The narrow path is the shipped smoke experiment:

bflow experiment run experiments/smoke/qwen3-06b-llm-d-smoke.yaml

Then follow the execution:

bflow watch <execution-name> --namespace benchflow

BenchFlow also supports matrix experiments by turning one or more profile fields into lists; the cluster then runs the cartesian product through child executions. rhoai and llm-d child executions can be admitted in parallel when target-cluster GPU capacity allows it. After all child executions finish, the parent prints a compact MLflow run summary. Once the parent has submitted child executions, canceling the parent is only best-effort; already queued or running children may need individual cancellation.

For the full command surface, RunPlan PipelineRun flow, matrix execution, and lower-level runtime commands, see docs/ADVANCED.md.

Known Limitations

• The current AIPerf support is intentionally narrow: the explicit aiperf-mooncake-trace, aiperf-mooncake-toolagent-trace, and aiperf-weka-trace benchmark profiles for existing chat-style endpoints. It is not generic AIPerf support.
• Comparison reports do not mix benchmark tools. bflow benchmark plot comparison can compare multiple GuideLLM runs or multiple AIPerf runs, but not both in one report.
• Legacy target clusters without BenchFlow platform state are adopted heuristically; the first mutating run after upgrading BenchFlow may reset and reinstall shared llm-d or RHOAI prerequisites.
• BenchFlow coordinates only the shared platform state it manages itself. Manual or out-of-band cluster mutations are not version-reconciled.
• llm-d matrix children depend on release-scoped chart values; use the current BenchFlow image when testing parallel llm-d runs.
• Matrix parent cancellation is best-effort after child executions have already been submitted; queued or running children may need to be cancelled individually.
• BenchFlow manages GuideLLM benchmark output paths itself. GUIDELLM_OUTPUT_DIR is set automatically during benchmark execution, and GUIDELLM_OUTPUT_PATH is not supported in benchmark environment overrides.