Performance evaluation framework for vLLM on CPU platforms — methodology, automation, and platform configurations for reproducible inference benchmarking.
The recommended entry point is the cpueval CLI — it wraps Ansible playbooks and suite scripts so you can run full test matrices without hand-writing playbook commands.
git clone https://github.com/redhat-et/vllm-cpu-perf-eval.git
cd vllm-cpu-perf-eval
# System packages, Ansible collections, and tab completion
./cpueval install
# Set up your test hosts
export DUT_HOSTNAME=<dut-host>
export LOADGEN_HOSTNAME=<loadgen-host>
# Verify environment, then run a quick sanity check
./cpueval doctor
./cpueval --suite chat-smoke --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 --cores 8
# Explore suites and view results
./cpueval list
./cpueval results --lastSee the Getting Started Guide and cpueval CLI reference for full setup instructions.
| Suite | Status | Entry point |
|---|---|---|
| Concurrent Load | Validated | ./cpueval --suite concurrent-load |
| RHAIIS Sweep | Validated | ./cpueval --suite rhaiis-sweep |
| Offline Batch | Validated | ./cpueval --suite offline-batch |
| Embedding | Validated | ./cpueval --suite embedding |
| Audio | Validated | ./cpueval --suite audio |
| Scalability | WIP | Ansible playbooks |
| Resource Contention | Planned | — |
Full suite reference, selection guide, and status details: docs/test-suites.md
Unsupported suites (e.g. scalability) are blocked by default. For development only,
pass -e "allow_unsupported_tests=true" to Ansible or set ALLOW_UNSUPPORTED_TESTS=true.
- cpueval CLI — Matrix-first benchmarking across 8 suites
- 3-phase testing — Baseline, realistic, and production methodology (details)
- Ansible automation — Distributed test execution on DUT + load generator
- Docker or Podman — Auto-detected container runtime with rootless Podman support
- MTEB integration — Embedding quality evaluation (guide)
- Results tooling — Streamlit dashboards and MLflow tracking
| Topic | Link |
|---|---|
| Documentation index | docs/index.md |
| Getting started | docs/getting-started.md |
| cpueval CLI | docs/cpueval-cli.md |
| Ansible playbooks | automation/test-execution/ansible/ansible.md |
| Model catalog | models/models.md |
| Methodology & metrics | docs/methodology/overview.md |
| Platform setup (Intel) | docs/platform-setup/x86/intel/deterministic-benchmarking.md |
| Path | Purpose |
|---|---|
automation/ |
Ansible playbooks, cpueval CLI, dashboards, MLflow |
docs/ |
Guides and methodology (source for the published site) |
models/ |
Model definitions and test matrices |
tests/ |
Per-suite methodology and scenario configs |
results/ |
Local benchmark output (gitignored) |
logs/ |
Execution logs (gitignored) |
- Hardware: Intel Xeon (Ice Lake+) or AMD EPYC; 64 GB+ RAM recommended
- OS: Ubuntu 22.04+, RHEL 9+, or Fedora 38+
- Software: Python 3.10+, Docker 24+ or Podman 4+, Ansible 2.14+, vLLM, GuideLLM
On the control machine, ./cpueval install installs Ansible, Galaxy collections,
and shell completion. See Getting Started for
control-machine and DUT setup.
- Fork the repository and create a feature branch
- Make your changes
- Run pre-commit checks:
pre-commit run --all-files - Open a pull request
pip install pre-commit
pre-commit install
pre-commit install --hook-type commit-msgLicensed under the Apache License, Version 2.0.
Model weights, container images, and other third-party assets referenced by this project remain under their respective licenses.