[Feature]: Parity with CUDA: vLLM router should have ROCm CI

[functionstackx]

🚀 The feature, motivation and pitch

hi @hongxiayang

+viz @powderluv @chunfangamd @andyluo7

PD disagg is already the current state of optimization that ppl run in prod. currently vLLM router only has CUDA GPU tests and doesn't have AMD GPU test. can u look into fixing this? AMD supports RIXL in vLLM router but there is no tests for that. it should also have tests in vLLM router for MoRI kvcache transfer once it implements that

router/.buildkite/pipeline.yml

Lines 193 to 276 in 4df9bbc

	- label: ":satellite: P/D Disaggregation Test (4 GPUs)"
	key: "pd-disagg-test"
	timeout_in_minutes: 60
	retry:
	automatic:
	- exit_status: "*"
	limit: 2
	manual:
	allowed: true
	reason: "GPU tests can be flaky - retry if needed"
	plugins:
	- docker#v5.11.0:
	image: "nvidia/cuda:12.9.1-devel-ubuntu22.04"
	workdir: /workdir
	volumes:
	- ".:/workdir"
	- "/root/.cache/huggingface:/root/.cache/huggingface"
	runtime: nvidia
	gpus: all
	shm-size: "16g"
	environment:
	- "HF_TOKEN"
	- "VLLM_USE_V1=1"
	- "VLLM_LOGGING_LEVEL=INFO"
	- "UCX_TLS=all"
	- "UCX_NET_DEVICES=all"
	propagate-environment: true
	command:
	- bash
	- -c
	- |
	set -euo pipefail
	# Verify HF_TOKEN is set (required for gated models like Llama)
	if [ -z "$${HF_TOKEN:-}" ]; then
	echo "ERROR: HF_TOKEN is not set. This is required for gated models."
	echo "Please configure HF_TOKEN as a Buildkite Secret in Organization Settings."
	exit 1
	fi
	echo "HF_TOKEN is configured."
	# Install system dependencies
	apt-get update && apt-get install -y \
	curl \
	python3 \
	python3-pip \
	pkg-config \
	libssl-dev \
	protobuf-compiler \
	git
	# Install uv for fast Python package management
	curl -LsSf https://astral.sh/uv/install.sh | sh
	source /root/.local/bin/env
	# Create Python virtual environment (as recommended by vLLM)
	uv venv --python 3.12 --seed
	source .venv/bin/activate
	# Install vLLM with CUDA 12.8 support
	uv pip install vllm --torch-backend=auto
	# Install NIXL library for P/D disaggregation
	uv pip install nixl
	# Install test dependencies
	uv pip install requests pytest openai
	# Install LM Evaluation Harness with API extras for accuracy testing
	uv pip install 'lm-eval[api]>=0.4.0' hf_transfer
	# Install Rust for building the router
	curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
	source /root/.cargo/env
	# Build the router
	cargo build --release
	export PATH="/workdir/target/release:$$PATH"
	# Run P/D disaggregation test
	cd py_test/e2e/pd_disagg_vllm
	bash ./run_accuracy_test.sh
	agents:
	queue: "gpu_4_queue"

Alternatives
ROCm doesn't improve their user experience & code quality

Alternatives

No response

Additional context

refile of vllm-project/vllm#38693

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[simondanielsson]

I have added e2e test for MoRI in #164.

Note that to merge this we need

1. a AMD GPU queue in buildkite we can use for this.
2. merge Update Dockerfile.rocm for AINIC & Thor NIC vllm#40453 - add bnxt RDMA userspace libs to vllm/vllm-openai-rocm

Would be great to understand how we can enable point 1 - can we use amd_mi250_4 (here)?. Point 2 is still WIP but soon being validated and tested.

[simondanielsson]

I defaulted to the amd_mi300_4 queue - let me know if we want to use a different one.

There is an alternative to the bnxt RDMA PR - to add the MoRI XGMI backend and use that in CI as added here: vllm-project/vllm#41753.

[functionstackx]

thanks @simondanielsson i was thinking it makes the most sense to test on RDMA since most disagg happens on AMD over the RDMA network right? i would assume that not a lot of ppl r doing disagg within the same node over XGMI

[simondanielsson]

Agree testing with RDMA would be ideal as you say, as that’s how most deployments are set up. Long term, CI should reflect this.

The benefit of (at least in the short term) leveraging the XGMI backend would be that it’s simpler to make it run in CI with the official vLLM images, in the sense that you don’t need RDMA userspace libs installed into the image (that also have to match the driver/NIC FW versions). Since MoRI is opaque with the XGMI/RDMA backend abstraction it’s from a vllm perspective not that different to use XGMI (even if it’s physically very different, of course).

[simondanielsson]

Got confirmation the amd_mi300_4 buildkite agent uses xGMI. Hence if we want, short term, we can add the xGMI backend to MoRI and use that in the vllm-router CI until vllm-project/vllm#40453 is merged.

[simondanielsson]

Adding CI for NIXL on ROCm here: #218

cc: @andyluo7 @functionstackx