NVIDIA Dynamo is a distributed inference serving framework. Its frontend
exposes an OpenAI-compatible HTTP API, including /v1/responses, and routes requests to backend workers. Dynamo ships
its own workers (vLLM, SGLang, TensorRT-LLM); the vLLM worker (python -m dynamo.vllm) embeds the vLLM engine, so a
Dynamo deployment is a complete inference stack on its own. You do not run vllm serve alongside it.
This guide records how to put Agentic API in front of a Dynamo deployment and what differs from pointing the gateway
at a standalone vllm serve.
The short version: nothing in Agentic API needs to change. Start the gateway with --llm-api-base pointing at the
Dynamo frontend and it works, including stateful previous_response_id chaining and client-executed function tools.
| Capability | Dynamo frontend | Agentic API adds |
|---|---|---|
POST /v1/responses, /v1/chat/completions, /v1/models, /health |
✅ | — |
| Reasoning / tool-call parsing | ✅ via --dyn-reasoning-parser / --dyn-tool-call-parser on the worker |
— |
previous_response_id |
❌ returns 501 Not Implemented (Validation: previous_response_id is not supported.) |
✅ Stores every response and rehydrates the full item history on each turn, so the upstream call is stateless |
| Gateway-executed built-in tools (web search, MCP), background execution, WebSocket transport | ❌ | ✅ |
Because Dynamo rejects previous_response_id, the gateway never forwards it. The second turn of a conversation reaches
Dynamo as one input array containing the earlier user message, the stored assistant message, and the new user
message. The replay tests in crates/agentic-server-core/tests/dynamo_cassette_test.rs assert exactly that shape.
Dynamo publishes wheels on PyPI. The [vllm] extra pulls in the vLLM version Dynamo's worker is built against, so use
a dedicated virtual environment, and pin the Dynamo release:
mkdir -p ~/dev/dynamo && cd ~/dev/dynamo
uv venv --python 3.12 .venv
VIRTUAL_ENV=$PWD/.venv uv pip install "ai-dynamo[vllm]==1.4.1"Pin the version. Dynamo's README suggests uv pip install --prerelease=allow "ai-dynamo[vllm]", but that flag lets uv
resolve any dependency to a pre-release and, with an unpinned ai-dynamo, installs the latest 1.5.0.devYYYYMMDD
build rather than a release. Check what you got with python -c 'import importlib.metadata as m; print(m.version("ai-dynamo"))'.
This guide was verified with ai-dynamo==1.4.1 (which installs vllm==0.26.0 and torch cu130) on an aarch64 host
with a single GB10 GPU. See Dynamo's release artifacts
and support matrix for the wheel/CUDA combinations of other
releases. File discovery can be used for a manually managed single-host setup. The Messages recordings used
etcd discovery because the file watcher did not deliver model registrations on the tested container host.
Run each in its own terminal (or tmux window). The frontend is the HTTP entry point (default port 8000, round-robin routing across whatever workers register); the worker loads the model. Models are resolved from the Hugging Face cache, so anything already downloaded for vLLM is reused.
# Frontend: OpenAI-compatible HTTP on :8000
.venv/bin/python -m dynamo.frontend --http-port 8000 --discovery-backend file
# Worker: vLLM engine managed by Dynamo
.venv/bin/python -m dynamo.vllm \
--model openai/gpt-oss-20b \
--discovery-backend file \
--kv-events-config '{"enable_kv_cache_events": false}' \
--dyn-reasoning-parser gpt_oss \
--dyn-tool-call-parser harmony \
--max-model-len 32768openai/gpt-oss-20b needs roughly 16 GB of GPU memory for weights plus KV cache, so it fits a single 24 GB GPU with
vLLM's default --gpu-memory-utilization 0.9. Lower that only when the GPU is shared (see below).
Flags worth knowing:
| Flag | Why |
|---|---|
--discovery-backend file |
Lets the frontend and worker find each other via /tmp/dynamo_store_kv instead of etcd. Pass it to both. |
--kv-events-config '{"enable_kv_cache_events": false}' |
Required for the vLLM worker without NATS. |
--dyn-reasoning-parser / --dyn-tool-call-parser |
The Dynamo frontend parses model output, not vLLM. vLLM's --reasoning-parser is ignored and --tool-call-parser / --enable-auto-tool-choice are rejected as unknown arguments. Without the --dyn-* flags, gpt-oss "analysis" text leaks into content and tool calls come back as plain text. gpt_oss + harmony is the verified pair for gpt-oss models; python -m dynamo.vllm --help lists the parsers for other model families. |
--gpu-memory-utilization |
Standard vLLM engine flag (the worker accepts vLLM engine arguments). It is a fraction of total device memory and must fit in the memory currently free, or the engine fails at startup. On a dedicated GPU keep the default; on a shared GPU size it to what is actually free (the recordings for this guide used 0.15 on a 121 GB unified-memory host that was also running another model). |
Confirm the worker registered and parsing works:
curl -s localhost:8000/v1/models | jq '.data[].id'
curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
"model": "openai/gpt-oss-20b",
"messages": [{"role": "user", "content": "Say hello in five words."}],
"max_tokens": 500
}' | jq '.choices[0].message | {content, reasoning_content}'content should hold the answer and reasoning_content the chain of thought. If the answer starts with analysis, the
worker is missing --dyn-reasoning-parser.
cargo build -p agentic-server --bins
./target/debug/agentic-server --llm-api-base http://127.0.0.1:8000Agentic API's startup probe uses Dynamo's /health, so no --skip-llm-ready-check is needed. Be aware of what that
probe means: Dynamo returns 200 {"status":"healthy", ...} as soon as the frontend's HTTP service is up, even with no
worker registered (the instances list is simply empty). It does not mean a model is loaded. Wait for the
per-model readiness endpoint before sending traffic:
curl -s localhost:8000/v1/models/openai%2Fgpt-oss-20b # 404 until the worker registers, then model metadataThe harness CLI works the same way: ./target/debug/agentic run codex --upstream http://127.0.0.1:8000.
R1=$(curl -s localhost:9000/v1/responses -H 'Content-Type: application/json' -d '{
"model": "openai/gpt-oss-20b",
"input": "Remember the word APPLE. Just say: OK",
"max_output_tokens": 2048
}')
echo "$R1" | jq -r '.output[] | select(.type=="message") | .content[0].text' # OK
curl -s localhost:9000/v1/responses -H 'Content-Type: application/json' -d "{
\"model\": \"openai/gpt-oss-20b\",
\"input\": \"What word did I ask you to remember? Reply with just the word.\",
\"previous_response_id\": \"$(echo "$R1" | jq -r .id)\",
\"max_output_tokens\": 2048
}" | jq -r '.output[] | select(.type=="message") | .content[0].text' # APPLE
curl -s localhost:9000/v1/responses -H 'Content-Type: application/json' -d '{
"model": "openai/gpt-oss-20b",
"input": "What is the current NVIDIA stock price? Use the tool.",
"max_output_tokens": 2048,
"tools": [{"type": "function", "name": "get_stock_price",
"description": "Get the latest stock price for a ticker symbol",
"parameters": {"type": "object", "properties": {"ticker": {"type": "string"}}, "required": ["ticker"]}}]
}' | jq '.output[] | select(.type=="function_call") | {name, arguments}'Expected: OK, then APPLE, then a get_stock_price call with {"ticker":"NVDA", ...}.
Sending the second request straight to Dynamo (port 8000) instead of the gateway fails with 501; that difference is
the value the gateway adds. The third request exercises a client-executed function tool: Dynamo returns the function
call and the application runs it.
The interactions above are recorded in crates/agentic-server-core/tests/cassettes/dynamo/ and replayed by
tests/dynamo_cassette_test.rs on every cargo test, so CI covers the Dynamo upstream without a GPU. The
dynamo-upstream CI job also runs scripts/validate-cassettes.py, a structural check over every recorded cassette. To re-record
against a live Dynamo (for example after a Dynamo release changes the response shape):
cd crates/agentic-server-core
DYNAMO_URL=http://127.0.0.1:8000 MODEL=openai/gpt-oss-20b \
bash tests/cassettes/record_dynamo_cassettes.shThe script records the second stateful turn from the hydrated item history the gateway would send (built from turn
1's recorded assistant message), because the recorder's own previous_response_id chaining cannot be used against a
stateless upstream.
| Symptom | Cause / fix |
|---|---|
unrecognized arguments: --tool-call-parser |
Use --dyn-tool-call-parser on the worker. |
Answer text begins with analysis…assistantfinal… |
Add --dyn-reasoning-parser gpt_oss (or the parser for your model). |
Free memory on device … is less than desired GPU memory utilization |
Lower --gpu-memory-utilization; it is a fraction of total memory. |
CUDA error: out of memory right after restarting a worker |
A previous dynamo.vllm process is still alive and holding memory; pkill -f "python -m dynamo.vllm" before relaunching. Closing its terminal or tmux window does not kill it. |
501 Validation: previous_response_id is not supported. |
You are calling Dynamo directly. Send the request to the gateway. |
Gateway logs LLM ready but requests fail with no model / model not found |
/health is green before the worker registers. Check /v1/models or /v1/models/{model} and the worker log. |
Installed version is 1.5.0.dev… |
--prerelease=allow with an unpinned ai-dynamo picked a dev build. Reinstall with "ai-dynamo[vllm]==1.4.1". |
The streaming and non-streaming Messages recordings were captured from Dynamo 1.4.1 with vLLM 0.26.0 on Linux
x86_64, an NVIDIA L40S 46,068 MiB GPU, and NVIDIA driver 580.159.04. The model snapshot was
6cee5e81ee83917806bbde320786a8fb61efebee. The checked-in acceptance test replays both real captures on every run.
The frontend must enable the experimental Anthropic endpoint with --enable-anthropic-api. The gateway sends
/v1/messages to that endpoint; it does not convert Messages requests into Chat Completions itself. Check this flag
with the pinned frontend's --help before recording. Keep the worker's --dyn-reasoning-parser gpt_oss and
--dyn-tool-call-parser harmony settings.
The regular Rust CI job runs this integration test through cargo test. It replays the checked-in Dynamo responses
through the production Messages tool loop and compares the next upstream request with the recorded history.
No GPU, running Dynamo service, Python environment, or external search service is required:
cargo test --locked -p agentic-server-core --test dynamo_messages_testThe fixed tool output comes from messages/tool_outputs.json; it is not a claim about today's Rust release.
The legacy vLLM preparation fixture omitted a thinking signature in its next request. Its test recovers that
expectation from the recorded event; the Dynamo acceptance test compares the recorded history directly.
Provision and start Dynamo separately, with the Messages endpoint and parsers described above. From the repository root, install the recorder dependencies in a separate environment:
uv venv --python 3.12 .venv-dynamo-recorder
uv pip install --python .venv-dynamo-recorder/bin/python \
-r crates/agentic-server-core/tests/cassettes/recorder-requirements.txt
PYTHON="$PWD/.venv-dynamo-recorder/bin/python" \
DYNAMO_URL=http://127.0.0.1:8000 MODEL=openai/gpt-oss-20b \
bash crates/agentic-server-core/tests/cassettes/record_dynamo_messages_cassettes.sh
cargo test --locked -p agentic-server-core --test dynamo_messages_testThe script records against an existing endpoint; it does not install or manage Dynamo. Both modes must pass
scenario and structural validation before replacing the destination files. Failed runs retain their staging
directory for diagnosis. Do not hand-edit captured YAML. The recorder preserves signature_delta content and
fails on malformed tool argument JSON instead of substituting an empty object.
The recording utilities have separate, GPU-independent tests for signature preservation, malformed arguments, two-round history, and preserving installed fixtures when recording fails. Run these when changing the recorder:
.venv-dynamo-recorder/bin/python -m unittest discover \
-s crates/agentic-server-core/tests/cassettes -p 'test_messages.py' -v