diff --git a/README.md b/README.md
index 11c30eac..c33356be 100644
--- a/README.md
+++ b/README.md
@@ -26,6 +26,7 @@
- [Quickstart](#quickstart)
- [Generator with Diffusers](#generator-with-diffusers)
- [Generator with vLLM-Omni](#generator-with-vllm-omni)
+ - [Generator with NIM](#generator-with-nim)
- [Reasoner with Transformers](#reasoner-with-transformers)
- [Reasoner with vLLM](#reasoner-with-vllm)
- [Reasoner with NIM](#reasoner-with-nim)
@@ -64,7 +65,7 @@ Cosmos 3 exposes two runtime surfaces:
- **World understanding:** Analyze videos and images for captions, temporal events, next actions, spatial grounding, physical plausibility, and causal outcomes.
- **World generation:** Produce images, videos, synchronized sound, and action-conditioned rollouts from text, image, video, or action inputs.
- **Action modeling:** Predict policy actions, inverse dynamics, and forward dynamics for robotics, camera motion, egocentric motion, and autonomous-driving settings.
-- **Research and production paths:** Use Diffusers and Transformers for Python-first development, then vLLM-Omni and vLLM for OpenAI-compatible serving.
+- **Research and production paths:** Use Diffusers and Transformers for Python-first development, vLLM-Omni and vLLM for OpenAI-compatible serving, and NIM containers for turnkey Reasoner serving or Generator deployment for text to video and image to video generations.
- **Post-training recipes:** Adapt vision, action, and reasoner workflows with Cosmos Framework training recipes and task-specific evaluation [Coming Soon].
### Model Architecture
@@ -204,13 +205,15 @@ Before running examples, create a Hugging Face access token and then authenticat
uvx hf@latest auth login
```
-Set `HF_HOME` if you want to use a shared cache or a disk with more space.
+Set `HF_HOME` if you want to use a shared cache or a disk with more space. NIM
+examples use an NGC API key (`NGC_API_KEY`) instead of Hugging Face
+authentication.
Generator requires the Guardrail. Request access to the gated
[nvidia/Cosmos-1.0-Guardrail](https://huggingface.co/nvidia/Cosmos-1.0-Guardrail)
-HF repository. To disable the guardrail, set `enable_safety_checker=False` (Diffusers),
-`guardrails: false` (vLLM-Omni `extra_params`/`extra_args`), or
-`--no-guardrails` (Cosmos Framework).
+HF repository for Hugging Face based Generator paths. To disable the guardrail,
+set `enable_safety_checker=False` (Diffusers), `guardrails: false` (vLLM-Omni
+`extra_params`/`extra_args`), or `--no-guardrails` (Cosmos Framework).
#### Generator with Diffusers
@@ -437,6 +440,90 @@ References:
+#### Generator with NIM
+
+
+Use the prebuilt Cosmos3-Generator NIM for turnkey T2V/I2V video generation.
+
+Use the `Cosmos3-Generator` NIM for turnkey Generator deployment through an NGC
+container. This NIM serves **Text2Video** and **Image2Video** only. It does not
+expose text-to-image, video-to-video, sound/audio generation, action modes, or
+transfer controls; use [Generator with vLLM-Omni](#generator-with-vllm-omni) or
+Cosmos Framework for those broader Generator workflows.
+
+The Generator NIM API differs from vLLM-Omni: send JSON requests to
+`POST /v1/infer`, and decode the JSON response field `b64_video` to get the MP4
+bytes. The NIM infers the mode automatically from request fields:
+
+| Mode | Request shape | Response |
+| --- | --- | --- |
+| Text2Video | non-empty `prompt`, no `image` | JSON with `b64_video` |
+| Image2Video | `image` provided, optional `prompt` | JSON with `b64_video` |
+
+Authenticate to NGC and launch the default Nano server:
+
+```shell
+export NGC_API_KEY=
+echo "$NGC_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin
+
+export LOCAL_NIM_CACHE="${LOCAL_NIM_CACHE:-$HOME/.cache/nim}"
+mkdir -p "$LOCAL_NIM_CACHE"
+chmod -R 777 "$LOCAL_NIM_CACHE" 2>/dev/null || true
+
+docker run --runtime=nvidia --gpus all \
+ --shm-size=32GB \
+ --ulimit nofile=65536:65536 \
+ -e NGC_API_KEY="$NGC_API_KEY" \
+ -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
+ -p 8000:8000 \
+ nvcr.io/nim/nvidia/cosmos3-generator:1.0.0
+```
+
+For the larger model, add `-e NIM_MODEL_SIZE=super`. The main launch-time knobs
+are `NIM_MODEL_SIZE=nano|super` (default `nano`),
+`NIM_PRECISION=bf16|fp8|nvfp4` (default `fp8`; `nvfp4` requires Blackwell),
+`NIM_PERF_PROFILE=latency|throughput` (default `latency`), and advanced
+`NIM_TAGS_SELECTOR` profile filters.
+
+Wait for readiness:
+
+```shell
+curl -fsS http://127.0.0.1:8000/v1/health/ready
+```
+
+Send a Text2Video request and decode the MP4:
+
+```shell
+curl -sS -X POST http://127.0.0.1:8000/v1/infer \
+ -H 'Accept: application/json' \
+ -H 'Content-Type: application/json' \
+ -d '{
+ "prompt": "A humanoid robot walks through a futuristic warehouse, inspecting shelves of mechanical components.",
+ "seed": 42,
+ "guidance_scale": 6.0,
+ "steps": 35,
+ "resolution": "256",
+ "num_output_frames": 25,
+ "fps": 24.0
+ }' | jq -r '.b64_video' | base64 -d > cosmos3_generator_nim_t2v.mp4
+```
+
+For Image2Video, provide `image` as raw base64, a `data:image/...;base64,...` URI,
+or a public URL when URL inputs are enabled.
+
+Request constraints include: `guidance_scale` in `[1.0, 7.0]`, `steps` in
+`[1, 100]`, `num_output_frames` on the `4k+1` cadence (`25, 29, 33, ...`) with
+per-tier caps (`256 <= 397`, `480 <= 297`, `720 <= 197`), and resolution keys
+`256`, `480`, `720` plus optional suffixes `_16_9`, `_1_1`, `_9_16`, `_4_3`, and
+`_3_4`.
+
+See the [Generator NIM cookbook](cookbooks/cosmos3/generator/audiovisual/run_with_nim.ipynb)
+for an end-to-end notebook that launches the container, polls readiness,
+inspects service metadata, runs T2V and I2V, decodes `b64_video`, and previews
+the generated MP4 files.
+
+
+
#### Reasoner with Transformers
@@ -446,8 +533,9 @@ Use Hugging Face Transformers for Python-first Reasoner inference. This path
loads only the Reasoner tower from the unified `nvidia/Cosmos3-Nano` or
`nvidia/Cosmos3-Super` checkpoint and returns text from text, image, or video
inputs. It does not load the Generator diffusion, audio, or action heads; use
-[Generator with Diffusers](#generator-with-diffusers) or
-[Generator with vLLM-Omni](#generator-with-vllm-omni) for non-text outputs.
+[Generator with Diffusers](#generator-with-diffusers),
+[Generator with vLLM-Omni](#generator-with-vllm-omni), or
+[Generator with NIM](#generator-with-nim) for supported non-text outputs.
Cosmos3 support first appears in the Transformers `v5.11.0` release tag. Install
Transformers `5.11.0` or newer:
@@ -555,7 +643,7 @@ installed. For an OpenAI-compatible server, use
Expand vLLM Reasoner setup, server launch, and configuration
-Use vLLM for Reasoner production inference behind an OpenAI-compatible chat-completions API. This path loads only the reasoner; for generation tasks that return images or video, use [Generator with vLLM-Omni](#generator-with-vllm-omni) instead.
+Use vLLM for Reasoner production inference behind an OpenAI-compatible chat-completions API. This path loads only the reasoner; for generation tasks that return images or video, use [Generator with vLLM-Omni](#generator-with-vllm-omni), or [Generator with NIM](#generator-with-nim) for turnkey T2V/I2V video generation only.
```shell
uv venv --python 3.13 --seed --managed-python
@@ -730,7 +818,8 @@ The Cosmos Framework requires `uv >= 0.11.3` (enforced via its `pyproject.toml`)
| Goal | Use | Notes |
| --- | --- | --- |
| Generator research or model development | Diffusers | Python-first path for inspecting and modifying generator behavior |
-| Generator production inference | vLLM-Omni | API path for image, video, sound, and action outputs |
+| Generator broader production/API serving | vLLM-Omni | API path for image, video, sound, and action outputs |
+| Generator turnkey deployment | NIM | Prebuilt NGC container for T2V/I2V video generation only; uses `/v1/infer` and returns JSON `b64_video` |
| Reasoner research or model development | Transformers | Python-first path for prompts, processors, and model behavior |
| Reasoner production inference | vLLM | OpenAI-compatible endpoint for text outputs from text and vision inputs |
| Reasoner turnkey deployment | NIM | Prebuilt, optimized OpenAI-compatible container — no vLLM/CUDA setup |
@@ -745,6 +834,7 @@ We are building examples that show Cosmos 3 capabilities end to end, including w
| Generator (audiovisual) with Diffusers | Generator | Text-to-image, plus text-to-video and image-to-video each with or without synchronized sound, via `Cosmos3OmniPipeline`. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_with_diffusers.ipynb) | [](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_with_diffusers.ipynb) |
| Generator (audiovisual) with Cosmos Framework | Generator | Text-to-image, plus text-to-video and image-to-video each with sound on or off, through the `cosmos_framework.scripts.inference` entrypoint. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_with_cosmos_framework.ipynb) | [](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_with_cosmos_framework.ipynb) |
| Generator (audiovisual) with vLLM-Omni | Generator | Text-to-image, plus text-to-video and image-to-video each with sound on or off, against an OpenAI-compatible vLLM-Omni server. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_with_vllm_omni.ipynb) | [](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_with_vllm_omni.ipynb) |
+| Generator (audiovisual) with NIM | Generator | Text2Video and Image2Video only, against the prebuilt `Cosmos3-Generator` NIM; requests use `POST /v1/infer` and decode JSON `b64_video` responses. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_with_nim.ipynb) | [](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_with_nim.ipynb) |
| Forward dynamics with Cosmos Framework | Generator | Forward dynamics: action-conditioned future-observation prediction for AV, DROID, and UMI, through the `cosmos_framework.scripts.inference` entrypoint. | [Notebook](cookbooks/cosmos3/generator/action/run_fd_with_cosmos_framework.ipynb) | [](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/action/run_fd_with_cosmos_framework.ipynb) |
| Forward dynamics with vLLM-Omni | Generator | Forward dynamics: action-conditioned future-observation prediction for AV, DROID, and UMI, against an OpenAI-compatible vLLM-Omni server. | [Notebook](cookbooks/cosmos3/generator/action/run_fd_with_vllm.ipynb) | [](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/action/run_fd_with_vllm.ipynb) |
| Inverse dynamics with Cosmos Framework | Generator | Inverse dynamics: ego-motion trajectory prediction from input AV video, through the `cosmos_framework.scripts.inference` entrypoint. | [Notebook](cookbooks/cosmos3/generator/action/run_id_with_cosmos_framework.ipynb) | [](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/action/run_id_with_cosmos_framework.ipynb) |
diff --git a/cookbooks/cosmos3/README.md b/cookbooks/cosmos3/README.md
index 4c1f9fdc..a6eaf9ce 100644
--- a/cookbooks/cosmos3/README.md
+++ b/cookbooks/cosmos3/README.md
@@ -11,7 +11,8 @@ backend you want to run and follow that one section.
| [Transformers](#transformers) | Hugging Face Transformers inference | Reasoner |
| [vLLM](#vllm) | OpenAI-compatible reasoning server (image/video understanding) | Reasoner |
| [vLLM-Omni](#vllm-omni) | OpenAI-compatible generation server (image/video/audio/action) | Generator (Audiovisual, Action) |
-| [NIM](#nim) | Prebuilt OpenAI-compatible reasoning server (image/video understanding); no venv | Reasoner |
+| [Reasoner NIM](#reasoner-nim) | Prebuilt OpenAI-compatible reasoning server (image/video understanding); no venv | Reasoner |
+| [Generator NIM](#generator-nim) | Prebuilt NGC container serving the Cosmos3 Generator for Text-to-Video and Image-to-Video inference | Generator (Audiovisual) |
## Prerequisites
@@ -30,9 +31,10 @@ backend you want to run and follow that one section.
To disable the guardrail, set `enable_safety_checker=False` (Diffusers), `guardrails: false`
(vLLM-Omni `extra_params`/`extra_args`), or
- `--no-guardrails` (Cosmos Framework).
+ `--no-guardrails` (Cosmos Framework). For Generator NIM set environment variables `NIM_ENABLE_TEXT_GUARDRAILS=0 NIM_ENABLE_VIDEO_GUARDRAILS=0`.
+- NIMs don't need Hugging Face access; instead, an NGC API key is required
+ (used as `NGC_API_KEY`). You can generate one on [build.nvidia.com](https://build.nvidia.com/) or [NGC](https://catalog.ngc.nvidia.com/), then run `docker login nvcr.io` once (username `$oauthtoken`, password = your key). This repository uses the Reasoner NIM image `nvcr.io/nim/nvidia/cosmos3-reasoner` and the Generator NIM image `nvcr.io/nim/nvidia/cosmos3-generator`.
- For the Cosmos Framework backend: access to `git@github.com:NVIDIA/cosmos-framework.git`.
-- For the NIM backend: an NGC API key (used as `NGC_API_KEY`), which you can generate on [build.nvidia.com](https://build.nvidia.com/nvidia/cosmos3-nano-reasoner) or [NGC](https://catalog.ngc.nvidia.com/orgs/nim/teams/nvidia/containers/cosmos3-reasoner), plus a one-time `docker login nvcr.io` (username `$oauthtoken`, password = your key). The HF login above is not needed for NIM.
- Enough local disk for the venv/image, the uv cache, and the model cache. Nano
downloads plus CUDA dependencies can take tens of GiB.
@@ -388,14 +390,28 @@ Ensure the server has enough GPUs for the product of enabled degrees
## NIM
+Prebuilt NGC containers for Cosmos3. Like vLLM-Omni, NIM runs from Docker, so
+there is no venv or `--torch-backend` to manage. Unlike the Hugging Face based
+backends, NIM authenticates with an NGC API key instead of a Hugging Face token
+(see [Prerequisites](#prerequisites)).
+
+Authenticate Docker to NGC once:
+
+```bash
+export NGC_API_KEY=
+echo "$NGC_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin
+```
+
+Both NIMs expose readiness at `GET /v1/health/ready` after model download,
+engine initialization, and warmup complete.
+
+### Reasoner NIM
+
A prebuilt container that serves the Reasoner over an OpenAI-compatible API for
-image and video understanding. Like vLLM-Omni this is a Docker image, so there is
-no venv or `--torch-backend` to manage; unlike the other backends it
-authenticates with an NGC API key instead of Hugging Face (see
-[Prerequisites](#prerequisites)).
+image and video understanding.
-Start a Nano server (publishes the OpenAI-compatible API on port 8000; the first
-run downloads the model into `~/.cache/nim`):
+Start a Nano Reasoner server (publishes the API on port 8000; the first run
+downloads the model into `~/.cache/nim`):
```bash
export NGC_API_KEY=
@@ -415,6 +431,62 @@ The container serves `nvidia/cosmos3-nano-reasoner` (or
`nvidia/cosmos3-super-reasoner`); pass that exact name as the request `model`, or
resolve it dynamically with `client.models.list()`.
+### Generator NIM
+
+A prebuilt container that serves **Cosmos3-Generator Text-to-Video and Image-to-Video
+only** through `POST /v1/infer`. The NIM infers mode from the request fields:
+non-empty `prompt` with no `image` means Text-to-Video; `image` provided means Image-to-Video. The
+response is JSON with a base64-encoded MP4 in `b64_video`.
+
+It does **not** expose text-to-image, video-to-video, sound/audio generation,
+action modes, or transfer controls. Use vLLM-Omni or Cosmos Framework for those
+broader Generator workflows.
+
+Start a Nano Generator server (default `NIM_MODEL_SIZE=nano`, `NIM_PRECISION=fp8`,
+`NIM_PERF_PROFILE=latency`):
+
+```bash
+export NGC_API_KEY=
+export LOCAL_NIM_CACHE="${LOCAL_NIM_CACHE:-$HOME/.cache/nim}"
+mkdir -p "$LOCAL_NIM_CACHE"
+chmod -R 777 "$LOCAL_NIM_CACHE" 2>/dev/null || true
+
+docker run --runtime=nvidia --gpus all \
+ --shm-size=32GB \
+ --ulimit nofile=65536:65536 \
+ -e NGC_API_KEY="$NGC_API_KEY" \
+ -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
+ -p 8000:8000 \
+ nvcr.io/nim/nvidia/cosmos3-generator:1.0.0
+```
+
+For **Cosmos3-Super Generator**, add `-e NIM_MODEL_SIZE=super`. Other selection
+knobs:
+
+| Env var | Values | Default | Use |
+| --- | --- | --- | --- |
+| `NIM_MODEL_SIZE` | `nano`, `super` | `nano` | Selects 8B Nano or 32B Super |
+| `NIM_PRECISION` | `bf16`, `fp8`, `nvfp4` | `fp8` | Selects precision; `nvfp4` requires Blackwell |
+| `NIM_PERF_PROFILE` | `latency`, `throughput` | `latency` | Optimizes profile selection objective |
+| `NIM_TAGS_SELECTOR` | comma-separated `key=value` filters | unset | Advanced profile pinning, e.g. `model_size=super,nim_tp=2` |
+
+A quick T2V smoke test:
+
+```bash
+curl -sS -X POST http://127.0.0.1:8000/v1/infer \
+ -H 'Accept: application/json' \
+ -H 'Content-Type: application/json' \
+ -d '{
+ "prompt": "A humanoid robot walks through a futuristic warehouse, inspecting shelves of mechanical components.",
+ "seed": 42,
+ "guidance_scale": 6.0,
+ "steps": 35,
+ "resolution": "256",
+ "num_output_frames": 25,
+ "fps": 24.0
+ }' | jq -r '.b64_video' | base64 -d > /tmp/cosmos3_generator_nim_t2v.mp4
+```
+
## Verify the environment
For the Cosmos Framework / Diffusers / vLLM venvs, check that PyTorch sees the GPU:
diff --git a/cookbooks/cosmos3/generator/audiovisual/README.md b/cookbooks/cosmos3/generator/audiovisual/README.md
index d80adad4..0385d1af 100644
--- a/cookbooks/cosmos3/generator/audiovisual/README.md
+++ b/cookbooks/cosmos3/generator/audiovisual/README.md
@@ -1,8 +1,12 @@
# Cosmos3 Generator Audiovisual Examples
Generate images and video (with optional audio) from text or image prompts with
-`Cosmos3-Nano` and `Cosmos3-Super`, across three inference backends. Sample
-prompts live under [`assets/`](./assets).
+`Cosmos3-Nano` and `Cosmos3-Super`, across Cosmos Framework, Diffusers,
+vLLM-Omni, and NIM backends. Sample prompts live under [`assets/`](./assets).
+
+> **NIM scope:** `Cosmos3-Generator` NIM currently exposes Text2Video and
+> Image2Video only. It does not expose text-to-image, video-to-video,
+> sound/audio generation, action modes, or transfer controls.
Environment setup for every backend is centralized in the shared
[Cosmos3 cookbooks environment setup](../../README.md) guide; each backend below
@@ -13,7 +17,11 @@ Generator requires the Guardrail. Request access to the gated
[nvidia/Cosmos-1.0-Guardrail](https://huggingface.co/nvidia/Cosmos-1.0-Guardrail)
HF repository before running these examples. To disable the guardrail, set
`enable_safety_checker=False` (Diffusers), `guardrails: false` (vLLM-Omni
-`extra_params`/`extra_args`), or `--no-guardrails` (Cosmos Framework).
+`extra_params`/`extra_args`), or `--no-guardrails` (Cosmos Framework). For
+Generator NIM set `NIM_ENABLE_TEXT_GUARDRAILS=0 NIM_ENABLE_VIDEO_GUARDRAILS=0`.
+
+NIM backends use NGC authentication instead of Hugging Face login; see the
+[Generator NIM setup](../../README.md#generator-nim) for details.
## Run with Cosmos Framework
@@ -184,3 +192,128 @@ the vLLM-Omni backend: it walks through text-to-image, text-to-video, and
image-to-video requests with audio on or off. Server launch options (Nano and
Super, tensor parallelism, layerwise offload, and CFG-parallel variants) live in
the [shared environment setup guide](../../README.md#vllm-omni).
+
+## Run with NIM
+
+### Quickstart
+
+Set up the environment: [Generator NIM setup](../../README.md#generator-nim).
+`Cosmos3-Generator` NIM is a prebuilt NGC container that serves Text2Video and
+Image2Video through `POST /v1/infer`. It returns JSON with a base64-encoded MP4
+in `b64_video`; unlike vLLM-Omni, it does not use `/v1/videos/sync` and does not
+return MP4 bytes directly.
+
+Authenticate Docker to NGC once:
+
+```bash
+export NGC_API_KEY=
+echo "$NGC_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin
+```
+
+Launch **Cosmos3-Nano** (default model size, FP8, latency profile):
+
+```bash
+export LOCAL_NIM_CACHE="${LOCAL_NIM_CACHE:-$HOME/.cache/nim}"
+mkdir -p "$LOCAL_NIM_CACHE"
+chmod -R 777 "$LOCAL_NIM_CACHE" 2>/dev/null || true
+
+docker run -it --rm --name cosmos3-generator \
+ --runtime=nvidia \
+ --gpus all \
+ --shm-size=32GB \
+ --ulimit nofile=65536:65536 \
+ -e NGC_API_KEY="$NGC_API_KEY" \
+ -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
+ -p 8000:8000 \
+ nvcr.io/nim/nvidia/cosmos3-generator:1.0.0
+```
+
+Launch **Cosmos3-Super** by adding `NIM_MODEL_SIZE=super`:
+
+```bash
+docker run -it --rm --name cosmos3-generator \
+ --runtime=nvidia \
+ --gpus all \
+ --shm-size=32GB \
+ --ulimit nofile=65536:65536 \
+ -e NGC_API_KEY="$NGC_API_KEY" \
+ -e NIM_MODEL_SIZE=super \
+ -e NIM_PRECISION=fp8 \
+ -e NIM_PERF_PROFILE=latency \
+ -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
+ -p 8000:8000 \
+ nvcr.io/nim/nvidia/cosmos3-generator:1.0.0
+```
+
+Wait until the readiness endpoint returns success:
+
+```bash
+curl -fsS http://127.0.0.1:8000/v1/health/ready
+```
+
+Send a Text2Video request. The NIM infers T2V from a non-empty `prompt` with no
+`image` field:
+
+```bash
+curl -sS -X POST http://127.0.0.1:8000/v1/infer \
+ -H 'Accept: application/json' \
+ -H 'Content-Type: application/json' \
+ -d '{
+ "prompt": "A humanoid robot walks through a futuristic warehouse, inspecting shelves of mechanical components. Photorealistic, cinematic lighting.",
+ "seed": 42,
+ "guidance_scale": 6.0,
+ "steps": 35,
+ "resolution": "256",
+ "num_output_frames": 25,
+ "fps": 24.0
+ }' | jq -r '.b64_video' | base64 -d > /tmp/cosmos3_nim_t2v.mp4
+```
+
+Send an Image2Video request with a local image encoded as a data URI:
+
+```bash
+IMG_B64=$(base64 -w 0 assets/images/image2video/humanoid_robot.jpg)
+cat > /tmp/cosmos3_nim_i2v.json < /tmp/cosmos3_nim_i2v.mp4
+```
+
+Key request fields and constraints:
+
+| Field | Constraint / default |
+| --- | --- |
+| `prompt` | Required for T2V; optional when `image` is provided; max 20000 chars |
+| `negative_prompt` | Optional; omitted means the server uses the Cosmos3 default |
+| `image` | I2V conditioning image; raw base64, `data:image/...;base64,...`, or public URL when URL inputs are enabled |
+| `guidance_scale` | `1.0` to `7.0`, default `6.0` |
+| `steps` | `1` to `100`, default `35` |
+| `resolution` | `256`, `480`, `720`, optionally with `_16_9`, `_1_1`, `_9_16`, `_4_3`, or `_3_4` |
+| `num_output_frames` | Must follow the 4k+1 cadence (`25, 29, 33, ...`); caps: `256 <= 397`, `480 <= 297`, `720 <= 197` |
+| `fps` | `1.0` to `60.0`, recommended `10` to `30`, default `24.0` |
+
+### Notebook walkthrough
+
+[`run_with_nim.ipynb`](./run_with_nim.ipynb) launches the NIM container,
+waits for readiness, inspects the service endpoints, sends T2V and I2V requests,
+decodes `b64_video`, and previews the generated MP4 files inline.
+
+### Limitations
+
+`Cosmos3-Generator` NIM currently exposes **Text2Video** and **Image2Video** only.
+It does **not** expose text-to-image, video-to-video, sound/audio generation,
+action modes, or transfer controls. For those broader Generator API workflows,
+use vLLM-Omni or Cosmos Framework as appropriate.
diff --git a/cookbooks/cosmos3/generator/audiovisual/run_with_nim.ipynb b/cookbooks/cosmos3/generator/audiovisual/run_with_nim.ipynb
new file mode 100644
index 00000000..bebed76a
--- /dev/null
+++ b/cookbooks/cosmos3/generator/audiovisual/run_with_nim.ipynb
@@ -0,0 +1,397 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "license-header",
+ "metadata": {},
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "intro",
+ "metadata": {},
+ "source": [
+ "### Cosmos3 Generator inference with NVIDIA NIM\n",
+ "\n",
+ "This notebook runs the prebuilt `Cosmos3-Generator` NIM container and sends video-generation requests to its HTTP API. It mirrors the Reasoner NIM cookbook structure, but uses the Generator NIM API contract:\n",
+ "\n",
+ "1. Launch `nvcr.io/nim/nvidia/cosmos3-generator:1.0.0`.\n",
+ "2. Wait for `GET /v1/health/ready`.\n",
+ "3. Inspect `/v1/models`, `/v1/metadata`, and `/v1/version`.\n",
+ "4. Send Text2Video and Image2Video requests to `POST /v1/infer`.\n",
+ "5. Decode the JSON response field `b64_video` into `.mp4` files and preview them.\n",
+ "\n",
+ "The NIM container downloads model artifacts from NGC into a local cache on first launch. You need an `NGC_API_KEY` and Docker login to `nvcr.io` before starting the container.\n",
+ "\n",
+ "Useful links:\n",
+ "- Container [page](https://catalog.ngc.nvidia.com/orgs/nim/nvidia/containers/cosmos3-generator/-) on NGC\n",
+ "- [Documentation](https://docs.nvidia.com/nim/cosmos/latest/introduction.html) for the Cosmos NIMs, each page have a table specific to Cosmos3-Generator NIM"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "capabilities",
+ "metadata": {},
+ "source": [
+ "## Capability scope\n",
+ "\n",
+ "`Cosmos3-Generator` NIM currently exposes only the two video-generation modes below. It infers the mode automatically from request fields; there is no explicit `mode` field.\n",
+ "\n",
+ "| Workflow | Request shape | Output | Supported here? |\n",
+ "| --- | --- | --- | --- |\n",
+ "| Text2Video | non-empty `prompt`, no `image` | JSON with `b64_video` | Yes |\n",
+ "| Image2Video | `image` plus optional `prompt` | JSON with `b64_video` | Yes |\n",
+ "| Text-to-image | — | image | No |\n",
+ "| Video-to-video | — | video | No |\n",
+ "| Sound/audio generation | — | video + audio | No |\n",
+ "| Action modes | — | video/action | No |\n",
+ "| Transfer controls | — | controlled video | No |\n",
+ "\n",
+ "The request schema used in this notebook is Cosmos3-specific: `prompt`, `negative_prompt`, `image`, `seed`, `guidance_scale`, `steps`, `resolution`, `num_output_frames`, and `fps`. Unknown request fields are rejected by the server."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "launch-intro",
+ "metadata": {},
+ "source": [
+ "## 1. Launch the NIM container\n",
+ "\n",
+ "Set `NGC_API_KEY` before launching Jupyter, or set it in the notebook environment. Authenticate Docker once with:\n",
+ "\n",
+ "```bash\n",
+ "echo \"$NGC_API_KEY\" | docker login nvcr.io --username '$oauthtoken' --password-stdin\n",
+ "```\n",
+ "\n",
+ "The default launch below uses Nano (`NIM_MODEL_SIZE=nano`), FP8 (`NIM_PRECISION=fp8`), and the latency profile (`NIM_PERF_PROFILE=latency`). Set `NIM_MODEL_SIZE=super` before running the launch cell to serve the larger model."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "helpers",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from pathlib import Path\n",
+ "import base64\n",
+ "import json\n",
+ "import mimetypes\n",
+ "import os\n",
+ "import subprocess\n",
+ "import time\n",
+ "\n",
+ "import requests\n",
+ "from IPython.display import Video, display\n",
+ "\n",
+ "\n",
+ "def find_repo_root() -> Path:\n",
+ " try:\n",
+ " return Path(\n",
+ " subprocess.check_output([\"git\", \"rev-parse\", \"--show-toplevel\"], text=True).strip()\n",
+ " ).resolve()\n",
+ " except Exception:\n",
+ " return Path.cwd().resolve()\n",
+ "\n",
+ "\n",
+ "COSMOS_ROOT = find_repo_root()\n",
+ "AUDIOVISUAL_ROOT = COSMOS_ROOT / \"cookbooks\" / \"cosmos3\" / \"generator\" / \"audiovisual\"\n",
+ "ASSETS_ROOT = AUDIOVISUAL_ROOT / \"assets\"\n",
+ "OUTPUT_DIR = AUDIOVISUAL_ROOT / \"outputs\" / \"cosmos3_generator_nim\"\n",
+ "OUTPUT_DIR.mkdir(parents=True, exist_ok=True)\n",
+ "\n",
+ "I2V_IMAGE = ASSETS_ROOT / \"images\" / \"image2video\" / \"humanoid_robot.jpg\"\n",
+ "assert I2V_IMAGE.exists(), I2V_IMAGE\n",
+ "\n",
+ "NIM_PORT = int(os.environ.get(\"NIM_PORT\", \"8000\"))\n",
+ "BASE_URL = f\"http://127.0.0.1:{NIM_PORT}\"\n",
+ "\n",
+ "\n",
+ "def file_data_uri(path: Path) -> str:\n",
+ " mime = mimetypes.guess_type(path.name)[0] or \"application/octet-stream\"\n",
+ " encoded = base64.b64encode(path.read_bytes()).decode(\"ascii\")\n",
+ " return f\"data:{mime};base64,{encoded}\"\n",
+ "\n",
+ "\n",
+ "def write_b64_video(response_json: dict, path: Path) -> Path:\n",
+ " data = response_json[\"b64_video\"]\n",
+ " path.write_bytes(base64.b64decode(data))\n",
+ " return path\n",
+ "\n",
+ "\n",
+ "def post_infer(payload: dict, output_name: str) -> Path:\n",
+ " response = requests.post(\n",
+ " f\"{BASE_URL}/v1/infer\",\n",
+ " json=payload,\n",
+ " headers={\"Accept\": \"application/json\", \"Content-Type\": \"application/json\"},\n",
+ " timeout=60 * 60,\n",
+ " )\n",
+ " response.raise_for_status()\n",
+ " output_path = OUTPUT_DIR / output_name\n",
+ " return write_b64_video(response.json(), output_path)\n",
+ "\n",
+ "\n",
+ "print(\"cosmos root:\", COSMOS_ROOT)\n",
+ "print(\"audiovisual root:\", AUDIOVISUAL_ROOT)\n",
+ "print(\"i2v image:\", I2V_IMAGE)\n",
+ "print(\"outputs:\", OUTPUT_DIR)\n",
+ "print(\"NIM base URL:\", BASE_URL)\n",
+ "print(\"I2V data URI prefix:\", file_data_uri(I2V_IMAGE)[:64] + \"...\")\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "launch-docker",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%%bash\n",
+ "set -euo pipefail\n",
+ "\n",
+ ": \"${NGC_API_KEY:?Set NGC_API_KEY before launching the Cosmos3-Generator NIM}\"\n",
+ "\n",
+ "export CONTAINER_NAME=\"${CONTAINER_NAME:-nvidia-cosmos3-generator}\"\n",
+ "export IMG_NAME=\"${IMG_NAME:-nvcr.io/nim/nvidia/cosmos3-generator:1.0.0}\"\n",
+ "export NIM_MODEL_SIZE=\"${NIM_MODEL_SIZE:-nano}\"\n",
+ "export NIM_PRECISION=\"${NIM_PRECISION:-fp8}\"\n",
+ "export NIM_PERF_PROFILE=\"${NIM_PERF_PROFILE:-latency}\"\n",
+ "export LOCAL_NIM_CACHE=\"${LOCAL_NIM_CACHE:-$HOME/.cache/nim}\"\n",
+ "export NIM_PORT=\"${NIM_PORT:-8000}\"\n",
+ "mkdir -p \"$LOCAL_NIM_CACHE\"\n",
+ "chmod -R 777 \"$LOCAL_NIM_CACHE\" 2>/dev/null || true\n",
+ "\n",
+ "# The container name must be free. If a previous run is still up, stop it first:\n",
+ "# docker stop nvidia-cosmos3-generator\n",
+ "docker run -d --rm --name=\"$CONTAINER_NAME\" \\\n",
+ " --runtime=nvidia \\\n",
+ " --gpus all \\\n",
+ " --shm-size=32GB \\\n",
+ " --ulimit nofile=65536:65536 \\\n",
+ " -e NGC_API_KEY=\"$NGC_API_KEY\" \\\n",
+ " -e NIM_MODEL_SIZE=\"$NIM_MODEL_SIZE\" \\\n",
+ " -e NIM_PRECISION=\"$NIM_PRECISION\" \\\n",
+ " -e NIM_PERF_PROFILE=\"$NIM_PERF_PROFILE\" \\\n",
+ " -v \"$LOCAL_NIM_CACHE:/opt/nim/.cache\" \\\n",
+ " -p \"${NIM_PORT}:8000\" \\\n",
+ " \"$IMG_NAME\"\n",
+ "\n",
+ "echo \"NIM container '$CONTAINER_NAME' launching on port $NIM_PORT\"\n",
+ "echo \"model_size=$NIM_MODEL_SIZE precision=$NIM_PRECISION profile=$NIM_PERF_PROFILE\"\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "wait-intro",
+ "metadata": {},
+ "source": [
+ "## 2. Wait for readiness\n",
+ "\n",
+ "The first run downloads model artifacts and may spend additional time compiling/warming the engine. The server is ready for inference when `GET /v1/health/ready` returns HTTP 200."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "wait-ready",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%%bash\n",
+ "set -euo pipefail\n",
+ "\n",
+ "PORT=\"${NIM_PORT:-8000}\"\n",
+ "CONTAINER_NAME=\"${CONTAINER_NAME:-nvidia-cosmos3-generator}\"\n",
+ "\n",
+ "echo \"Waiting for Cosmos3-Generator NIM on port ${PORT}...\"\n",
+ "\n",
+ "for i in $(seq 1 3600); do\n",
+ " if curl -fsS \"http://127.0.0.1:${PORT}/v1/health/ready\" >/dev/null 2>&1; then\n",
+ " echo \"NIM server is ready.\"\n",
+ " exit 0\n",
+ " fi\n",
+ " if ! docker ps --format '{{.Names}}' | grep -q \"^${CONTAINER_NAME}$\"; then\n",
+ " echo \"Container '${CONTAINER_NAME}' is no longer running. Recent logs:\"\n",
+ " docker logs --tail 160 \"$CONTAINER_NAME\" 2>&1 || true\n",
+ " exit 1\n",
+ " fi\n",
+ " if (( i % 30 == 0 )); then\n",
+ " echo \"Still waiting... recent logs:\"\n",
+ " docker logs --tail 20 \"$CONTAINER_NAME\" 2>&1 || true\n",
+ " fi\n",
+ " sleep 2\n",
+ "done\n",
+ "\n",
+ "echo \"Timed out waiting for NIM server. Recent logs:\"\n",
+ "docker logs --tail 160 \"$CONTAINER_NAME\" 2>&1 || true\n",
+ "exit 1\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "inspect-intro",
+ "metadata": {},
+ "source": [
+ "## 3. Inspect the running service\n",
+ "\n",
+ "`Cosmos3-Generator` exposes standard NIM health/metadata endpoints plus `/v1/version` and a live OpenAPI schema at `/openapi.json`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "inspect-service",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "for endpoint in [\"/v1/models\", \"/v1/metadata\", \"/v1/version\"]:\n",
+ " response = requests.get(f\"{BASE_URL}{endpoint}\", timeout=30)\n",
+ " response.raise_for_status()\n",
+ " print(f\"\\n### {endpoint}\")\n",
+ " print(json.dumps(response.json(), indent=2)[:4000])\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "schema-intro",
+ "metadata": {},
+ "source": [
+ "## 4. Request parameters\n",
+ "\n",
+ "Useful constraints for `POST /v1/infer`:\n",
+ "\n",
+ "| Field | Constraint / default |\n",
+ "| --- | --- |\n",
+ "| `guidance_scale` | float in `[1.0, 7.0]`, default `6.0` |\n",
+ "| `steps` | int in `[1, 100]`, default `35` |\n",
+ "| `resolution` | `256`, `480`, `720`, optionally suffixed with `_16_9`, `_1_1`, `_9_16`, `_4_3`, `_3_4`; bare tiers mean 16:9 |\n",
+ "| `num_output_frames` | must follow the `4k+1` cadence: `25, 29, 33, ...`; per-tier caps are `256 <= 397`, `480 <= 297`, `720 <= 197` |\n",
+ "| `fps` | float in `[1.0, 60.0]`, recommended `10` to `30`, default `24.0` |\n",
+ "\n",
+ "The examples use `resolution=\"256\"` and `num_output_frames=25` so a smoke test completes faster. Increase these values for quality once the workflow is verified."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "t2v-intro",
+ "metadata": {},
+ "source": [
+ "## 5. Text2Video\n",
+ "\n",
+ "For Text2Video, provide a non-empty `prompt` and omit `image`. The response is JSON; decode `response[\"b64_video\"]` to get the MP4 bytes."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "t2v-request",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "t2v_payload = {\n",
+ " \"prompt\": \"A humanoid robot walks through a futuristic warehouse, inspecting shelves of mechanical components. Photorealistic, cinematic lighting.\",\n",
+ " \"seed\": 42,\n",
+ " \"guidance_scale\": 6.0,\n",
+ " \"steps\": 35,\n",
+ " \"resolution\": \"256\",\n",
+ " \"num_output_frames\": 25,\n",
+ " \"fps\": 24.0,\n",
+ "}\n",
+ "\n",
+ "t2v_path = post_infer(t2v_payload, \"cosmos3_generator_nim_t2v.mp4\")\n",
+ "print(\"wrote\", t2v_path, t2v_path.stat().st_size, \"bytes\")\n",
+ "display(Video(str(t2v_path), embed=True, html_attributes=\"controls loop\"))\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "i2v-intro",
+ "metadata": {},
+ "source": [
+ "## 6. Image2Video\n",
+ "\n",
+ "For Image2Video, provide an `image` field. The field accepts raw base64, a `data:image/...;base64,...` URI, or a public URL when URL inputs are enabled. This example uses a checked-in local image encoded as a data URI, so the NIM container does not need to see the host filesystem."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "i2v-request",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "i2v_payload = {\n",
+ " \"prompt\": \"The humanoid robot performs a controlled standing backflip in a modern living room, then lands steadily on both feet.\",\n",
+ " \"image\": file_data_uri(I2V_IMAGE),\n",
+ " \"seed\": 123,\n",
+ " \"guidance_scale\": 6.0,\n",
+ " \"steps\": 35,\n",
+ " \"resolution\": \"256\",\n",
+ " \"num_output_frames\": 25,\n",
+ " \"fps\": 24.0,\n",
+ "}\n",
+ "\n",
+ "i2v_path = post_infer(i2v_payload, \"cosmos3_generator_nim_i2v.mp4\")\n",
+ "print(\"wrote\", i2v_path, i2v_path.stat().st_size, \"bytes\")\n",
+ "display(Video(str(i2v_path), embed=True, html_attributes=\"controls loop\"))\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "playback-note",
+ "metadata": {},
+ "source": [
+ "## Playback note\n",
+ "\n",
+ "The generated `b64_video` payload is an MP4 container. Some environments may not preview VP9-in-MP4 inline. If the notebook preview does not play, verify the written file with a player such as `ffplay`, `mpv`, or IINA, or re-encode externally to H.264."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "cleanup-intro",
+ "metadata": {},
+ "source": [
+ "## 7. Optional cleanup\n",
+ "\n",
+ "Stop the detached container when you are done. The cached artifacts stay in `LOCAL_NIM_CACHE` for future launches."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "cleanup",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%%bash\n",
+ "set -euo pipefail\n",
+ "CONTAINER_NAME=\"${CONTAINER_NAME:-nvidia-cosmos3-generator}\"\n",
+ "docker stop \"$CONTAINER_NAME\"\n"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.12.13"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/inference_benchmarks.md b/inference_benchmarks.md
index 1311ddd2..8f70a64e 100644
--- a/inference_benchmarks.md
+++ b/inference_benchmarks.md
@@ -36,7 +36,7 @@ Generator results are published incrementally from internal benchmark runs. **Em
These tables report **Cosmos3-Nano** generator latency in seconds for **image-to-video (i2v)**, **text-to-image (t2i)**, and **text-to-video (t2v)** - the primary vision-generation modes of the omni-model. Benchmarks use BF16 precision, batch size 1, and identical prompts, seeds, and sampler settings across engines where noted below. Video workloads follow the standard Cosmos3 generation profile (189 frames at 24 FPS unless a resolution tier limits frame count).
-Three integration paths are compared. **PyTorch** reports average generation (sampling) time from OSS reference inference with CUDA Graphs enabled where supported. **vLLM-Omni** reports total pipeline time at **720p** on supported GPUs. **Diffusers** reports end-to-end generation time through the Hugging Face `Cosmos3OmniPipeline` without custom CUDA graphs at **256p/1**, **480p/1**, and **720p/1** (320×192, 832×480, and 1280×720). Empty cells indicate a run has not been completed for that GPU, engine, resolution, or tensor-parallel width - tables are filled in as benchmark campaigns finish.
+Four integration paths are compared. **PyTorch** reports average generation (sampling) time from OSS reference inference with CUDA Graphs enabled where supported. **vLLM-Omni** reports total pipeline time at **720p** on supported GPUs. **Diffusers** reports end-to-end generation time through the Hugging Face `Cosmos3OmniPipeline` without custom CUDA graphs at **256p/1**, **480p/1**, and **720p/1** (320×192, 832×480, and 1280×720). **NIM** reports end-to-end request latency using NIM latency profiles with FP8 precision, including request processing, video generation, output encoding, and returning the response. Empty cells indicate a run has not been completed for that GPU, engine, resolution, or tensor-parallel width - tables are filled in as benchmark campaigns finish.
### Text-to-Video (t2v)
@@ -45,27 +45,35 @@ Three integration paths are compared. **PyTorch** reports average generation (sa
| **RTX PRO 6000 Blackwell** | PyTorch | 13.95 | | 4.90 | 180.81 | | | 786.37 | 225.45 | 127.57 |
| | vLLM-Omni | 10.65 | 5.06 | 3.78 | 105.61 | 35.93 | 23.76 | 369.67 | 114.30 | 68.66 |
| | Diffusers | 11.20 | | | 112.00 | | | 392.00 | | |
+| | NIM | 7.93 | 4.69 | 4.00 | 82.69 | 33.57 | 24.49 | 318.69 | 107.83 | 68.50 |
| **H20** | PyTorch | 30.57 | | | 257.51 | | | 931.39 | 268.88 | 157.71 |
| | vLLM-Omni | 28.58 | 10.20 | 7.70 | 256.97 | 77.42 | 47.53 | 929.81 | 260.75 | 148.46 |
| | Diffusers | 30.20 | | | 258.00 | | | 926.00 | | |
+| | NIM | 17.81 | 7.84 | 6.14 | 192.07 | 62.80 | 41.67 | 771.37 | 223.24 | 132.68 |
| **H100 NVL** | PyTorch | 10.03 | 4.27 | 3.95 | 84.12 | 29.18 | 21.46 | 297.27 | 94.15 | 61.63 |
| | vLLM-Omni | 9.25 | 3.68 | 3.15 | 80.75 | 27.48 | 18.77 | 311.13 | 88.25(*) | 54.01(*) |
| | Diffusers | 11.00 | | | 90.00 | | | 324.20 | | |
+| | NIM | 7.09 | 3.77 | 3.65 | 68.01 | 25.67 | 20.51 | 267.73 | 86.18 | 57.38 |
| **H200 NVL** | PyTorch | 8.17 | | | 69.79 | | | 244.39 | 77.35 | 45.70 |
| | vLLM-Omni | 7.44 | 3.27 | 2.33 | 64.58 | 21.31 | 12.92 | 240.05 | 69.63 | 39.17 |
| | Diffusers | 9.00 | | | 74.00 | | | 276.20 | | |
+| | NIM | 5.87 | 3.38 | 3.00 | 57.10 | 21.74 | 15.04 | 229.63 | 71.32 | 43.34 |
| **H100 80GB HBM3** | PyTorch | 7.61 | 3.50 | 3.17 | 59.83 | 21.23 | 14.37 | 207.78 | 66.94 | 41.81 |
| | vLLM-Omni | 6.97 | 3.45 | 3.49 | 58.17 | 19.95 | 13.46 | 202.29 | 62.82 | 37.80 |
| | Diffusers | 9.00 | | | 68.00 | | | 240.00 | | |
+| | NIM | 5.72 | 3.36 | 3.12 | 51.73 | 20.26 | 14.82 | 199.46 | 65.32 | 41.66 |
| **H200 141GB HBM3** | PyTorch | 7.53 | 3.34 | 3.19 | 60.18 | 20.84 | 13.97 | 214.28 | 67.48 | 41.26 |
| | vLLM-Omni | 6.79 | 3.25 | 3.42 | 58.14 | 19.77 | 12.97 | 208.36 | 63.27 | 37.49 |
| | Diffusers | 9.00 | | | 67.00 | | | 239.60 | | |
+| | NIM | 5.72 | 3.26 | 3.10 | 51.90 | 20.09 | 14.61 | 200.63 | 64.57 | 40.70 |
| **B200** | PyTorch | 4.56 | 2.78 | 2.79 | 33.20 | 13.20 | 9.69 | 114.85 | 39.75 | 26.27 |
| | vLLM-Omni | 4.03 | 2.43 | 3.49 | 32.04 | 12.63 | 10.09 | 107.84 | 35.29 | 22.87 |
| | Diffusers | 7.00 | | | 36.80 | | | 117.00 | | |
+| | NIM | 3.72 | 2.74 | 2.89 | 26.68 | 12.66 | 10.10 | 93.33 | 35.04 | 24.64 |
| **B300** | PyTorch | | | | | | | | | |
| | vLLM-Omni | 4.46 | 4.11 | 5.44 | 32.18 | 13.83 | 11.57 | 102.10 | 35.68 | 24.33 |
| | Diffusers | 39.40 | | | 63.40 | | | 139.40 | | |
+| | NIM | 4.51 | 3.54 | 4.11 | 27.49 | 14.51 | 11.86 | 90.39 | 37.28 | 26.19 |
### Image-to-Video (i2v)
@@ -74,27 +82,35 @@ Three integration paths are compared. **PyTorch** reports average generation (sa
| **RTX PRO 6000 Blackwell** | PyTorch | | | | 182.14 | | | 788.80 | 226.25 | 127.79 |
| | vLLM-Omni | 11.04 | 5.48 | 4.24 | 107.77 | 38.05 | 25.95 | 375.01 | 119.27 | 73.57 |
| | Diffusers | 12.00 | | | 112.00 | | | 397.00 | | |
+| | NIM | 8.23 | 5.28 | 4.64 | 84.58 | 35.74 | 26.63 | 326.96 | 112.27 | 73.31 |
| **H20** | PyTorch | 31.36 | | | 257.10 | | | 933.07 | 268.99 | 158.10 |
| | vLLM-Omni | 29.50 | 11.26 | 8.64 | 261.56 | 81.93 | 52.06 | 940.16 | 271.37 | 158.76 |
| | Diffusers | 31.00 | | | 258.00 | | | 925.00 | | |
+| | NIM | 18.67 | 9.04 | 7.38 | 195.10 | 67.64 | 46.30 | 774.88 | 233.92 | 143.15 |
| **H100 NVL** | PyTorch | 10.19 | 4.31 | 3.99 | 84.50 | 28.69 | 21.52 | 298.57 | 95.76 | 60.58 |
| | vLLM-Omni | 9.62 | 4.11 | 3.63 | 82.61 | 29.35 | 20.73 | 286.33 | 92.23(*) | 58.02(*) |
| | Diffusers | 11.00 | | | 91.00 | | | 325.20 | | |
+| | NIM | 7.39 | 4.36 | 4.28 | 69.39 | 27.67 | 22.43 | 272.29 | 90.26 | 61.55 |
| **H200 NVL** | PyTorch | 8.27 | | | 69.99 | | | 246.62 | 77.69 | 45.99 |
| | vLLM-Omni | 7.83 | 3.69 | 2.78 | 66.39 | 22.93 | 14.58 | 243.52 | 73.26 | 42.86 |
| | Diffusers | 9.00 | | | 74.00 | | | 275.20 | | |
+| | NIM | 6.25 | 3.97 | 3.60 | 58.64 | 23.33 | 16.75 | 232.47 | 75.13 | 47.22 |
| **H100 80GB HBM3** | PyTorch | 7.64 | 3.47 | 3.21 | 59.95 | 21.40 | 14.43 | 207.87 | 67.52 | 41.66 |
| | vLLM-Omni | 7.37 | 3.81 | 3.97 | 59.77 | 21.68 | 15.12 | 205.97 | 66.52 | 41.51 |
| | Diffusers | 9.00 | | | 68.00 | | | 239.80 | | |
+| | NIM | 6.08 | 3.89 | 3.71 | 53.02 | 22.20 | 16.61 | 202.59 | 69.02 | 45.16 |
| **H200 141GB HBM3** | PyTorch | 7.65 | 3.37 | 3.17 | 60.51 | 21.01 | 14.07 | 214.80 | 67.14 | 41.00 |
| | vLLM-Omni | 7.28 | 3.63 | 3.83 | 59.64 | 21.35 | 14.67 | 209.65 | 66.65 | 40.77 |
| | Diffusers | 9.00 | | | 67.20 | | | 240.00 | | |
+| | NIM | 6.04 | 3.80 | 3.65 | 53.25 | 21.89 | 16.23 | 203.66 | 68.30 | 44.29 |
| **B200** | PyTorch | 4.60 | 2.77 | 2.81 | | 13.07 | 9.66 | 113.90 | 40.01 | 26.58 |
| | vLLM-Omni | 4.33 | 2.77 | 3.84 | 33.09 | 13.79 | 11.39 | 110.19 | 37.76 | 25.68 |
| | Diffusers | | | | | | | 116.00 | | |
+| | NIM | 4.05 | 3.43 | 3.60 | 27.72 | 14.02 | 11.43 | 95.57 | 37.76 | 27.29 |
| **B300** | PyTorch | | | | | | | | | |
| | vLLM-Omni | 5.61 | 4.67 | 5.90 | 33.45 | 15.06 | 13.13 | 104.75 | 38.27 | 26.87 |
| | Diffusers | 28.60 | | | 65.60 | | | 139.60 | | |
+| | NIM | 4.50 | 4.96 | 5.03 | 28.90 | 16.06 | 13.25 | 92.59 | 39.49 | 29.00 |
### Text-to-Image (t2i)
@@ -131,13 +147,14 @@ Three integration paths are compared. **PyTorch** reports average generation (sa
3. vLLM-Omni numbers are for the upcoming public release in the vLLM-Omni repo; subject to change before GA. Values marked with (*) are pre-release vLLM-Omni measurements on H100 NVL and may change before GA.
4. Diffusers numbers use the HuggingFace `diffusers` integration without custom CUDA graphs; reported at 256p/1, 480p/1, and 720p/1 (single-GPU only).
5. PyTorch numbers report average generation (sampling) time from OSS inference benchmarking.
-6. At 256p, multi-GPU configurations on B300 may underperform single-GPU due to small-workload TP overhead; single-GPU is recommended at this resolution.
+6. At 256p, multi-GPU configurations on B300 may underperform single-GPU due to small-workload TP overhead; single-GPU is recommended at this resolution.
+7. NIM numbers use latency profiles with FP8 precision and report end-to-end `Request Latency s`, including request processing, video generation, output encoding, and returning the response.
## Cosmos3-Super Generator
These tables report **Cosmos3-Super** generator latency in seconds for **image-to-video (i2v)**, **text-to-image (t2i)**, and **text-to-video (t2v)**. The 32B checkpoint targets higher-quality world generation; expect longer runtimes than Nano at the same resolution. Benchmarks use BF16 precision, batch size 1, and matched prompts, seeds, and sampler settings. Video workloads follow the standard Cosmos3 profile (189 frames at 24 FPS where applicable).
-As with Nano, three engines are tracked: **PyTorch** (OSS generation/sampling time), **vLLM-Omni** (total pipeline time at 720p on supported GPUs), and **Diffusers** (Hugging Face `Cosmos3OmniPipeline` end-to-end time at **256p/1**, **480p/1**, and **720p/1** — 320×192, 832×480, and 1280×720). Super coverage is narrower than Nano in early releases - for example, vLLM-Omni and Diffusers runs exist primarily on B200 and select H200 configurations. **Empty cells are pending measurements**, not unsupported configurations.
+As with Nano, four engines are tracked: **PyTorch** (OSS generation/sampling time), **vLLM-Omni** (total pipeline time at 720p on supported GPUs), **Diffusers** (Hugging Face `Cosmos3OmniPipeline` end-to-end time at **256p/1**, **480p/1**, and **720p/1** — 320×192, 832×480, and 1280×720), and **NIM** (end-to-end request latency using latency profiles with FP8 precision, including request processing, video generation, output encoding, and returning the response). Super coverage is narrower than Nano in early releases - for example, vLLM-Omni and Diffusers runs exist primarily on B200 and select H200 configurations. **Empty cells are pending measurements**, not unsupported configurations.
### Text-to-Video (t2v)
@@ -146,27 +163,35 @@ As with Nano, three engines are tracked: **PyTorch** (OSS generation/sampling ti
| **RTX PRO 6000 Blackwell** | PyTorch | | | | | | | | 789.03 | 427.16 |
| | vLLM-Omni | | | | | | | | | |
| | Diffusers | | | | | | | | | |
+| | NIM | | 12.65 | 13.99 | | 104.25 | 99.05 | | 350.74 | 286.02 |
| **H20** | PyTorch | | | | | | | | | 492.41 |
| | vLLM-Omni | | | | | | | | | |
| | Diffusers | | | | | | | | | |
+| | NIM | | 20.07 | 12.95 | | 192.45 | 110.71 | | 734.37 | 395.56 |
| **H100 NVL** | PyTorch | | | 16.83 | | 101.27 | 64.14 | | 330.04 | 186.19 |
| | vLLM-Omni | | | | | | | | | |
| | Diffusers | | | | | | | | | |
+| | NIM | | 8.77 | 12.73 | | 73.37 | 66.07 | | 267.64 | 197.32 |
| **H200 NVL** | PyTorch | | | | | | | | 258.34 | 139.37 |
| | vLLM-Omni | 27.54 | | 5.06 | 252.33 | | 36.66 | 911.49 | 245.51 | 123.85 |
| | Diffusers | 33.00 | | | 286.80 | | | 1036.00 | | |
+| | NIM | 17.13 | 6.79 | 4.43 | 200.00 | 58.55 | 32.87 | 811.41 | 223.00 | 117.98 |
| **H100 80GB HBM3** | PyTorch | | | | | | | | | |
| | vLLM-Omni | | | | | | | | | |
| | Diffusers | | | | | | | | | |
+| | NIM | | 6.98 | 5.89 | | 55.10 | 35.52 | | 198.13 | 114.92 |
| **H200 141GB HBM3** | PyTorch | | 14.82 | 11.82 | | 70.27 | 41.78 | | 224.43 | 123.49 |
| | vLLM-Omni | 25.61 | | 5.87 | 219.11 | | 35.26 | 769.63 | 212.30 | 111.94 |
| | Diffusers | 31.00 | | | 251.60 | | | 886.20 | | |
+| | NIM | 15.95 | 6.14 | 4.28 | 174.71 | 52.94 | 30.94 | 695.89 | 194.34 | 106.16 |
| **B200** | PyTorch | | 5.59 | 4.09 | 114.38 | 35.73 | 21.39 | 407.50 | 118.38 | 65.93 |
| | vLLM-Omni | 13.84 | | 4.76 | 114.08 | | 22.09 | 390.28 | 113.31 | 62.11 |
| | Diffusers | | | | 127.20 | | | 414.40 | | |
+| | NIM | 9.09 | 4.26 | 3.38 | 82.39 | 27.83 | 17.74 | 314.68 | 92.25 | 53.43 |
| **B300** | PyTorch | | | | | | | | | |
| | vLLM-Omni | 14.57 | | 6.68 | 109.03 | | 22.67 | 366.66 | 108.58 | 60.73 |
| | Diffusers | 54.20 | | | 155.40 | | | 424.80 | | |
+| | NIM | 9.67 | 5.23 | 5.19 | 79.73 | 28.97 | 18.39 | 292.35 | 92.31 | 54.07 |
### Image-to-Video (i2v)
@@ -175,27 +200,35 @@ As with Nano, three engines are tracked: **PyTorch** (OSS generation/sampling ti
| **RTX PRO 6000 Blackwell** | PyTorch | | | | | | | | 795.14 | 427.96 |
| | vLLM-Omni | | | | | | | | | |
| | Diffusers | | | | | | | | | |
+| | NIM | | 13.17 | 14.48 | | 106.50 | 100.23 | | 356.60 | 289.93 |
| **H20** | PyTorch | | | | | | | | 931.74 | |
| | vLLM-Omni | | | | | | | | | |
| | Diffusers | | | | | | | | | |
+| | NIM | | 21.16 | 14.06 | | 196.86 | 114.49 | | 745.12 | 405.68 |
| **H100 NVL** | PyTorch | | 20.85 | 16.96 | | 99.56 | | | 331.40 | 186.47 |
| | vLLM-Omni | | | | | | | | | |
| | Diffusers | | | | | | | | | |
+| | NIM | | 9.32 | 13.30 | | 75.31 | 67.34 | | 271.46 | 201.39 |
| **H200 NVL** | PyTorch | | | | | | | | 265.33 | 138.31 |
| | vLLM-Omni | 27.90 | | 5.52 | 254.29 | | 38.51 | 915.05 | 248.89 | 127.32 |
| | Diffusers | 33.00 | | | 287.20 | | | 1034.60 | | |
+| | NIM | 17.51 | 7.43 | 5.04 | 201.45 | 60.15 | 34.48 | 817.35 | 226.38 | 121.35 |
| **H100 80GB HBM3** | PyTorch | | | | | | | | | |
| | vLLM-Omni | | | | | | | | | |
| | Diffusers | | | | | | | | | |
+| | NIM | | 7.50 | 6.49 | | 56.33 | 36.81 | | 200.97 | 118.77 |
| **H200 141GB HBM3** | PyTorch | | 14.87 | 11.80 | | 70.45 | 42.10 | | 224.36 | 123.57 |
| | vLLM-Omni | 25.47 | | 6.32 | 220.70 | | 36.90 | 766.33 | 215.03 | 117.52 |
| | Diffusers | 31.00 | | | 249.20 | | | 879.20 | | |
+| | NIM | 16.39 | 6.74 | 4.90 | 175.95 | 54.46 | 32.51 | 699.13 | 197.96 | 109.55 |
| **B200** | PyTorch | 14.71 | 5.63 | 4.12 | | 35.70 | 21.25 | 397.31 | 117.98 | 65.91 |
| | vLLM-Omni | 14.13 | | 5.31 | 115.17 | | 23.26 | 393.02 | 115.69 | 64.82 |
| | Diffusers | 19.20 | | | | | | 414.80 | | |
+| | NIM | 9.36 | 4.83 | 4.12 | 83.19 | 29.09 | 19.14 | 316.76 | 94.65 | 55.92 |
| **B300** | PyTorch | | | | | | | | | |
| | vLLM-Omni | 14.14 | | 7.19 | 111.42 | | 23.91 | 368.73 | 111.41 | 63.25 |
| | Diffusers | 54.20 | | | 151.80 | | | 425.00 | | |
+| | NIM | 9.73 | 5.58 | 5.94 | 80.51 | 30.17 | 20.62 | 294.11 | 93.77 | 56.76 |
### Text-to-Image (t2i)
@@ -232,7 +265,8 @@ As with Nano, three engines are tracked: **PyTorch** (OSS generation/sampling ti
3. vLLM-Omni numbers are for the upcoming public release in the vLLM-Omni repo; subject to change before GA. Current vLLM-Omni coverage is B200 at 720p.
4. Diffusers numbers use the HuggingFace `diffusers` integration without custom CUDA graphs; reported at 256p/1, 480p/1, and 720p/1 (single-GPU only).
5. At 256p, multi-GPU configurations on B300 may underperform single-GPU due to small-workload TP overhead; single-GPU is recommended at this resolution.
-6. PyTorch numbers report average generation (sampling) time from OSS inference benchmarking.
+6. PyTorch numbers report average generation (sampling) time from OSS inference benchmarking.
+7. NIM numbers use latency profiles with FP8 precision and report end-to-end `Request Latency s`, including request processing, video generation, output encoding, and returning the response.
## Cosmos3-Nano Reasoner