vLLM: Phi-4-multimodal-instruct-NVFP4 fails with missing scipy in 26.02-py3 container

[jaycoolslm]

Description

The vLLM playbook lists Phi-4-multimodal-instruct-NVFP4 as supported with nvcr.io/nvidia/vllm:26.02-py3, but serving it fails because the container does not have scipy installed, which is required by Phi-4's custom processor code.

Steps to reproduce

1. Download the model:

   hf download nvidia/Phi-4-multimodal-instruct-NVFP4 --local-dir ~/models/Phi-4-multimodal-instruct-NVFP4

2. Serve with vLLM (as per playbook, with --trust-remote-code as noted):

   docker run --gpus all -p 8001:8001 \
     -v ~/models/Phi-4-multimodal-instruct-NVFP4:/models/Phi-4-multimodal-instruct-NVFP4 \
     nvcr.io/nvidia/vllm:26.02-py3 \
     vllm serve /models/Phi-4-multimodal-instruct-NVFP4 \
       --port 8001 \
       --trust-remote-code \
       --gpu-memory-utilization 0.7

3. Observe the error.

Error

ImportError: This modeling file requires the following packages that were not found in your environment: scipy. Run `pip install scipy`

Full traceback shows:

• vLLM resolves the architecture as Phi4MMForCausalLM successfully
• --trust-remote-code is accepted
• Failure occurs when loading the HuggingFace processor via AutoProcessor.from_pretrained()
• The custom processor code (transformers.dynamic_module_utils) checks imports and finds scipy missing from the container

Environment

• Container: nvcr.io/nvidia/vllm:26.02-py3
• vLLM version: 0.15.1+befbc472
• Device: DGX Spark (GB10, 128GB unified memory)
• Model: nvidia/Phi-4-multimodal-instruct-NVFP4 (downloaded locally)

Expected behaviour

The container should include all dependencies required to serve models listed in the playbook's Model Support Matrix.

Suggested fix

Add scipy to the container's installed packages, or document it as a required manual install step for Phi-4-multimodal models.

[eluinstra]

Maybe you can try to use the latest nvidia vllm image. I had no problem starting it with the following command:

docker run --rm -it --gpus all --ipc=host --network host \
-e HF_TOKEN=$HF_TOKEN \
-v ~/.cache/huggingface:/root/.cache/huggingface \
nvcr.io/nvidia/vllm:26.05-py3 \
vllm serve nvidia/Phi-4-multimodal-instruct-NVFP4 \
  --served-model-name phi-4-multimodal \
  --host 0.0.0.0 --port 8080 \
  --trust-remote-code \
  --gpu-memory-utilization 0.25