[Bug]: Wrong output with tp4 and granite 8b

[RajalakshmiSR]

Describe the bug

torch spyre commit - 88db7467c51e2ffcfa34e11b41c5577cc0db9aa9
spyre-inference - 0.1.0.dev537+main.gebf8ca03

How to reproduce

vllm serve /models/granite-3.3-8b-instruct --max-model-len 2048 --max-num-seqs 1 --num-gpu-blocks-override 2049 --port 8000 -tp 4

curl http://127.0.0.1:36550/v1/completions -H 'Content-Type: application/json' -d '{ "model": "/models/granite-3.3-8b-instruct", "prompt": "write a python program for bubble sort", "max_tokens": 128, "temperature": 1 }'

< junk result>

Additional context

Looks good with tp1.

Checklist

• I have searched for similar issues.

[dmholtz]

I cannot reproduce this issue on Z. The generated python program looks plausible.

System Setup:

• spyre-inference - 0.1.0.dev537+main.gebf8ca03
• all other dependencies synced from uv.lock and spyre-rpms.lock

[RajalakshmiSR]

The issue is resolved in the latest image.

[jjhursey]

Cross-reference for future torch-spyre/torch-spyre#4871