Please use params.SetInputs for phi3v. AppendTokens is not supported for this model type. #1151
Comments
### Fix SetInputs for Multi-Modal Models #### Description This PR addresses an issue where `SetInputs`, which is used with multi-modal models like phi3v and whisper, was broken. The error encountered was: ``` RuntimeError: Please use params.SetInputs for phi3v. AppendTokens is not supported for this model type. ``` #### Root Cause The error was caused because `AppendTokens` is an API only meant to be used by models that support it. However, in the case of multi modal models, it was called internally during the construction of the generator to process inputs, leading to the runtime error. #### Fix This PR avoids calling `AppendTokens` internally, thereby fixing the issue with `SetInputs` on multi-modal models. It also fixes this issue: #1151.
|
Thanks for reporting this issue. The linked PR fixes this error. |
|
Hello @kunal-vaishnavi, thanks for your prompt help!! |
|
Hello @kunal-vaishnavi, I mentioned you in my blog post. Thanks a lot for your help!! https://www.hackster.io/shahizat/running-microsoft-s-phi-3-5-vision-on-nvidia-jetson-platform-8c69a6 |
Thank you for the mention! Great blog post! |
|
Hi @kunal-vaishnavi, I'm happy to inform you that @johnnynunez has successfully dockerized the onnxruntime-genai. And it's now ready to be used within the @dusty-nv 's jetson-containers project, thanks to his efforts. |
That's great to see! As a heads up, a new stable release is expected to be published this month. |
|
hello @kunal-vaishnavi, Thanks for your information. If possible, you can join our Jetson AI Lab Research Group Discord channel: https://discord.gg/N4E5UCb2 ,I'd be glad to see you there. @dusty-nv and @johnnynunez are also there. |
Greetings to all,
I was running an inference of the Phi-3.5-vision-instruct-onnx model on the NVIDIA Jetson AGX Orin 64GB Developer Kit, following the guide provided by @kunal-vaishnavi. Thank you, I was successful in building ONNX Runtime GenAI on NVIDIA JetPack 6.1, but I encountered the issue below.
I ran the command:
python phi3v.py -m ./gpu/gpu-int4-rtn-block-32/ -p cudaThe log output was:
Loading model... Model loaded Image Path (comma separated; leave empty if no image): test2.jpg Using image: test2.jpg Prompt: describe Processing images and prompt... Generating response... Traceback (most recent call last): File "/home/jetson/Projects/tensorrtllm/Phi-3.5-vision-instruct-onnx/phi3v.py", line 141, in <module> run(args) File "/home/jetson/Projects/tensorrtllm/Phi-3.5-vision-instruct-onnx/phi3v.py", line 99, in run generator = og.Generator(model, params) RuntimeError: Please use params.SetInputs for phi3v. AppendTokens is not supported for this model type.Thanks in advance for your help!
The text was updated successfully, but these errors were encountered: