Error loading vision model in 0.51

[elephantpanda]

c# 0.51 Directml

OnnxRuntimeGenAIException: Error encountered while parsing 'D:\Phi3OnnxVision\genai_config.json' JSON Error: Unknown value: visual_features at line 48 index 53

Didn't have this error when running in 0.40

[kunal-vaishnavi]

New ONNX models for Phi-3 vision and Phi-3.5 vision have been uploaded to support no-image, single-image, and multi-image scenarios. You can download the new ONNX models to use with ONNX Runtime GenAI v0.5.1.

[elephantpanda]

Downloaded the new model.
After it loads I get the error:
OnnxRuntimeGenAIException: [ImageDecoder]: Failed when CoInitialize.
when it does this line:
processor = new MultiModalProcessor(model);

Previously there were onnx models specifically for DML or CUDA. Not sure if that is a reason.

OnnxRuntimeGenAIException: [ImageDecoder]: Failed when CoInitialize.
Microsoft.ML.OnnxRuntimeGenAI.Result.VerifySuccess (System.IntPtr nativeResult) (at D:/a/_work/1/onnxruntime-genai/src/csharp/Result.cs:25)
Microsoft.ML.OnnxRuntimeGenAI.MultiModalProcessor..ctor (Microsoft.ML.OnnxRuntimeGenAI.Model model) (at D:/a/_work/1/onnxruntime-genai/src/csharp/MultiModalProcessor.cs:15)
Main.Start () (at Assets/Main.cs:37)

[kunal-vaishnavi]

OnnxRuntimeGenAIException: [ImageDecoder]: Failed when CoInitialize.
Microsoft.ML.OnnxRuntimeGenAI.Result.VerifySuccess (System.IntPtr nativeResult) (at D:/a/_work/1/onnxruntime-genai/src/csharp/Result.cs:25)
Microsoft.ML.OnnxRuntimeGenAI.MultiModalProcessor..ctor (Microsoft.ML.OnnxRuntimeGenAI.Model model) (at D:/a/_work/1/onnxruntime-genai/src/csharp/MultiModalProcessor.cs:15)
Main.Start () (at Assets/Main.cs:37)

Can you try running the phi3v.py example script to verify the models can run with Python? This will help narrow down whether the problem is specific to the C# example or not.

Previously there were onnx models specifically for DML or CUDA. Not sure if that is a reason.

The uploaded ONNX model in the GPU folder should work for both CUDA and DML. We're working on adding the support needed to run the same ONNX model on CPU, CUDA, and DML.

[skyline75489]

@elephantpanda What version of Windows are you using? We currently rely on WIC for image decoding. The failure seems to be related to initializing WIC.

[elephantpanda]

@elephantpanda What version of Windows are you using? We currently rely on WIC for image decoding. The failure seems to be related to initializing WIC.

I am using a Windows 10 cloud computer to run this. So I don't really have control over upgrading this to Windows 11.

BTW, I didn't get this bug in 0.4.0 I believe so I think this may be a new thing.

If I am to release software to the public which runs GenAI I would expect it to work on Windows 10 and Windows 11. If it is relying on Windows 11 only libraries that doesn't seem great. Unless I can add those libraries to my software.

Although WIC says it works from Vista onwards so I don't think that would be the problem.

A nice error message saying "This doesn't work on Windows 10" or "missing library" would be a small nicety.

BTW, I am running this in Unity, so if it is relying on an external image library I might have to add that to my project if I knew what was needed.

[skyline75489]

@elephantpanda Yes, the WIC dependency was newly introduced in 0.5.0. And since it works with Vista+, Windows 10 shouldn't be a problem. Could you please check if these libraries are available in your environment? Especially windowscodecs.dll which should be loaded at runtime.