phi3.5 genai converted model output garbage results with input length around 3000 and 8000.

[yufang67]

Describe the bug
Currently, i use onnxgenai==0.4.0 converted phi_3_5_mini_instruct (fp16 and cuda) and run the infer with onnxgenai on A100 80G.
I observed for some input length around 3000 (8000), i got result length up to the fixed max_length and the results are full of "\n" .

for example, i fixed the max_length is 12K, if the input is 3424 and the output gives 8576 and the output is filled with followings:

n0.\n.\n0.\n.\n.\n0.\n\n\n\n\n\n2.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n.\n.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n.\n2.\n\n\n\n\n2.\n2.\n\n\n.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n.\n\n\n\n\n\n\n.\n\n\n\n\n\n\n\n\n\n\n.\n\n\n\n\n\n.\n0.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n2.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n0.

I compared with the transformer API, i didn't get this kind of results with same model.

Any clue of this issue ? (I have seen for vLLM/ transfomers, there exists an issue, https://huggingface.co/microsoft/Phi-3-mini-128k-instruct/discussions/85, vllm-project/vllm#8254 )
Thanks

[natke]

Thanks for reporting this. Does it matter which prompt you use? Or any long prompt is producing this output?

[yufang67]

It seems only related to length of the prompt, i got several ones around 3000 length have this issue, like 3824, 3613, 3918... And i have also some samples are correct with 4000 and 5000 length.
Not sure if its related to the issue (link above) where it happens the input length <4096 but input + output >4096. But as in onnx we use max_length, we don't have control of the output length.

[natke]

Thank you. Can you share the prompts that produce garbage? The 3000 length and the 8000 length, so that we can repro.

[yufang67]

Sorry, i cant provide the prompt, because its customer data.

[natke]

No problem. I did reproduce garbage output for a prompt length of 3402. We are investigating.

[natke]

We are investigating a fix for this issue

[yufang67]

any updates ? Thanks

[ajindal1]

sorry haven't made much progress on this till now, will prioritize it this week. Thanks.

[yufang67]

Hi @ajindal1 ,
any clue ?
We found similar issue on Biomistral model with fp16 onnx conversion. The input length is around 6K, 7K.
Not sure if its the same issue

[yufang67]

Hi @ajindal1 ,
any update ?
Thanks

[ajindal1]

Hi @yufang67,
Apologies for the late response as I was traveling, looking into this today.

[ajindal1]

Adding this fix into GenAI huggingface/transformers#33129, it should resolve the issue.

[yufang67]

Great, thanks. How can i use this latest fix ? Or there will have a new release soon.
Thanks

[yufang67]

Hi @ajindal1 ,
I still get this issue with onnxruntime_genai 0.5.1. could you confirm that the fix is included in 0.5.1 version ?
Thanks

[ajindal1]

Sorry the fix is not yet available, we are working on the fix and will be part of next release (0.6.0) or you can use the main branch (build from source) for the fix once it is added. I will update once it is complete.