Add model support for Phi 3.5 MoE

[svaruag]

Motivation

Add support for Phi 3.5 MoE

Modifications

Updated mixtral.py with a custom routing function specific to phi3moe

Checklist

• Format your code according to the Contributor Guide.
• Add unit tests as outlined in the Contributor Guide.
• Update documentation as needed, including docstrings or example tutorials.

[svaruag]

@merrymercy I'm still making progress on this and would appreciate your input. I'm using vllm's phi3 moe implementation as a base, and I'm adjusting the parts that differ from mixtral, specifically the custom top k routing mechanism and layer norm. However, I'm encountering some gibberish outputs during generation. Do you have any insights on debugging this further? Thanks a lot!

[merrymercy]

I am not super familiar with this model. Can you try this interactive debugging approach and compare the activations layer-by-layer? We typically use this method to add new models

https://sgl-project.github.io/references/supported_models.html#interactive-debugging

[svaruag]

Thanks @merrymercy, I was able to fix the problem here (was missing the residuals), will push in the fixes shortly, along with the testing code etc.