Support Several MLLM Models

[BUAADreamer]

What does this PR do?

This PR is working!!

If you are interested, you can use my branch https://github.com/BUAADreamer/LLaMA-Factory for now.

Support models:

• Video-LLaVA/LLaVA-Video-Next with videos and images as inputs
  • sft with only video/image inputs (mllm_demo/video_demo)
  • sft with video/image mix inputs (visual_mix_demo)
• LLaVA-Next
• Idefics-2

Features:

• fine-tuning: sft/ppo/dpo/kto/orpo/simpo
• inference: add video inference

Before submitting

• Did you read the contributor guideline?

[Davidchu11381]

Hi! By LlaVA-NEXT did you mean specifically this model? Similarly, does LlaVA-NEXT-Video refer to this model?

If not, would you be able to clarify which model you referred to? If yes, please let me know when they're good to use. Thank you!

[BUAADreamer]

All models are hf official models
for LLaVA, you could refer to https://huggingface.co/llava-hf
for idefics2, you could refer to https://huggingface.co/HuggingFaceM4

[Davidchu11381]

Thanks for the clarification! Please let me know when you support the LlaVA-Next-Video model. Thank you!

[BUAADreamer]

This PR has not been merged into main, so every feature in this PR can not be tried yet using latest llama-factory.
You could wait until hiyouga releases the new version of llama-factory or this PR is closed!
@Davidchu11381 Please delete the above two comments so as not to mislead others!! Thanks!! 🤗

[Davidchu11381]

I did! Thank you for your help! Could you also ping hiyouga for this issue? Sorry for bothering you. My lab (USC ISI) is working on something and we would really appreciate your help if we could use your tool to use LlaVA-Next. Please let me know when this is done :D Thanks again!

[JianbangZ]

How about miniCPM V2.5?

[BUAADreamer]

How about miniCPM V2.5?

On the road!