MLLM text encoder

[Bionagato]

Hi, this model is really incredible, but it is really hard to control the outputs. When will you release the MLLM Text Encoder? It seems necessary to control the model's outputs. With Llama, it is really hard, if not impossible, to get videos as good as those published on the project page. It would also be good to have some examples of how the videos in the dataset were captioned to get an idea of how to format the prompts.

[Gobz]

Seconding this, seems to be one of the most important tools to get the most out of the model.

[breadbrowser]

the text Text Encoder is t5 flan xxl

[win10ogod]

the text Text Encoder is t5 flan xxl

MLLM is based on causal attention while T5-XXL utilizes bidirectional attention
From this we can see that it should not be t5!

[kathrinawu]

Please refer to this https://github.com/Tencent/HunyuanVideo/blob/main/ckpts/README.md

[Bionagato]

Please refer to this https://github.com/Tencent/HunyuanVideo/blob/main/ckpts/README.md

I read the document, but it doesn't mention a release date for HunyuanMLLM or include examples of how the videos were captioned. It just states "At this stage, we have not yet released HunyuanMLLM".