Any way to apply Lora result of training script to Diffusers model?

[congdm]

Hi, at the moment I noticed that the transformer model in huggingface-cli Tencent-Hunyuan/HunyuanDiT has different structure than the transformer model in Diffusers pipeline. This makes me cannot apply lora such as links to Diffusers pipeline.

For example, the transformer in Tencent-Hunyuan/HunyuanDiT has fused qkv layer while the one in diffusers is unfused. They also different modules naming too.
In long run, having to train lora separately for both types of transformer is very impractical.

[xljh0520]

Thank you for your attention to our work.
Due to certain compatibility adjustments and modifications made by the diffuser, there may be issues with the model state_dict keys not matching. We have provided a code example in lora README that demonstrates how to merge LoRA weights through keyword matching.
We hope this can help you.