Question about stage2 training. Audio_projection layer doesn't seem to be trained successfully.

[arceus-jia]

I'm doing some similar training but I'm having some problems and would like to ask for your help
I see that you trained audio_projection/motion_module/attn2 for stage2 training.
And When I'm training, it seemed like it was mostly the motion_module that worked, and the audio related layers don't seem to be trained successfully.
The result is that even though the video is smooth, different audio inputs come out with the same mouth movements.
What are your tips for handling this? I've increased the weight of the mouth loss but that doesn't seem to work either.

Thanks.

[Lanzl-lab]

I'm doing some similar training but I'm having some problems and would like to ask for your help I see that you trained audio_projection/motion_module/attn2 for stage2 training. And When I'm training, it seemed like it was mostly the motion_module that worked, and the audio related layers don't seem to be trained successfully. The result is that even though the video is smooth, different audio inputs come out with the same mouth movements. What are your tips for handling this? I've increased the weight of the mouth loss but that doesn't seem to work either.

Thanks.

Could you please let me see your training code, I have met some trouble in training?

[piwawa]

Talk is cheap, show me your code.

[zhangjun001]

I'm doing some similar training but I'm having some problems and would like to ask for your help I see that you trained audio_projection/motion_module/attn2 for stage2 training. And When I'm training, it seemed like it was mostly the motion_module that worked, and the audio related layers don't seem to be trained successfully. The result is that even though the video is smooth, different audio inputs come out with the same mouth movements. What are your tips for handling this? I've increased the weight of the mouth loss but that doesn't seem to work either.

Thanks.

We are working on an easy-to-train framework and will update it later.

[tiankuan93]

I'm doing some similar training but I'm having some problems and would like to ask for your help I see that you trained audio_projection/motion_module/attn2 for stage2 training. And When I'm training, it seemed like it was mostly the motion_module that worked, and the audio related layers don't seem to be trained successfully. The result is that even though the video is smooth, different audio inputs come out with the same mouth movements. What are your tips for handling this? I've increased the weight of the mouth loss but that doesn't seem to work either.
Thanks.

Could you please let me see your training code, I have met some trouble in training?

Our training code has been released, and we hope it will be helpful to you.