Use dynamic learning rate decay for convergence

[begeekmyfriend]

The evaluation sounds better than that with fixed learning rate.

Signed-off-by: begeekmyfriend begeekmyfriend@gmail.com

[seungwonpark]

Hi, your code looks great, and thanks for kindly sending PR!
Can you please show the audio samples you got (w/ the number of epochs) for comparison?

[begeekmyfriend]

melgan_eval_mandarin.zip
I have synthesized voices from 4 anchors (1 male and 3 females). And the checkpoint is only at epoch 375 and still under training. I think it helps convergence.

[delip]

Shouldn't this be init_val / final_val?

[begeekmyfriend]

The learning rate decays. You might write a demo for testing.

init_val = 1e-4
final_val = 1e-5

[casper-hansen]

According to the following source it's "Minimum learning rate value as a fraction of learning_rate."
https://docs.w3cub.com/tensorflow~python/tf/train/cosine_decay/

Given the values, it looks like it's correct. The naming is just off - it should be the smallest value in the numerator and largest value in the denominator.

[bob80333]

Is this different from pytorch's built-in CosineAnnealingLR?

[Liujingxiu23]

@begeekmyfriend
Your tried different lr and found that the cos-lr is the best? And why it is suitable in melgan?
I am confused, when should we use unchanged lr, when should we use lr with decline，for example exponential decline in tacotron, and when shoud we use cos-lr.

[begeekmyfriend]

It is just a preference. Pick it or other as you like.

[Liujingxiu23]

@begeekmyfriend Thank you for your quick reply. I used your branch of tacotron and found it is one of the best among a lot of code branchs. I will try the cos-lr as well as the apex in tfgan.