Audio normalization in train/val/test set

[thanhtvt]

Hi @hkchengrex,

Congratulations on your great work! I really enjoyed reading your paper and running your well-organized codebase. I noticed one thing: in the snippet below, it seems that you're normalizing the audio from the training set but not from the validation set. I understand why this might be the case for the test set, since it's not used for generation/evaluation anyway, but could you clarify why the validation set is treated differently?

MMAudio/training/extract_video_training_latents.py

Lines 54 to 75 in ec6ab44

	data_cfg = {
	'example': {
	'root': './training/example_videos',
	'subset_name': './training/example_video.tsv',
	'normalize_audio': True,
	},
	# 'train': {
	# 'root': '../data/video',
	# 'subset_name': './sets/vgg3-train.tsv',
	# 'normalize_audio': True,
	# },
	# 'test': {
	# 'root': '../data/video',
	# 'subset_name': './sets/vgg3-test.tsv',
	# 'normalize_audio': False,
	# },
	# 'val': {
	# 'root': '../data/video',
	# 'subset_name': './sets/vgg3-val.tsv',
	# 'normalize_audio': False,
	# },
	}

[hkchengrex]

Thanks for your interest in our project! This is a good question. One reason is that we want the validation set to be more like the test set -- since we cannot normalize the audio in the test set, we also do not normalize it for the validation set. There could also be arguments for do normalizing the audio in the validation set (e.g., such that it better reflects convergence). We did not ablate this choice.

[thanhtvt]

When I followed your settings and trained the model on VGGSound only, I couldn’t generate meaningful audio—the output was very noisy. I also noticed that the model barely generalizes to the validation set: the loss decreases only slightly before rising again quite rapidly (see attached). Could you provide some guidance on this? I kept the training settings the same, except that I reduced the number of training steps to 150k and set the learning rate schedule to [100k, 120k].

[thanhtvt]

Also, do you still have the generated samples from when you trained on VGGSound only? I’d like to use them as references.

[hkchengrex]

For reference, you can take a look at this issue: #49 for the loss curve

If you don't train on the audio-text datasets, it overfits fairly quickly. We apply early stop as mentioned in the paper (by looking at the validation performance).

[hkchengrex]

Also, do you still have the generated samples from when you trained on VGGSound only? I’d like to use them as references.

I don't have those samples anymore.

[thanhtvt]

Thanks for the quick reply! Issue #49 discusses training the 44k-large version, while I’m currently using the 16k-small one. Are there any differences in the loss curves between the two?

[hkchengrex]

The screenshot in that post is from an ablation of the 16k-small model, but the loss trend should be similar.