100% training WER on custom data

[Ashbajawed]

Hello,

I am finetuning the QuartznetBase-en5x5 model on my own dataset, but I am getting trainingWER 100% and training loss is flatted after few steps although val_WER and val_loss is decreasing gradually

Dataset Details
train Dataset consists of 169630 files totaling 410.18 hours
Val_ Dataset consists of 25036 files totaling 84.82 hours
Audio data is of length between( 5-60secs) and made sure that there are no samples without text output

I'm using the file speech_to_text.py just made a little changes and added the following line before 'trainer.fit'

asr_model.restore_from("QuartzNet15x5Base-En.nemo")

other details are:
trainer.max_epochs=200
model.train_ds.batch_size=16
model.validation_ds.batch_size=16
model.optim.lr=0.00015
+model.validation_ds.num_workers=2
+model.train_ds.num_workers=10
+trainer.precision=16

nemo version = 1.0.0rc1

. Could you please troubleshoot where could the problem be?

[ulucsahin]

It is normal to get 100% WER for first few epochs. How many epochs did train?

[YovelRom]

Also, there might be a mismatch between the datasets. For example, is the sampling rate in your data the same as it is in the LibriSpeech dataset?

[Ashbajawed]

around 50 epoch, after 50 epochs its crashing

It is normal to get 100% WER for first few epochs. How many epochs did train?

[Ashbajawed]

Also, there might be a mismatch between the datasets. For example, is the sampling rate in your data the same as it is in the LibriSpeech dataset?

sample rate is same but when I make a test run on lesser data i.e. around 200 hours at that time it was decreasing

Blue = ~200hours data
pink = ~400 hours data

[okuchaiev]

60s is unnecessarily long. We typically use max 16.7s. Btw, by default, it may ignore everything longer than that.
Such long fragments doesn't help with training too much but will require large amount of GPU ram.

Also, what is your character set in your transcriptions? For QuartzNet, it is lowercase En letters and apostrophe ('), no punctuation or numbers.

Try lowering your learning rate and increasing batch size too.

[titu1994]

Several things - disable spec augment, reduce Weight Decay to 0 and see if the model can learn with no regularization first.
The fact that 200 hours works and 400 doesn't means that your learning rate is too high - with 200 hours you have roughly half the number of steps per epoch so warmup and all other parameters should be adjusted.