NaN losses

[DandiC]

I'm training the model with different datasets and image sizes. I have noticed that if I use images larger than 64x64, all the losses become NaN after a certain point. It seems like it happens earlier as the image size increase (for instance, it happens after 77 epochs with 128x128 images and after 3 epochs with 1024x1024 images). Do you happen to know why this is happening and do you have any advice to address it?

Thanks!

[FIlipHand]

I had the same problem multiple times on my own. It's happening because of something known as gradient explosion, which means that loss values are to big and they are overflowing therefore you are getting nones. There are some things you can try to apply and they are written here: https://machinelearningmastery.com/exploding-gradients-in-neural-networks/