Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Loss is nan on validation #98

Open
maiiabocharova opened this issue Sep 3, 2021 · 2 comments
Open

Loss is nan on validation #98

maiiabocharova opened this issue Sep 3, 2021 · 2 comments
Labels
bug Something isn't working

Comments

@maiiabocharova
Copy link

maiiabocharova commented Sep 3, 2021

I have created a custom dataset using labelImg, everything looks fine, but while training I get Loss: nan after every epoch.

I reduced the size of the dataset for better manual checking, checked the names of the files, everything is fine, but the model is not training.

Thought that maybe the thing was that I was using png format and not jpg, but converting my files and labels didn't solve the issue.

Can you help me? Maybe I can send you the dataset? (just 30 images and 30 labels for easier debugging)

@maiiabocharova maiiabocharova added the bug Something isn't working label Sep 3, 2021
@alankbi
Copy link
Owner

alankbi commented Jan 7, 2022

Do any of the suggestions in #91 and #36 work? This issue occasionally comes up for some people and I'm not sure the best solution besides maybe double-checking/recreating environments to see if that helps.

@Tanmay-Trantor
Copy link

Just decrease the learning rate, it might getting large gradient descent. In my case I reduced it to 0.001

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants