Could you share distill log for each dataset ?

[JarveeLee]

As title, for nuScenes, ScanNet, Matterport, it would be great if you can share the logs of distilling, which can help me to debug the training process ...

[pengsongyou]

Sorry I did not take the logs out from my internship, so I don't really have it anymore. Maybe you can specify what kind of issues you are encountering?

[JarveeLee]

I am distilling on my own data.
Occasionally I get 'WARNING:tensorboardX.x2num:NaN or Inf found in input tensor.
' then the loss is Nan as well. Restart then NaN would disappear ...

I have no idea how to explore this bug ..... It would be great if you can suggest me to inspect something inside my data. I just replace the data_dir in config, while the other remains the same as nuScenes.

[pengsongyou]

Emmmmm, I never encountered such issue before. Make sure that your data preprocessing is correct, and the codes you changed do not divide by 0 or so.

…

On Thu, Jun 1, 2023, 5:37 AM Li Jiahui ***@***.***> wrote: I am distilling on my own data. Occasionally I get 'WARNING:tensorboardX.x2num:NaN or Inf found in input tensor. ' then the loss is Nan as well. Restart then NaN would disappear ... I have no idea how to explore this bug ..... [image: image] <https://user-images.githubusercontent.com/21051247/242471283-39d08935-d716-4636-b460-bac19a23e3b8.png> — Reply to this email directly, view it on GitHub <#30 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADNJ3IXU57USQ4NADIXD4Y3XJAE6HANCNFSM6AAAAAAYVKLH5E> . You are receiving this because you commented.Message ID: ***@***.***>

[Outlying3720]

I meet the same problem too when I run distill on Matterport or ScanNet datasets.

NaN or Inf found in input

Could anyone have some solution?

I try to add gradient clip to max 1 or lower the lr. But still no help.

What's more, I found it will be overfitting and val loss will be NaN. But the training process will not be interrupted.

I run code on Pytorch 2.0 with 4090 GPU. Maybe the Pytorch upgrade caused this problem?

[pengsongyou]

@wengmingao I will investigate this from my end and share the information to you later.