CUDA error: device-side assert triggered

[stealth0414]

when i train my own model in epoch 100,it has error in loss so that stop training
/pytorch/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [1237,0,0], thread: [25,0,0] Assertion input_val >= zero && input_val <= one failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [1254,0,0], thread: [15,0,0] Assertion input_val >= zero && input_val <= one failed.
it looks like in l1_loss.py line 11
if mask_sum.item() == 0:
how to fix it plz,thanks

[goseesomething]

try edit learning rate params. It's because the dataset is difference and the params is not the best conditions, that's why the calculation between mask and pred coming out NAN value

[MxxM-max]

请问您解决了吗，我也遇到同样的问题