HAVE ANYONE MEET SUCH ERROR WHEN TRAINING ON OWN DATASET

[lonelyqian]

error log：

  File "train_net_sod.py", line 408, in <module>
    launch(
  File "/media/data/QFS/detectron2/detectron2/engine/launch.py", line 84, in launch
    main_func(*args)
  File "train_net_sod.py", line 402, in main
    return trainer.train()
  File "/media/data/QFS/detectron2/detectron2/engine/defaults.py", line 486, in train
    super().train(self.start_iter, self.max_iter)
  File "/media/data/QFS/detectron2/detectron2/engine/train_loop.py", line 155, in train
    self.run_step()
  File "/media/data/QFS/detectron2/detectron2/engine/defaults.py", line 496, in run_step
    self._trainer.run_step()
  File "/media/data/QFS/detectron2/detectron2/engine/train_loop.py", line 495, in run_step
    loss_dict = self.model(data)
  File "/root/miniconda3/envs/mask2former2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/media/data/QFS/Mask2Former-main/mask2former/maskformer_model.py", line 198, in forward
    outputs = self.sem_seg_head(features)
  File "/root/miniconda3/envs/mask2former2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/media/data/QFS/Mask2Former-main/mask2former/modeling/meta_arch/mask_former_head.py", line 116, in forward
    return self.layers(features, mask)
  File "/media/data/QFS/Mask2Former-main/mask2former/modeling/meta_arch/mask_former_head.py", line 119, in layers
    mask_features, transformer_encoder_features, multi_scale_features = self.pixel_decoder.forward_features(features)
  File "/root/miniconda3/envs/mask2former2/lib/python3.8/site-packages/torch/cuda/amp/autocast_mode.py", line 141, in decorate_autocast
    return func(*args, **kwargs)
  File "/media/data/QFS/Mask2Former-main/mask2former/modeling/pixel_decoder/msdeformattn.py", line 324, in forward_features
    y, spatial_shapes, level_start_index = self.transformer(srcs, pos)
  File "/root/miniconda3/envs/mask2former2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/media/data/QFS/Mask2Former-main/mask2former/modeling/pixel_decoder/msdeformattn.py", line 87, in forward
    memory = self.encoder(src_flatten, spatial_shapes, level_start_index, valid_ratios, lvl_pos_embed_flatten, mask_flatten)
  File "/root/miniconda3/envs/mask2former2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/media/data/QFS/Mask2Former-main/mask2former/modeling/pixel_decoder/msdeformattn.py", line 159, in forward
    output = layer(output, pos, reference_points, spatial_shapes, level_start_index, padding_mask)
  File "/root/miniconda3/envs/mask2former2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/media/data/QFS/Mask2Former-main/mask2former/modeling/pixel_decoder/msdeformattn.py", line 124, in forward
    src2 = self.self_attn(self.with_pos_embed(src, pos), reference_points, src, spatial_shapes, level_start_index, padding_mask)
  File "/root/miniconda3/envs/mask2former2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/media/data/QFS/Mask2Former-main/mask2former/modeling/pixel_decoder/ops/modules/ms_deform_attn.py", line 124, in forward
    output = self.output_proj(output)
  File "/root/miniconda3/envs/mask2former2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/root/miniconda3/envs/mask2former2/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 96, in forward
    return F.linear(input, self.weight, self.bias)
  File "/root/miniconda3/envs/mask2former2/lib/python3.8/site-packages/torch/nn/functional.py", line 1847, in linear
    return torch._C._nn.linear(input, weight, bias)
RuntimeError: CUDA error: an illegal memory access was encountered


environment：
pytorch                   2.0.0           py3.9_cuda11.8_cudnn8.7.0_0    pytorch
torchvision               0.15.0               py39_cu118                         pytorch
cudatoolkit               11.8.0               h6a678d5_0                        default

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0

NVIDIA-SMI 545.23.08              Driver Version: 545.23.08    CUDA Version: 12.3 

need some help

[lonelyqian]

by the way, there is no such error when I run the demo