SystemError: (Fatal) DataLoader process (pid 2176640) exited is killed by signal: Aborted. (at /paddle/paddle/fluid/imperative/data_loader.cc:189)这个问题咋解决

[Kitzzaaa]

问题确认 Search before asking

• 我已经搜索过问题，但是没有找到解答。I have searched the question and found no related answer.

请提出你的问题 Please ask your question

File "/home/gpu03/PaddleYOLO-release-2.6/tools/train.py", line 138, in run
trainer = Trainer(cfg, mode='train')
File "/home/gpu03/PaddleYOLO-release-2.6/ppdet/engine/trainer.py", line 92, in init
self.model = create(cfg.architecture)
File "/home/gpu03/PaddleYOLO-release-2.6/ppdet/core/workspace.py", line 255, in create
cls_kwargs.update(cls.from_config(config, **kwargs))
File "/home/gpu03/PaddleYOLO-release-2.6/ppdet/modeling/architectures/yolov8.py", line 56, in from_config
backbone = create(cfg['backbone'])
File "/home/gpu03/PaddleYOLO-release-2.6/ppdet/core/workspace.py", line 292, in create
return cls(**cls_kwargs)
File "/home/gpu03/PaddleYOLO-release-2.6/ppdet/modeling/backbones/yolov8_csp_darknet.py", line 159, in init
self.stem = Conv(
File "/home/gpu03/PaddleYOLO-release-2.6/ppdet/modeling/backbones/csp_darknet.py", line 64, in init
self.conv = nn.Conv2D(
File "/home/gpu03/Anaconda3/envs/paddleyolo2/lib/python3.10/site-packages/paddle/nn/layer/conv.py", line 690, in init
super().init(
File "/home/gpu03/Anaconda3/envs/paddleyolo2/lib/python3.10/site-packages/paddle/nn/layer/conv.py", line 156, in init
self.weight = self.create_parameter(
File "/home/gpu03/Anaconda3/envs/paddleyolo2/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 781, in create_parameter
return self._helper.create_parameter(
File "/home/gpu03/Anaconda3/envs/paddleyolo2/lib/python3.10/site-packages/paddle/base/layer_helper_base.py", line 430, in create_parameter
return self.main_program.global_block().create_parameter(
File "/home/gpu03/Anaconda3/envs/paddleyolo2/lib/python3.10/site-packages/paddle/base/framework.py", line 4381, in create_parameter
initializer(param, self)
File "/home/gpu03/Anaconda3/envs/paddleyolo2/lib/python3.10/site-packages/paddle/nn/initializer/initializer.py", line 40, in call
return self.forward(param, block)
File "/home/gpu03/Anaconda3/envs/paddleyolo2/lib/python3.10/site-packages/paddle/nn/initializer/normal.py", line 75, in forward
out_var = _C_ops.gaussian(
File "/home/gpu03/Anaconda3/envs/paddleyolo2/lib/python3.10/site-packages/paddle/io/multiprocess_utils.py", line 133, in handler
core._throw_error_if_process_failed()
SystemError: (Fatal) DataLoader process (pid 2176640) exited is killed by signal: Aborted. (at /paddle/paddle/fluid/imperative/data_loader.cc:189)

I0113 15:40:51.830221 2176593 process_group_nccl.cc:132] ProcessGroupNCCL destruct