pytorch1.9.1-->1.12.1 RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

[pedroHuang123]

Thanks for your error report and we appreciate it a lot.

---

2023-02-08 18:10:06,678 - mmflow - INFO - workflow: [('train', 1)], max: 200000 iters
2023-02-08 18:10:06,679 - mmflow - INFO - Checkpoints will be saved to D:\work\OpenmmLab\mmflow\mmflow_log_kd_t35_0206_test by HardDiskBackend.
D:\software\miniconda3\lib\site-packages\torch\functional.py:478: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\TensorShape.cpp:2895.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
Traceback (most recent call last):
File "D:\software\Pycharm\PyCharm Community Edition 2022.1.1\plugins\python-ce\helpers\pydev\pydevd.py", line 1491, in exec
pydev_imports.execfile(file, globals, locals) # execute the script
File "D:\software\Pycharm\PyCharm Community Edition 2022.1.1\plugins\python-ce\helpers\pydev_pydev_imps_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "D:/KdWork/OpenmmLab/mmflow/tools/train.py", line 208, in
main()
File "D:/KdWork/OpenmmLab/mmflow/tools/train.py", line 197, in main
train_model(
File "D:\KdWork\OpenmmLab\mmflow\mmflow\apis\train.py", line 238, in train_model
runner.run(data_loaders, cfg.workflow)
File "D:\software\miniconda3\lib\site-packages\mmcv\runner\iter_based_runner.py", line 144, in run
iter_runner(iter_loaders[i], **kwargs)
File "D:\software\miniconda3\lib\site-packages\mmcv\runner\iter_based_runner.py", line 70, in train
self.call_hook('after_train_iter')
File "D:\software\miniconda3\lib\site-packages\mmcv\runner\base_runner.py", line 317, in call_hook
getattr(hook, fn_name)(self)
File "D:\software\miniconda3\lib\site-packages\mmcv\runner\hooks\optimizer.py", line 61, in after_train_iter
runner.outputs['loss'].backward()
File "D:\software\miniconda3\lib\site-packages\torch_tensor.py", line 396, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "D:\software\miniconda3\lib\site-packages\torch\autograd_init.py", line 173, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [2, 128, 46, 96]], which is output 0 of ReluBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).