RuntimeError: CUDA Error encountered in <function Lib.bn_mean_var_cuda at 0x7f96e5e9d2f0>

[hustcc19860606]

Hello, i use the command 'python3 train.py --data-dir ./dataset/cityscapes/ --random-mirror --random-scale --restore-from ./dataset/resnet101-imagenet.pth --gpu 4,5,6,7 --learning-rate 1e-2 --input-size 769,769 --weight-decay 1e-4 --batch-size 8 --num-steps 60000 --recurrence 2', and has some wrong as follows:
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:52: UserWarning: size_average and reduce args will be deprecated, please use reduction='elementwise_mean' instead.
warnings.warn(warning.format(ret))
481950 images are loaded!
Traceback (most recent call last):
File "train.py", line 245, in
main()
File "train.py", line 209, in main
preds = model(images, args.recurrence)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/data_parallel.py", line 123, in forward
outputs = self.parallel_apply(replicas, inputs, kwargs)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/data_parallel.py", line 133, in parallel_apply
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/parallel_apply.py", line 77, in parallel_apply
raise output
File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/parallel_apply.py", line 53, in _worker
output = module(*input, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/home/cc/networks/ccnet.py", line 196, in forward
x = self.relu1(self.bn1(self.conv1(x)))
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/home/cc/libs/bn.py", line 184, in forward
self.activation, self.slope)
File "/home/cc/libs/functions.py", line 183, in forward
_check(_ext.bn_mean_var_cuda, x, mean, var)
File "/home/cc/libs/functions.py", line 16, in _check
raise RuntimeError("CUDA Error encountered in {}".format(fn))
RuntimeError: CUDA Error encountered in <function Lib.bn_mean_var_cuda at 0x7f96e5e9d2f0>
Can you give some suggestions?
@speedinghzl @honghuis

[hustcc19860606]

And I follow your readme to compile Inplace-abn and criss-cross attention:
root@a55bfbbee40c:/home/cc# cd libs
root@a55bfbbee40c:/home/cc/libs# sh build.sh
root@a55bfbbee40c:/home/cc/libs# python3 build.py
generating /tmp/tmpcqj20vvw/__ext.c
setting the current directory to '/tmp/tmpcqj20vvw'
running build_ext
building '__ext' extension
creating home
creating home/cc
creating home/cc/libs
creating home/cc/libs/src
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/local/lib/python3.6/dist-packages/torch/utils/ffi/../../lib/include -I/usr/local/lib/python3.6/dist-packages/torch/utils/ffi/../../lib/include/TH -I/usr/local/lib/python3.6/dist-packages/torch/utils/ffi/../../lib/include/THC -I/usr/local/cuda/include -I/usr/include/python3.6m -c __ext.c -o ./__ext.o -std=c99 -std=c++11
cc1: warning: command line option '-std=c++11' is valid for C++/ObjC++ but not for C
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/local/lib/python3.6/dist-packages/torch/utils/ffi/../../lib/include -I/usr/local/lib/python3.6/dist-packages/torch/utils/ffi/../../lib/include/TH -I/usr/local/lib/python3.6/dist-packages/torch/utils/ffi/../../lib/include/THC -I/usr/local/cuda/include -I/usr/include/python3.6m -c /home/cc/libs/src/lib_cffi.cpp -o ./home/cc/libs/src/lib_cffi.o -std=c99 -std=c++11
cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
cc1plus: warning: command line option '-std=c99' is valid for C/ObjC but not for C++
x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 ./__ext.o ./home/cc/libs/src/lib_cffi.o /home/cc/libs/src/bn.o -o ./__ext.so
root@a55bfbbee40c:/home/cc/libs# cd ../cc_attention
root@a55bfbbee40c:/home/cc/cc_attention# sh build.sh
root@a55bfbbee40c:/home/cc/cc_attention# python3 build.py
generating /tmp/tmpbc6m099s/__ext.c
setting the current directory to '/tmp/tmpbc6m099s'
running build_ext
building '__ext' extension
creating home
creating home/cc
creating home/cc/cc_attention
creating home/cc/cc_attention/src
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/local/lib/python3.6/dist-packages/torch/utils/ffi/../../lib/include -I/usr/local/lib/python3.6/dist-packages/torch/utils/ffi/../../lib/include/TH -I/usr/local/lib/python3.6/dist-packages/torch/utils/ffi/../../lib/include/THC -I/usr/local/cuda/include -I/usr/include/python3.6m -c __ext.c -o ./__ext.o -std=c99 -std=c++11
cc1: warning: command line option '-std=c++11' is valid for C++/ObjC++ but not for C
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/local/lib/python3.6/dist-packages/torch/utils/ffi/../../lib/include -I/usr/local/lib/python3.6/dist-packages/torch/utils/ffi/../../lib/include/TH -I/usr/local/lib/python3.6/dist-packages/torch/utils/ffi/../../lib/include/THC -I/usr/local/cuda/include -I/usr/include/python3.6m -c /home/cc/cc_attention/src/lib_cffi.cpp -o ./home/cc/cc_attention/src/lib_cffi.o -std=c99 -std=c++11
cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
cc1plus: warning: command line option '-std=c99' is valid for C/ObjC but not for C++
x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 ./__ext.o ./home/cc/cc_attention/src/lib_cffi.o /home/cc/cc_attention/src/ca.o -o ./__ext.so
Is it right?

[speedinghzl]

Hi @hustcc19860606 Maybe the pure-python or >Pytorch 1.5 could solve your problem.