Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cuda out of memory when running tools/test.py #215

Open
FatmaMazen2021 opened this issue Feb 22, 2022 · 1 comment
Open

Cuda out of memory when running tools/test.py #215

FatmaMazen2021 opened this issue Feb 22, 2022 · 1 comment

Comments

@FatmaMazen2021
Copy link

FatmaMazen2021 commented Feb 22, 2022

Dear sir,
Thank you very much for sharing your work
I have trained a model using a lightweight version of solov2
When trying to test it by running tools/test.py, I got CUDA out of memory only after processing half images of my test set as shown in the attached message
Could you kindly help me to solve this issue?
loading annotations into memory...
Done (t=0.16s)
creating index...
index created!
[ ] 0/8512, elapsed: 0s, ETA:/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:3613: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
"See the documentation of nn.Upsample for details.".format(mode)
/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:3658: UserWarning: The default behavior for interpolate/upsample with float scale_factor changed in 1.6.0 to align with other frameworks/libraries, and now uses scale_factor directly, instead of relying on the computed output size. If you wish to restore the old behavior, please set recompute_scale_factor=True. See the documentation of nn.Upsample for details.
"The default behavior for interpolate/upsample with float scale_factor changed "
[>>>>>>>>>>>>>>>>> ] 4833/8512, 28.7 task/s, elapsed: 168s, ETA: 128sTraceback (most recent call last):
File "/kaggle/working/SOLO/tools/test.py", line 282, in
main()
File "/kaggle/working/SOLO/tools/test.py", line 241, in main
outputs = single_gpu_test(model, data_loader, args.show)
File "/kaggle/working/SOLO/tools/test.py", line 26, in single_gpu_test
result = model(return_loss=False, rescale=not show, **data)
File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 166, in forward
return self.module(*inputs[0], **kwargs[0])
File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/kaggle/working/SOLO/mmdet/core/fp16/decorators.py", line 49, in new_func
return old_func(*args, **kwargs)
File "/kaggle/working/SOLO/mmdet/models/detectors/base.py", line 144, in forward
return self.forward_test(img, img_meta, **kwargs)
File "/kaggle/working/SOLO/mmdet/models/detectors/base.py", line 127, in forward_test
return self.simple_test(imgs[0], img_metas[0], **kwargs)
File "/kaggle/working/SOLO/mmdet/models/detectors/single_stage_ins.py", line 92, in simple_test
seg_result = self.bbox_head.get_seg(*seg_inputs)
File "/kaggle/working/SOLO/mmdet/models/anchor_heads/solov2_head.py", line 381, in get_seg
featmap_size, img_shape, ori_shape, scale_factor, cfg, rescale)
File "/kaggle/working/SOLO/mmdet/models/anchor_heads/solov2_head.py", line 482, in get_seg_single
seg_masks = seg_masks > cfg.mask_thr
RuntimeError: CUDA out of memory. Tried to allocate 40.00 MiB (GPU 0; 15.90 GiB total capacity; 14.73 GiB already allocated; 31.75 MiB free; 14.97 GiB reserved in total by PyTorch)

@FatmaMazen2021
Copy link
Author

I have solved the problem by using test_ins.py instead of test.py since I have only one gpu

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant