How to disable MXNET_CUDNN_AUTOTUNE_DEFAULT and bucketing log message without turning off MXNET_CUDNN_AUTOTUNE_DEFAULT? #20557

yxchng · 2017-10-03T08:51:33Z

yxchng
Oct 3, 2017

16:49:37] src/operator/././cudnn_algoreg-inl.h:112: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
[16:49:37] src/operator/././cudnn_algoreg-inl.h:117: If you see this message in the middle of training, you are probably using bucketing. Consider setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disabl
e cudnn tuning.
[16:49:42] src/operator/././cudnn_algoreg-inl.h:112: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
[16:49:42] src/operator/././cudnn_algoreg-inl.h:117: If you see this message in the middle of training, you are probably using bucketing. Consider setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disabl
e cudnn tuning.
[16:49:42] src/operator/././cudnn_algoreg-inl.h:112: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
[16:49:42] src/operator/././cudnn_algoreg-inl.h:117: If you see this message in the middle of training, you are probably using bucketing. Consider setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disabl
e cudnn tuning.
[16:49:42] src/operator/././cudnn_algoreg-inl.h:112: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
[16:49:42] src/operator/././cudnn_algoreg-inl.h:117: If you see this message in the middle of training, you are probably using bucketing. Consider setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disabl
e cudnn tuning.
[16:49:45] src/operator/././cudnn_algoreg-inl.h:112: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
[16:49:45] src/operator/././cudnn_algoreg-inl.h:117: If you see this message in the middle of training, you are probably using bucketing. Consider setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disabl
e cudnn tuning.
[16:49:47] src/operator/././cudnn_algoreg-inl.h:112: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
[16:49:47] src/operator/././cudnn_algoreg-inl.h:117: If you see this message in the middle of training, you are probably using bucketing. Consider setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disabl
e cudnn tuning.
[16:49:49] src/operator/././cudnn_algoreg-inl.h:112: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
[16:49:49] src/operator/././cudnn_algoreg-inl.h:117: If you see this message in the middle of training, you are probably using bucketing. Consider setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disabl
e cudnn tuning.
[16:49:49] src/operator/././cudnn_algoreg-inl.h:112: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
[16:49:49] src/operator/././cudnn_algoreg-inl.h:117: If you see this message in the middle of training, you are probably using bucketing. Consider setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disabl
e cudnn tuning.
[16:49:51] src/operator/././cudnn_algoreg-inl.h:112: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
[16:49:51] src/operator/././cudnn_algoreg-inl.h:117: If you see this message in the middle of training, you are probably using bucketing. Consider setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disabl
e cudnn tuning.
[16:49:52] src/operator/././cudnn_algoreg-inl.h:112: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
[16:49:52] src/operator/././cudnn_algoreg-inl.h:117: If you see this message in the middle of training, you are probably using bucketing. Consider setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disabl
e cudnn tuning.
[16:49:52] src/operator/././cudnn_algoreg-inl.h:112: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
[16:49:52] src/operator/././cudnn_algoreg-inl.h:117: If you see this message in the middle of training, you are probably using bucketing. Consider setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disabl
e cudnn tuning.
[16:49:53] src/operator/././cudnn_algoreg-inl.h:112: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
[16:49:53] src/operator/././cudnn_algoreg-inl.h:117: If you see this message in the middle of training, you are probably using bucketing. Consider setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disabl
e cudnn tuning.
[16:49:54] src/operator/././cudnn_algoreg-inl.h:112: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
[16:49:54] src/operator/././cudnn_algoreg-inl.h:117: If you see this message in the middle of training, you are probably using bucketing. Consider setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disabl
e cudnn tuning.
[16:49:55] src/operator/././cudnn_algoreg-inl.h:112: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
[16:49:55] src/operator/././cudnn_algoreg-inl.h:117: If you see this message in the middle of training, you are probably using bucketing. Consider setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disabl
e cudnn tuning.
[16:49:55] src/operator/././cudnn_algoreg-inl.h:112: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
[16:49:55] src/operator/././cudnn_algoreg-inl.h:117: If you see this message in the middle of training, you are probably using bucketing. Consider setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disabl
e cudnn tuning.

Answered by joanshen0508

Aug 26, 2021

you should revise your model file (*.json). Modify "cudnn_tune": "limited_workspace", to "cudnn_tune": "None",
add
import os
os.environ['MXNET_CUDNN_AUTOTUNE_DEFAULT'] = '0'
it works for me.

View full answer

yxchng · 2017-10-03T09:41:13Z

yxchng
Oct 3, 2017
Author

I tried exporting or using os.environ but they don't work. The code i am using is https://github.com/pangyupo/mxnet_mtcnn_face_detection

0 replies

szha · 2017-10-05T04:43:18Z

szha
Oct 5, 2017
Collaborator

#7988 ought to do it.

0 replies

yxchng · 2017-10-05T13:51:30Z

yxchng
Oct 5, 2017
Author

@szha I don't quite understand how to use #7988. How should I solve the problem?

0 replies

szha · 2017-10-05T16:43:36Z

szha
Oct 5, 2017
Collaborator

Sorry for the confusion. What I meant was that once that PR is merged the problem should be resolved. Let's wait for that PR to get in first.

0 replies

stupidZZ · 2017-10-13T13:31:06Z

stupidZZ
Oct 13, 2017

I have the same problems for that.

0 replies

kli-casia · 2017-12-05T03:16:02Z

kli-casia
Dec 5, 2017

@yxchng did you save this problem, i am also using mtcnn mxnet

0 replies

guker · 2017-12-11T08:45:10Z

guker
Dec 11, 2017

export MXNET_CUDNN_AUTOTUNE_DEFALUT=0, it is ok

0 replies

jrzaurin · 2017-12-29T21:00:52Z

jrzaurin
Dec 29, 2017

Hi there

I am on Ubuntu 16.04.3 using conda 4.4.2 and Python 2.7.14

I see the same error even after export MXNET_CUDNN_AUTOTUNE_DEFALUT=0

ubuntu@ip-10-0-0-14:$ export MXNET_CUDNN_AUTOTUNE_DEFALUT=0
ubuntu@ip-10-0-0-14:$ python fine_tune_vgg.py --vgg nets/vgg16 --checkpoints checkpoints --prefix vggnet
[20:56:31] src/io/iter_image_recordio_2.cc:153: ImageRecordIOParser2: ../datasets/rec/train.rec, use 1 threads for decoding..
[20:56:36] src/io/iter_image_recordio_2.cc:153: ImageRecordIOParser2: ../datasets/rec/val.rec, use 1 threads for decoding..
[INFO] loading pre-trained model...
[20:56:37] src/nnvm/legacy_json_util.cc:190: Loading symbol saved by previous version v0.8.0. Attempting to upgrade...
[20:56:37] src/nnvm/legacy_json_util.cc:198: Symbol successfully upgraded!
[INFO] training network...
[20:56:37] src/operator/././cudnn_algoreg-inl.h:112: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)

Any news on how to solve this?

0 replies

szha · 2017-12-29T23:26:23Z

szha
Dec 29, 2017
Collaborator

@jrzaurin This bug should have already been fixed by the PR mentioned above. Which version of mxnet are you using? Would you be able to upgrade to 1.0?

0 replies

jrzaurin · 2017-12-29T23:49:17Z

jrzaurin
Dec 29, 2017

@szha

In fact I am using

In [2]: mxnet.__version__
Out[2]: '0.11.0'

which I installed via:
pip install mxnet-cu80==0.11.0

(opencv and graphviz where already installed)

I upgraded to cuDNN v6.0 but I doubt that has something to do (?)

Overall: Ubuntu 16.04.3, conda 4.4.2, Python 2.7.14 and mxnet 0.11 and I see the same message (I am running it as I write and is there) and it runs very very slow on a tiny dataset (Stanford Cars Dataset)

so...any idea is appreciated, Thanks!

0 replies

szha · 2017-12-29T23:53:31Z

szha
Dec 29, 2017
Collaborator

The latest version of mxnet is mxnet-cu80==1.0.0.post2. Would you be able to try out this version?

0 replies

jrzaurin · 2017-12-30T00:28:48Z

jrzaurin
Dec 30, 2017

@szha

Hey thanks!

I did not know about the mxnet-cu80==1.0.0, updated on my p2 instance (Tesla K80) and my mac and the message is gone, so I assume that problem solved 👍

Still runs very slow, but I am sure this is something on my side...so, regarding to the MXNET_CUDNN_AUTOTUNE_DEFALUT message, the solution was to upgrade:

pip install mxnet-cu80==1.0.0

and the message disappeared

thanks!

EDIT
@szha

just updated to mxnet-cu80==1.0.1 and:

train_net.py:92: DeprecationWarning: mxnet.model.FeedForward has been deprecated. Please use mxnet.mod.Module instead.
  begin_epoch=10)
/home/ubuntu/anaconda2/lib/python2.7/site-packages/mxnet/model.py:573: DeprecationWarning: Calling initializer with init(str, NDArray) has been deprecated.please use init(mx.init.InitDesc(...), NDArray) instead.
  self.initializer(k, v)
/home/ubuntu/anaconda2/lib/python2.7/site-packages/mxnet/model.py:579: DeprecationWarning: Calling initializer with init(str, NDArray) has been deprecated.please use init(mx.init.InitDesc(...), NDArray) instead.
  self.initializer(k, v)
[16:21:55] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:107: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)

The first 3 lines are just DeprecationWarning, but the final one is again the MXNET_CUDNN_AUTOTUNE_DEFAULT message even though is set to 0. Anyway, All seems to work fine, and it runs in decent times.

I am just saying in case is useful.

0 replies

lanking520 · 2018-04-02T20:47:22Z

lanking520
Apr 2, 2018
Collaborator

@yxchng , is this still an issue for you?

0 replies

yxchng · 2018-04-03T15:01:25Z

yxchng
Apr 3, 2018
Author

@lanking520 nope thanks

0 replies

canpeng78 · 2018-04-14T13:59:00Z

canpeng78
Apr 14, 2018

I'm using mxnet 1.2.0, cuda 9.0 and cudnn 7.0, and still encounter this problem. So what's the final solution? (resetting MXNET_CUDNN_AUTOTUNE_DEFAULT doesn't work).
Thanks!

0 replies

arisliang · 2018-04-18T01:27:20Z

arisliang
Apr 18, 2018

While auto tuning, it gives an error message and still running. Is this normal? mx 1.2.0, cuda 9.0, ubuntu 16.04

2605:2644:0418/091830.114475:ERROR:upload_data_presenter.cc(73)] Not implemented reached in virtual void extensions::RawDataPresenter::FeedNext(const net::UploadElementReader &)

After export MXNET_CUDNN_AUTOTUNE_DEFAULT = 0, this error was gone.

0 replies

HX1079299660 · 2018-11-16T09:07:13Z

HX1079299660
Nov 16, 2018

I have the same problems for now

0 replies

sbharadwajj · 2019-02-05T11:01:01Z

sbharadwajj
Feb 5, 2019

The problem still persists after export MXNET_CUDNN_AUTOTUNE_DEFAULT = 0, running on mxnet 1.3.1, cuda 9.0, ubuntu 18.
Help would be really appreciated.

0 replies

apeforest · 2019-02-21T00:05:59Z

apeforest
Feb 21, 2019
Collaborator

Can you check in your model, if some convolution layer has cudnn_tune not set to off. If so, please update your model by setting all the cudnn_tune param all the layers to off. Thanks.

0 replies

joanshen0508 · 2021-08-26T08:55:56Z

joanshen0508
Aug 26, 2021

you should revise your model file (*.json). Modify "cudnn_tune": "limited_workspace", to "cudnn_tune": "None",
add
import os
os.environ['MXNET_CUDNN_AUTOTUNE_DEFAULT'] = '0'
it works for me.

0 replies

Rakesh-John · 2022-05-31T12:12:04Z

Rakesh-John
May 31, 2022

It is pretty simple.

If you are using Windows:

Edit the system evironmental variables (Add a new variable: MXNET_CUDNN_AUTOTUNE_DEFAULT and set it to 0).
You may also need to restart the visual studio code.

Check it out!

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to disable MXNET_CUDNN_AUTOTUNE_DEFAULT and bucketing log message without turning off MXNET_CUDNN_AUTOTUNE_DEFAULT? #20557

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 21 comments

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

How to disable MXNET_CUDNN_AUTOTUNE_DEFAULT and bucketing log message without turning off MXNET_CUDNN_AUTOTUNE_DEFAULT? #20557

Replies: 21 comments

yxchng Oct 3, 2017 Author

szha Oct 5, 2017 Collaborator

yxchng Oct 5, 2017 Author

szha Oct 5, 2017 Collaborator

szha Dec 29, 2017 Collaborator

szha Dec 29, 2017 Collaborator

lanking520 Apr 2, 2018 Collaborator

yxchng Apr 3, 2018 Author

apeforest Feb 21, 2019 Collaborator

yxchng
Oct 3, 2017
Author

szha
Oct 5, 2017
Collaborator

yxchng
Oct 5, 2017
Author

szha
Oct 5, 2017
Collaborator

szha
Dec 29, 2017
Collaborator

szha
Dec 29, 2017
Collaborator

lanking520
Apr 2, 2018
Collaborator

yxchng
Apr 3, 2018
Author

apeforest
Feb 21, 2019
Collaborator