Error using Tensorflow 1.2.0 - Conv2DCustomBackpropInputOp only supports NHWC.

[ProgramItUp]

This is a great model, thanks for publishing it.

After upgrading Tensorflow to version 1.2.0 with:
pip install tensorflow --upgrade

I'm getting the following error:

InvalidArgumentError (see above for traceback): Conv2DCustomBackpropInputOp only supports NHWC.
         [[Node: gradients/D/Conv_20/convolution_grad/Conv2DBackpropInput = Conv2DBackpropInput[T=DT_FLOAT, data_format="NCHW", padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/cpu:0"](gradients/D/Conv_20/convolution_grad/Shape, D/Conv_20/weights/read, gradients/D/Conv_20/add_grad/tuple/control_dependency)]]

I'm getting the attached error, any thoughts on how to bring the model up to the current version of Tensorflow?

Full trace:

$ python main.py --dataset=CelebA --use_gpu=True
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcudnn.so.6 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcurand.so.8.0 locally
attempting to open ['data/CelebA/splits/train/139407.jpg', 'data/CelebA/splits/train/157712.jpg', 
.........
'data/CelebA/splits/train/116308.jpg', 'data/CelebA/splits/train/123878.jpg', 'data/CelebA/splits/train/155290.jpg', 'data/CelebA/splits/train/034270.jpg']
2017-06-17 02:38:27.935960: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-17 02:38:27.936041: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-17 02:38:27.936063: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-06-17 02:38:27.936083: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-17 02:38:27.936123: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
[*] MODEL dir: logs/CelebA_0617_023824
[*] PARAM path: logs/CelebA_0617_023824/params.json
  0%|                                                                                                                                                                               | 0/500000 [00:00<?, ?it/s]2017-06-17 02:38:30.372334: E tensorflow/core/common_runtime/executor.cc:644] Executor failed to create kernel. Invalid argument: Conv2DCustomBackpropInputOp only supports NHWC.
         [[Node: gradients/D/Conv_20/convolution_grad/Conv2DBackpropInput = Conv2DBackpropInput[T=DT_FLOAT, data_format="NCHW", padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/cpu:0"](gradients/D/Conv_20/convolution_grad/Shape, D/Conv_20/weights/read, gradients/D/Conv_20/add_grad/tuple/control_dependency)]]

Traceback (most recent call last):
  File "main.py", line 43, in <module>
    main(config)
  File "main.py", line 35, in main
    trainer.train()
  File "/home/medgar/models/GAN/carpedm20_BEGAN-tensorflow/BEGAN-tensorflow/trainer.py", line 140, in train
    result = self.sess.run(fetch_dict)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 789, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 997, in _run
    feed_dict_string, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1132, in _do_run
    target_list, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1152, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Conv2DCustomBackpropInputOp only supports NHWC.
         [[Node: gradients/D/Conv_20/convolution_grad/Conv2DBackpropInput = Conv2DBackpropInput[T=DT_FLOAT, data_format="NCHW", padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/cpu:0"](gradients/D/Conv_20/convolution_grad/Shape, D/Conv_20/weights/read, gradients/D/Conv_20/add_grad/tuple/control_dependency)]]

Caused by op u'gradients/D/Conv_20/convolution_grad/Conv2DBackpropInput', defined at:
  File "main.py", line 43, in <module>
    main(config)
  File "main.py", line 31, in main
    trainer = Trainer(config, data_loader)
  File "/home/medgar/models/GAN/carpedm20_BEGAN-tensorflow/BEGAN-tensorflow/trainer.py", line 92, in __init__
    self.build_model()
  File "/home/medgar/models/GAN/carpedm20_BEGAN-tensorflow/BEGAN-tensorflow/trainer.py", line 199, in build_model
    d_optim = d_optimizer.minimize(self.d_loss, var_list=self.D_var)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/optimizer.py", line 315, in minimize
    grad_loss=grad_loss)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/optimizer.py", line 386, in compute_gradients
    colocate_gradients_with_ops=colocate_gradients_with_ops)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gradients_impl.py", line 540, in gradients
    grad_scope, op, func_call, lambda: grad_fn(op, *out_grads))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gradients_impl.py", line 346, in _MaybeCompile
    return grad_fn()  # Exit early
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gradients_impl.py", line 540, in <lambda>
    grad_scope, op, func_call, lambda: grad_fn(op, *out_grads))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/nn_grad.py", line 445, in _Conv2DGrad
    op.get_attr("data_format")),
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 488, in conv2d_backprop_input
    data_format=data_format, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2506, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1269, in __init__
    self._traceback = _extract_stack()

...which was originally created as op u'D/Conv_20/convolution', defined at:
  File "main.py", line 43, in <module>
    main(config)
[elided 1 identical lines from previous traceback]
  File "/home/medgar/models/GAN/carpedm20_BEGAN-tensorflow/BEGAN-tensorflow/trainer.py", line 92, in __init__
    self.build_model()
  File "/home/medgar/models/GAN/carpedm20_BEGAN-tensorflow/BEGAN-tensorflow/trainer.py", line 180, in build_model
    self.conv_hidden_num, self.data_format)
  File "/home/medgar/models/GAN/carpedm20_BEGAN-tensorflow/BEGAN-tensorflow/models.py", line 50, in DiscriminatorCNN
    out = slim.conv2d(x, input_channel, 3, 1, activation_fn=None, data_format=data_format)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 181, in func_with_args
    return func(*args, **current_args)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/layers/python/layers/layers.py", line 947, in convolution
    outputs = layer.apply(inputs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/layers/base.py", line 492, in apply
    return self.__call__(inputs, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/layers/base.py", line 441, in __call__
    outputs = self.call(inputs, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/layers/convolutional.py", line 158, in call
    data_format=utils.convert_data_format(self.data_format, self.rank + 2))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/nn_ops.py", line 670, in convolution
    op=op)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/nn_ops.py", line 338, in with_space_to_batch
    return op(input, num_spatial_dims, padding)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/nn_ops.py", line 662, in op
    name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/nn_ops.py", line 131, in _non_atrous_convolution
    name=name)

InvalidArgumentError (see above for traceback): Conv2DCustomBackpropInputOp only supports NHWC.
         [[Node: gradients/D/Conv_20/convolution_grad/Conv2DBackpropInput = Conv2DBackpropInput[T=DT_FLOAT, data_format="NCHW", padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/cpu:0"](gradients/D/Conv_20/convolution_grad/Shape, D/Conv_20/weights/read, gradients/D/Conv_20/add_grad/tuple/control_dependency)]]

[MichaelOVertolli]

You could go through the code and swap everything to 'NHWC' as follows: tf.transpose(obj, [0, 2, 3, 1])

There's also code in trainer.py that does equivalent: to_nhwc(image, data_format)

[sukkyusun]

i have same problem

@MichaelOVertolli i can't get your solution.

please help me how can i do following

i think code is trying to change NCHW format to NHWC but it 'didn't work in gpu mode for some reason
how can i change related to NHWC?
<<
You could go through the code and swap everything to 'NHWC' as follows: tf.transpose(obj, [0, 2, 3, 1])

There's also code in trainer.py that does equivalent: to_nhwc(image, data_format)>>

[barisgecer]

I had the same problem and fixed it by doing the following two changes. I hope it works for you too:

1. in config.py line 64

def get_config():
config, unparsed = parser.parse_known_args()
if config.use_gpu:
data_format = 'NHWC' #'NCHW'
else:
data_format = 'NHWC'
setattr(config, 'data_format', data_format)
return config, unparsed

2. in trainer.py line 259

    for key, img in items.items():
       if img is None:
           continue
       #if img.shape[3] in [1, 3]:
       #    img = img.transpose([0, 3, 1, 2])