Training efficientdet-d0 with custom datasets

[molyswu]

Tred to training with custom datasets :

python main.py --mode=train_and_eval --num_classes=8 --training_file_pattern=tfrecord/train/train*.tfrecord --validation_file_pattern=tfrecord/val/val*.tfrecord --val_json_file=annotations/instances_minival2014.json --model_name=efficientdet-d0 --model_dir=/tmp/efficientdet-d0-scratch --ckpt=efficientdet-d0 --train_batch_size=8 --eval_batch_size=8 --eval_samples=1024 --num_examples_per_epoch=15717 --num_epochs=1 --hparams="use_bfloat16=false,num_classes=8,moving_average_decay=0" --use_tpu=False

15 11:29:47.601560 140068141688640 utils.py:107] skip fpn_cells/cell_2/fnode6/op_after_combine11/bn/gamma/Momentum (fpn_cells/cell_2/fnode6/op_after_combine11/bn/gamma/Momentum) -- not in ckpt
I0415 11:29:47.601620 140068141688640 utils.py:107] skip fpn_cells/cell_2/fnode6/op_after_combine11/bn/beta/Momentum (fpn_cells/cell_2/fnode6/op_after_combine11/bn/beta/Momentum) -- not in ckpt
I0415 11:29:47.601680 140068141688640 utils.py:107] skip fpn_cells/cell_2/fnode7/WSM/Momentum (fpn_cells/cell_2/fnode7/WSM/Momentum) -- not in ckpt
I0415 11:29:47.601739 140068141688640 utils.py:107] skip fpn_cells/cell_2/fnode7/WSM_1/Momentum (fpn_cells/cell_2/fnode7/WSM_1/Momentum) -- not in ckpt
I0415 11:29:47.601799 140068141688640 utils.py:107] skip fpn_cells/cell_2/fnode7/op_after_combine12/conv/depthwise_kernel/Momentum (fpn_cells/cell_2/fnode7/op_after_combine12/conv/depthwise_kernel/Momentum) -- not in ckpt
I0415 11:29:47.601859 140068141688640 utils.py:107] skip fpn_cells/cell_2/fnode7/op_after_combine12/conv/pointwise_kernel/Momentum (fpn_cells/cell_2/fnode7/op_after_combine12/conv/pointwise_kernel/Momentum) -- not in ckpt
I0415 11:29:47.601919 140068141688640 utils.py:107] skip fpn_cells/cell_2/fnode7/op_after_combine12/conv/bias/Momentum (fpn_cells/cell_2/fnode7/op_after_combine12/conv/bias/Momentum) -- not in ckpt
I0415 11:29:47.601978 140068141688640 utils.py:107] skip fpn_cells/cell_2/fnode7/op_after_combine12/bn/gamma/Momentum (fpn_cells/cell_2/fnode7/op_after_combine12/bn/gamma/Momentum) -- not in ckpt
I0415 11:29:47.602038 140068141688640 utils.py:107] skip fpn_cells/cell_2/fnode7/op_after_combine12/bn/beta/Momentum (fpn_cells/cell_2/fnode7/op_after_combine12/bn/beta/Momentum) -- not in ckpt
I0415 11:29:47.602097 140068141688640 utils.py:107] skip class_net/class-0/depthwise_kernel/Momentum (class_net/class-0/depthwise_kernel/Momentum) -- not in ckpt
I0415 11:29:47.602158 140068141688640 utils.py:107] skip class_net/class-0/pointwise_kernel/Momentum (class_net/class-0/pointwise_kernel/Momentum) -- not in ckpt
I0415 11:29:47.602234 140068141688640 utils.py:107] skip class_net/class-0/bias/Momentum (class_net/class-0/bias/Momentum) -- not in ckpt
I0415 11:29:47.602357 140068141688640 utils.py:107] skip class_net/class-0-bn-3/gamma/Momentum (class_net/class-0-bn-3/gamma/Momentum) -- not in ckpt
I0415 11:29:47.602454 140068141688640 utils.py:107] skip class_net/class-0-bn-3/beta/Momentum (class_net/class-0-bn-3/beta/Momentum) -- not in ckpt
I0415 11:29:47.602572 140068141688640 utils.py:107] skip class_net/class-1/depthwise_kernel/Momentum (class_net/class-1/depthwise_kernel/Momentum) -- not in ckpt
I0415 11:29:47.602673 140068141688640 utils.py:107] skip class_net/class-1/pointwise_kernel/Momentum (class_net/class-1/pointwise_kernel/Momentum) -- not in ckpt
I0415 11:29:47.602771 140068141688640 utils.py:107] skip class_net/class-1/bias/Momentum (class_net/class-1/bias/Momentum) -- not in ckpt
I0415 11:29:47.602869 140068141688640 utils.py:107] skip class_net/class-1-bn-3/gamma/Momentum (class_net/class-1-bn-3/gamma/Momentum) -- not in ckpt
I0415 11:29:47.602966 140068141688640 utils.py:107] skip class_net/class-1-bn-3/beta/Momentum (class_net/class-1-bn-3/beta/Momentum) -- not in ckpt
I0415 11:29:47.603064 140068141688640 utils.py:107] skip class_net/class-2/depthwise_kernel/Momentum (class_net/class-2/depthwise_kernel/Momentum) -- not in ckpt
I0415 11:29:47.603162 140068141688640 utils.py:107] skip class_net/class-2/pointwise_kernel/Momentum (class_net/class-2/pointwise_kernel/Momentum) -- not in ckpt
I0415 11:29:47.603262 140068141688640 utils.py:107] skip class_net/class-2/bias/Momentum (class_net/class-2/bias/Momentum) -- not in ckpt
I0415 11:29:47.603360 140068141688640 utils.py:107] skip class_net/class-2-bn-3/gamma/Momentum (class_net/class-2-bn-3/gamma/Momentum) -- not in ckpt
I0415 11:29:47.603464 140068141688640 utils.py:107] skip class_net/class-2-bn-3/beta/Momentum (class_net/class-2-bn-3/beta/Momentum) -- not in ckpt
I0415 11:29:47.603558 140068141688640 utils.py:96] skip class_net/class-predict/depthwise_kernel/Momentum -- excluded by ./class-predict/.
I0415 11:29:47.603650 140068141688640 utils.py:96] skip class_net/class-predict/pointwise_kernel/Momentum -- excluded by ./class-predict/.
I0415 11:29:47.603742 140068141688640 utils.py:96] skip class_net/class-predict/bias/Momentum -- excluded by ./class-predict/.
I0415 11:29:47.603844 140068141688640 utils.py:107] skip class_net/class-0-bn-4/gamma/Momentum (class_net/class-0-bn-4/gamma/Momentum) -- not in ckpt
I0415 11:29:47.603933 140068141688640 utils.py:107] skip class_net/class-0-bn-4/beta/Momentum (class_net/class-0-bn-4/beta/Momentum) -- not in ckpt
I0415 11:29:47.604003 140068141688640 utils.py:107] skip class_net/class-1-bn-4/gamma/Momentum (class_net/class-1-bn-4/gamma/Momentum) -- not in ckpt
I0415 11:29:47.604066 140068141688640 utils.py:107] skip class_net/class-1-bn-4/beta/Momentum (class_net/class-1-bn-4/beta/Momentum) -- not in ckpt
I0415 11:29:47.604130 140068141688640 utils.py:107] skip class_net/class-2-bn-4/gamma/Momentum (class_net/class-2-bn-4/gamma/Momentum) -- not in ckpt
I0415 11:29:47.604193 140068141688640 utils.py:107] skip class_net/class-2-bn-4/beta/Momentum (class_net/class-2-bn-4/beta/Momentum) -- not in ckpt
I0415 11:29:47.604268 140068141688640 utils.py:107] skip class_net/class-0-bn-5/gamma/Momentum (class_net/class-0-bn-5/gamma/Momentum) -- not in ckpt
I0415 11:29:47.604344 140068141688640 utils.py:107] skip class_net/class-0-bn-5/beta/Momentum (class_net/class-0-bn-5/beta/Momentum) -- not in ckpt
I0415 11:29:47.604443 140068141688640 utils.py:107] skip class_net/class-1-bn-5/gamma/Momentum (class_net/class-1-bn-5/gamma/Momentum) -- not in ckpt
I0415 11:29:47.604542 140068141688640 utils.py:107] skip class_net/class-1-bn-5/beta/Momentum (class_net/class-1-bn-5/beta/Momentum) -- not in ckpt
I0415 11:29:47.604641 140068141688640 utils.py:107] skip class_net/class-2-bn-5/gamma/Momentum (class_net/class-2-bn-5/gamma/Momentum) -- not in ckpt
I0415 11:29:47.604739 140068141688640 utils.py:107] skip class_net/class-2-bn-5/beta/Momentum (class_net/class-2-bn-5/beta/Momentum) -- not in ckpt
I0415 11:29:47.604839 140068141688640 utils.py:107] skip class_net/class-0-bn-6/gamma/Momentum (class_net/class-0-bn-6/gamma/Momentum) -- not in ckpt
I0415 11:29:47.604938 140068141688640 utils.py:107] skip class_net/class-0-bn-6/beta/Momentum (class_net/class-0-bn-6/beta/Momentum) -- not in ckpt
I0415 11:29:47.605038 140068141688640 utils.py:107] skip class_net/class-1-bn-6/gamma/Momentum (class_net/class-1-bn-6/gamma/Momentum) -- not in ckpt
I0415 11:29:47.605138 140068141688640 utils.py:107] skip class_net/class-1-bn-6/beta/Momentum (class_net/class-1-bn-6/beta/Momentum) -- not in ckpt
I0415 11:29:47.605237 140068141688640 utils.py:107] skip class_net/class-2-bn-6/gamma/Momentum (class_net/class-2-bn-6/gamma/Momentum) -- not in ckpt
I0415 11:29:47.605336 140068141688640 utils.py:107] skip class_net/class-2-bn-6/beta/Momentum (class_net/class-2-bn-6/beta/Momentum) -- not in ckpt
I0415 11:29:47.605421 140068141688640 utils.py:107] skip class_net/class-0-bn-7/gamma/Momentum (class_net/class-0-bn-7/gamma/Momentum) -- not in ckpt
I0415 11:29:47.605488 140068141688640 utils.py:107] skip class_net/class-0-bn-7/beta/Momentum (class_net/class-0-bn-7/beta/Momentum) -- not in ckpt
I0415 11:29:47.605551 140068141688640 utils.py:107] skip class_net/class-1-bn-7/gamma/Momentum (class_net/class-1-bn-7/gamma/Momentum) -- not in ckpt
I0415 11:29:47.605614 140068141688640 utils.py:107] skip class_net/class-1-bn-7/beta/Momentum (class_net/class-1-bn-7/beta/Momentum) -- not in ckpt
I0415 11:29:47.605678 140068141688640 utils.py:107] skip class_net/class-2-bn-7/gamma/Momentum (class_net/class-2-bn-7/gamma/Momentum) -- not in ckpt
I0415 11:29:47.605741 140068141688640 utils.py:107] skip class_net/class-2-bn-7/beta/Momentum (class_net/class-2-bn-7/beta/Momentum) -- not in ckpt
I0415 11:29:47.605806 140068141688640 utils.py:107] skip box_net/box-0/depthwise_kernel/Momentum (box_net/box-0/depthwise_kernel/Momentum) -- not in ckpt
I0415 11:29:47.605870 140068141688640 utils.py:107] skip box_net/box-0/pointwise_kernel/Momentum (box_net/box-0/pointwise_kernel/Momentum) -- not in ckpt
I0415 11:29:47.605935 140068141688640 utils.py:107] skip box_net/box-0/bias/Momentum (box_net/box-0/bias/Momentum) -- not in ckpt
I0415 11:29:47.605999 140068141688640 utils.py:107] skip box_net/box-0-bn-3/gamma/Momentum (box_net/box-0-bn-3/gamma/Momentum) -- not in ckpt
I0415 11:29:47.606063 140068141688640 utils.py:107] skip box_net/box-0-bn-3/beta/Momentum (box_net/box-0-bn-3/beta/Momentum) -- not in ckpt
I0415 11:29:47.606127 140068141688640 utils.py:107] skip box_net/box-1/depthwise_kernel/Momentum (box_net/box-1/depthwise_kernel/Momentum) -- not in ckpt
I0415 11:29:47.606190 140068141688640 utils.py:107] skip box_net/box-1/pointwise_kernel/Momentum (box_net/box-1/pointwise_kernel/Momentum) -- not in ckpt
I0415 11:29:47.606254 140068141688640 utils.py:107] skip box_net/box-1/bias/Momentum (box_net/box-1/bias/Momentum) -- not in ckpt
I0415 11:29:47.606318 140068141688640 utils.py:107] skip box_net/box-1-bn-3/gamma/Momentum (box_net/box-1-bn-3/gamma/Momentum) -- not in ckpt
I0415 11:29:47.606381 140068141688640 utils.py:107] skip box_net/box-1-bn-3/beta/Momentum (box_net/box-1-bn-3/beta/Momentum) -- not in ckpt
I0415 11:29:47.606445 140068141688640 utils.py:107] skip box_net/box-2/depthwise_kernel/Momentum (box_net/box-2/depthwise_kernel/Momentum) -- not in ckpt
I0415 11:29:47.606509 140068141688640 utils.py:107] skip box_net/box-2/pointwise_kernel/Momentum (box_net/box-2/pointwise_kernel/Momentum) -- not in ckpt
I0415 11:29:47.606585 140068141688640 utils.py:107] skip box_net/box-2/bias/Momentum (box_net/box-2/bias/Momentum) -- not in ckpt
I0415 11:29:47.606654 140068141688640 utils.py:107] skip box_net/box-2-bn-3/gamma/Momentum (box_net/box-2-bn-3/gamma/Momentum) -- not in ckpt
I0415 11:29:47.606718 140068141688640 utils.py:107] skip box_net/box-2-bn-3/beta/Momentum (box_net/box-2-bn-3/beta/Momentum) -- not in ckpt
I0415 11:29:47.606782 140068141688640 utils.py:107] skip box_net/box-predict/depthwise_kernel/Momentum (box_net/box-predict/depthwise_kernel/Momentum) -- not in ckpt
I0415 11:29:47.606846 140068141688640 utils.py:107] skip box_net/box-predict/pointwise_kernel/Momentum (box_net/box-predict/pointwise_kernel/Momentum) -- not in ckpt
I0415 11:29:47.606911 140068141688640 utils.py:107] skip box_net/box-predict/bias/Momentum (box_net/box-predict/bias/Momentum) -- not in ckpt
I0415 11:29:47.606975 140068141688640 utils.py:107] skip box_net/box-0-bn-4/gamma/Momentum (box_net/box-0-bn-4/gamma/Momentum) -- not in ckpt
I0415 11:29:47.607038 140068141688640 utils.py:107] skip box_net/box-0-bn-4/beta/Momentum (box_net/box-0-bn-4/beta/Momentum) -- not in ckpt
I0415 11:29:47.607116 140068141688640 utils.py:107] skip box_net/box-1-bn-4/gamma/Momentum (box_net/box-1-bn-4/gamma/Momentum) -- not in ckpt
I0415 11:29:47.607175 140068141688640 utils.py:107] skip box_net/box-1-bn-4/beta/Momentum (box_net/box-1-bn-4/beta/Momentum) -- not in ckpt
I0415 11:29:47.607234 140068141688640 utils.py:107] skip box_net/box-2-bn-4/gamma/Momentum (box_net/box-2-bn-4/gamma/Momentum) -- not in ckpt
I0415 11:29:47.607294 140068141688640 utils.py:107] skip box_net/box-2-bn-4/beta/Momentum (box_net/box-2-bn-4/beta/Momentum) -- not in ckpt
I0415 11:29:47.607353 140068141688640 utils.py:107] skip box_net/box-0-bn-5/gamma/Momentum (box_net/box-0-bn-5/gamma/Momentum) -- not in ckpt
I0415 11:29:47.607413 140068141688640 utils.py:107] skip box_net/box-0-bn-5/beta/Momentum (box_net/box-0-bn-5/beta/Momentum) -- not in ckpt
I0415 11:29:47.607472 140068141688640 utils.py:107] skip box_net/box-1-bn-5/gamma/Momentum (box_net/box-1-bn-5/gamma/Momentum) -- not in ckpt
I0415 11:29:47.607531 140068141688640 utils.py:107] skip box_net/box-1-bn-5/beta/Momentum (box_net/box-1-bn-5/beta/Momentum) -- not in ckpt
I0415 11:29:47.607589 140068141688640 utils.py:107] skip box_net/box-2-bn-5/gamma/Momentum (box_net/box-2-bn-5/gamma/Momentum) -- not in ckpt
I0415 11:29:47.607648 140068141688640 utils.py:107] skip box_net/box-2-bn-5/beta/Momentum (box_net/box-2-bn-5/beta/Momentum) -- not in ckpt
I0415 11:29:47.607713 140068141688640 utils.py:107] skip box_net/box-0-bn-6/gamma/Momentum (box_net/box-0-bn-6/gamma/Momentum) -- not in ckpt
I0415 11:29:47.607773 140068141688640 utils.py:107] skip box_net/box-0-bn-6/beta/Momentum (box_net/box-0-bn-6/beta/Momentum) -- not in ckpt
I0415 11:29:47.607832 140068141688640 utils.py:107] skip box_net/box-1-bn-6/gamma/Momentum (box_net/box-1-bn-6/gamma/Momentum) -- not in ckpt
I0415 11:29:47.607892 140068141688640 utils.py:107] skip box_net/box-1-bn-6/beta/Momentum (box_net/box-1-bn-6/beta/Momentum) -- not in ckpt
I0415 11:29:47.607951 140068141688640 utils.py:107] skip box_net/box-2-bn-6/gamma/Momentum (box_net/box-2-bn-6/gamma/Momentum) -- not in ckpt
I0415 11:29:47.608011 140068141688640 utils.py:107] skip box_net/box-2-bn-6/beta/Momentum (box_net/box-2-bn-6/beta/Momentum) -- not in ckpt
I0415 11:29:47.608071 140068141688640 utils.py:107] skip box_net/box-0-bn-7/gamma/Momentum (box_net/box-0-bn-7/gamma/Momentum) -- not in ckpt
I0415 11:29:47.608131 140068141688640 utils.py:107] skip box_net/box-0-bn-7/beta/Momentum (box_net/box-0-bn-7/beta/Momentum) -- not in ckpt
I0415 11:29:47.608190 140068141688640 utils.py:107] skip box_net/box-1-bn-7/gamma/Momentum (box_net/box-1-bn-7/gamma/Momentum) -- not in ckpt
I0415 11:29:47.608249 140068141688640 utils.py:107] skip box_net/box-1-bn-7/beta/Momentum (box_net/box-1-bn-7/beta/Momentum) -- not in ckpt
I0415 11:29:47.608309 140068141688640 utils.py:107] skip box_net/box-2-bn-7/gamma/Momentum (box_net/box-2-bn-7/gamma/Momentum) -- not in ckpt
I0415 11:29:47.608368 140068141688640 utils.py:107] skip box_net/box-2-bn-7/beta/Momentum (box_net/box-2-bn-7/beta/Momentum) -- not in ckpt
INFO:tensorflow:Done calling model_fn.
I0415 11:29:49.044274 140068141688640 estimator.py:1171] Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
I0415 11:29:49.045373 140068141688640 basic_session_run_hooks.py:546] Create CheckpointSaverHook.
WARNING:tensorflow:Issue encountered when serializing edsummaries.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
'tuple' object has no attribute 'name'
W0415 11:29:50.779332 140068141688640 meta_graph.py:437] Issue encountered when serializing edsummaries.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
'tuple' object has no attribute 'name'
INFO:tensorflow:Graph was finalized.
I0415 11:29:52.982281 140068141688640 monitored_session.py:246] Graph was finalized.
2020-04-15 11:29:52.982752: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-04-15 11:29:53.010787: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 2596920000 Hz
2020-04-15 11:29:53.013378: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f61c0000b20 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-04-15 11:29:53.013423: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-04-15 11:29:53.272964: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x556164d73ab0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-04-15 11:29:53.273009: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla M60, Compute Capability 5.2
2020-04-15 11:29:53.273020: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (1): Tesla M60, Compute Capability 5.2
2020-04-15 11:29:53.273936: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
vice: 1, name: Tesla M60, pci bus id: 0000:07:00.0, compute capability: 5.2)
INFO:tensorflow:Restoring parameters from /tmp/efficientdet-d0-scratch/model.ckpt-6248
I0415 11:29:53.287226 140068141688640 saver.py:1293] Restoring parameters from /tmp/efficientdet-d0-scratch/model.ckpt-6248
INFO:tensorflow:training_loop marked as finished
I0415 11:29:56.248856 140068141688640 error_handling.py:115] training_loop marked as finished
WARNING:tensorflow:Reraising captured error
W0415 11:29:56.249081 140068141688640 error_handling.py:149] Reraising captured error
Traceback (most recent call last):
File "/home/yy/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/home/yy/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/home/yy/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [72] rhs shape= [180]
[[{{node save/Assign_222}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/ai/.local/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1299, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/home/yy/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 958, in run
run_metadata_ptr)
File "/home/yy/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1181, in _run
feed_dict_tensor, options, run_metadata)
File "/home/yy/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1359, in _do_run
run_metadata)

[mingxingtan]

Hi @molyswu looks like this is an old issue that should have been fixed a while ago. Could you check out the latest code and retry it?

Since I can run it well in colab now, I am going to close this issue.

[adarsh0906]

2020-05-22 16:29:19.583405: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
WARNING:tensorflow:From main.py:229: The name tf.estimator.tpu.TPUConfig is deprecated. Please use tf.compat.v1.estimator.tpu.TPUConfig instead.

W0522 16:29:22.477168 140217329719168 module_wrapper.py:138] From main.py:229: The name tf.estimator.tpu.TPUConfig is deprecated. Please use tf.compat.v1.estimator.tpu.TPUConfig instead.

WARNING:tensorflow:From main.py:234: The name tf.estimator.tpu.InputPipelineConfig is deprecated. Please use tf.compat.v1.estimator.tpu.InputPipelineConfig instead.

W0522 16:29:22.477524 140217329719168 module_wrapper.py:138] From main.py:234: The name tf.estimator.tpu.InputPipelineConfig is deprecated. Please use tf.compat.v1.estimator.tpu.InputPipelineConfig instead.

WARNING:tensorflow:From main.py:237: The name tf.estimator.tpu.RunConfig is deprecated. Please use tf.compat.v1.estimator.tpu.RunConfig instead.

W0522 16:29:22.477794 140217329719168 module_wrapper.py:138] From main.py:237: The name tf.estimator.tpu.RunConfig is deprecated. Please use tf.compat.v1.estimator.tpu.RunConfig instead.

WARNING:tensorflow:From main.py:354: The name tf.estimator.tpu.TPUEstimator is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimator instead.

W0522 16:29:22.478182 140217329719168 module_wrapper.py:138] From main.py:354: The name tf.estimator.tpu.TPUEstimator is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimator instead.

WARNING:tensorflow:eval_on_tpu ignored because use_tpu is False.
W0522 16:29:22.478967 140217329719168 tpu_context.py:218] eval_on_tpu ignored because use_tpu is False.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py:1666: calling BaseResourceVariable.init (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
W0522 16:29:22.497153 140217329719168 deprecation.py:506] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py:1666: calling BaseResourceVariable.init (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
W0522 16:29:22.497616 140217329719168 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
2020-05-22 16:29:22.514062: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-05-22 16:29:22.563861: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:29:22.564640: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:00:04.0 name: Tesla K80 computeCapability: 3.7
coreClock: 0.8235GHz coreCount: 13 deviceMemorySize: 11.17GiB deviceMemoryBandwidth: 223.96GiB/s
2020-05-22 16:29:22.564693: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-22 16:29:22.808575: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-22 16:29:22.975975: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-05-22 16:29:23.012252: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-05-22 16:29:23.313184: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-05-22 16:29:23.348768: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-05-22 16:29:23.888002: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-22 16:29:23.888264: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:29:23.889110: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:29:23.889816: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
WARNING:tensorflow:From /content/automl/efficientdet/dataloader.py:329: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE) instead. If sloppy execution is desired, use tf.data.Options.experimental_deterministic.
W0522 16:29:23.953376 140217329719168 deprecation.py:323] From /content/automl/efficientdet/dataloader.py:329: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE) instead. If sloppy execution is desired, use tf.data.Options.experimental_deterministic.
WARNING:tensorflow:From /content/automl/efficientdet/utils.py:320: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
W0522 16:29:36.019176 140217329719168 deprecation.py:323] From /content/automl/efficientdet/utils.py:320: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/profiler/internal/flops_registry.py:142: tensor_shape_from_node_def_name (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.tensor_shape_from_node_def_name
W0522 16:29:39.917539 140217329719168 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/profiler/internal/flops_registry.py:142: tensor_shape_from_node_def_name (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.tensor_shape_from_node_def_name
Parsing Inputs...
WARNING:tensorflow:From /content/automl/efficientdet/efficientdet_arch.py:138: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras.layers.Conv2D instead.
W0522 16:29:40.535063 140217329719168 deprecation.py:323] From /content/automl/efficientdet/efficientdet_arch.py:138: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras.layers.Conv2D instead.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/layers/convolutional.py:424: Layer.apply (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version.
Instructions for updating:
Please use layer.__call__ method instead.
W0522 16:29:40.538732 140217329719168 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/layers/convolutional.py:424: Layer.apply (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version.
Instructions for updating:
Please use layer.__call__ method instead.
WARNING:tensorflow:From /content/automl/efficientdet/efficientdet_arch.py:164: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0522 16:29:40.621974 140217329719168 deprecation.py:323] From /content/automl/efficientdet/efficientdet_arch.py:164: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
WARNING:tensorflow:From /content/automl/efficientdet/efficientdet_arch.py:682: separable_conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras.layers.SeparableConv2D instead.
W0522 16:29:40.652706 140217329719168 deprecation.py:323] From /content/automl/efficientdet/efficientdet_arch.py:682: separable_conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras.layers.SeparableConv2D instead.
Parsing Inputs...
Parsing Inputs...
WARNING:tensorflow:From /content/automl/efficientdet/det_model_fn.py:645: The name tf.estimator.tpu.TPUEstimatorSpec is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimatorSpec instead.

W0522 16:30:03.451806 140217329719168 module_wrapper.py:138] From /content/automl/efficientdet/det_model_fn.py:645: The name tf.estimator.tpu.TPUEstimatorSpec is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimatorSpec instead.

2020-05-22 16:30:12.299059: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 2300000000 Hz
2020-05-22 16:30:12.299335: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x23e1b80 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-05-22 16:30:12.299375: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-05-22 16:30:12.389253: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:30:12.390139: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x23e19c0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-05-22 16:30:12.390183: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla K80, Compute Capability 3.7
2020-05-22 16:30:12.391280: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:30:12.391961: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:00:04.0 name: Tesla K80 computeCapability: 3.7
coreClock: 0.8235GHz coreCount: 13 deviceMemorySize: 11.17GiB deviceMemoryBandwidth: 223.96GiB/s
2020-05-22 16:30:12.392049: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-22 16:30:12.392115: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-22 16:30:12.392167: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-05-22 16:30:12.392205: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-05-22 16:30:12.392245: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-05-22 16:30:12.392284: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-05-22 16:30:12.392324: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-22 16:30:12.392442: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:30:12.393222: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:30:12.393880: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-05-22 16:30:12.398628: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-22 16:30:18.007722: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-05-22 16:30:18.007792: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0
2020-05-22 16:30:18.007811: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N
2020-05-22 16:30:18.008335: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:30:18.009186: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:30:18.009890: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2020-05-22 16:30:18.009957: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10634 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)
2020-05-22 16:31:05.522371: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:05.522384: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:05.567831: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:05.572154: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:05.590671: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:05.604242: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:05.624042: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:05.633405: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:05.652860: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:05.658814: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:05.681913: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[35] = [35] does not index into param shape [21,1]
2020-05-22 16:31:05.686797: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:05.708839: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:05.729091: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:05.738880: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:05.759249: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [9,1]
2020-05-22 16:31:05.777218: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:05.797153: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:05.816734: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[12] = [12] does not index into param shape [10,1]
2020-05-22 16:31:05.838102: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:05.846522: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:05.866338: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:05.882193: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:05.893664: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [9,1]
2020-05-22 16:31:05.913585: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:05.932648: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[12] = [12] does not index into param shape [10,1]
2020-05-22 16:31:05.950592: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:05.956992: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [12,1]
2020-05-22 16:31:05.974346: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:05.995711: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:06.004369: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:06.022955: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:06.038539: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [15,1]
2020-05-22 16:31:06.057798: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:06.075323: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:06.093582: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:06.099683: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[41] = [41] does not index into param shape [16,1]
2020-05-22 16:31:06.126675: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:06.154887: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:06.191888: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [12,1]
2020-05-22 16:31:06.199997: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:06.221942: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[6] = [6] does not index into param shape [5,1]
2020-05-22 16:31:06.239163: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:06.245122: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:06.271038: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:06.275912: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:06.297768: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:06.309857: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [9,1]
2020-05-22 16:31:06.348741: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:06.350590: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[26] = [26] does not index into param shape [8,1]
2020-05-22 16:31:06.387384: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:06.388206: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:06.412207: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:06.422883: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:06.452221: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:06.544299: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:06.567363: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[12] = [12] does not index into param shape [10,1]
2020-05-22 16:31:06.585986: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[41] = [41] does not index into param shape [16,1]
2020-05-22 16:31:06.612142: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:06.625769: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[12] = [12] does not index into param shape [10,1]
2020-05-22 16:31:06.652096: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:06.667538: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:06.680634: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[41] = [41] does not index into param shape [16,1]
2020-05-22 16:31:06.714303: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:06.737168: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:06.742907: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [12,1]
2020-05-22 16:31:06.763202: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:06.786136: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:06.804955: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:06.825781: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:06.834186: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:06.860635: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [15,1]
2020-05-22 16:31:06.874033: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [12,1]
2020-05-22 16:31:06.890233: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:06.901098: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:06.922681: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:06.932385: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:06.947384: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:06.967061: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [15,1]
2020-05-22 16:31:06.976468: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:06.998789: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:07.001308: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:07.032170: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [7,1]
2020-05-22 16:31:07.052380: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:07.086655: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:07.094043: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [12,1]
2020-05-22 16:31:07.118464: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:07.127837: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [7,1]
2020-05-22 16:31:07.145598: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [7,1]
2020-05-22 16:31:07.160204: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:07.180313: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:07.186828: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:07.207349: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[35] = [35] does not index into param shape [21,1]
2020-05-22 16:31:07.215665: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:07.241168: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [7,1]
2020-05-22 16:31:07.243895: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:07.267392: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:07.283598: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[12] = [12] does not index into param shape [10,1]
2020-05-22 16:31:07.294371: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:07.316382: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[35] = [35] does not index into param shape [21,1]
2020-05-22 16:31:07.326330: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [15,1]
2020-05-22 16:31:07.339827: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:07.368194: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [15,1]
2020-05-22 16:31:07.371477: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [9,1]
2020-05-22 16:31:07.406615: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [9,1]
2020-05-22 16:31:07.411576: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:07.434634: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [4,1]
2020-05-22 16:31:07.445323: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:07.462931: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:07.465411: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[41] = [41] does not index into param shape [16,1]
2020-05-22 16:31:07.500710: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:07.501345: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:07.520868: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:07.541397: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:07.544464: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [12,1]
2020-05-22 16:31:07.567987: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [7,1]
2020-05-22 16:31:07.589679: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[6] = [6] does not index into param shape [5,1]
2020-05-22 16:31:07.621578: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:07.661384: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[26] = [26] does not index into param shape [8,1]
2020-05-22 16:31:07.674124: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:07.681213: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:07.703896: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [7,1]
2020-05-22 16:31:07.709902: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
WARNING:tensorflow:Reraising captured error
W0522 16:31:08.010707 140217329719168 error_handling.py:149] Reraising captured error
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: indices[25] = [25] does not index into param shape [17,1]
[[{{node parser/GatherNd}}]]
[[IteratorGetNext]]
[[IteratorGetNext/_11169]]
(1) Invalid argument: indices[25] = [25] does not index into param shape [17,1]
[[{{node parser/GatherNd}}]]
[[IteratorGetNext]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "main.py", line 397, in
app.run(main)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "main.py", line 364, in main
steps=int(FLAGS.num_examples_per_epoch / FLAGS.train_batch_size))
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3083, in train
rendezvous.raise_errors()
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/tpu/error_handling.py", line 150, in raise_errors
six.reraise(typ, value, traceback)
File "/usr/local/lib/python3.6/dist-packages/six.py", line 693, in reraise
raise value
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3078, in train
saving_listeners=saving_listeners)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 349, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1182, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1215, in _train_model_default
saving_listeners)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1518, in _train_with_estimator_spec
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 778, in run
run_metadata=run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1283, in run
run_metadata=run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1384, in run
raise six.reraise(*original_exc_info)
File "/usr/local/lib/python3.6/dist-packages/six.py", line 693, in reraise
raise value
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1369, in run
return self._sess.run(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1442, in run
run_metadata=run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1200, in run
return self._sess.run(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 958, in run
run_metadata_ptr)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1181, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1359, in _do_run
run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: indices[25] = [25] does not index into param shape [17,1]
[[{{node parser/GatherNd}}]]
[[IteratorGetNext]]
[[IteratorGetNext/_11169]]
(1) Invalid argument: indices[25] = [25] does not index into param shape [17,1]
[[{{node parser/GatherNd}}]]
[[IteratorGetNext]]
0 successful operations.
0 derived errors ignored.

i tried to train for 4 classes with following tfrecors in zip but i get the above error if my methos are wrong please guide me
vehicle detection and counting.zip

[adarsh0906]

2020-05-22 16:29:19.583405: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
WARNING:tensorflow:From main.py:229: The name tf.estimator.tpu.TPUConfig is deprecated. Please use tf.compat.v1.estimator.tpu.TPUConfig instead.

W0522 16:29:22.477168 140217329719168 module_wrapper.py:138] From main.py:229: The name tf.estimator.tpu.TPUConfig is deprecated. Please use tf.compat.v1.estimator.tpu.TPUConfig instead.

WARNING:tensorflow:From main.py:234: The name tf.estimator.tpu.InputPipelineConfig is deprecated. Please use tf.compat.v1.estimator.tpu.InputPipelineConfig instead.

W0522 16:29:22.477524 140217329719168 module_wrapper.py:138] From main.py:234: The name tf.estimator.tpu.InputPipelineConfig is deprecated. Please use tf.compat.v1.estimator.tpu.InputPipelineConfig instead.

WARNING:tensorflow:From main.py:237: The name tf.estimator.tpu.RunConfig is deprecated. Please use tf.compat.v1.estimator.tpu.RunConfig instead.

W0522 16:29:22.477794 140217329719168 module_wrapper.py:138] From main.py:237: The name tf.estimator.tpu.RunConfig is deprecated. Please use tf.compat.v1.estimator.tpu.RunConfig instead.

WARNING:tensorflow:From main.py:354: The name tf.estimator.tpu.TPUEstimator is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimator instead.

W0522 16:29:22.478182 140217329719168 module_wrapper.py:138] From main.py:354: The name tf.estimator.tpu.TPUEstimator is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimator instead.

WARNING:tensorflow:eval_on_tpu ignored because use_tpu is False.
W0522 16:29:22.478967 140217329719168 tpu_context.py:218] eval_on_tpu ignored because use_tpu is False.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py:1666: calling BaseResourceVariable.init (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
W0522 16:29:22.497153 140217329719168 deprecation.py:506] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py:1666: calling BaseResourceVariable.init (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
W0522 16:29:22.497616 140217329719168 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
2020-05-22 16:29:22.514062: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-05-22 16:29:22.563861: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:29:22.564640: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:00:04.0 name: Tesla K80 computeCapability: 3.7
coreClock: 0.8235GHz coreCount: 13 deviceMemorySize: 11.17GiB deviceMemoryBandwidth: 223.96GiB/s
2020-05-22 16:29:22.564693: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-22 16:29:22.808575: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-22 16:29:22.975975: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-05-22 16:29:23.012252: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-05-22 16:29:23.313184: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-05-22 16:29:23.348768: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-05-22 16:29:23.888002: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-22 16:29:23.888264: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:29:23.889110: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:29:23.889816: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
WARNING:tensorflow:From /content/automl/efficientdet/dataloader.py:329: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE) instead. If sloppy execution is desired, use tf.data.Options.experimental_deterministic.
W0522 16:29:23.953376 140217329719168 deprecation.py:323] From /content/automl/efficientdet/dataloader.py:329: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE) instead. If sloppy execution is desired, use tf.data.Options.experimental_deterministic.
WARNING:tensorflow:From /content/automl/efficientdet/utils.py:320: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
W0522 16:29:36.019176 140217329719168 deprecation.py:323] From /content/automl/efficientdet/utils.py:320: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/profiler/internal/flops_registry.py:142: tensor_shape_from_node_def_name (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.tensor_shape_from_node_def_name
W0522 16:29:39.917539 140217329719168 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/profiler/internal/flops_registry.py:142: tensor_shape_from_node_def_name (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.tensor_shape_from_node_def_name
Parsing Inputs...
WARNING:tensorflow:From /content/automl/efficientdet/efficientdet_arch.py:138: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras.layers.Conv2D instead.
W0522 16:29:40.535063 140217329719168 deprecation.py:323] From /content/automl/efficientdet/efficientdet_arch.py:138: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras.layers.Conv2D instead.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/layers/convolutional.py:424: Layer.apply (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version.
Instructions for updating:
Please use layer.__call__ method instead.
W0522 16:29:40.538732 140217329719168 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/layers/convolutional.py:424: Layer.apply (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version.
Instructions for updating:
Please use layer.__call__ method instead.
WARNING:tensorflow:From /content/automl/efficientdet/efficientdet_arch.py:164: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0522 16:29:40.621974 140217329719168 deprecation.py:323] From /content/automl/efficientdet/efficientdet_arch.py:164: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
WARNING:tensorflow:From /content/automl/efficientdet/efficientdet_arch.py:682: separable_conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras.layers.SeparableConv2D instead.
W0522 16:29:40.652706 140217329719168 deprecation.py:323] From /content/automl/efficientdet/efficientdet_arch.py:682: separable_conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras.layers.SeparableConv2D instead.
Parsing Inputs...
Parsing Inputs...
WARNING:tensorflow:From /content/automl/efficientdet/det_model_fn.py:645: The name tf.estimator.tpu.TPUEstimatorSpec is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimatorSpec instead.

W0522 16:30:03.451806 140217329719168 module_wrapper.py:138] From /content/automl/efficientdet/det_model_fn.py:645: The name tf.estimator.tpu.TPUEstimatorSpec is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimatorSpec instead.

2020-05-22 16:30:12.299059: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 2300000000 Hz
2020-05-22 16:30:12.299335: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x23e1b80 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-05-22 16:30:12.299375: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-05-22 16:30:12.389253: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:30:12.390139: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x23e19c0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-05-22 16:30:12.390183: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla K80, Compute Capability 3.7
2020-05-22 16:30:12.391280: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:30:12.391961: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:00:04.0 name: Tesla K80 computeCapability: 3.7
coreClock: 0.8235GHz coreCount: 13 deviceMemorySize: 11.17GiB deviceMemoryBandwidth: 223.96GiB/s
2020-05-22 16:30:12.392049: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-22 16:30:12.392115: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-22 16:30:12.392167: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-05-22 16:30:12.392205: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-05-22 16:30:12.392245: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-05-22 16:30:12.392284: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-05-22 16:30:12.392324: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-22 16:30:12.392442: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:30:12.393222: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:30:12.393880: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-05-22 16:30:12.398628: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-22 16:30:18.007722: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-05-22 16:30:18.007792: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0
2020-05-22 16:30:18.007811: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N
2020-05-22 16:30:18.008335: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:30:18.009186: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:30:18.009890: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2020-05-22 16:30:18.009957: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10634 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)
2020-05-22 16:31:05.522371: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:05.522384: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:05.567831: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:05.572154: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:05.590671: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:05.604242: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:05.624042: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:05.633405: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:05.652860: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:05.658814: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:05.681913: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[35] = [35] does not index into param shape [21,1]
2020-05-22 16:31:05.686797: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:05.708839: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:05.729091: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:05.738880: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:05.759249: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [9,1]
2020-05-22 16:31:05.777218: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:05.797153: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:05.816734: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[12] = [12] does not index into param shape [10,1]
2020-05-22 16:31:05.838102: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:05.846522: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:05.866338: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:05.882193: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:05.893664: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [9,1]
2020-05-22 16:31:05.913585: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:05.932648: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[12] = [12] does not index into param shape [10,1]
2020-05-22 16:31:05.950592: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:05.956992: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [12,1]
2020-05-22 16:31:05.974346: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:05.995711: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:06.004369: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:06.022955: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:06.038539: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [15,1]
2020-05-22 16:31:06.057798: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:06.075323: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:06.093582: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:06.099683: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[41] = [41] does not index into param shape [16,1]
2020-05-22 16:31:06.126675: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:06.154887: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:06.191888: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [12,1]
2020-05-22 16:31:06.199997: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:06.221942: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[6] = [6] does not index into param shape [5,1]
2020-05-22 16:31:06.239163: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:06.245122: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:06.271038: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:06.275912: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:06.297768: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:06.309857: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [9,1]
2020-05-22 16:31:06.348741: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:06.350590: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[26] = [26] does not index into param shape [8,1]
2020-05-22 16:31:06.387384: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:06.388206: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:06.412207: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:06.422883: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:06.452221: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:06.544299: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:06.567363: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[12] = [12] does not index into param shape [10,1]
2020-05-22 16:31:06.585986: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[41] = [41] does not index into param shape [16,1]
2020-05-22 16:31:06.612142: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:06.625769: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[12] = [12] does not index into param shape [10,1]
2020-05-22 16:31:06.652096: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:06.667538: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:06.680634: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[41] = [41] does not index into param shape [16,1]
2020-05-22 16:31:06.714303: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:06.737168: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:06.742907: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [12,1]
2020-05-22 16:31:06.763202: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:06.786136: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:06.804955: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:06.825781: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:06.834186: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:06.860635: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [15,1]
2020-05-22 16:31:06.874033: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [12,1]
2020-05-22 16:31:06.890233: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:06.901098: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:06.922681: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:06.932385: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:06.947384: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:06.967061: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [15,1]
2020-05-22 16:31:06.976468: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:06.998789: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:07.001308: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:07.032170: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [7,1]
2020-05-22 16:31:07.052380: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:07.086655: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:07.094043: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [12,1]
2020-05-22 16:31:07.118464: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:07.127837: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [7,1]
2020-05-22 16:31:07.145598: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [7,1]
2020-05-22 16:31:07.160204: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:07.180313: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:07.186828: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:07.207349: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[35] = [35] does not index into param shape [21,1]
2020-05-22 16:31:07.215665: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:07.241168: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [7,1]
2020-05-22 16:31:07.243895: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:07.267392: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:07.283598: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[12] = [12] does not index into param shape [10,1]
2020-05-22 16:31:07.294371: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:07.316382: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[35] = [35] does not index into param shape [21,1]
2020-05-22 16:31:07.326330: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [15,1]
2020-05-22 16:31:07.339827: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:07.368194: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [15,1]
2020-05-22 16:31:07.371477: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [9,1]
2020-05-22 16:31:07.406615: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [9,1]
2020-05-22 16:31:07.411576: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:07.434634: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [4,1]
2020-05-22 16:31:07.445323: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:07.462931: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:07.465411: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[41] = [41] does not index into param shape [16,1]
2020-05-22 16:31:07.500710: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:07.501345: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:07.520868: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:07.541397: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:07.544464: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [12,1]
2020-05-22 16:31:07.567987: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [7,1]
2020-05-22 16:31:07.589679: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[6] = [6] does not index into param shape [5,1]
2020-05-22 16:31:07.621578: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:07.661384: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[26] = [26] does not index into param shape [8,1]
2020-05-22 16:31:07.674124: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:07.681213: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:07.703896: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [7,1]
2020-05-22 16:31:07.709902: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
WARNING:tensorflow:Reraising captured error
W0522 16:31:08.010707 140217329719168 error_handling.py:149] Reraising captured error
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: indices[25] = [25] does not index into param shape [17,1]
[[{{node parser/GatherNd}}]]
[[IteratorGetNext]]
[[IteratorGetNext/_11169]]
(1) Invalid argument: indices[25] = [25] does not index into param shape [17,1]
[[{{node parser/GatherNd}}]]
[[IteratorGetNext]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "main.py", line 397, in
app.run(main)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "main.py", line 364, in main
steps=int(FLAGS.num_examples_per_epoch / FLAGS.train_batch_size))
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3083, in train
rendezvous.raise_errors()
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/tpu/error_handling.py", line 150, in raise_errors
six.reraise(typ, value, traceback)
File "/usr/local/lib/python3.6/dist-packages/six.py", line 693, in reraise
raise value
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3078, in train
saving_listeners=saving_listeners)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 349, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1182, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1215, in _train_model_default
saving_listeners)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1518, in _train_with_estimator_spec
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 778, in run
run_metadata=run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1283, in run
run_metadata=run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1384, in run
raise six.reraise(*original_exc_info)
File "/usr/local/lib/python3.6/dist-packages/six.py", line 693, in reraise
raise value
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1369, in run
return self._sess.run(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1442, in run
run_metadata=run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1200, in run
return self._sess.run(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 958, in run
run_metadata_ptr)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1181, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1359, in _do_run
run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: indices[25] = [25] does not index into param shape [17,1]
[[{{node parser/GatherNd}}]]
[[IteratorGetNext]]
[[IteratorGetNext/_11169]]
(1) Invalid argument: indices[25] = [25] does not index into param shape [17,1]
[[{{node parser/GatherNd}}]]
[[IteratorGetNext]]
0 successful operations.
0 derived errors ignored.

i tried to train for 4 classes with following tfrecors in zip but i get the above error if my methos are wrong please guide me
vehicle detection and counting.zip

sorry i found the problem that was causing the error i rectified it the problem is with tfrecords file

[Li505358678]

2020-05-22 16:29:19.583405: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
WARNING:tensorflow:From main.py:229: The name tf.estimator.tpu.TPUConfig is deprecated. Please use tf.compat.v1.estimator.tpu.TPUConfig instead.
W0522 16:29:22.477168 140217329719168 module_wrapper.py:138] From main.py:229: The name tf.estimator.tpu.TPUConfig is deprecated. Please use tf.compat.v1.estimator.tpu.TPUConfig instead.
WARNING:tensorflow:From main.py:234: The name tf.estimator.tpu.InputPipelineConfig is deprecated. Please use tf.compat.v1.estimator.tpu.InputPipelineConfig instead.
W0522 16:29:22.477524 140217329719168 module_wrapper.py:138] From main.py:234: The name tf.estimator.tpu.InputPipelineConfig is deprecated. Please use tf.compat.v1.estimator.tpu.InputPipelineConfig instead.
WARNING:tensorflow:From main.py:237: The name tf.estimator.tpu.RunConfig is deprecated. Please use tf.compat.v1.estimator.tpu.RunConfig instead.
W0522 16:29:22.477794 140217329719168 module_wrapper.py:138] From main.py:237: The name tf.estimator.tpu.RunConfig is deprecated. Please use tf.compat.v1.estimator.tpu.RunConfig instead.
WARNING:tensorflow:From main.py:354: The name tf.estimator.tpu.TPUEstimator is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimator instead.
W0522 16:29:22.478182 140217329719168 module_wrapper.py:138] From main.py:354: The name tf.estimator.tpu.TPUEstimator is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimator instead.
WARNING:tensorflow:eval_on_tpu ignored because use_tpu is False.
W0522 16:29:22.478967 140217329719168 tpu_context.py:218] eval_on_tpu ignored because use_tpu is False.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py:1666: calling BaseResourceVariable.init (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
W0522 16:29:22.497153 140217329719168 deprecation.py:506] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py:1666: calling BaseResourceVariable.init (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
W0522 16:29:22.497616 140217329719168 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
2020-05-22 16:29:22.514062: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-05-22 16:29:22.563861: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:29:22.564640: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:00:04.0 name: Tesla K80 computeCapability: 3.7
coreClock: 0.8235GHz coreCount: 13 deviceMemorySize: 11.17GiB deviceMemoryBandwidth: 223.96GiB/s
2020-05-22 16:29:22.564693: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-22 16:29:22.808575: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-22 16:29:22.975975: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-05-22 16:29:23.012252: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-05-22 16:29:23.313184: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-05-22 16:29:23.348768: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-05-22 16:29:23.888002: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-22 16:29:23.888264: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:29:23.889110: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:29:23.889816: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
WARNING:tensorflow:From /content/automl/efficientdet/dataloader.py:329: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE) instead. If sloppy execution is desired, use tf.data.Options.experimental_deterministic.
W0522 16:29:23.953376 140217329719168 deprecation.py:323] From /content/automl/efficientdet/dataloader.py:329: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE) instead. If sloppy execution is desired, use tf.data.Options.experimental_deterministic.
WARNING:tensorflow:From /content/automl/efficientdet/utils.py:320: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
W0522 16:29:36.019176 140217329719168 deprecation.py:323] From /content/automl/efficientdet/utils.py:320: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/profiler/internal/flops_registry.py:142: tensor_shape_from_node_def_name (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.tensor_shape_from_node_def_name
W0522 16:29:39.917539 140217329719168 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/profiler/internal/flops_registry.py:142: tensor_shape_from_node_def_name (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.tensor_shape_from_node_def_name
Parsing Inputs...
WARNING:tensorflow:From /content/automl/efficientdet/efficientdet_arch.py:138: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras.layers.Conv2D instead.
W0522 16:29:40.535063 140217329719168 deprecation.py:323] From /content/automl/efficientdet/efficientdet_arch.py:138: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras.layers.Conv2D instead.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/layers/convolutional.py:424: Layer.apply (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version.
Instructions for updating:
Please use layer.__call__ method instead.
W0522 16:29:40.538732 140217329719168 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/layers/convolutional.py:424: Layer.apply (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version.
Instructions for updating:
Please use layer.__call__ method instead.
WARNING:tensorflow:From /content/automl/efficientdet/efficientdet_arch.py:164: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0522 16:29:40.621974 140217329719168 deprecation.py:323] From /content/automl/efficientdet/efficientdet_arch.py:164: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
WARNING:tensorflow:From /content/automl/efficientdet/efficientdet_arch.py:682: separable_conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras.layers.SeparableConv2D instead.
W0522 16:29:40.652706 140217329719168 deprecation.py:323] From /content/automl/efficientdet/efficientdet_arch.py:682: separable_conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras.layers.SeparableConv2D instead.
Parsing Inputs...
Parsing Inputs...
WARNING:tensorflow:From /content/automl/efficientdet/det_model_fn.py:645: The name tf.estimator.tpu.TPUEstimatorSpec is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimatorSpec instead.
W0522 16:30:03.451806 140217329719168 module_wrapper.py:138] From /content/automl/efficientdet/det_model_fn.py:645: The name tf.estimator.tpu.TPUEstimatorSpec is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimatorSpec instead.
2020-05-22 16:30:12.299059: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 2300000000 Hz
2020-05-22 16:30:12.299335: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x23e1b80 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-05-22 16:30:12.299375: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-05-22 16:30:12.389253: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:30:12.390139: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x23e19c0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-05-22 16:30:12.390183: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla K80, Compute Capability 3.7
2020-05-22 16:30:12.391280: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:30:12.391961: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:00:04.0 name: Tesla K80 computeCapability: 3.7
coreClock: 0.8235GHz coreCount: 13 deviceMemorySize: 11.17GiB deviceMemoryBandwidth: 223.96GiB/s
2020-05-22 16:30:12.392049: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-22 16:30:12.392115: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-22 16:30:12.392167: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-05-22 16:30:12.392205: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-05-22 16:30:12.392245: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-05-22 16:30:12.392284: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-05-22 16:30:12.392324: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-22 16:30:12.392442: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:30:12.393222: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:30:12.393880: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-05-22 16:30:12.398628: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-22 16:30:18.007722: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-05-22 16:30:18.007792: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0
2020-05-22 16:30:18.007811: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N
2020-05-22 16:30:18.008335: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:30:18.009186: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-22 16:30:18.009890: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2020-05-22 16:30:18.009957: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10634 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)
2020-05-22 16:31:05.522371: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:05.522384: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:05.567831: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:05.572154: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:05.590671: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:05.604242: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:05.624042: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:05.633405: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:05.652860: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:05.658814: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:05.681913: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[35] = [35] does not index into param shape [21,1]
2020-05-22 16:31:05.686797: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:05.708839: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:05.729091: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:05.738880: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:05.759249: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [9,1]
2020-05-22 16:31:05.777218: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:05.797153: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:05.816734: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[12] = [12] does not index into param shape [10,1]
2020-05-22 16:31:05.838102: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:05.846522: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:05.866338: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:05.882193: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:05.893664: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [9,1]
2020-05-22 16:31:05.913585: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:05.932648: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[12] = [12] does not index into param shape [10,1]
2020-05-22 16:31:05.950592: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:05.956992: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [12,1]
2020-05-22 16:31:05.974346: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:05.995711: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:06.004369: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:06.022955: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:06.038539: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [15,1]
2020-05-22 16:31:06.057798: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:06.075323: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:06.093582: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:06.099683: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[41] = [41] does not index into param shape [16,1]
2020-05-22 16:31:06.126675: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:06.154887: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:06.191888: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [12,1]
2020-05-22 16:31:06.199997: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:06.221942: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[6] = [6] does not index into param shape [5,1]
2020-05-22 16:31:06.239163: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:06.245122: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:06.271038: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:06.275912: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:06.297768: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:06.309857: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [9,1]
2020-05-22 16:31:06.348741: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:06.350590: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[26] = [26] does not index into param shape [8,1]
2020-05-22 16:31:06.387384: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:06.388206: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:06.412207: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:06.422883: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:06.452221: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:06.544299: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:06.567363: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[12] = [12] does not index into param shape [10,1]
2020-05-22 16:31:06.585986: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[41] = [41] does not index into param shape [16,1]
2020-05-22 16:31:06.612142: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:06.625769: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[12] = [12] does not index into param shape [10,1]
2020-05-22 16:31:06.652096: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:06.667538: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:06.680634: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[41] = [41] does not index into param shape [16,1]
2020-05-22 16:31:06.714303: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:06.737168: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:06.742907: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [12,1]
2020-05-22 16:31:06.763202: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:06.786136: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:06.804955: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:06.825781: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:06.834186: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:06.860635: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [15,1]
2020-05-22 16:31:06.874033: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [12,1]
2020-05-22 16:31:06.890233: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:06.901098: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:06.922681: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:06.932385: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:06.947384: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:06.967061: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [15,1]
2020-05-22 16:31:06.976468: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
2020-05-22 16:31:06.998789: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:07.001308: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:07.032170: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [7,1]
2020-05-22 16:31:07.052380: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:07.086655: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:07.094043: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [12,1]
2020-05-22 16:31:07.118464: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:07.127837: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [7,1]
2020-05-22 16:31:07.145598: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [7,1]
2020-05-22 16:31:07.160204: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:07.180313: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:07.186828: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:07.207349: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[35] = [35] does not index into param shape [21,1]
2020-05-22 16:31:07.215665: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:07.241168: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [7,1]
2020-05-22 16:31:07.243895: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:07.267392: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[5] = [5] does not index into param shape [4,1]
2020-05-22 16:31:07.283598: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[12] = [12] does not index into param shape [10,1]
2020-05-22 16:31:07.294371: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:07.316382: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[35] = [35] does not index into param shape [21,1]
2020-05-22 16:31:07.326330: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [15,1]
2020-05-22 16:31:07.339827: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[19] = [19] does not index into param shape [10,1]
2020-05-22 16:31:07.368194: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [15,1]
2020-05-22 16:31:07.371477: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [9,1]
2020-05-22 16:31:07.406615: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [9,1]
2020-05-22 16:31:07.411576: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[11] = [11] does not index into param shape [8,1]
2020-05-22 16:31:07.434634: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [4,1]
2020-05-22 16:31:07.445323: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:07.462931: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:07.465411: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[41] = [41] does not index into param shape [16,1]
2020-05-22 16:31:07.500710: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[7] = [7] does not index into param shape [4,1]
2020-05-22 16:31:07.501345: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [12,1]
2020-05-22 16:31:07.520868: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:07.541397: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:07.544464: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23] = [23] does not index into param shape [12,1]
2020-05-22 16:31:07.567987: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [7,1]
2020-05-22 16:31:07.589679: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[6] = [6] does not index into param shape [5,1]
2020-05-22 16:31:07.621578: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[28] = [28] does not index into param shape [16,1]
2020-05-22 16:31:07.661384: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[26] = [26] does not index into param shape [8,1]
2020-05-22 16:31:07.674124: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[33] = [33] does not index into param shape [15,1]
2020-05-22 16:31:07.681213: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[49] = [49] does not index into param shape [38,1]
2020-05-22 16:31:07.703896: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[9] = [9] does not index into param shape [7,1]
2020-05-22 16:31:07.709902: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[25] = [25] does not index into param shape [17,1]
WARNING:tensorflow:Reraising captured error
W0522 16:31:08.010707 140217329719168 error_handling.py:149] Reraising captured error
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: indices[25] = [25] does not index into param shape [17,1]
[[{{node parser/GatherNd}}]]
[[IteratorGetNext]]
[[IteratorGetNext/_11169]]
(1) Invalid argument: indices[25] = [25] does not index into param shape [17,1]
[[{{node parser/GatherNd}}]]
[[IteratorGetNext]]
0 successful operations.
0 derived errors ignored.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "main.py", line 397, in
app.run(main)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "main.py", line 364, in main
steps=int(FLAGS.num_examples_per_epoch / FLAGS.train_batch_size))
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3083, in train
rendezvous.raise_errors()
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/tpu/error_handling.py", line 150, in raise_errors
six.reraise(typ, value, traceback)
File "/usr/local/lib/python3.6/dist-packages/six.py", line 693, in reraise
raise value
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3078, in train
saving_listeners=saving_listeners)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 349, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1182, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1215, in _train_model_default
saving_listeners)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1518, in _train_with_estimator_spec
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 778, in run
run_metadata=run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1283, in run
run_metadata=run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1384, in run
raise six.reraise(*original_exc_info)
File "/usr/local/lib/python3.6/dist-packages/six.py", line 693, in reraise
raise value
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1369, in run
return self._sess.run(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1442, in run
run_metadata=run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1200, in run
return self._sess.run(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 958, in run
run_metadata_ptr)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1181, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1359, in _do_run
run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: indices[25] = [25] does not index into param shape [17,1]
[[{{node parser/GatherNd}}]]
[[IteratorGetNext]]
[[IteratorGetNext/_11169]]
(1) Invalid argument: indices[25] = [25] does not index into param shape [17,1]
[[{{node parser/GatherNd}}]]
[[IteratorGetNext]]
0 successful operations.
0 derived errors ignored.
i tried to train for 4 classes with following tfrecors in zip but i get the above error if my methos are wrong please guide me
vehicle detection and counting.zip

sorry i found the problem that was causing the error i rectified it the problem is with tfrecords file

what's wrong with your tfrecords file? I have questions just like yours

[hchauhan123]

@adarsh0906 @molyswu I have a similar error with my tfrecord SKU110K dataset on efficientdet model. Can I get more insight in to how you resolved it. So I can resolve my issue: #1173