Running a network for the first time takes very long

[uschmidt83]

Hi,

when I run a network for the first time (I tried the deconvolution demo), loading the network takes a very long time (e.g. 193 seconds, see below). Most users will probably suspect that it's broken.
After that initial run, loading the network is reasonably quick (a few seconds at most). Is this a caching issue?

Best,
Uwe

$ ./ImageJ-linux64 --java-home /sw/apps/jdk/current
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
[INFO] imagej-tensorflow version: 1.0.1
[INFO] tensorflow version: 1.6.0
[INFO] The current library path is: LD_LIBRARY_PATH=/sw/apps/cuda/9.0.176/lib64:/sw/apps/cuda/9.0.176/lib:/home/uschmidt/sw/local/lib:/home/uschmidt/tmp/new_fiji/Fiji.app/lib/linux64:/home/uschmidt/tmp/new_fiji/Fiji.app/mm/linux64
[INFO] loading model net_tubulin from source http://csbdeep.bioimagecomputing.com/model-tubulin.zip
[INFO] TensorFlow model cache: /home/uschmidt/tmp/new_fiji/Fiji.app/models
2018-11-19 16:19:18.450296: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX
2018-11-19 16:19:18.451773: I tensorflow/cc/saved_model/loader.cc:240] Loading SavedModel with tags: { serve }; from: /home/uschmidt/tmp/new_fiji/Fiji.app/models/net_tubulin
2018-11-19 16:19:18.631059: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-11-19 16:19:18.631955: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1212] Found device 0 with properties:
name: GeForce GTX TITAN X major: 5 minor: 2 memoryClockRate(GHz): 1.076
pciBusID: 0000:05:00.0
totalMemory: 11.93GiB freeMemory: 11.71GiB
2018-11-19 16:19:18.632026: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1312] Adding visible gpu devices: 0
2018-11-19 16:22:32.125101: I tensorflow/core/common_runtime/gpu/gpu_device.cc:993] Creating TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 11339 MB memory) -> physical GPU (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:05:00.0, compute capability: 5.2)
2018-11-19 16:22:32.283239: I tensorflow/cc/saved_model/loader.cc:159] Restoring SavedModel bundle.
2018-11-19 16:22:32.329594: I tensorflow/cc/saved_model/loader.cc:194] Running LegacyInitOp on SavedModel bundle.
2018-11-19 16:22:32.330448: I tensorflow/cc/saved_model/loader.cc:289] SavedModel load for tags { serve }; Status: success. Took 193878674 microseconds.
[...]