Checkpointing with collections raising exception in Keras

[skottapa]

Hi,

A TF newbie here. Am trying to use a pre-trained Keras Resnet-50 model on very large biological images(3000x4000 pix) of larvae. Since the image size is huge have to resort to gradient checkpointing.

have done the requisite monkey patching

import memory_saving_gradients K.__dict__['gradients']=memory_saving_gradients.gradients_collection

I have manually defined a collection of all the checkpoint nodes like so

[tf.add_to_collection("checkpoints",base_model.get_layer(i).get_output_at(0)) for i in ["add_4","add_8","add_12","add_16"]]

However I get the following exception

File "/home/satish/anaconda3/envs/tensorflow/lib/python3.6/site-packages/keras/optimizers.py", line 244, in get_updates grads = self.get_gradients(loss, params) File "/home/satish/anaconda3/envs/tensorflow/lib/python3.6/site-packages/keras/optimizers.py", line 78, in get_gradients grads = K.gradients(loss, params) File "/home/satish/anaconda3/envs/tensorflow/lib/python3.6/site-packages/memory_saving_gradients.py", line 31, in gradients_collection return gradients(ys, xs, grad_ys, checkpoints='collection', **kwargs) File "/home/satish/anaconda3/envs/tensorflow/lib/python3.6/site-packages/memory_saving_gradients.py", line 185, in gradients raise Exception('no checkpoints nodes found or given as input! ') Exception: no checkpoints nodes found or given as input!

Not sure why the collection is deemed empty.

Would greatly appreciate any feedback.

Thanks
Satish

[skottapa]

Unfortunately no. I just shifted from 1080 to K80, and was able to fit my model in, though it was a lot slower. Need to get back to it, time permitting Get Outlook for Android<https://aka.ms/ghei36>

…

________________________________ From: Shyam Saladi <[email protected]> Sent: Tuesday, October 16, 2018 2:51:37 AM To: openai/gradient-checkpointing Cc: skottapa; Mention Subject: Re: [openai/gradient-checkpointing] Checkpointing with collections raising exception in Keras (#28) @skottapa<https://github.com/skottapa> Were you able to figure out the issue with here and fix it? I plan on using gradient checkpointing with a large model in the near future. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#28 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AIfvPh4hd22hZbByJ9Y282STq-QSZMdIks5ulPxhgaJpZM4Vv0Da>.