Implementation of Calvo3 (or Paco's method) in local_fast_trainer

[napulen]

This new method is an extension of the Calvo2 method (using random uniformly distributed samples from 5 images to generate training batches).

According to @fjcastellanos, this is what is needed for Calvo3 (Paco's method):

The main change that I have to make in the code is to deal with the path of the images instead of preloading all the images before. However, as there are dependencies with Rodan, I 'm not sure how to test the code

@fjcastellanos, based on my understanding of the code, the changes you propose should be implemented on local_fast_trainer.py file, in a new branch derived from develop.

That script is independent of Rodan and can be trained locally, so it will allow you to iterate quickly over your changes.

@kemalkongar has volunteered to help you oversee those changes if you need it.

Preferably, push your changes to your branch frequently, and we (e.g., @kemalkongar or myself) can jump in to peer-review the code or help if needed.

[fjcastellanos]

The thing is that I have to change code in training_engine_sae.py, which contains the generator function for extracting samples. However, that module is used by other two modules:
->fast_calvo_trainer.py
->local_fast_trained.py.

So, I have to modify these 3 files to include this feature. I could test the local script, but the fast_calvo_trainer.py module uses the rodan library. For that, I mentioned about the rodan dependencies.

[kemalkongar]

If all the changes will be in training_engine_sae and the driver file, I would recommend creating 2 new files (maybe called new_local and new_sae) based on the existing ones so we can test the changes independently of their effects on Rodan code.

Alternatively, you’re welcome to change all the current files in a branch other than develop, and we’ll simply PR them when they’re complete.

I think both are fine, depends on which workflow you find more appropriate.

[fjcastellanos]

I think that a new branch and modifying the code in the current files will be better than having duplicated files, because also I would need a new file for training_engine_sae.py since the current scripts use the old version of that module.

[kemalkongar]

Sure, makes sense. We’ll work on implementing the changes inside the Rodan job wrapper once the local version is working and pushed to develop.

[fjcastellanos]

Hi!, in the branch "sample_generator" we have now a version of the Paco's method. I have prepared two manners of using the code. Also, I have included instructions in the README.md file (I did not delete the previous information just in case you need it).
Please, I would like to ask you if you could test the program. I have tested it in local and in Compute Canada with Tensorflow 2.5, and it seems working, but I would like to have more testers if possible. And of course, if you have any suggestion for improving the program, let me know, please.
Thank you.

[kemalkongar]

I'll test it locally today and try to get a CC setup going as well. If all this good, we can work on implementing it into a Rodan job and pushing it onto staging.