Hi @dianemarquette,
I am open to supporting it, although resume study might be harder than in seems (in general resume is not something supported by any Hydra Sweeeper right now).

In any case, we do not have the cycles for it, which means this will happen if someone from the community wants to work toward it.

Supporting gc_after_trial seems like it should be straight forward though.

@omry Thanks for your quick reply. Any idea when gc_after_trial could be supported?

It can be supported after sends a pull request to add support for it.
As I said, this is not a high pri. you can either wait and hope someone eventually do it, or alternatively you can try to do it yourself.

Ok, thanks for the clarifcation :)

@dianemarquette as of now resuming trials is somewhat supported by the optuna optimizer, by setting a storage backend:

hydra.sweeper.study_name=my_trial
hydra.sweeper.storage=sqlite:///my_trial.sqlite

But this will start the job numbering always from scratch and will therefore overwrite the output directories of individual jobs.

@dianemarquette as of now resuming trials is somewhat supported by the optuna optimizer, by setting a storage backend:

hydra.sweeper.study_name=my_trial hydra.sweeper.storage=sqlite:///my_trial.sqlite

But this will start the job numbering always from scratch and will therefore overwrite the output directories of individual jobs.

what's more, it did be able to resume study but will inevitably launch multiple replicated runs for a specific params combination, since it will still run 80 times (let's assume for a grid search, there are 80 exps in total.) without launching those exps that have been successfully excecuted.

and yes, gc collect is a feature i want too, cos now, no matter how i set the n_jobs or pre_dispatch params, the finished jobs will still exists and will exit until next group of parallel trials finish.

Hydra has callbacks which can probably be used for it.
See this.

As far as I can tell we only need to add gc_after_trial to the correct config. I did that but when I run pytest I get two errors because the hydra/sweeper is not as expected.

You can find my code here. I do not have much experience with pytest that's why it's hard for me to debug the error.

Would be great if someone could help me 😅

Also I am not sure how to write a test that actually tests what I coded, since gc.collect() does not return anything. I managed to modify a test and added gc_after_trial and the config got build correctly. But we would need a test that actually loads a model with cuda right?
If we do not want to test it then the question becomes if we actually call the optuna implementation def _optimize() (see) and if it is enough to add the key to the config.