Killed message with no error.

[praksharma]

I am using Hydra-zen to run a Joblib multithread sweeper. The code works fine with Problem A. But when I use it on Problem B, it gives me a killed message as follows:

[s.1915438@scs2044(sunbird) 4. 3d problem]$ python optimisation_main.py 
Importing packages
Using backend: pytorch

Loading the config file using Hydra-Zen
Start Joblib multithread sweeper
[2022-12-20 16:57:50,033][HYDRA] Joblib.Parallel(n_jobs=1,backend=loky,prefer=processes,require=None,verbose=0,timeout=None,pre_dispatch=2*n_jobs,batch_size=auto,temp_folder=None,max_nbytes=None,mmap_mode=r) is launching 24 jobs
[2022-12-20 16:57:50,033][HYDRA] Launching jobs, sweep output dir : multirun/2022-12-20/16-57-48
[2022-12-20 16:57:50,033][HYDRA]        #0 : +lr=0.0001 +activation=elu
[2022-12-20 16:57:50,033][HYDRA]        #1 : +lr=0.0001 +activation=relu
[2022-12-20 16:57:50,033][HYDRA]        #2 : +lr=0.0001 +activation=selu
[2022-12-20 16:57:50,033][HYDRA]        #3 : +lr=0.0001 +activation=sigmoid
[2022-12-20 16:57:50,033][HYDRA]        #4 : +lr=0.0001 +activation=silu
[2022-12-20 16:57:50,033][HYDRA]        #5 : +lr=0.0001 +activation=sin
[2022-12-20 16:57:50,033][HYDRA]        #6 : +lr=0.0001 +activation=swish
[2022-12-20 16:57:50,033][HYDRA]        #7 : +lr=0.0001 +activation=tanh
[2022-12-20 16:57:50,033][HYDRA]        #8 : +lr=0.001 +activation=elu
[2022-12-20 16:57:50,033][HYDRA]        #9 : +lr=0.001 +activation=relu
[2022-12-20 16:57:50,033][HYDRA]        #10 : +lr=0.001 +activation=selu
[2022-12-20 16:57:50,033][HYDRA]        #11 : +lr=0.001 +activation=sigmoid
[2022-12-20 16:57:50,033][HYDRA]        #12 : +lr=0.001 +activation=silu
[2022-12-20 16:57:50,033][HYDRA]        #13 : +lr=0.001 +activation=sin
[2022-12-20 16:57:50,034][HYDRA]        #14 : +lr=0.001 +activation=swish
[2022-12-20 16:57:50,034][HYDRA]        #15 : +lr=0.001 +activation=tanh
[2022-12-20 16:57:50,034][HYDRA]        #16 : +lr=0.01 +activation=elu
[2022-12-20 16:57:50,034][HYDRA]        #17 : +lr=0.01 +activation=relu
[2022-12-20 16:57:50,034][HYDRA]        #18 : +lr=0.01 +activation=selu
[2022-12-20 16:57:50,034][HYDRA]        #19 : +lr=0.01 +activation=sigmoid
[2022-12-20 16:57:50,034][HYDRA]        #20 : +lr=0.01 +activation=silu
[2022-12-20 16:57:50,034][HYDRA]        #21 : +lr=0.01 +activation=sin
[2022-12-20 16:57:50,034][HYDRA]        #22 : +lr=0.01 +activation=swish
[2022-12-20 16:57:50,034][HYDRA]        #23 : +lr=0.01 +activation=tanh
Killed

When i dig in, I found that, the code works fine until I use scipy.interpolate.Rbf. This Rbf function works just fine without hydra-zen i.e. when I run the python files by defining variables manually.

I can't see any error. What could be the reason? Is there some sort of verbose which is hiding the error? Is there any known conflict with scipy.interpolate.Rbf?

[rsokl]

It looks like scipy.interpolate.Rbf is legacy code. Can you try using scipy.interpolate.RBFInterpolator instead? Before digging in any further, let's see if this fixes the issue on its own.

[praksharma]

Ok I figured out whats wrong. Running multiple RBF interpolation using joblib fills the RAM, but the error message is not very clear. I reduced the number of jobs to 2. It works fine.

[jgbos]

@praksharma To possibly get better error messages from Hydra, try using the Hydra callback, LogJobReturnCallback:

• code: https://github.com/facebookresearch/hydra/blob/main/hydra/experimental/callbacks.py#L14
• docs: https://hydra.cc/docs/experimental/callbacks/

Works great for my Slurm jobs.

[rsokl]

See #359 (reply in thread) for the solution