Support SLURM array jobs in babs-submit

[tsalo]

Summary

It looks like BABS currently submits a separate job for each subject/session, but an array job would be nice. Proposed by @mattcieslak.

[mattcieslak]

@djarecka my memory is fuzzy here - did you at one point start working on this? I remember talking about it

[djarecka]

I think at the end we decided to leave it for later, so I haven't added. But I would be happy to take a look

[tientong98]

Hi @djarecka, I'd like to work on adding SLURM array job submission to BABS. @mattcieslak mentioned that you worked on this before, so I want to reach out to see what's the best way for us to work together, maybe I can start by learning from the branch you were working on for this issue?

[djarecka]

Hi @tientong98 , thanks for reaching. I've been just checking my old branches. And it doesn't look like the work was ever finished, and it was before some other changes to the main babs code, so it's hard to understand changes between branches..

If you want to work on this, perhaps we can have a quick chat next week, or I can point you on github to the places I believe should be changed. And if you already started working, I'm happy to review and test. Let me know what would work best for you!

[tientong98]

1. Change to solely array submission, can submit an array of 1 job, but if max_index > 1, will not submit as separate jobs any more
2. Deprecate SGE

if people use --all, give a warning about hard limit of number of task per job
remove SGE

check status -- grep subid and sesid (because log files don't have this appended in file name)

babs.py:
generate_job_submit_template
generate_bash_participant_job
babs_submit

babs_status:
if_request_resubmit_this_job --> if_request_resubmit_this_task
df_resubmit_job_specific --> df_resubmit_task_specific

utils.py:
submit_one_job --> submit_array
prepare_job_ind_list --> prepare_job_array_df
df_update_one_job --> df_submit_update and df_status_update
subset by i_task instead of i_job

create_job_status_csv
read_job_status_csv

cli.py:
babs_submit_main
babs_status_main