Batch timeseries analysis in slurm

[ledm]

Closes #118

It's looking like this script is working now.

This PR adds a slurm queue based batch parallel processing of single job timeseries tool.

It's got the following features:

• Loads from existing input_yaml files.
• Run with a single command
• Fully parallelises the slowest part of the analysis_comparison tool: the single job analysis.
• Won't submit the same jobID if it's already existing*

• However, if two analyses call the same jobID with different suites (ie one has bgc and one has physics), it will only run the first one.

Need to do:

• Documentation in README.md
• Is it @DrYool proof?

[valeriupredoi]

hey bud, nice and fast turnaround - I am a bit concerned about the actual submission process (call to subprocess, see me comment) - I'd also think you should add job requirements, no? Also, I'd rename it to batch so and so, since it's not quite parallel-parallel 😁

[valeriupredoi]

I'd defo encase this in a try/except with except fishing for some key elements in stderr, or just printing the whole stderr to screen; if you don't pipe stderr as out it'll be hidden, and the user won't know why their jobs have not been submitted when they thought they have

[valeriupredoi]

oh and also - this things really do deserve a test, not thru and thru with SLURM submission, but everything up to that. I can write the test when it's about ready 👍

[valeriupredoi]

let's make sure the piping is done correctly; also, have you tried this in practice? We don't need any special env to pass to sbatch do we? Like any special environment variables

[ledm]

let's make sure the piping is done correctly; also, have you tried this in practice? We don't need any special env to pass to sbatch do we? Like any special environment variables

I've been using this for a few days and it works on jasmin. Your amendment to the subprocess also works too.

If batch_timeseries fails you get normal python errors. If it fails inside inside the sbatch script, then you get error messages in the places that we tell it to fail.

[valeriupredoi]

yeh that's how we want it to behave, so stdout can be piped to eg a file. Looks good, bud! Let me write a test for it!

[ledm]

I'm not ready to merge. Still need to add documentation & maybe get @DrYool to try it.

[ledm]

The next question I have:

• Do we want to make this the default behaviour? This would mean that we run this script from inside analysis_compare when the -s flag is absent instead of calling the analysis_timeseries command?

[ledm]

Basically, the process for adding a new job, input.yml:

1. analysis_compare -s -y input.yml: This generates the job download commands, which will run overnight. It also creates an html report, but it breaks if there's no data downloaded yet.
2. Wait overnight for data to download on mass.
3. batch_timeseries -y input.yml: This submits the job timeseries onto the processing nodes.
4. analysis_compare -s -y input.yml: This job creates the html report.
5. ./rsync_to_esmeval.sh: this copies to html to the web visible location on disk.

I suspect that these can be merged into fewer commands!

[valeriupredoi]

@ledm I added some test gubbins, make sure to pull or merge so there are no conflictseses

[ledm]

The logo in the README points towards a file on the main branch, but of course it's not available yet until this PR is merged.

[ledm]

We're not actually using config_user here, but I'm keeping it for completeness, just in case.

[ledm]

Okay @valeriupredoi, I'm happy with this now.

[valeriupredoi]

all good by me too, bud! Go ahead and merge when you good 🍺