feat: Run detector timelines on SLURM

[mfmceneaney]

No description provided.

[mfmceneaney]

Redid this PR after closing #289 and fixing branch history.

[c-dilks]

Thanks a lot for doing this; it will be very useful for chefs (and developers), and will significantly speed things up.

I have some comments below, which hopefully aren't too much work to address.

Also I think for now it's great that the default behavior is the same; after we do some more testing we can switch the default to be Slurm.

[c-dilks]

Chefs won't have write access to $TIMELINESRC, use ./slurm instead (for consistency with run-monitoring.sh)

Also, some files created in this directory will overwrite those from step 1, if Slurm is used there too (e.g., job.$dataset.detectors.list). Perhaps an easy way to avoid this conflict is to set slurmDir to be different for the two steps, such as ./slurm/step1 for run-monitoring.sh and ./slurm/step2 here.

[mfmceneaney]

I went with the second option using ./slurm/step1 for and ./slurm/step2 but would still like to check that this runs fine.

[mfmceneaney]

I ended up having to set the output slurm directory to $outputDir/slurm/step2 in run-detector-timelines.sh since the script changes directories. It might also be wise to use this full path in run-monitoring.sh too even though that script does not change directories in a way that it would affect whether ./slurm or $outputDir/slurm would be identical.

[c-dilks]

$CLASSPATH is necessary for now, but is removed in #293. Depending on whether we merge this PR or #293 first, we'll need to remember to deal with this (though if we forget, the script will just fail, reporting $CLASSPATH as unbound).

[mfmceneaney]

Ok, thanks! Will keep an eye on this.

[c-dilks]

Slurm will handle the logging, automatically splitting stdout and stderr.

Suggested change

	java $TIMELINE_JAVA_OPTS $run_detectors_script $key $inputDir > $logFile.out 2> $logFile.err || touch $logFile.fail
	java $TIMELINE_JAVA_OPTS $run_detectors_script $key $inputDir

You may also remove the logFile=$logDir/$key from a few lines above.

Later below, in the "error checking" part, we'll need to figure out how to read the Slurm error logs... or just tell the user to check them for themselves...

If we do end up reading the Slurm error logs, we'll need to use the job ID or something, so in the case where the user runs this script on twice, the correct set of log files is used.

[mfmceneaney]

For now I just removed what you suggested and in the documentation I just told the user to check for the job errors following the directions in step 1.

[mfmceneaney]

And I also removed the extra log file definition.

[c-dilks]

Could you also update the documentation how to use these new options?

• doc/chef_guide.md: supposed to be as terse as possible
• doc/procedure.md: where you don't have to be terse (in fact, chef_guide.md was created because procedure.md was too verbose...)

[mfmceneaney]

Just added a paragraph in both of these files, but let me know if I was too verbose in the chefs' documentation.

[mfmceneaney]

@c-dilks Thanks for looking through this in detail! I made some more commits in response to your comments but let me know if anything else looks awry.