Testing ILoTSS airflow workflow

[twshimwell]

Perform a complete test of the present airflow widefield workflow (see image) on a few fields.

[twshimwell]

A few issues encountered so far:

• downloading often goes to partial success and needs repeating. @tikk3r has fixed this in the widefield DAG.

• delay calibration for one of the test fields picked up a complex source and didnt work too well. Experimenting with adding --no-use-vlass --starting-skymodel {outdir}/*fits --do-auto-delay-selection to the flocs-run vlbi delay-calibration allowing it to examine the different targets in the delay_calibrators.csv. Also required --vlass added to the lofar-vlbi-plot.

• various library issues at the delay calibration filter_files_to_mses.py step which is presently run outside the singularity (required facet-selfcal install and the joblib library)

[twshimwell]

• dag could be altered so that calibrator processing could begin before target downloading is complete and the calibration and target could download in parallel.

• corrupt msfiles can still stay around if the workflow fails and gets resumed because it doesnt recheck existing msfiles. Would probably be good to check all existing msfiles before the download starts if it is resuming. Also maybe to do a final run through at the end of the download step before moving onto the next step.

[twshimwell]

• delay calibration fails with vlass models. Thought this is because of the naming of the VLASS models not having the ILTJ name for the source. Output from delay-calibration is like: WARNING: Difficulty with parsing the source name form L618618_vlass.fits Parsed into L618618_vlass.fits. But then it stops.