Question about applying calibration solutions

[spectram]

According to the information in this link for the L1 visibilities:

The generated calibration solutions are stored in the observation meta-data and can optionally be applied using katdal or when exporting to MSv2.

But when I push the data using the default calibrated setting, a corrected column is not obtained, unless the -m flag is enabled. Does this mean that the solutions are applied directly to the data column? Is there a way to keep the data column intact and have the solutions applied to only the corrected column?

[spectram]

And also are the solutions only applied to the target field as pointed out in caracal issue #1454

[spectram]

The WEIGHTS_SPECTRUM and SIGMA_SPECTRUM columns of data obtained with the 'default calibrated' setting also appear to be '0' and 'inf' (See discussion in caracal issue #1451)

[ludwigschwardt]

Hi Sriram,

Does this mean that the solutions are applied directly to the data column?

Yes. This is mostly to reduce the size / download time.

Is there a way to keep the data column intact and have the solutions applied to only the corrected column?

I like your suggestion. The -m flag predates the applycal machinery by several years, but they could certainly work better together. I would probably keep the model data column equal to zero.

Are the solutions only applied to the target field?

I'm somewhat surprised by this. As far as I remember the L1 solutions like G are applied everywhere, while the L2 selfcal solutions like GPHASE are only applied to the target. Maybe there was a mixup of some sort there?

I'm also don't expect the weights and sigma columns to be empty. If there were network issues downloading the dataset, the weights could end up zero but then the data_lost flag should also be set.

Could you please supply the capture block ID and flags passed to mvftoms, if possible?

[spectram]

Hi Ludwig,

Thanks for your response.

I'm also don't expect the weights and sigma columns to be empty. If there were network issues downloading the dataset, the weights could end up zero but then the data_lost flag should also be set.

The data was pushed to Ilifu from the backend, I suspect there could have been network issues. The behaviours also seem to be consistent across multiple datasets. The weights seem to be modified at some stage.

Could you please supply the capture block ID and flags passed to mvftoms, if possible?

Due to a bug in the UI of the archive, I am unable to push the data by myself. (jira service ticket) . I can request this information from ilifu staff who, initiate the push from the backend. Otherwise, I can also provide the observation ID.

[ludwigschwardt]

Thanks, the service ticket has all the info I need. I'll check whether the raw data suffers from the same weight=0 issues.

[Jordatious]

Hi @ludwigschwardt, as we discussed previously, if possible, I think it would great if using the mvftoms.py --applycal parameter in conjunction with the -m, --model-data parameter caused the data column to remain as the level 0 data with no solutions applied (as is the case when not using the --applycal parameter), and the solutions to be applied to the CORRECTED_DATA column. Similarly, it would be great if any model data general by the SDP pipelines (e.g. from cross-cal and/or selfcal, according to the --applycal selection) be written to the MODEL_DATA column.

For now, since I think it doesn't make sense to use the -m, --model-data parameter during raw data transfers, I've added its removal from the SARAO archive here, as a sub-task for an upcoming archive UI upgrade.

[ludwigschwardt]

Agreed Jordan. I still have to see whether the dataset contains sufficient information to reconstruct the model data column...