Selective target download using mvftoms, all data is flagged.

[physicistsam]

Hi @ludwigschwardt and @Jordatious,

Maybe I am doing something very silly. I wanted to dowload some particular fileds from an observation. I am using the script mvftoms.py. I tried with two different commads they are as follows:

1) python3 ../mvftoms.py --target J1939-6342,J1130-1449,J1550+0527,HydraA -o import_test.ms --flags cam,data_lost,ingest_rfi /<rdb_path>/1745785447_sdp_l0.full.rdb
2) python3 ../mvftoms.py --full_pol --target-tag='fluxcal' --target-tag='gaincal' --target-tag='polcal' --target-tag='target' /<rdb_path>/1745354701_sdp_l0.full.rdb

Then I ran a loop to find flag fraction in these two cases:

vis='import_test.ms'#os.getcwd()+'/'+block+'_sdp_l0.ms'
ii = 0
for ii in range (1, 10):
    invis = vis
    sumr = ct.flagdata(vis=invis, mode='summary',scan='%s'%ii)
    print(sumr)
    dsum = pd.DataFrame.from_dict(sumr['antenna'])
    dsum.loc['flag_frac'] = dsum.loc['flagged']/dsum.loc['total']
    # plt.subplots(figsize=(6, 2))
    ax = dsum.transpose().plot.bar(y='flag_frac',rot=45, figsize=(16, 4), legend=False, ylabel='Flag Fraction', title='Scan = %s'%ii)
    plt.tight_layout()
    fig = ax.get_figure()
    fig.savefig('scan_%s.png'%ii)
    
    del sumr, dsum, ax 

However, both the cases I am finiding 98-100% data is flagged.

I tried with three more observations, same results. Can you please point me, what am I doing wrong.

[ludwigschwardt]

Hi @physicistsam,

It could be due to poor network conditions when you ran mvftoms.py. The script retries downloading a chunk if it fails but eventually gives up and flags it as data_lost. This could potentially flag a large percentage of the data.

Have a look at your visibility data. If it has a high proportion of zeros, excessive data_lost would be my bet.

It is possible to tweak the retries for mvftoms.py, but requires modifying the script. Have a look at the retries parameter of katdal.chunkstore_s3.S3ChunkStore:

    retries : int or tuple of 2 ints or :class:`urllib3.util.retry.Retry`, optional
        Number of connect / read retries, either a single value for both or
        custom values as (connect, read) tuple, or a `Retry` object for full
        customisation (including status retries).

This can be passed to katdal.open in the script as well.

[physicistsam]

Thanks @ludwigschwardt for the quick reply. Indeed most of the visibilities are zero. I will tweak the retries, and update you.

[ludwigschwardt]

Have you considered the standard way of making MSes from the archive? That at least has the promise of more reliable chunk downloads to a server right next to the archive, and then you can ensure that the MS download transferred all the expected bits.

[physicistsam]

Yes I have submitted a request there too, but even if I select the targets, it shows that 3 TB data will be produced.

[ludwigschwardt]

It looks like that size estimate ignores the --target and --scans parameters... I'll report the bug (if it hasn't been reported already).

It should definitely be smaller than 2.94 TB.

[physicistsam]

Yes, I dowloaded the data, it was smaller about 300 GB.

We plan to download a lot of data for processing, it would be helpful if we can have a tool that can directly work on ilifu.

[ludwigschwardt]

For that mvftoms might be adequate since ilifu should have a fast link to the archive (or have you been running on ilifu before when it flagged the data?).

[physicistsam]

Yes, everything is done on ilifu.

[physicistsam]

@ludwigschwardt I guess if we download the data in ilifu using mvf_download.py and then convert it using mvftoms.py. What will be the arguments for that. Do you have any examples somewhere for that.

Because the all-data-flagged issue still persists.

[ludwigschwardt]

From mvf_download.py --help:

usage: mvf_download.py [-h] [--select JSON] [--workers N] source dest [rclone options]

Download MVFv4 dataset (or a subset of chunks) from S3 to disk using rclone.

You need rclone (https://rclone.org/downloads/) if it is not on your system.
It is a single executable file that you could download to your user account.
Just ensure that it is on your PATH; no need to configure it any further.

Run the script like this:

  mvf_download.py https://archive/1698676533/1698676533_sdp_l0.full.rdb?token=<> dest

Data will appear in three subdirectories in the specified output directory as

  dest/1698676533/...
  dest/1698676533-sdp-l0/...
  dest/1698676533-sdp-l1-flags/...

Open the local dataset like this:

  d = katdal.open("dest/1698676533/1698676533_sdp_l0.full.rdb")

If the script crashes or you terminate it, you can just run it again and
it will carry on, fixing any half-downloaded chunks along the way. If it
completes, you can be sure that all your data is safely downloaded.

BONUS: you can even copy just parts of the data (e.g. the tracks and not the
slews). This works as long as your selection picks out a subset of the chunks
but leaves the chunks themselves intact. It is well suited for time-based
selections.

Because MeerKAT data is chunked first in time and then in frequency, but not
in correlation product, this won't help to select a subset of antennas or
baselines or autocorrelations, as that would require breaking up chunks into
smaller chunks. For that, consider using the mvf_copy.py script instead, which
is also useful if you want to copy a subset of data from disk to disk.

Note that you have to pass a JSON object (which resembles a Python dict) as a
string to the --select argument. The "dict" contains keyword arguments meant
for the DataSet.select() method. It's important to note that the strings in
the dict need double quotes (") while the entire string has to be encapsulated
in single quotes ('). Some examples:

  mvf_download.py url directory --select='{"scans": "track"}'
  mvf_download.py url directory --select='{"scans": 1}'
  mvf_download.py url directory --select='{"scans": [0, 1, 2]}'
  mvf_download.py url directory --select='{"targets": "J1939-6342"}'

The chunks that are not copied will appear as "lost" data in the downloaded
dataset, but that is fine. If you apply the same selection, you won't see it.

positional arguments:
  source             Dataset URL (including token if needed)
  dest               Output directory

options:
  -h, --help         show this help message and exit
  --select SELECT    Kwargs for katdal.DataSet.select as a JSON object
  --workers WORKERS  Number of rclone threads for parallel I/O [16]

Any extra script options are passed to rclone.

[physicistsam]

@ludwigschwardt Thanks for the reply but, how do I convert this to MS format. That was the question.

[ludwigschwardt]

Just run mvftoms.py as before on the RDB file. It looks for the data in the adjacent ../<cbid>-sdp-l0 directory created by mvf_download.py and won't download it from the archive then.

[physicistsam]

Okay, let me try this.

[Jordatious]

Hi all, I'm not sure why I was tagged here, but just to mention that if this is done on ilifu, it would be best to use the "push to IDIA" feature of the MeerKAT archive, as that will use Globus and a direct network link, so will transfer the data much faster. @ludwigschwardt, is a warning displayed when there are transfer issues, telling the user that the data is being flagged as data_lost? It could be a bit misleading having data flagged just due to transfer issues.

[ludwigschwardt]

@ludwigschwardt, is a warning displayed when there are transfer issues, telling the user that the data is being flagged as data_lost? It could be a bit misleading having data flagged just due to transfer issues.

Nope... It's been on my radar since 2021 (see SPR1-891) 😂