Gaintables cannot be written in Python 3 mode due to current casacore implementation issues

[o-smirnov]

Are there still casacore issues blocking this?

[caseyjlaw]

I'd like to understand this issue. Can someone tag the other relevant discussions here? Also, is the recommended workaround to reinstall with the entire python2 stack?

[o-smirnov]

Well there was this: #269

But I do have a nagging suspicion we can just disable the warning and let it write the damn tables and see if anything breaks. There was a nasty interregnum when python3-casacore had some teething issues, and that's when @bennahugo put this in. I believe the upstream issues have now been solved.

[bennahugo]

it has been an issue on my list. Unfortunately haven't got back to this yet as I'm trying to roll a release for killms at the moment. You can try removing the warnings, however it may be that you need to add UTF-8 encoding to the columns meant to contain casacore::string types like the antenna name column, etc. Unfortunately you cannot roll back to python 2.7. We no longer support it. In the interm you can always plot the cubical native paramdb tables from ipython?

[bennahugo]

An example is here:
https://gist.github.com/bennahugo/921f931973b159b7f19348b46b38d836

[caseyjlaw]

Thanks for the quick feedback, all.
I have commented out the lines that stop execution of saving calibration information. After rebuilding, I demonstrated that I can run cubical and that it saves paramdb files. I can read them into an ipython session and make plots of amplitude and phase of bandpasses. Is that sufficient to test this issue?
Incidentally, I'm a bit confused as to why casacore is required to write these paramdb files. They seem to be python data structures, so how is casacore used?

[JSKenyon]

The paramdb objects can be written without casacore. The original issue was that we also support(ed) writing to casa gain tables. It is that functionality which broke due to upstream issues. I will take a look and see if the functionality can be restored now that things have settled down upstream. In the meantime, using the paramdb objects as you seem to be doing will work just fine.

…

On Tue, 30 Mar 2021, 00:49 Casey Law, ***@***.***> wrote: Thanks for the quick feedback, all. I have commented out the lines that stop execution of saving calibration information. After rebuilding, I demonstrated that I can run cubical and that it saves paramdb files. I can read them into an ipython session and make plots of amplitude and phase of bandpasses. Is that sufficient to test this issue? Incidentally, I'm a bit confused as to why casacore is required to write these paramdb files. They seem to be python data structures, so what how is casacore used? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#440 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABSHDWL5NPFU4SWY4QREQN3TGD7WHANCNFSM4YS74PZA> .

[o-smirnov]

Yep, paramdb is the native format and it is always written. The CASA table export is the extra feature that's been disabled. @JSKenyon if you can take a look what's up there, that'd be great.

@caseyjlaw another plotting option (for parmdb files) are the plot-xxx-solutions scripts in CubiCal.

[JSKenyon]

There are definitely still problems to be solved. I removed all the checks and run into conformance errors. I am doing a little digging now to see if I can figure out if there is an easy fix.

[JSKenyon]

Ah - I can see where at least one conformance error comes from. It looks like some of the subtables in the casa gain table end up with the wrong number of rows e.g. 40 rows get added, but then a putcol is attempted with 28 rows. Trying to figure out why it happens.

[JSKenyon]

@bennahugo I think you may be better suited to chase this further. What I have found is that the blank table actually contains a non-zero number of rows. Thus, when we do table.addrows, we actually end up with excess rows and conformance errors when we attempt to write to the table. I am not sure if the blank table just needs to be replaced/fixed (something you are more familiar with than me).

[JSKenyon]

For reference, the problems seem to be caused by code like:

CubiCal/cubical/database/casa_db_adaptor.py

Line 61 in 57740a4

t.addrows(nrows=len(db.anttype))

[bennahugo]

I'll take a look after I released a new ddf and kms. It could be that this version of casacore needs the rows removed first

…

On Tue, 30 Mar 2021, 10:06 JSKenyon, ***@***.***> wrote: For reference, the problems seem to be caused by code like: https://github.com/ratt-ru/CubiCal/blob/57740a4c35396d043ecdd38fed2e08601db44788/cubical/database/casa_db_adaptor.py#L61 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#440 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB4RE6VGTYKUTZGIMXVR4WTTGGBATANCNFSM4YS74PZA> .

[JSKenyon]

Just a heads up that #448 may fix this for anyone wanting to take it for a spin.