undefined symbol: register_derivedmscal when importing casacore.tables.

[maxnoe]

Trying to install the whole LOFAR software on an ubuntu 20.04 workstation.

1. Create and activate a virtualenv using the system python 3.8 in /opt/LOFAR/python:

$ python3 -m venv --prompt lofar /opt/LOFAR/python
$ source /opt/LOFAR/python/bin/activate

2. Install python requirements and ipython:

$ pip install numpy wheel ipython

3. Download casacore data into /opt/LOFAR/casa/data

4. Building and install casacore:

$ curl -L https://github.com/casacore/casacore/archive/refs/tags/v3.4.0.tar.gz | tar xz
$ cmake -S . -B build -DCMAKE_INSTALL_PREFIX=/opt/LOFAR/casa/3.4.0 -DUSE_OPENMP=ON -DUSE_HDF5=ON -DBUILD_PYTHON=OFF -DBUILD_PYTHON3=ON -DUSE_THREADS=ON -DCMAKE_BUILD_TYPE=RelWithDebInfo -DDATA_DIR=/opt/LOFAR/casa/data
$ cmake --build build -- -j24
$ cmake --install build

5. Download and install dysco:

$ curl -L https://github.com/aroffringa/dysco/archive/refs/tags/v1.2.tar.gz | tar xz
$ cd dysco-1.2
$ cmake -S. -B build -DCMAKE_INSTALL_PREFIX=/opt/LOFAR/casa/3.4.0 -DCMAKE_BUILD_TYPE=RelWithDebInfo
$ cmake --build build -- -j24
$ cmake --install build

6. Install python-casacore in --no-binary-mode:

$ export CFLAGS="--std=c++11 -L/opt/LOFAR/casa/3.4.0/lib -I/opt/LOFAR/casa/3.4.0/include"
$ export LDFLAGS="-Wl,-rpath,/opt/LOFAR/casa/3.4.0/lib" 
$ pip install --no-binary python-casacore --no-cache python-casacore 

I then get:

In [1]: from casacore import tables
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Input In [1], in <cell line: 1>()
----> 1 from casacore import tables

File /opt/LOFAR/python/lib/python3.8/site-packages/casacore/tables/__init__.py:60, in <module>
      1 # __init__.py: Top level .py file for python table interface
      2 # Copyright (C) 2006
      3 # Associated Universities, Inc. Washington DC, USA.
   (...)
     25 #
     26 # $Id: __init__.py,v 1.6 2006/11/06 01:54:21 gvandiep Exp $
     28 """Python interface to the Casacore tables module.
     29 
     30 A `casacore table <../../casacore/doc/html/group__Tables__module.html>`_
   (...)     57 
     58 """
---> 60 from .msutil import *
     61 from .table import table
     62 from .table import default_ms

File /opt/LOFAR/python/lib/python3.8/site-packages/casacore/tables/msutil.py:29, in <module>     27 import numpy as np
     28 import six
---> 29 from casacore.tables.table import (table, taql,
     30                                    _required_ms_desc,
     31                                    _complete_ms_desc)
     32 from casacore.tables.tableutil import (makescacoldesc, makearrcoldesc,
     33                                        makecoldesc, maketabdesc)
     36 def required_ms_desc(table=None):

File /opt/LOFAR/python/lib/python3.8/site-packages/casacore/tables/table.py:40, in <module>
     25 """Access to Casacore tables.
     26 
     27 The :class:`table` class is the main class to access a table. Its constructor
   (...)
     36 
     37 """
     39 from six import string_types
---> 40 from ._tables import (Table,
     41                       _default_ms,
     42                       _default_ms_subtable,
     43                       _required_ms_desc,
     44                       _complete_ms_desc)
     46 from .tablehelper import (_add_prefix, _remove_prefix, _do_remove_prefix,
     47                           _format_row)
     48 import six

ImportError: /opt/LOFAR/python/lib/python3.8/site-packages/casacore/tables/_tables.cpython-38-x86_64-linux-gnu.so: undefined symbol: register_derivedmscal

Any help would be much appreciated!

[StFroese]

This may be related

python-casacore/src/tables.cc

Lines 57 to 61 in f722ae9

	// Register the TaQL meas and mscal functions.
	// Normally they are loaded as a shared library, but that cannot
	// be done if the program is built statically.
	register_meas();
	register_derivedmscal();

[maxnoe]

Ah, dam. There was a hint about adding -DBUILD_SHARED_LIBS somewhere. I guess we need to add this to the casacore cmake call.

[StFroese]

There's a hint to use this flag for the AOFlagger install on the LOFAR software page here

Note: use the -DBUILD_SHARED_LIBS=ON flag! See http://sourceforge.net/p/aoflagger

[StFroese]

I rebuilt it from scratch with -DBUILD_SHARED_LIBS=ON for the casacore cmake but the error still remains.

[StFroese]

I think I figured out a fix for this.

1. libcasa_derivedmscal.so doesn't show up in the _tables.cpython-38-x86_64-linux-gnu.so

(lofar) root@e5b-dell-09:/opt/LOFAR# ldd /opt/LOFAR/python/lib/python3.8/site-packages/casacore/tables/_tables.cpython-38-x86_64-linux-gnu.so
        linux-vdso.so.1 (0x00007fff39f8a000)
        libcasa_meas.so.6 => /opt/LOFAR/casa/3.4.0/lib/libcasa_meas.so.6 (0x00007f3d6cb5b000)
        libcasa_ms.so.6 => /opt/LOFAR/casa/3.4.0/lib/libcasa_ms.so.6 (0x00007f3d6c66a000)
        libcasa_tables.so.6 => /opt/LOFAR/casa/3.4.0/lib/libcasa_tables.so.6 (0x00007f3d6bef6000)
        libboost_python38.so.1.71.0 => /lib/x86_64-linux-gnu/libboost_python38.so.1.71.0 (0x00007f3d6be97000)
        libcasa_python3.so.6 => /opt/LOFAR/casa/3.4.0/lib/libcasa_python3.so.6 (0x00007f3d6be3c000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f3d6bc5a000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f3d6bc3d000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3d6ba4b000)
        libcasa_measures.so.6 => /opt/LOFAR/casa/3.4.0/lib/libcasa_measures.so.6 (0x00007f3d6b725000)
        libcasa_casa.so.6 => /opt/LOFAR/casa/3.4.0/lib/libcasa_casa.so.6 (0x00007f3d6b2e7000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f3d6b198000)
        libgomp.so.1 => /lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f3d6b156000)
        libcasa_scimath.so.6 => /opt/LOFAR/casa/3.4.0/lib/libcasa_scimath.so.6 (0x00007f3d6afa0000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f3d6af7d000)
        liblapack.so.3 => /lib/x86_64-linux-gnu/liblapack.so.3 (0x00007f3d6a8d9000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f3d6cd06000)
        libpython3.8.so.1.0 => /lib/x86_64-linux-gnu/libpython3.8.so.1.0 (0x00007f3d6a383000)
        libhdf5_serial.so.103 => /lib/x86_64-linux-gnu/libhdf5_serial.so.103 (0x00007f3d6a006000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f3d69ffe000)
        libcasa_scimath_f.so.6 => /opt/LOFAR/casa/3.4.0/lib/libcasa_scimath_f.so.6 (0x00007f3d69fd7000)
        libfftw3f_threads.so.3 => /lib/x86_64-linux-gnu/libfftw3f_threads.so.3 (0x00007f3d69fcd000)
        libfftw3_threads.so.3 => /lib/x86_64-linux-gnu/libfftw3_threads.so.3 (0x00007f3d69fc3000)
        libfftw3f.so.3 => /lib/x86_64-linux-gnu/libfftw3f.so.3 (0x00007f3d69db3000)
        libfftw3.so.3 => /lib/x86_64-linux-gnu/libfftw3.so.3 (0x00007f3d69bab000)
        libblas.so.3 => /lib/x86_64-linux-gnu/libblas.so.3 (0x00007f3d69b3e000)
        libgfortran.so.5 => /lib/x86_64-linux-gnu/libgfortran.so.5 (0x00007f3d69876000)
        libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f3d69848000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f3d6982c000)
        libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f3d69827000)
        libsz.so.2 => /lib/x86_64-linux-gnu/libsz.so.2 (0x00007f3d69820000)
        libquadmath.so.0 => /lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007f3d697d6000)
        libaec.so.0 => /lib/x86_64-linux-gnu/libaec.so.0 (0x00007f3d697cd000)

2. It seems to be that there is a default flag added to ld called --as-needed on Ubuntu which results in libraries not beeing linked (take a look here)
3. Defining export LDFLAGS="-Wl,-rpath,/opt/LOFAR/casa/3.4.0/lib,--no-as-needed -lcasa_derivedmscal" as recommended here resolves the error. The library now shows up

(lofar) root@e5b-dell-09:/opt/LOFAR# ldd /opt/LOFAR/python/lib/python3.8/site-packages/casacore/tables/_tables.cpython-38-x86_64-linux-gnu.so
        linux-vdso.so.1 (0x00007fffe7aca000)
    --> libcasa_derivedmscal.so.6 => /opt/LOFAR/casa/3.4.0/lib/libcasa_derivedmscal.so.6 (0x00007f114aba3000)
        libcasa_meas.so.6 => /opt/LOFAR/casa/3.4.0/lib/libcasa_meas.so.6 (0x00007f114aa98000)
        libcasa_ms.so.6 => /opt/LOFAR/casa/3.4.0/lib/libcasa_ms.so.6 (0x00007f114a5a7000)
        libcasa_tables.so.6 => /opt/LOFAR/casa/3.4.0/lib/libcasa_tables.so.6 (0x00007f1149e33000)
        libboost_python38.so.1.71.0 => /lib/x86_64-linux-gnu/libboost_python38.so.1.71.0 (0x00007f1149dd4000)
        libcasa_python3.so.6 => /opt/LOFAR/casa/3.4.0/lib/libcasa_python3.so.6 (0x00007f1149d79000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f1149b95000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1149a46000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f1149a2b000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1149a08000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1149816000)
        libcasa_measures.so.6 => /opt/LOFAR/casa/3.4.0/lib/libcasa_measures.so.6 (0x00007f11494f0000)
        libcasa_casa.so.6 => /opt/LOFAR/casa/3.4.0/lib/libcasa_casa.so.6 (0x00007f11490b0000)
        libgomp.so.1 => /lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f114906e000)
        libcasa_scimath.so.6 => /opt/LOFAR/casa/3.4.0/lib/libcasa_scimath.so.6 (0x00007f1148eba000)
        liblapack.so.3 => /lib/x86_64-linux-gnu/liblapack.so.3 (0x00007f1148816000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f114acee000)
        libpython3.8.so.1.0 => /lib/x86_64-linux-gnu/libpython3.8.so.1.0 (0x00007f11482c0000)
        libhdf5_serial.so.103 => /lib/x86_64-linux-gnu/libhdf5_serial.so.103 (0x00007f1147f41000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1147f3b000)
        libcasa_scimath_f.so.6 => /opt/LOFAR/casa/3.4.0/lib/libcasa_scimath_f.so.6 (0x00007f1147f14000)
        libfftw3f_threads.so.3 => /lib/x86_64-linux-gnu/libfftw3f_threads.so.3 (0x00007f1147f0a000)
        libfftw3_threads.so.3 => /lib/x86_64-linux-gnu/libfftw3_threads.so.3 (0x00007f1147f00000)
        libfftw3f.so.3 => /lib/x86_64-linux-gnu/libfftw3f.so.3 (0x00007f1147cee000)
        libfftw3.so.3 => /lib/x86_64-linux-gnu/libfftw3.so.3 (0x00007f1147ae8000)
        libblas.so.3 => /lib/x86_64-linux-gnu/libblas.so.3 (0x00007f1147a7b000)
        libgfortran.so.5 => /lib/x86_64-linux-gnu/libgfortran.so.5 (0x00007f11477b3000)
        libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f1147785000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f1147769000)
        libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f1147762000)
        libsz.so.2 => /lib/x86_64-linux-gnu/libsz.so.2 (0x00007f114775d000)
        libquadmath.so.0 => /lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007f1147713000)
        libaec.so.0 => /lib/x86_64-linux-gnu/libaec.so.0 (0x00007f114770a000)

and the module can be imported

In [1]: from casacore import tables

In [2]: tables
Out[2]: <module 'casacore.tables' from '/opt/LOFAR/python/lib/python3.8/site-packages/casacore/tables/__init__.py'>

In [3]: tables.table('/home/stefan/Documents/PhD/RADIO/data/LOFAR/CVnI/L652360_SB001_uv.pre-cal_12BF8C5ACt_121MHz.pre-cal.ms.archive')
Successful readonly open of default-locked table /home/stefan/Documents/PhD/RADIO/data/LOFAR/CVnI/L652360_SB001_uv.pre-cal_12BF8C5ACt_121MHz.pre-cal.ms.archive: 25 columns, 6798145 rows
Out[3]: <casacore.tables.table.table at 0x7ff425d540e0>

[tammojan]

Wow that's a nasty flag. Thanks for figuring it out! So do you think we should put this flag in the setup.py of python-casacore?

[maxnoe]

From troubleshooting "as-nneded":

This usually means that a library was not linked to another library, but was using it, and then relying on the final executable to link them together