Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OpenFOAM-8: symbol not found in flat namespace '_MPI_Abort' #76

Open
abzrg opened this issue Jan 8, 2022 · 22 comments
Open

OpenFOAM-8: symbol not found in flat namespace '_MPI_Abort' #76

abzrg opened this issue Jan 8, 2022 · 22 comments

Comments

@abzrg
Copy link

abzrg commented Jan 8, 2022

Installation finished without any error. Every utility or solver I run, aborts. Am I missing something?

output of uname -a:

Darwin machine.local 21.2.0 Darwin Kernel Version 21.2.0: Sun Nov 28 20:29:10 PST 2021; root:xnu-8019.61.5~1/RELEASE_ARM64_T8101 arm64

Complete Error:

% icoFoam
dyld[83421]: symbol not found in flat namespace '_MPI_Abort'
[1]    83421 abort      icoFoam
@abzrg
Copy link
Author

abzrg commented Jan 8, 2022

I found the culprit. In etc/config.sh/mpi there is a line:

libDir=`mpicc --showme:link | sed -e 's/.*-L\([^ ]*\).*/\1/'`

It simply returns one of the directories that are to be linked

/opt/homebrew/opt/libevent/lib

whereas there is also another directory that need to be adde:

% mpicc --showme:link
-L/opt/homebrew/Cellar/open-mpi/4.1.2/lib -L/opt/homebrew/opt/libevent/lib -lmpi

Maybe some kind of regex problem (?)...

So I added the /opt/homebrew/Cellar/open-mpi/4.1.2/lib with _foamAddLib:

~~~
libDir=`mpicc --showme:link | sed -e 's/.*-L\([^ ]*\).*/\1/'`
libDir2="/opt/homebrew/Cellar/open-mpi/4.1.2/lib"
~~~
_foamAddLib     $libDir
_foamAddLib     $libDir2
~~~

Thank you for maintaining this repository. I really appreciate your work.

@abzrg abzrg closed this as completed Jan 8, 2022
@mrklein
Copy link
Owner

mrklein commented Jan 8, 2022

Thanks for the update. Yes, the problem is that on macOS there are two folders in --showme:link, while on Linux it is single folder. Folder lookup code should be corrected.

@massisenergy
Copy link

I found the culprit. In etc/config.sh/mpi there is a line:

libDir=`mpicc --showme:link | sed -e 's/.*-L\([^ ]*\).*/\1/'`

It simply returns one of the directories that are to be linked

/opt/homebrew/opt/libevent/lib

whereas there is also another directory that need to be adde:

% mpicc --showme:link
-L/opt/homebrew/Cellar/open-mpi/4.1.2/lib -L/opt/homebrew/opt/libevent/lib -lmpi

Maybe some kind of regex problem (?)...

So I added the /opt/homebrew/Cellar/open-mpi/4.1.2/lib with _foamAddLib:

~~~
libDir=`mpicc --showme:link | sed -e 's/.*-L\([^ ]*\).*/\1/'`
libDir2="/opt/homebrew/Cellar/open-mpi/4.1.2/lib"
~~~
_foamAddLib     $libDir
_foamAddLib     $libDir2
~~~

Thank you for maintaining this repository. I really appreciate your work.

Thanks for posting this 👍🏽 @reverseila . I think I also have the same issue, so wanted to follow your solution. System: M1Pro with 32Gigs of RAM. Tried OpenFOAM 9 (similar issue) then found this and trying OpenFOAM 8. Part of my etc/config.sh/mpi:

~~~
 45     libDir=`mpicc --showme:link | sed -e 's/.*-L\([^ ]*\).*/\1/'`
 46     libDir2="/opt/homebrew/Cellar/open-mpi/4.1.2/lib" #custom addition
 47
 48     # Bit of a hack: strip off 'lib' and hope this is the path to openmpi
 49     # include files and libraries.
 50     _foam_on_macos || export MPI_ARCH_PATH="${libDir%/*}"
 51
 52     _foamAddLib     $libDir
 53     _foamAddLib     $libDir2 #custom addition
 54
~~~

I also got the same output as you:

$ mpicc --showme:link | sed -e 's/.*-L\([^ ]*\).*/\1/'
/opt/homebrew/opt/libevent/lib
$ mpicc --showme:link
-L/opt/homebrew/Cellar/open-mpi/4.1.2/lib -L/opt/homebrew/opt/libevent/lib -lmpi

After the modification, I start the build with Allwmake. But then it returns the same error:

blockMesh
dyld[75517]: symbol not found in flat namespace '_MPI_Abort'
[1]    75517 abort      blockMesh -help

Can you please help me, I think I'm missing something trivial, but can't figure out myself 🙁

@abzrg
Copy link
Author

abzrg commented Jan 13, 2022

LOL. This is crazy. I've just noticed a strange typo. Not sure where it came from.

Ok, basically do this and you'll be fine.

case "$WM_MPLlB" in

Instead of an uppercase I between L and B, I somehow magically replaced it with an l. So basically, it evaluates to an empty string, and in this case it matches the * branch and ends up setting the FOAM_MPI to dummy... :)

Although it seems wrong, it works so far for me. Maybe I tweaked something else somewhere. But let me know If there is any problem. And sorry for your frustration.

Note that I built OpenFOAM before changing these settings, So I guess you don't need to recompile it. Just test it with those changes.

Thanks for testing it @massisenergy.

So I think the issue need to be reopen.

@abzrg abzrg reopened this Jan 13, 2022
@abzrg
Copy link
Author

abzrg commented Jan 13, 2022

taking a look at _foamAddLib function in etc/config.sh/functions

# ~~~
# Prefix to LD_LIBRARY_PATH
_foamAddLib()
{
    while [ $# -ge 1 ]
    do
        export LD_LIBRARY_PATH=$1:$LD_LIBRARY_PATH
        shift
    done
}
# ~~~

Fixing my typo, having WM_MPLIB set to SYSTEMOPENMPI, and checking the value of LD_LIBRARY_PATH after source, I'll see

/platforms/darwin64Clang/gperftools-svn/lib:

and that comes from config.sh/gperftools line 39.

I manually added those two directories to the beginning of LD_LIBRARY_PATH but no cigar.

Another thing I did was checking that whether the script executed the function or not. So I put two echo commands before and after the _foamAddLib execution. Sourcing bashrc didn't print anything into stdout!

It seems that redirection to stdout is thrown away. I redirect echo command to a debug file to inspect. I've noticed that the SYSTEMOPENMPI branch under the case command is fully executed. However, those two directories that we passed to _foamAddLib does not appear in LD_LIBRARY_PATH.

#~~~
    libDir=`mpicc --showme:link | sed -e 's/.*-L\([^ ]*\).*/\1/'`
    libDir2="/opt/homebrew/Cellar/open-mpi/4.1.2/lib"

    # Bit of a hack: strip off 'lib' and hope this is the path to openmpi
    # include files and libraries.
    _foam_on_macos || export MPI_ARCH_PATH="${libDir%/*}"

    echo "before" > ~/debug
    echo "libDir: $libDir" >> ~/debug
    echo "libDir2: $libDir2" >> ~/debug
    echo "LD_LIBRARY_PATH: $LD_LIBRARY_PATH" >> ~/debug
    _foamAddLib     $libDir
    _foamAddLib     $libDir2
    echo "after" >> ~/debug
    echo "LD_LIBRARY_PATH: $LD_LIBRARY_PATH" >> ~/debug
    unset libDir
    unset libDir2
    ;;
#~~~

Now, sourcing etc/bashrc spits following into the ~/debug

before
libDir: /opt/homebrew/opt/libevent/lib
libDir2: /opt/homebrew/Cellar/open-mpi/4.1.2/lib
LD_LIBRARY_PATH: /platforms/darwin64Clang/gperftools-svn/lib:
after
LD_LIBRARY_PATH: /platforms/darwin64Clang/gperftools-svn/lib:

So it seems that _foamAddLib failed to do its job.

Okay, adding an export command after the _foamAddLib should fix it. But it doesn't. So it seems that is not the root of evil. However, I believe something happens in this branch of case command that if it didn't, everything would be fine

@mrklein
Copy link
Owner

mrklein commented Jan 13, 2022

@reverseila if you take a look at etc/bashrc (line 167):

. $WM_PROJECT_DIR/etc/config.sh/functions

_foam_on_macos && . $WM_PROJECT_DIR/etc/config.sh/mac/functions

So, on macOS additional functions file is sourced (which redefines _foamAddLib function).

LD_LIBRARY_PATH is not used on macOS, variable name is DYLD_LIBRARY_PATH.

These two lines are not touched by patch:

_foamSource `$WM_PROJECT_DIR/bin/foamEtcFile config.sh/ensight`
_foamSource `$WM_PROJECT_DIR/bin/foamEtcFile config.sh/gperftools`

The last does LD_LIBRARY_PATH additions. Yet, as the variable is ignored, I do not worry about it.

@mrklein
Copy link
Owner

mrklein commented Jan 13, 2022

I look at etc/config.sh/mpi file after fresh clone and patch application, there WM_MPLIB is written correctly with I, not l.

@abzrg
Copy link
Author

abzrg commented Jan 13, 2022

I look at etc/config.sh/mpi file after fresh clone and patch application, there WM_MPLIB is written correctly with I, not l.

@mrklein Yes, as I said, it was a typo on my side.

@abzrg
Copy link
Author

abzrg commented Jan 13, 2022

@mrklein Now I see.

$ type _foamAddLib
_foamAddLib is a shell function from /Users/ali/OpenFOAM/OpenFOAM-8/etc/config.sh/mac/functions

So Now, If I do:

#~~~
    libDir=`mpicc --showme:link | sed -e 's/.*-L\([^ ]*\).*/\1/'`
    libDir2="/opt/homebrew/Cellar/open-mpi/4.1.2/lib"

    # Bit of a hack: strip off 'lib' and hope this is the path to openmpi
    # include files and libraries.
    _foam_on_macos || export MPI_ARCH_PATH="${libDir%/*}"

    echo "before" > ~/debug
    echo "libDir: $libDir" >> ~/debug
    echo "libDir2: $libDir2" >> ~/debug
    echo "DYLD_LIBRARY_PATH: $DYLD_LIBRARY_PATH" >> ~/debug
    _foamAddLib     $libDir
    _foamAddLib     $libDir2
    echo "after" >> ~/debug
    echo "DYLD_LIBRARY_PATH: $DYLD_LIBRARY_PATH" >> ~/debug
    unset libDir
    unset libDir2
    ;;
#~~~

I get following:

before
libDir: /opt/homebrew/opt/libevent/lib
libDir2: /opt/homebrew/Cellar/open-mpi/4.1.2/lib
DYLD_LIBRARY_PATH: /Users/ali/OpenFOAM/ali-8/platforms/darwin64ClangDPInt32Opt/lib:/Users/ali/OpenFOAM/site/8/platforms/darwin64ClangDPInt32Opt/lib:/Users/ali/OpenFOAM/OpenFOAM-8/platforms/darwin64ClangDPInt32Opt/lib:/Users/ali/OpenFOAM/OpenFOAM-8/platforms/darwin64ClangDPInt32Opt/lib/dummy:
after
DYLD_LIBRARY_PATH: /opt/homebrew/Cellar/open-mpi/4.1.2/lib:/opt/homebrew/opt/libevent/lib:/Users/ali/OpenFOAM/ali-8/platforms/darwin64ClangDPInt32Opt/lib:/Users/ali/OpenFOAM/site/8/platforms/darwin64ClangDPInt32Opt/lib:/Users/ali/OpenFOAM/OpenFOAM-8/platforms/darwin64ClangDPInt32Opt/lib:/Users/ali/OpenFOAM/OpenFOAM-8/platforms/darwin64ClangDPInt32Opt/lib/dummy:

Which is what we actually want. However that does not fix the linking problem. Still I get the following abortion error:

dyld[14516]: symbol not found in flat namespace '_MPI_Abort'
[1]    14516 abort      icoFoam

Now, let's compare it with the case it works (that magical typo). In that scenario, we don't have those _foamAddLib commands, and checking DYLD_LIBRARY_PATH,

/Users/ali/OpenFOAM/ali-8/platforms/darwin64ClangDPInt32Opt/lib:/Users/ali/OpenFOAM/site/8/platforms/darwin64ClangDPInt32Opt/lib:/Users/ali/OpenFOAM/OpenFOAM-8/platforms/darwin64ClangDPInt32Opt/lib:/Users/ali/OpenFOAM/OpenFOAM-8/platforms/darwin64ClangDPInt32Opt/lib/dummy

We don't have those two directories here anymore. But It fixes that linking issue. So What's wrong?

@mrklein
Copy link
Owner

mrklein commented Jan 13, 2022

My guess would be: if you remove OpenMPI paths from DYLD_LIBRARY_PATH, OpenFOAM simply does not load MPI libraries. So, errors, caused by MPI libraries, are gone. Yet, you can not execute solvers in parallel.

@abzrg
Copy link
Author

abzrg commented Jan 13, 2022

Yes, you are right. I can confirm that. I cannot decompose mesh using decomposePar utility.

@kazumamatata
Copy link

Is there any update about the issue?
Using OpenFOAM-10 the error still persist..

@mrklein
Copy link
Owner

mrklein commented Jul 16, 2023

Since, I do not have access to Apple's ARM hardware, I cannot do anything about the issue.

Could you post output of:

  • file /opt/homebrew/openmpi/lib/libmpi.dylib
  • nm /opt/homebrew/openmpi/lib/libmpi.dylib | grep _MPI_Abort

@kazumamatata
Copy link

Here are the output of your suggested commands:

$ file /opt/homebrew/openmpi/lib/libmpi.dylib
/opt/homebrew/openmpi/lib/libmpi.dylib: cannot open `/opt/homebrew/openmpi/lib/libmpi.dylib' (No such file or directory)
$ nm /opt/homebrew/openmpi/lib/libmpi.dylib | grep _MPI_Abort
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/nm: error: /opt/homebrew/openmpi/lib/libmpi.dylib: No such file or directory

@kazumamatata
Copy link

I found the files on a different location:

$ file /opt/homebrew/Cellar/open-mpi/4.1.5/lib/libmpi.dylib
/opt/homebrew/Cellar/open-mpi/4.1.5/lib/libmpi.dylib: Mach-O 64-bit dynamically linked shared library arm64
$ nm /opt/homebrew/Cellar/open-mpi/4.1.5/lib/libmpi.dylib | grep _MPI_Abort
000000000002e298 T _MPI_Abort

@mrklein
Copy link
Owner

mrklein commented Jul 17, 2023

Thank you for the reply. So, library is there and it has mentioned symbol. Could you post output of the following commands (after setting up OpenFOAM environment):

  • export | grep DYLD
  • nm -m $FOAM_LIBBIN/openmpi-system/libPstream.dylib | grep Abort

@kazumamatata
Copy link

Here are the outputs:

$ export | grep DYLD
DYLD_LIBRARY_PATH=/Users/kunihara/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32Opt/lib/openmpi-system:/opt/homebrew/opt/libevent/lib:/opt/homebrew/Cellar/open-mpi/4.1.4_2/lib:/Users/kunihara/OpenFOAM/kunihara-10/platforms/darwin64ClangDPInt32Opt/lib:/Users/kunihara/OpenFOAM/site/10/platforms/darwin64ClangDPInt32Opt/lib:/Users/kunihara/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32Opt/lib:/Users/kunihara/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32Opt/lib/dummy
FOAM_DYLD_LIBRARY_PATH=/Users/kunihara/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32Opt/lib/openmpi-system:/opt/homebrew/opt/libevent/lib:/opt/homebrew/Cellar/open-mpi/4.1.4_2/lib:/Users/kunihara/OpenFOAM/kunihara-10/platforms/darwin64ClangDPInt32Opt/lib:/Users/kunihara/OpenFOAM/site/10/platforms/darwin64ClangDPInt32Opt/lib:/Users/kunihara/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32Opt/lib:/Users/kunihara/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32Opt/lib/dummy
$ nm -m $FOAM_LIBBIN/openmpi-system/libPstream.dylib | grep Abort
                 (undefined) external _MPI_Abort (dynamically looked up)

@mrklein
Copy link
Owner

mrklein commented Jul 18, 2023

The only difference I see from the output on my Intel macBook is (also you have different versions of open-mpi in this (4.1.4_2) and previous (4.1.5) messages):

nm -m $FOAM_LIBBIN/openmpi-system/libPstream.dylib | grep Abort
                 (undefined) external _MPI_Abort (from libmpi.40)

So, on Intel location of _MPI_Abort symbol is resolved, while on ARM it is not.

Could you post output of:

otool -L $FOAM_LIBBIN/openmpi-system/libPstream.dylib

@kazumamatata
Copy link

Sorry for my delayed answer. This is the output

$ otool -L $FOAM_LIBBIN/openmpi-system/libPstream.dylib
/Users/username/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32Opt/lib/openmpi-system/libPstream.dylib:
	/Users/username/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32Opt/lib/openmpi-system/libPstream.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1300.23.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.100.3)

@mrklein
Copy link
Owner

mrklein commented Jul 26, 2023

So, basically MPI libraries are not linked in your case. Here is my output:

% otool -L openmpi-system/libPstream.dylib 
openmpi-system/libPstream.dylib:
	/Volumes/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32Opt/lib/openmpi-system/libPstream.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/local/opt/open-mpi/lib/libmpi.40.dylib (compatibility version 71.0.0, current version 71.5.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1500.65.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)

Could you post command line used to link Pstream dynamic library? The simples way would be to delete $FOAM_LIBBIN/openmpi-system/libPstream.dylib and run Allwmake script. On my laptop output is:

xcrun c++ -arch x86_64 -std=c++14 -DLIB_NAME=libPstream.dylib -Ddarwin64 -DWM_ARCH_OPTION=64 -DWM_DP -DWM_LABEL_SIZE=32 -Wall -Wextra -Wno-unused-parameter -Wno-overloaded-virtual -Wno-unused-variable -Wno-unused-local-typedef -Wno-invalid-offsetof -Wno-deprecated-register -Wno-undefined-var-template -Wno-unqualified-std-cast-call -O3  -DNoRepository -ftemplate-depth-100 -DOMPI_SKIP_MPICXX -I/usr/local/Cellar/open-mpi/4.1.5/include -IlnInclude -I. -I/Volumes/OpenFOAM/OpenFOAM-10/src/OpenFOAM/lnInclude -I/Volumes/OpenFOAM/OpenFOAM-10/src/OSspecific/POSIX/lnInclude   -fPIC -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -Wl,-dylib,-undefined,dynamic_lookup,-w /Volumes/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32OptSYSTEMOPENMPI/src/Pstream/mpi/UOPwrite.o /Volumes/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32OptSYSTEMOPENMPI/src/Pstream/mpi/UIPread.o /Volumes/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32OptSYSTEMOPENMPI/src/Pstream/mpi/UPstream.o /Volumes/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32OptSYSTEMOPENMPI/src/Pstream/mpi/PstreamGlobals.o -L/Volumes/OpenFOAM/OpenFOAM-11/platforms/darwin64ClangDPInt32OptSYSTEMOPENMPI/lib \
	    -L/usr/local/Cellar/open-mpi/4.1.5/lib -L/usr/local/opt/libevent/lib -lmpi  -o /Volumes/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32Opt/lib/openmpi-system/libPstream.dylib

@kazumamatata
Copy link

I deleted the file $FOAM_LIBBIN/openmpi-system/libPstream.dylib and run Allwmake as suggested. Here is the output of it.

I noticed that the option -DLIB_NAME=libPstream.dylib is not present in my case..

xcrun c++ -arch x86_64 -std=c++14 -Ddarwin64 -DWM_ARCH_OPTION=64 -DWM_DP -DWM_LABEL_SIZE=32 -Wall -Wextra -Wno-unused-parameter -Wno-overloaded-virtual -Wno-unused-variable -Wno-unused-local-typedef -Wno-invalid-offsetof -Wno-deprecated-register -Wno-undefined-var-template -Wno-deprecated-copy -O3  -DNoRepository -ftemplate-depth-100 -DOMPI_SKIP_MPICXX -I/opt/homebrew/Cellar/open-mpi/4.1.5/include -IlnInclude -I. -I/Users/username/OpenFOAM/OpenFOAM-10/src/OpenFOAM/lnInclude -I/Users/username/OpenFOAM/OpenFOAM-10/src/OSspecific/POSIX/lnInclude   -fPIC -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -Wl,-dylib,-undefined,dynamic_lookup,-w /Users/username/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32OptSYSTEMOPENMPI/src/Pstream/mpi/UOPwrite.o /Users/username/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32OptSYSTEMOPENMPI/src/Pstream/mpi/UIPread.o /Users/username/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32OptSYSTEMOPENMPI/src/Pstream/mpi/UPstream.o /Users/username/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32OptSYSTEMOPENMPI/src/Pstream/mpi/PstreamGlobals.o -L/Users/username/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32OptSYSTEMOPENMPI/lib \
	    -L/opt/homebrew/Cellar/open-mpi/4.1.5/lib -L/opt/homebrew/opt/libevent/lib -lmpi  -o /Users/username/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32Opt/lib/openmpi-system/libPstream.dylib

@mrklein
Copy link
Owner

mrklein commented Jul 27, 2023

Thank you for the output. You are mixing architectures and, I think, this causes the problem.

OpenMPI libraries are arm64 binaries (as you posted here: #76 (comment)), while you compile the code for x64_64 architecture (xcrun c++ -arch x86_64 in your last message). Guess, that is why OpenMPI libraries are simply ignored.

Quick solution is to edit $WM_DIR/rules/darwin64Clang/c++ and change the line:

xcrun c++ -arch x86_64 -std=c++14

to

xcrun c++ -arch arm64 -std=c++14

Do the same thing with $WM_DIR/rules/darwin64Clang/c and recompile OpenFOAM.

More complicated solution would be to create separate set of rules in $WM_DIR/rules/darwinArm64Clang and set WM_ARCH variable to darwinArm64. Guess, I will add this to the patches as soon as I get Apple ARM hardware.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants