using Apple clang with USE_OPENMP=1

[dimpase]

Currently a macOS toolchain with Apple's clang 16+ and gfortran fails with make build if USE_OPENMP=1 is set.

The 1st reason for this is that clang rejects -fopenmp, for it needs -Xclang -fopenmp option, as written by R Project in https://mac.r-project.org/openmp/

This was discovered while I bumped into a Homebrew problem, described in Homebrew/homebrew-core#209091

Is there any interest in looking into it further? In principle, R Project might have a point here.

[martin-frbg]

I'm not entirely sure - according to the R link, one would need to obtain a third-party libomp for linking to work with the Apple version of LLVM, so it is far from a standard environment (and apparently one actively discouraged by Apple for whatever reason). At the very least, I'd need to check if a non-Apple clang would cope with the extra -Xclang

And as I understand it, your initial problem arises from trying to use the pkgconfig file from&for a gcc build to obtain your CFLAGS setting - I think the build would probably work correctly if you would run it as make CFLAGS="-Xclang -fopenmp" ... ?

[dimpase]

Apple wants to push users away from OSS into using their own libraries and tools. Just as the botch Python they ship, etc.

I have no idea how much more performance one gets on macOS from openblas by enabling openmp.

Current Homebrew openblas is built with GNU OpenMP (libgomp). I have no idea about it being compatible with clang's idea of OpenMP. If it is compatible, one can think of using GNU OpenMP with clang, no need for R Project's implementation of OpenMP on macOS.

[rgommers]

I don't think a change needs to be made here. As described in the linked R doc, the reason it doesn't work is because Apple has disabled OpenMP on purpose and doesn't ship libomp.dylib. So out of the box nothing will work here, and making the compile succeed without libomp.dylib present is a footgun. If a user or packager goes through the trouble of providing their own libomp build, then they can also be made responsible for passing -Xclang as part of that hacking around.

Conda-forge is shipping OpenBLAS built with OPENMP=1 without problems by the way (with their own Clang builds): https://github.com/conda-forge/openblas-feedstock

[dimpase]

Homebrew does ship libomp, so it might make perfect sense to try using it with openblas Homebrew.

I just don't know how flexible openblas cmake build is to accommodate such a setup.

[dimpase]

OK, I am making progress with building openblas with OpenMP support from LLVM's libomp, as installed by Homebrew,
using cmake build. Good news that it knows the correct flags -Xclang -fopenmp.

There are couple of issues/bugs I encountered in the build, see below. Overcoming them, I ran ctest on the result, all 116 tests passed.
Here is the linkage of the resulting dylib:

% otool -L lib/libopenblas.dylib 
lib/libopenblas.dylib:
        @rpath/libopenblas.0.dylib (compatibility version 0.0.0, current version 0.3.0)
        /opt/homebrew/opt/libomp/lib/libomp.dylib (compatibility version 5.0.0, current version 5.0.0)
        /opt/homebrew/opt/gcc/lib/gcc/current/libgfortran.5.dylib (compatibility version 6.0.0, current version 6.0.0)
        /opt/homebrew/opt/gcc/lib/gcc/current/libgomp.1.dylib (compatibility version 2.0.0, current version 2.0.0)
        /opt/homebrew/opt/gcc/lib/gcc/current/libquadmath.0.dylib (compatibility version 1.0.0, current version 1.0.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1351.0.0)

I called cmake as follows, in clean build/ directory

cmake .. -G Ninja -DCMAKE_BUILD_TYPE=Release  -DBUILD_SHARED_LIBS=ON -DUSE_OPENMP=1

1. cmake build does not try to get the information where to look for omp.h from FindOpenMP. I made a crude patch to fix it for the time being, but there must be a proper cmake way for this.

--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -105,6 +105,7 @@ message(WARNING "CMake support is experimental. It does not yet support all buil
 
 if (USE_OPENMP)
   find_package(OpenMP REQUIRED)
+  include_directories("/opt/homebrew/include")
 endif ()
 
 include("${PROJECT_SOURCE_DIR}/cmake/utils.cmake")

2. Apple's clang appears not to look at the current directory for include files. In particular it does not find lapacke_mangling.h which is generated at some point. I worked it around by manually copying it into one of the directories in -I options, but, again, there must be a proper cmake way.

---

Would you be interested in a PR fixing 1 and/or 2?

[dimpase]

PS. There is also one more quirk to be fixed: openblas.pc is not make quite correct, I got

libdir=/usr/local/lib
libnameprefix=
libnamesuffix=
libsuffix=
includedir=/usr/local/include/openblas

openblas_config=USE_64BITINT= NO_CBLAS= NO_LAPACK= NO_LAPACKE= DYNAMIC_ARCH=OFF DYNAMIC_OLDER=OFF NO_AFFINITY=1 USE_OPENMP=1 VORTEX MAX_THREADS=10 
Name: OpenBLAS
Description: OpenBLAS is an optimized BLAS library based on GotoBLAS2 1.13 BSD version
Version: 0.3.29
URL: https://github.com/OpenMathLib/OpenBLAS
Libs: -L${libdir} -l${libnameprefix}openblas${libnamesuffix}${libsuffix} 
Cflags: -I${includedir} -Xclang -fopenmp 

Here libdir and includedir have totally wrong values

Although cflags are fine :-)

EDIT - or probably there is nothing to fix here, it's just default values, which can be changed by passing appropriate parameters to cmake.

[martin-frbg]

I wonder what is the point of getting only libomp from homebrew while continuing to use the Apple-flavored clang, when I guess you could get the entire toolchain from homebrew and solve the include path issue that way ?

[dimpase]

include_directory(".") solves 2, and fixing 1 property is useful for a not so unusual case when the location of libomp one wants to use is non-standard.

Also, Homebrew does not mandate a full Homebrew toolchain.

[rgommers]

The output of otool -L lib/libopenblas.dylib shows that both libgomp and libomp are linked. That won't work at all. @dimpase also looking at the conversation in Homebrew/homebrew-core#209091, I think your ask here is based on a misunderstanding of how things should work. You never ever want to link two OpenMP runtimes, and ideally never have more than one in your whole software stack.

Homebrew builds with GCC, so it will be pulling in libgomp. Hence that's what should be used.

[dimpase]

Thus we see that it's possible to trick openblas' cmake build into producing an incorrect libopenblas, isn't it?
(And then tests don't show this, probably because the C side of openmp is not tested.)

[rgommers]

I'm not sure what the last comment means - you can always "trick" builds of complex packages to have a non-optimal or non-functional outcome. I think this issue should be resolved by either closing without a change, or adding a note to the Install section of the docs, something like: "Apple discourages use of OpenMP when using the Apple Clang compiler; if you use that compiler then prefer building with the default pthreads. If you really want to use OPENMP=1 with Apple Clang, you should add -Xclang -fopenmp to CFLAGS and ensure you do this for other packages in your environment that use OpenMP as well."

[dimpase]

@rgommers can one use gfortan with libomp? If so, we could have a functional build of openblas with Apple clang and gfortan.

It does not seem to work the other way around, using only libgomp with Apple clang (and gfortan), as there is some kind of incompatibility - it doesn't seem to be possible to build a working executable (say, of a standard OpenMP example) using Apple clang and libgomp.

[rgommers]

can one use gfortan with libomp?

I'm not sure, I can't remember anyone trying or any project shipping binaries built this way.

I am still unclear on why you're so interested in this, it just seems like a bad idea to do this kind of mixing.

[isuruf]

We do in conda-forge. Just need a symlink from libgomp.so to libomp.so.

[dimpase]

can one use gfortan with libomp?

I'm not sure, I can't remember anyone trying or any project shipping binaries built this way.

I am still unclear on why you're so interested in this, it just seems like a bad idea to do this kind of mixing.

because mixing different C/C++ toolchains in one project is not much better than mixing different OpenMP libraries.

For Homebrew, and on *BSDs, clang+gfortran is the usual toolchain. Using libgomp, as the current Homebrew package does, does not allow any OpenMP use on the C side.
Thus why not use libomp instead?

[dimpase]

We do in conda-forge. Just need a symlink from libgomp.so to libomp.so.

that's a hack. Is there a nice, proper way to call cmake to achieve the same effect?

[martin-frbg]

converting this topic to a discussion - I do not get the impression that there is anything here that OpenBLAS could or should be changing

[dimpase]

Thanks, @martin-frbg for keeping this open. I still think that OpenBLAS should build on macOS (or other clang+gfortran toolchains - e.g. that's the normal toolchain on FreeBSD and OpenBSD, as well as on Android, ChromeOS, OpenMandriva, and Chimera Linux) without any drastic measures like patching config files or making libgomp a link to libomp. The platforms listed above aren't some "exotic" platforms, they are rather popular ones.

And at least as far as the cmake build is going, a fix for this would merely allow configuring Apple clang CFLAGS (if it's used/applicable to the platform), as well when gfortran is used in linking more, it should have -fopenmp flag (which triggers linking to libgomp) with -L<path to libomp> -lomp.

Would you consider a PR implementing changes to the cmake config to accommodate the latter?

[ywwry66]

I was able to build openblas with the cmake options -DUSE_OPENMP=1, -DOpenMP_C_FLAGS="-Xpreprocessor -fopenmp -I$(brew --prefix libomp)/include", -DOpenMP_C_LIB_NAMES=omp, and -DOpenMP_omp_LIBRARY="-L$(brew --prefix libomp)/lib -lomp". However, I couldn't find a way to prevent it from linking to libgomp.dylib. Not sure why passing -lomp to the linker through make versus cmake leads to different results.

[dimpase]

I was able to build openblas with the cmake options -DUSE_OPENMP=1, -DOpenMP_C_FLAGS="-Xpreprocessor -fopenmp -I$(brew --prefix libomp)/include", -DOpenMP_C_LIB_NAMES=omp, and -DOpenMP_omp_LIBRARY="-L$(brew --prefix libomp)/lib -lomp". However, I couldn't find a way to prevent it from linking to libgomp.dylib. Not sure why passing -lomp to the linker through make versus cmake leads to different results.

IMHO linking to libgomp.dylib happens when you call gfortran -fopenmp in link mode.
At least this is what my experiments on Linux tells me

[ywwry66]

Yes, you are right. Looks like the only way to preventing linking to libgomp is: using-fopenmp for compiler mode while using -lomp without -fopenmp for linker mode. I did some testing and found that multi-threading works properly for OpenMP Fortran code in this way. However, this seems very tricky to implement.

[dimpase]

"very tricky" looks like an overstatement.

If you talk about makefiles, it's just a different rule that can be made for linking with gfortran using libomp.

similarly with cmake - compute the variables to be used for gfortran linking (one for the call to gcc, one for the extra -L and -lomp flags to be passed - it might be needed to keep them apart.)

I'd even consider reaching upstream (gcc) to tell them to stop hardcoding libgomp into gfortran -fopenmp.

[ywwry66]

I have found a solution based on the cmake infrastructure without needing to manually tweak the flags. Will open a PR.