Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build concept? #602

Open
thompsonbry opened this issue Nov 20, 2024 · 38 comments
Open

Build concept? #602

thompsonbry opened this issue Nov 20, 2024 · 38 comments

Comments

@thompsonbry
Copy link

From a download of master, the following does not succeed.
cmake -DSTANDALONE=ON .

The detailed output is below.

Is the expectation that the user should check out each project into the corresponding 3rd_party library before invoking cmake at the top level? Or am I missing something and there is an incantation which will do all of this automagically?

Thanks,
Bryan

-- Warning diagnostics are not fatal
-- Using AVX2 instructions on x86_64
-- Code coverage reporting not enabled
-- Enabling IPO/LTO for release config
CMake Error at CMakeLists.txt:329 (add_subdirectory):
  The source directory

    /local/home/bryant/p8/src/Neptune-Playground/src/hash/unodb/unodb-master/3rd_party/googletest

  does not contain a CMakeLists.txt file.


-- Enabling LTO for Google Benchmark
CMake Error at CMakeLists.txt:353 (add_subdirectory):
  The source directory

    /local/home/bryant/p8/src/Neptune-Playground/src/hash/unodb/unodb-master/3rd_party/benchmark

  does not contain a CMakeLists.txt file.


CMake Error at /usr/local/share/cmake-3.25/Modules/ExternalProject.cmake:3115 (message):
  No download info given for '3rd_party_deepstate' and its source directory:

   /local/home/bryant/p8/src/Neptune-Playground/src/hash/unodb/unodb-master/3rd_party/deepstate

  is not an existing non-empty directory.  Please specify one of:

   * SOURCE_DIR with an existing non-empty directory
   * DOWNLOAD_COMMAND
   * URL
   * GIT_REPOSITORY
   * SVN_REPOSITORY
   * HG_REPOSITORY
   * CVS_REPOSITORY and CVS_MODULE
Call Stack (most recent call first):
  /usr/local/share/cmake-3.25/Modules/ExternalProject.cmake:4185 (_ep_add_download_command)
  CMakeLists.txt:374 (ExternalProject_Add)
@laurynas-biveinis
Copy link
Owner

Some 3rd party dependencies are vendored as Git submodules, so you have to do git submodule update --init first.

@thompsonbry
Copy link
Author

Ok. Here is my incantation.

git clone https://github.com/laurynas-biveinis/unodb.git
cd unodb
git submodule update --init
export CC=/usr/bin/gcc10-cc
export CXX=/usr/bin/gcc10-c++
cmake -DSTANDALONE=ON .
make

This fails as follows:

[ 11%] Building C object CMakeFiles/deepstate32.dir/src/lib/DeepState.c.o
In file included from /usr/include/features.h:447,
                 from /usr/include/assert.h:35,
                 from /local/home/bryant/p8/src/Neptune-Playground/src/hash/unodb/3rd_party/deepstate/src/include/deepstate/DeepState.h:20,
                 from /local/home/bryant/p8/src/Neptune-Playground/src/hash/unodb/3rd_party/deepstate/src/lib/DeepState.c:17:
/usr/include/gnu/stubs.h:7:11: fatal error: gnu/stubs-32.h: No such file or directory
    7 | # include <gnu/stubs-32.h>
      |           ^~~~~~~~~~~~~~~~
compilation terminated.

The full output of the command sequence is below.

cmake -DSTANDALONE=ON .

Cloning into 'unodb'...
remote: Enumerating objects: 7592, done.
remote: Counting objects: 100% (1109/1109), done.
remote: Compressing objects: 100% (462/462), done.
remote: Total 7592 (delta 662), reused 1055 (delta 628), pack-reused 6483 (from 1)
Receiving objects: 100% (7592/7592), 3.68 MiB | 21.30 MiB/s, done.
Resolving deltas: 100% (4862/4862), done.
Submodule '3rd_party/GSL' (https://github.com/microsoft/GSL.git) registered for path '3rd_party/GSL'
Submodule '3rd_party/benchmark' (https://github.com/google/benchmark.git) registered for path '3rd_party/benchmark'
Submodule '3rd_party/deepstate' (https://github.com/trailofbits/deepstate.git) registered for path '3rd_party/deepstate'
Submodule '3rd_party/googletest' (https://github.com/google/googletest.git) registered for path '3rd_party/googletest'
Cloning into '/local/home/bryant/p8/src/Neptune-Playground/src/hash/unodb/3rd_party/GSL'...
Cloning into '/local/home/bryant/p8/src/Neptune-Playground/src/hash/unodb/3rd_party/benchmark'...
Cloning into '/local/home/bryant/p8/src/Neptune-Playground/src/hash/unodb/3rd_party/deepstate'...
Cloning into '/local/home/bryant/p8/src/Neptune-Playground/src/hash/unodb/3rd_party/googletest'...
Submodule path '3rd_party/GSL': checked out '82ba6c77ceb70cf92add22ca9ecaf0fc7cac99fa'
Submodule path '3rd_party/benchmark': checked out '12235e24652fc7f809373e7c11a5f73c5763fc4c'
Submodule path '3rd_party/deepstate': checked out 'b0b42eb1d49f9e302dbc49e4b4effbe23568e5cd'
Submodule path '3rd_party/googletest': checked out 'b514bdc898e2951020cbdca1304b75f5950d1f59'
zsh: command not found: #
zsh: command not found: #
-- The CXX compiler identification is GNU 10.5.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/gcc10-c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Warning diagnostics are not fatal
-- Using AVX2 instructions on x86_64
-- Code coverage reporting not enabled
-- Enabling IPO/LTO for release config
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE
-- Found Boost: /usr/local/lib/cmake/Boost-1.85.0/BoostConfig.cmake (found version "1.85.0")
-- The C compiler identification is GNU 10.5.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/gcc10-cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Enabling LTO for Google Benchmark
-- Failed to find LLVM FileCheck
-- Found Git: /usr/bin/git (found version "2.40.1")
-- Google Benchmark version: v1.9.0, normalized to 1.9.0
-- Looking for shm_open in rt
-- Looking for shm_open in rt - found
-- Performing Test HAVE_CXX_FLAG_WALL
-- Performing Test HAVE_CXX_FLAG_WALL - Success
-- Performing Test HAVE_CXX_FLAG_WEXTRA
-- Performing Test HAVE_CXX_FLAG_WEXTRA - Success
-- Performing Test HAVE_CXX_FLAG_WSHADOW
-- Performing Test HAVE_CXX_FLAG_WSHADOW - Success
-- Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL
-- Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL - Success
-- Performing Test HAVE_CXX_FLAG_WOLD_STYLE_CAST
-- Performing Test HAVE_CXX_FLAG_WOLD_STYLE_CAST - Success
-- Performing Test HAVE_CXX_FLAG_WCONVERSION
-- Performing Test HAVE_CXX_FLAG_WCONVERSION - Success
-- Performing Test HAVE_CXX_FLAG_WERROR
-- Performing Test HAVE_CXX_FLAG_WERROR - Success
-- Performing Test HAVE_CXX_FLAG_WSUGGEST_OVERRIDE
-- Performing Test HAVE_CXX_FLAG_WSUGGEST_OVERRIDE - Success
-- Performing Test HAVE_CXX_FLAG_PEDANTIC
-- Performing Test HAVE_CXX_FLAG_PEDANTIC - Success
-- Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS
-- Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS - Success
-- Performing Test HAVE_CXX_FLAG_WSHORTEN_64_TO_32
-- Performing Test HAVE_CXX_FLAG_WSHORTEN_64_TO_32 - Failed
-- Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING
-- Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING - Success
-- Performing Test HAVE_CXX_FLAG_WNO_DEPRECATED_DECLARATIONS
-- Performing Test HAVE_CXX_FLAG_WNO_DEPRECATED_DECLARATIONS - Success
-- Performing Test HAVE_CXX_FLAG_WNO_DEPRECATED
-- Performing Test HAVE_CXX_FLAG_WNO_DEPRECATED - Success
-- Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING
-- Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING - Success
-- Performing Test HAVE_CXX_FLAG_WD654
-- Performing Test HAVE_CXX_FLAG_WD654 - Failed
-- Performing Test HAVE_CXX_FLAG_WTHREAD_SAFETY
-- Performing Test HAVE_CXX_FLAG_WTHREAD_SAFETY - Failed
-- Performing Test HAVE_CXX_FLAG_FLTO
-- Performing Test HAVE_CXX_FLAG_FLTO - Success
-- Performing Test HAVE_CXX_FLAG_WNO_LTO_TYPE_MISMATCH
-- Performing Test HAVE_CXX_FLAG_WNO_LTO_TYPE_MISMATCH - Success
-- Performing Test HAVE_CXX_FLAG_COVERAGE
-- Performing Test HAVE_CXX_FLAG_COVERAGE - Success
-- Compiling and running to test HAVE_STD_REGEX
-- Performing Test HAVE_STD_REGEX -- success
-- Compiling and running to test HAVE_GNU_POSIX_REGEX
-- Performing Test HAVE_GNU_POSIX_REGEX -- failed to compile
-- Compiling and running to test HAVE_POSIX_REGEX
-- Performing Test HAVE_POSIX_REGEX -- success
-- Compiling and running to test HAVE_STEADY_CLOCK
-- Performing Test HAVE_STEADY_CLOCK -- success
-- Compiling and running to test HAVE_PTHREAD_AFFINITY
-- Performing Test HAVE_PTHREAD_AFFINITY -- failed to compile
-- Not using clang-tidy due to non-clang compiler being used
-- cppcheck not found
-- cpplint not found
-- User-set CMake options:
-- STANDALONE: ON
-- AVX2: ON
-- COVERAGE: OFF
-- GCOV_PATH:
-- SANITIZE_ADDRESS: OFF
-- SANITIZE_THREAD: OFF
-- SANITIZE_UB: OFF
-- STATIC_ANALYSIS: OFF
-- CPPCHECK_AGGRESSIVE: OFF
-- IWYU: OFF
-- Configuring done
-- Generating done
-- Build files have been written to: /local/home/bryant/p8/src/Neptune-Playground/src/hash/unodb

(24-11-21 15:15:18) <0> [/local/home/bryant/p8/src/Neptune-Playground/src/hash/unodb]
dev-dsk-bryant-2b-a6dd7191 % make
[  1%] Creating directories for '3rd_party_deepstate'
[  2%] No download step for '3rd_party_deepstate'
[  3%] No update step for '3rd_party_deepstate'
[  4%] No patch step for '3rd_party_deepstate'
[  5%] Performing configure step for '3rd_party_deepstate'
-- The C compiler identification is GNU 10.5.0
-- The CXX compiler identification is GNU 10.5.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/gcc10-cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/gcc10-c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Deprecation Warning at CMakeLists.txt:16 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


-- Configuring done
-- Generating done
-- Build files have been written to: /local/home/bryant/p8/src/Neptune-Playground/src/hash/unodb/3rd_party/deepstate
[  6%] Performing build step for '3rd_party_deepstate'
[  1%] Building C object CMakeFiles/deepstate.dir/src/lib/DeepState.c.o
[  3%] Building C object CMakeFiles/deepstate.dir/src/lib/Log.c.o
[  5%] Building C object CMakeFiles/deepstate.dir/src/lib/Option.c.o
[  7%] Building C object CMakeFiles/deepstate.dir/src/lib/Stream.c.o
[  9%] Linking C static library libdeepstate.a
[  9%] Built target deepstate
[ 11%] Building C object CMakeFiles/deepstate32.dir/src/lib/DeepState.c.o
In file included from /usr/include/features.h:447,
                 from /usr/include/assert.h:35,
                 from /local/home/bryant/p8/src/Neptune-Playground/src/hash/unodb/3rd_party/deepstate/src/include/deepstate/DeepState.h:20,
                 from /local/home/bryant/p8/src/Neptune-Playground/src/hash/unodb/3rd_party/deepstate/src/lib/DeepState.c:17:
/usr/include/gnu/stubs.h:7:11: fatal error: gnu/stubs-32.h: No such file or directory
    7 | # include <gnu/stubs-32.h>
      |           ^~~~~~~~~~~~~~~~
compilation terminated.
make[5]: *** [CMakeFiles/deepstate32.dir/src/lib/DeepState.c.o] Error 1
make[4]: *** [CMakeFiles/deepstate32.dir/all] Error 2
make[3]: *** [all] Error 2
make[2]: *** [3rd_party_deepstate-prefix/src/3rd_party_deepstate-stamp/3rd_party_deepstate-build] Error 2
make[1]: *** [CMakeFiles/3rd_party_deepstate.dir/all] Error 2
make: *** [all] Error 2

@thompsonbry
Copy link
Author

This appears to be a glibc-devel issue? However I observe

sudo yum install glibc-devel
Package glibc-devel-2.26-64.amzn2.0.2.x86_64 already installed and latest version

@laurynas-biveinis
Copy link
Owner

For some reason, DeepState requires 32-bit version of libc dev files: glibc-devel.i686

@laurynas-biveinis
Copy link
Owner

If that doesn't work for some reason, the easiest would be to disable DeepState build in CMake files, fuzzing the library probably isn't your top priority right now

@thompsonbry
Copy link
Author

Ok. Let me try that. The 32-bit version is not in the configured list of repositories....

@thompsonbry
Copy link
Author

I don't see the option to disable that in the README.md?

@laurynas-biveinis
Copy link
Owner

Yeah, I was referring to "files" as in editing CMakeLists.txt, because there is no option for that yet

@thompsonbry
Copy link
Author

thompsonbry commented Nov 21, 2024

So maybe https://github.com/laurynas-biveinis/unodb/blob/master/CMakeLists.txt#L370

if(NOT (is_darwin AND (is_gxx OR SANITIZE_ADDRESS OR SANITIZE_THREAD))

change to

if(FALSE AND NOT (is_darwin AND (is_gxx OR SANITIZE_ADDRESS OR SANITIZE_THREAD))

?

Or maybe a few lines down

if(NOT (is_darwin AND (is_gxx OR SANITIZE_ADDRESS OR SANITIZE_THREAD))
    AND NOT is_windows AND NOT is_arm)
  # ThreadSanitizer is not compatible with libfuzzer and LLVM 11 linker crashes
  # on libfuzzer release build
  if(FALSE) # is_clang AND NOT(is_release) AND NOT(SANITIZE_THREAD))
    set(LIBFUZZER_AVAILABLE TRUE)
    set(BUILD_DEEPSTATE_LIBFUZZER "-DDEEPSTATE_LIBFUZZER=ON")
  else()
    set(LIBFUZZER_AVAILABLE FALSE)
    set(BUILD_DEEPSTATE_LIBFUZZER "-DDEEPSTATE_LIBFUZZER=OFF")
  endif()

@thompsonbry
Copy link
Author

I can't figure out any way to really clean out the generated makefiles. Just fyi. The following does not seem to be enough. So I am just redoing things from the git clone.

cmake --build . --target clean

@laurynas-biveinis
Copy link
Owner

So maybe https://github.com/laurynas-biveinis/unodb/blob/master/CMakeLists.txt#L370

if(NOT (is_darwin AND (is_gxx OR SANITIZE_ADDRESS OR SANITIZE_THREAD))

change to

if(FALSE AND NOT (is_darwin AND (is_gxx OR SANITIZE_ADDRESS OR SANITIZE_THREAD))

?

This one

@laurynas-biveinis
Copy link
Owner

I can't figure out any way to really clean out the generated makefiles. Just fyi. The following does not seem to be enough. So I am just redoing things from the git clone.

cmake --build . --target clean

I never tested in-source builds, and always do mkdir build && cd build && cmake ..

@thompsonbry
Copy link
Author

ack. i will try that.

still hitting the deepstate dependency on that library with that edit to CMakeLists.txt.

@thompsonbry
Copy link
Author

Updated incantation works around the lack of the glibc-devel 32-bit library. I will update again once I have this working.

git clone https://github.com/laurynas-biveinis/unodb.git
cd unodb
git submodule update --init
export CC=/usr/bin/gcc10-cc
export CXX=/usr/bin/gcc10-c++
mkdir build && cd build
# EDIT ../CMakeLists.txt and disable deepstate fuzzing (requires glibc-devel which I can't find for AL2, AL2023)
# Replace
if(NOT (is_darwin AND (is_gxx OR SANITIZE_ADDRESS OR SANITIZE_THREAD))
    AND NOT is_windows AND NOT is_arm)
# With
if(FALSE)
#
cmake -DSTANDALONE=ON ..
make

Hits an issue with boost backtrace.

dev-dsk-bryant-2b-a6dd7191 % make
[  1%] Building CXX object CMakeFiles/unodb_qsbr.dir/qsbr.cpp.o
In file included from /usr/local/include/boost/stacktrace/detail/frame_unwind.ipp:25,
                 from /usr/local/include/boost/stacktrace/frame.hpp:59,
                 from /usr/local/include/boost/stacktrace.hpp:15,
                 from /local/home/bryant/p8/src/Neptune-Playground/src/hash/unodb/assert.hpp:17,
                 from /local/home/bryant/p8/src/Neptune-Playground/src/hash/unodb/qsbr.hpp:42,
                 from /local/home/bryant/p8/src/Neptune-Playground/src/hash/unodb/qsbr.cpp:7:
/usr/local/include/boost/stacktrace/detail/libbacktrace_impls.hpp:23:13: fatal error: backtrace.h: No such file or directory
   23 | #   include <backtrace.h>
      |             ^~~~~~~~~~~~~
compilation terminated.
make[2]: *** [CMakeFiles/unodb_qsbr.dir/qsbr.cpp.o] Error 1
make[1]: *** [CMakeFiles/unodb_qsbr.dir/all] Error 2
make: *** [all] Error 2

@thompsonbry
Copy link
Author

partial worked around for backtrace.h

git clone https://github.com/ianlancetaylor/libbacktrace.git
cd libbacktrace
./configure
sudo make install
# Note: installed into /usr/local/lib

Build progresses further, but fails on -lbacktrace.

[ 58%] Linking CXX executable micro_benchmark_key_prefix
/usr/bin/gcc10-ld: error: cannot find -lbacktrace
/usr/bin/gcc10-ld: error: cannot find -lbacktrace
lto-wrapper: warning: using serial compilation of 2 LTRANS jobs
/usr/local/include/boost/stacktrace/detail/libbacktrace_impls.hpp:104: error: undefined reference to 'backtrace_create_state'
/usr/local/include/boost/stacktrace/detail/libbacktrace_impls.hpp:127: error: undefined reference to 'backtrace_pcinfo'
/usr/local/include/boost/stacktrace/detail/libbacktrace_impls.hpp:135: error: undefined reference to 'backtrace_syminfo'
collect2: error: ld returned 1 exit status
make[2]: *** [benchmark/micro_benchmark_key_prefix] Error 1
make[1]: *** [benchmark/CMakeFiles/micro_benchmark_key_prefix.dir/all] Error 2
make: *** [all] Error 2

@laurynas-biveinis
Copy link
Owner

laurynas-biveinis commented Nov 22, 2024

I have spun up an Amazon Linux 2023 instance to come up with the recipe there

@thompsonbry
Copy link
Author

thompsonbry commented Nov 22, 2024 via email

@laurynas-biveinis
Copy link
Owner

laurynas-biveinis commented Nov 22, 2024

On AL2023 I got a successful build with:
disabling DeepState in CMakeLists.txt as discussed above

dnf install git gcc g++ cmake boost-devel
git clone https://github.com/ianlancetaylor/libbacktrace
(cd libbacktrace && ./configure && make && sudo make install)

then out-of-source build with -DSTANDALONE=ON -DCMAKE_BUILD_TYPE=Debug

@laurynas-biveinis
Copy link
Owner

Release configuration works too. I'll see if I can test AL2

@laurynas-biveinis
Copy link
Owner

OK, so AL2 packages Boost 1.53, which does not contain Boost.Stacktrace and your logs above suggest custom-compiled Boost. I believe for me the best way forward is to make that dependency optional.

@laurynas-biveinis
Copy link
Owner

laurynas-biveinis commented Nov 22, 2024

For AL2, try the branch at #603, which will not try to use Boost.Stacktrace if it's not present (this of course will require removing your, presumably custom built, Boost)

(Manually disabling DeepState is still required on that branch)

@laurynas-biveinis
Copy link
Owner

For DeepState, I am testing #604

@thompsonbry
Copy link
Author

thompsonbry commented Nov 22, 2024

For AL2, try the branch at #603, which will not try to use Boost.Stacktrace if it's not present (this of course will require removing your, presumably custom built, Boost)

(Manually disabling DeepState is still required on that branch)

Build still fails with DeepState disabled on the #603 branch

[ 57%] Building CXX object benchmark/CMakeFiles/micro_benchmark_key_prefix.dir/micro_benchmark_key_prefix.cpp.o
[ 58%] Linking CXX executable micro_benchmark_key_prefix
/usr/bin/gcc10-ld: error: cannot find -lbacktrace
/usr/bin/gcc10-ld: error: cannot find -lbacktrace
lto-wrapper: warning: using serial compilation of 2 LTRANS jobs
/usr/local/include/boost/stacktrace/detail/libbacktrace_impls.hpp:104: error: undefined reference to 'backtrace_create_state'
/usr/local/include/boost/stacktrace/detail/libbacktrace_impls.hpp:127: error: undefined reference to 'backtrace_pcinfo'
/usr/local/include/boost/stacktrace/detail/libbacktrace_impls.hpp:135: error: undefined reference to 'backtrace_syminfo'
collect2: error: ld returned 1 exit status
make[2]: *** [benchmark/micro_benchmark_key_prefix] Error 1
make[1]: *** [benchmark/CMakeFiles/micro_benchmark_key_prefix.dir/all] Error 2
make: *** [all] Error 2
 git status
On branch optional-boost-stacktrace
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   CMakeLists.txt

no changes added to commit (use "git add" and/or "git commit -a")
git diff
diff --git a/CMakeLists.txt b/CMakeLists.txt
index fbb1509..364786f 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -367,8 +367,9 @@ RESTORE_CXX_FLAGS_FOR_SUBDIR()
 # - on ARM
 # - with GCC under macOS due to https://github.com/trailofbits/deepstate/issues/374
 # - under macOS with ASan or TSan enabled
-if(NOT (is_darwin AND (is_gxx OR SANITIZE_ADDRESS OR SANITIZE_THREAD))
-    AND NOT is_windows AND NOT is_arm)
+#if(NOT (is_darwin AND (is_gxx OR SANITIZE_ADDRESS OR SANITIZE_THREAD))
+#    AND NOT is_windows AND NOT is_arm)
+if(FALSE)
   # ThreadSanitizer is not compatible with libfuzzer and LLVM 11 linker crashes
   # on libfuzzer release build
   if(is_clang AND NOT(is_release) AND NOT(SANITIZE_THREAD))

My process was:

git pull
git checkout -b optional-boost-stacktrace
# Edit CMakeLists.txt per above
cd .../build
rm -rf *
cmake -DSTANDALONE=ON ..
make

The cmake output is:

cmake -DSTANDALONE=ON ..
-- The CXX compiler identification is GNU 10.5.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/gcc10-c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Warning diagnostics are not fatal
-- Using AVX2 instructions on x86_64
-- Code coverage reporting not enabled
-- Enabling IPO/LTO for release config
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE
-- Found Boost: /usr/local/lib/cmake/Boost-1.85.0/BoostConfig.cmake (found version "1.85.0")
-- The C compiler identification is GNU 10.5.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/gcc10-cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Enabling LTO for Google Benchmark
-- Failed to find LLVM FileCheck
-- Found Git: /usr/bin/git (found version "2.40.1")
-- Google Benchmark version: v1.9.0, normalized to 1.9.0
-- Looking for shm_open in rt
-- Looking for shm_open in rt - found
-- Performing Test HAVE_CXX_FLAG_WALL
-- Performing Test HAVE_CXX_FLAG_WALL - Success
-- Performing Test HAVE_CXX_FLAG_WEXTRA
-- Performing Test HAVE_CXX_FLAG_WEXTRA - Success
-- Performing Test HAVE_CXX_FLAG_WSHADOW
-- Performing Test HAVE_CXX_FLAG_WSHADOW - Success
-- Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL
-- Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL - Success
-- Performing Test HAVE_CXX_FLAG_WOLD_STYLE_CAST
-- Performing Test HAVE_CXX_FLAG_WOLD_STYLE_CAST - Success
-- Performing Test HAVE_CXX_FLAG_WCONVERSION
-- Performing Test HAVE_CXX_FLAG_WCONVERSION - Success
-- Performing Test HAVE_CXX_FLAG_WERROR
-- Performing Test HAVE_CXX_FLAG_WERROR - Success
-- Performing Test HAVE_CXX_FLAG_WSUGGEST_OVERRIDE
-- Performing Test HAVE_CXX_FLAG_WSUGGEST_OVERRIDE - Success
-- Performing Test HAVE_CXX_FLAG_PEDANTIC
-- Performing Test HAVE_CXX_FLAG_PEDANTIC - Success
-- Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS
-- Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS - Success
-- Performing Test HAVE_CXX_FLAG_WSHORTEN_64_TO_32
-- Performing Test HAVE_CXX_FLAG_WSHORTEN_64_TO_32 - Failed
-- Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING
-- Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING - Success
-- Performing Test HAVE_CXX_FLAG_WNO_DEPRECATED_DECLARATIONS
-- Performing Test HAVE_CXX_FLAG_WNO_DEPRECATED_DECLARATIONS - Success
-- Performing Test HAVE_CXX_FLAG_WNO_DEPRECATED
-- Performing Test HAVE_CXX_FLAG_WNO_DEPRECATED - Success
-- Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING
-- Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING - Success
-- Performing Test HAVE_CXX_FLAG_WD654
-- Performing Test HAVE_CXX_FLAG_WD654 - Failed
-- Performing Test HAVE_CXX_FLAG_WTHREAD_SAFETY
-- Performing Test HAVE_CXX_FLAG_WTHREAD_SAFETY - Failed
-- Performing Test HAVE_CXX_FLAG_FLTO
-- Performing Test HAVE_CXX_FLAG_FLTO - Success
-- Performing Test HAVE_CXX_FLAG_WNO_LTO_TYPE_MISMATCH
-- Performing Test HAVE_CXX_FLAG_WNO_LTO_TYPE_MISMATCH - Success
-- Performing Test HAVE_CXX_FLAG_COVERAGE
-- Performing Test HAVE_CXX_FLAG_COVERAGE - Success
-- Compiling and running to test HAVE_STD_REGEX
-- Performing Test HAVE_STD_REGEX -- success
-- Compiling and running to test HAVE_GNU_POSIX_REGEX
-- Performing Test HAVE_GNU_POSIX_REGEX -- failed to compile
-- Compiling and running to test HAVE_POSIX_REGEX
-- Compiling and running to test HAVE_STEADY_CLOCK
-- Performing Test HAVE_STEADY_CLOCK -- success
-- Compiling and running to test HAVE_PTHREAD_AFFINITY
-- Performing Test HAVE_PTHREAD_AFFINITY -- failed to compile
-- Not using clang-tidy due to non-clang compiler being used
-- cppcheck not found
-- cpplint not found
-- User-set CMake options:
-- STANDALONE: ON
-- MAINTAINER_MODE: OFF
-- AVX2: ON
-- COVERAGE: OFF
-- GCOV_PATH:
-- SANITIZE_ADDRESS: OFF
-- SANITIZE_THREAD: OFF
-- SANITIZE_UB: OFF
-- STATIC_ANALYSIS: OFF
-- CPPCHECK_AGGRESSIVE: OFF
-- IWYU: OFF
-- Configuring done
-- Generating done
-- Build files have been written to: /local/home/bryant/p8/src/Neptune-Playground/src/hash/unodb/build

@laurynas-biveinis
Copy link
Owner

How does the CMake output look like?

@thompsonbry
Copy link
Author

updated the comment above with the cmake output.

@laurynas-biveinis
Copy link
Owner

This is not that branch - git checkout -b optional-boost-stacktrace created a new local branch with this name

@thompsonbry
Copy link
Author

doh. ok. let me fix that :-).

@thompsonbry
Copy link
Author

Sorry. Multi-tasking. Here is the complete output using a new checkout. Still fails on -lbacktrace


(24-11-22 19:00:40) <0> [~/p8/src/Neptune-Playground/src/hash]
dev-dsk-bryant-2b-a6dd7191 % rm -rf unodb

(24-11-22 19:00:46) <0> [~/p8/src/Neptune-Playground/src/hash]
dev-dsk-bryant-2b-a6dd7191 % git clone https://github.com/laurynas-biveinis/unodb.git

Cloning into 'unodb'...
remote: Enumerating objects: 7606, done.
remote: Counting objects: 100% (1123/1123), done.
remote: Compressing objects: 100% (469/469), done.
remote: Total 7606 (delta 671), reused 1067 (delta 635), pack-reused 6483 (from 1)
Receiving objects: 100% (7606/7606), 3.69 MiB | 23.76 MiB/s, done.
Resolving deltas: 100% (4871/4871), done.

(24-11-22 19:00:58) <0> [~/p8/src/Neptune-Playground/src/hash]
dev-dsk-bryant-2b-a6dd7191 % git checkout optional-boost-stacktrace

(24-11-22 19:02:01) <130> [~/p8/src/Neptune-Playground/src/hash]
dev-dsk-bryant-2b-a6dd7191 % cd unodb

(24-11-22 19:02:05) <0> [~/p8/src/Neptune-Playground/src/hash/unodb]
dev-dsk-bryant-2b-a6dd7191 % git checkout optional-boost-stacktrace
branch 'optional-boost-stacktrace' set up to track 'origin/optional-boost-stacktrace'.
Switched to a new branch 'optional-boost-stacktrace'

(24-11-22 19:02:13) <0> [~/p8/src/Neptune-Playground/src/hash/unodb]
dev-dsk-bryant-2b-a6dd7191 % git submodule update --init

Submodule '3rd_party/GSL' (https://github.com/microsoft/GSL.git) registered for path '3rd_party/GSL'
Submodule '3rd_party/benchmark' (https://github.com/google/benchmark.git) registered for path '3rd_party/benchmark'
Submodule '3rd_party/deepstate' (https://github.com/trailofbits/deepstate.git) registered for path '3rd_party/deepstate'
Submodule '3rd_party/googletest' (https://github.com/google/googletest.git) registered for path '3rd_party/googletest'
Cloning into '/local/home/bryant/p8/src/Neptune-Playground/src/hash/unodb/3rd_party/GSL'...
Cloning into '/local/home/bryant/p8/src/Neptune-Playground/src/hash/unodb/3rd_party/benchmark'...
Cloning into '/local/home/bryant/p8/src/Neptune-Playground/src/hash/unodb/3rd_party/deepstate'...
Cloning into '/local/home/bryant/p8/src/Neptune-Playground/src/hash/unodb/3rd_party/googletest'...
Submodule path '3rd_party/GSL': checked out '82ba6c77ceb70cf92add22ca9ecaf0fc7cac99fa'
Submodule path '3rd_party/benchmark': checked out '12235e24652fc7f809373e7c11a5f73c5763fc4c'
Submodule path '3rd_party/deepstate': checked out 'b0b42eb1d49f9e302dbc49e4b4effbe23568e5cd'
Submodule path '3rd_party/googletest': checked out 'b514bdc898e2951020cbdca1304b75f5950d1f59'

(24-11-22 19:02:28) <0> [~/p8/src/Neptune-Playground/src/hash/unodb]
dev-dsk-bryant-2b-a6dd7191 % export CC=/usr/bin/gcc10-cc
export CXX=/usr/bin/gcc10-c++

(24-11-22 19:02:41) <0> [~/p8/src/Neptune-Playground/src/hash/unodb]
dev-dsk-bryant-2b-a6dd7191 % mkdir build

(24-11-22 19:03:36) <0> [~/p8/src/Neptune-Playground/src/hash/unodb]
dev-dsk-bryant-2b-a6dd7191 % cd build

(24-11-22 19:03:38) <0> [~/p8/src/Neptune-Playground/src/hash/unodb/build]
dev-dsk-bryant-2b-a6dd7191 % cmake -DSTANDALONE=ON ..

-- The CXX compiler identification is GNU 10.5.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/gcc10-c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Warning diagnostics are not fatal
-- Using AVX2 instructions on x86_64
-- Code coverage reporting not enabled
-- Enabling IPO/LTO for release config
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE
-- Found Boost: /usr/local/lib/cmake/Boost-1.85.0/BoostConfig.cmake (found version "1.85.0")
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/gcc10-cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Enabling LTO for Google Benchmark
-- Failed to find LLVM FileCheck
-- Found Git: /usr/bin/git (found version "2.40.1")
-- Google Benchmark version: v1.9.0, normalized to 1.9.0
-- Looking for shm_open in rt
-- Looking for shm_open in rt - found
-- Performing Test HAVE_CXX_FLAG_WALL
-- Performing Test HAVE_CXX_FLAG_WALL - Success
-- Performing Test HAVE_CXX_FLAG_WEXTRA
-- Performing Test HAVE_CXX_FLAG_WEXTRA - Success
-- Performing Test HAVE_CXX_FLAG_WSHADOW
-- Performing Test HAVE_CXX_FLAG_WSHADOW - Success
-- Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL
-- Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL - Success
-- Performing Test HAVE_CXX_FLAG_WOLD_STYLE_CAST
-- Performing Test HAVE_CXX_FLAG_WOLD_STYLE_CAST - Success
-- Performing Test HAVE_CXX_FLAG_WCONVERSION
-- Performing Test HAVE_CXX_FLAG_WCONVERSION - Success
-- Performing Test HAVE_CXX_FLAG_WERROR
-- Performing Test HAVE_CXX_FLAG_WERROR - Success
-- Performing Test HAVE_CXX_FLAG_WSUGGEST_OVERRIDE
-- Performing Test HAVE_CXX_FLAG_WSUGGEST_OVERRIDE - Success
-- Performing Test HAVE_CXX_FLAG_PEDANTIC
-- Performing Test HAVE_CXX_FLAG_PEDANTIC - Success
-- Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS
-- Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS - Success
-- Performing Test HAVE_CXX_FLAG_WSHORTEN_64_TO_32
-- Performing Test HAVE_CXX_FLAG_WSHORTEN_64_TO_32 - Failed
-- Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING
-- Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING - Success
-- Performing Test HAVE_CXX_FLAG_WNO_DEPRECATED_DECLARATIONS
-- Performing Test HAVE_CXX_FLAG_WNO_DEPRECATED_DECLARATIONS - Success
-- Performing Test HAVE_CXX_FLAG_WNO_DEPRECATED
-- Performing Test HAVE_CXX_FLAG_WNO_DEPRECATED - Success
-- Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING
-- Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING - Success
-- Performing Test HAVE_CXX_FLAG_WD654
-- Performing Test HAVE_CXX_FLAG_WD654 - Failed
-- Performing Test HAVE_CXX_FLAG_WTHREAD_SAFETY
-- Performing Test HAVE_CXX_FLAG_WTHREAD_SAFETY - Failed
-- Performing Test HAVE_CXX_FLAG_FLTO
-- Performing Test HAVE_CXX_FLAG_FLTO - Success
-- Performing Test HAVE_CXX_FLAG_WNO_LTO_TYPE_MISMATCH
-- Performing Test HAVE_CXX_FLAG_WNO_LTO_TYPE_MISMATCH - Success
-- Performing Test HAVE_CXX_FLAG_COVERAGE
-- Performing Test HAVE_CXX_FLAG_COVERAGE - Success
-- Compiling and running to test HAVE_STD_REGEX
-- Performing Test HAVE_STD_REGEX -- success
-- Compiling and running to test HAVE_GNU_POSIX_REGEX
-- Performing Test HAVE_GNU_POSIX_REGEX -- failed to compile
-- Compiling and running to test HAVE_POSIX_REGEX
-- Performing Test HAVE_POSIX_REGEX -- success
-- Compiling and running to test HAVE_STEADY_CLOCK
-- Performing Test HAVE_STEADY_CLOCK -- success
-- Compiling and running to test HAVE_PTHREAD_AFFINITY
-- Performing Test HAVE_PTHREAD_AFFINITY -- failed to compile
-- Not using clang-tidy due to non-clang compiler being used
-- cppcheck not found
-- cpplint not found
-- User-set CMake options:
-- STANDALONE: ON
-- MAINTAINER_MODE: OFF
-- AVX2: ON
-- COVERAGE: OFF
-- GCOV_PATH:
-- SANITIZE_ADDRESS: OFF
-- SANITIZE_THREAD: OFF
-- SANITIZE_UB: OFF
-- STATIC_ANALYSIS: OFF
-- CPPCHECK_AGGRESSIVE: OFF
-- IWYU: OFF
-- Configuring done
-- Generating done
-- Build files have been written to: /home/bryant/p8/src/Neptune-Playground/src/hash/unodb/build

(24-11-22 19:03:50) <0> [~/p8/src/Neptune-Playground/src/hash/unodb/build]
dev-dsk-bryant-2b-a6dd7191 % make

[  1%] Building CXX object CMakeFiles/unodb_qsbr.dir/qsbr.cpp.o
[  2%] Building CXX object CMakeFiles/unodb_qsbr.dir/qsbr_ptr.cpp.o
[  4%] Linking CXX static library libunodb_qsbr.a
[  4%] Built target unodb_qsbr
[  5%] Building CXX object CMakeFiles/unodb.dir/art.cpp.o
[  6%] Building CXX object CMakeFiles/unodb.dir/art_common.cpp.o
[  8%] Building CXX object CMakeFiles/unodb.dir/olc_art.cpp.o
[  9%] Building CXX object CMakeFiles/unodb.dir/art_internal.cpp.o
[ 10%] Linking CXX static library libunodb.a
[ 10%] Built target unodb
[ 12%] Building CXX object CMakeFiles/unodb_test.dir/test_heap.cpp.o
[ 13%] Linking CXX static library libunodb_test.a
[ 13%] Built target unodb_test
[ 14%] Building CXX object 3rd_party/googletest/googletest/CMakeFiles/gtest.dir/src/gtest-all.cc.o
[ 16%] Linking CXX static library ../../../lib/libgtest.a
[ 16%] Built target gtest
[ 17%] Building CXX object 3rd_party/googletest/googlemock/CMakeFiles/gmock.dir/src/gmock-all.cc.o
[ 18%] Linking CXX static library ../../../lib/libgmock.a
[ 18%] Built target gmock
[ 20%] Building CXX object 3rd_party/googletest/googlemock/CMakeFiles/gmock_main.dir/src/gmock_main.cc.o
[ 21%] Linking CXX static library ../../../lib/libgmock_main.a
[ 21%] Built target gmock_main
[ 22%] Building CXX object 3rd_party/googletest/googletest/CMakeFiles/gtest_main.dir/src/gtest_main.cc.o
[ 24%] Linking CXX static library ../../../lib/libgtest_main.a
[ 24%] Built target gtest_main
[ 25%] Building CXX object 3rd_party/benchmark/src/CMakeFiles/benchmark.dir/benchmark.cc.o
[ 26%] Building CXX object 3rd_party/benchmark/src/CMakeFiles/benchmark.dir/benchmark_api_internal.cc.o
[ 28%] Building CXX object 3rd_party/benchmark/src/CMakeFiles/benchmark.dir/benchmark_name.cc.o
[ 29%] Building CXX object 3rd_party/benchmark/src/CMakeFiles/benchmark.dir/benchmark_register.cc.o
[ 30%] Building CXX object 3rd_party/benchmark/src/CMakeFiles/benchmark.dir/benchmark_runner.cc.o
[ 32%] Building CXX object 3rd_party/benchmark/src/CMakeFiles/benchmark.dir/check.cc.o
[ 33%] Building CXX object 3rd_party/benchmark/src/CMakeFiles/benchmark.dir/colorprint.cc.o
[ 34%] Building CXX object 3rd_party/benchmark/src/CMakeFiles/benchmark.dir/commandlineflags.cc.o
[ 36%] Building CXX object 3rd_party/benchmark/src/CMakeFiles/benchmark.dir/complexity.cc.o
[ 37%] Building CXX object 3rd_party/benchmark/src/CMakeFiles/benchmark.dir/console_reporter.cc.o
[ 38%] Building CXX object 3rd_party/benchmark/src/CMakeFiles/benchmark.dir/counter.cc.o
[ 40%] Building CXX object 3rd_party/benchmark/src/CMakeFiles/benchmark.dir/csv_reporter.cc.o
[ 41%] Building CXX object 3rd_party/benchmark/src/CMakeFiles/benchmark.dir/json_reporter.cc.o
[ 42%] Building CXX object 3rd_party/benchmark/src/CMakeFiles/benchmark.dir/perf_counters.cc.o
[ 44%] Building CXX object 3rd_party/benchmark/src/CMakeFiles/benchmark.dir/reporter.cc.o
[ 45%] Building CXX object 3rd_party/benchmark/src/CMakeFiles/benchmark.dir/statistics.cc.o
[ 46%] Building CXX object 3rd_party/benchmark/src/CMakeFiles/benchmark.dir/string_util.cc.o
[ 48%] Building CXX object 3rd_party/benchmark/src/CMakeFiles/benchmark.dir/sysinfo.cc.o
[ 49%] Building CXX object 3rd_party/benchmark/src/CMakeFiles/benchmark.dir/timers.cc.o
[ 50%] Linking CXX static library libbenchmark.a
[ 50%] Built target benchmark
[ 52%] Building CXX object 3rd_party/benchmark/src/CMakeFiles/benchmark_main.dir/benchmark_main.cc.o
[ 53%] Linking CXX static library libbenchmark_main.a
[ 53%] Built target benchmark_main
[ 54%] Building CXX object benchmark/CMakeFiles/micro_benchmark_utils.dir/micro_benchmark_utils.cpp.o
[ 56%] Linking CXX static library libmicro_benchmark_utils.a
[ 56%] Built target micro_benchmark_utils
[ 57%] Building CXX object benchmark/CMakeFiles/micro_benchmark_key_prefix.dir/micro_benchmark_key_prefix.cpp.o
[ 58%] Linking CXX executable micro_benchmark_key_prefix
/usr/bin/gcc10-ld: error: cannot find -lbacktrace
/usr/bin/gcc10-ld: error: cannot find -lbacktrace
lto-wrapper: warning: using serial compilation of 2 LTRANS jobs
/usr/local/include/boost/stacktrace/detail/libbacktrace_impls.hpp:104: error: undefined reference to 'backtrace_create_state'
/usr/local/include/boost/stacktrace/detail/libbacktrace_impls.hpp:127: error: undefined reference to 'backtrace_pcinfo'
/usr/local/include/boost/stacktrace/detail/libbacktrace_impls.hpp:135: error: undefined reference to 'backtrace_syminfo'
collect2: error: ld returned 1 exit status
make[2]: *** [benchmark/micro_benchmark_key_prefix] Error 1
make[1]: *** [benchmark/CMakeFiles/micro_benchmark_key_prefix.dir/all] Error 2
make: *** [all] Error 2

(24-11-22 19:04:48) <2> [~/p8/src/Neptune-Playground/src/hash/unodb/build]
dev-dsk-bryant-2b-a6dd7191 % git status
On branch optional-boost-stacktrace
Your branch is up to date with 'origin/optional-boost-stacktrace'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   ../CMakeLists.txt

no changes added to commit (use "git add" and/or "git commit -a")

dev-dsk-bryant-2b-a6dd7191 % git diff
diff --git a/CMakeLists.txt b/CMakeLists.txt
index cf0bae2..8bf5d8b 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -377,8 +377,9 @@ RESTORE_CXX_FLAGS_FOR_SUBDIR()
 # - on ARM
 # - with GCC under macOS due to https://github.com/trailofbits/deepstate/issues/374
 # - under macOS with ASan or TSan enabled
-if(NOT (is_darwin AND (is_gxx OR SANITIZE_ADDRESS OR SANITIZE_THREAD))
-    AND NOT is_windows AND NOT is_arm)
+#if(NOT (is_darwin AND (is_gxx OR SANITIZE_ADDRESS OR SANITIZE_THREAD))
+#    AND NOT is_windows AND NOT is_arm)
+if(FALSE)
   # ThreadSanitizer is not compatible with libfuzzer and LLVM 11 linker crashes
   # on libfuzzer release build
   if(is_clang AND NOT(is_release) AND NOT(SANITIZE_THREAD))

@laurynas-biveinis
Copy link
Owner

Something's not right, the CMake output does not have the added Boost.Stacktrace line. What does git log -1 --oneline say?

In any case I have also merged that branch to master, you might want try to pull and retry

@thompsonbry
Copy link
Author

git log -1 --oneline
6a80242 (HEAD -> optional-boost-stacktrace, origin/optional-boost-stacktrace) Make Boost.Stacktrace optional

@laurynas-biveinis
Copy link
Owner

I have no explanation. The commit is correct, but the CMake output does not match that commit - there should be one of the two added lines in the output, but there aren't:

unodb/CMakeLists.txt

Lines 304 to 308 in 3045f0a

if(HAS_BOOST_STACKTRACE_HPP)
message(STATUS "boost/stacktrace.hpp found, using Boost.Stacktrace")
else()
message(STATUS "boost/stacktrace.hpp not found, not using Boost.Stacktrace")
endif()

How does the updated master branch look now?

@thompsonbry
Copy link
Author

Pulled from master. rm -rf build/*

Same outcome (build fail for -lbacktrace).

I do see one of those messages in the output

-- boost/stacktrace.hpp found, using Boost.Stacktrace

Maybe we are looking for the wrong header file? Or maybe we should look for the library rather than the header file?

@thompsonbry
Copy link
Author

Trying this hack

set(HAS_BOOST_STACKTRACE_HPP,false)
#find_file(HAS_BOOST_STACKTRACE_HPP boost/stacktrace.hpp
#  PATHS "${Boost_INCLUDE_DIRS}")

With this, the build proceeds.

@thompsonbry
Copy link
Author

and the build completes!

[ 96%] Building CXX object test/CMakeFiles/test_art_oom.dir/test_art_oom.cpp.o
[ 97%] Linking CXX executable test_art_oom
[ 97%] Built target test_art_oom
[ 98%] Building CXX object test/CMakeFiles/test_qsbr_oom.dir/test_qsbr_oom.cpp.o
[100%] Linking CXX executable test_qsbr_oom
[100%] Built target test_qsbr_oom

So the issue is what we are "finding" to determine that the backtrace library is not available.

By the way, here is what I did to install boost.

wget boost-from-url
tar xvfz boost-tarball

See https://www.boost.org/doc/libs/1_86_0/more/getting_started/unix-variants.html#prepare-to-use-a-boost-library-binary

cd into boost download package

./bootstrap.sh

(sudo) ./b2 install

Not the exact commands, but how I did it.

@laurynas-biveinis
Copy link
Owner

Ah, I see now. The check I added handles the case of installed AL2 Boost package not having Boost.Stacktrace but not custom-built Boost. If you don't need it for other things, just replace your build with boost-devel package.

Time permitting, I will see if I can work out correct link libs for AL2 with custom-built Boost

@thompsonbry
Copy link
Author

Thank you VERY MUCH for your help with the build process! I will follow up on the thread-safe scan in December.

@thompsonbry
Copy link
Author

I aggressively blew away my boost 1.8.5 install in /usr/local/{lib,header} and installed boost via yum as

sudo yum -y install boost-devel

I now have a build which works with only the single override to disable deepstate in CMakeLists.txt

git diff
diff --git a/CMakeLists.txt b/CMakeLists.txt
index cf0bae2..8bf5d8b 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -377,8 +377,9 @@ RESTORE_CXX_FLAGS_FOR_SUBDIR()
 # - on ARM
 # - with GCC under macOS due to https://github.com/trailofbits/deepstate/issues/374
 # - under macOS with ASan or TSan enabled
-if(NOT (is_darwin AND (is_gxx OR SANITIZE_ADDRESS OR SANITIZE_THREAD))
-    AND NOT is_windows AND NOT is_arm)
+#if(NOT (is_darwin AND (is_gxx OR SANITIZE_ADDRESS OR SANITIZE_THREAD))
+#    AND NOT is_windows AND NOT is_arm)
+if(FALSE)
   # ThreadSanitizer is not compatible with libfuzzer and LLVM 11 linker crashes
   # on libfuzzer release build
   if(is_clang AND NOT(is_release) AND NOT(SANITIZE_THREAD))

@thompsonbry
Copy link
Author

make test passes

make test
Running tests...
Test project /home/bryant/p8/src/Neptune-Playground/src/hash/unodb/build
    Start 1: test_qsbr_ptr
1/6 Test #1: test_qsbr_ptr ....................   Passed    0.00 sec
    Start 2: test_qsbr
2/6 Test #2: test_qsbr ........................   Passed    0.01 sec
    Start 3: test_art
3/6 Test #3: test_art .........................   Passed    0.12 sec
    Start 4: test_art_concurrency
4/6 Test #4: test_art_concurrency .............   Passed    0.43 sec
    Start 5: test_art_oom
5/6 Test #5: test_art_oom .....................   Passed    0.06 sec
    Start 6: test_qsbr_oom
6/6 Test #6: test_qsbr_oom ....................   Passed    0.00 sec

100% tests passed, 0 tests failed out of 6

Total Test time (real) =   0.63 sec

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants