Compiler errors building examples with CUDA 11.5

[bryanpeele]

I attempted to build nvbench with the following setup and was faced with compiler errors.

• nvbench commit: a171514 (latest commit to main)
• nvcc: Cuda compilation tools, release 11.5, V11.5.119
• gcc: (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0
• cmake: version 3.30.2 (note: the docs state the minimum version as CMake > 2.23.1, but the CMakeLists.txt is coded to expect 3.23.1)

I configured cmake with -DNVBench_ENABLE_EXAMPLES=ON.

After doing so, my cmake output is:

bryan@bryan-linux-desktop:~/Projects/NVBENCH/nvbench/build$ cmake ..
-- Testing CXX17 Support: TRUE
-- Testing CXX20 Support: TRUE
-- Testing CUDA17 Support: TRUE
-- Testing CUDA20 Support: FALSE
-- Creating symlink from /home/bryan/Projects/NVBENCH/nvbench/compile_commands.json to /home/bryan/Projects/NVBENCH/nvbench/build/compile_commands.json...
-- CPM: Adding package [email protected] (9.1.0)
CMake Warning (dev) at /usr/local/share/cmake-3.30/Modules/FetchContent.cmake:1953 (message):
  Calling FetchContent_Populate(fmt) is deprecated, call
  FetchContent_MakeAvailable(fmt) instead.  Policy CMP0169 can be set to OLD
  to allow FetchContent_Populate(fmt) to be called directly for now, but the
  ability to call it with declared details will be removed completely in a
  future version.
Call Stack (most recent call first):
  build/cmake/CPM_0.38.5.cmake:1004 (FetchContent_Populate)
  build/cmake/CPM_0.38.5.cmake:798 (cpm_fetch_package)
  build/cmake/CPM_0.38.5.cmake:306 (CPMAddPackage)
  build/_deps/rapids-cmake-src/rapids-cmake/cpm/find.cmake:176 (CPMFindPackage)
  build/_deps/rapids-cmake-src/rapids-cmake/cpm/fmt.cmake:66 (rapids_cpm_find)
  cmake/NVBenchDependencies.cmake:10 (rapids_cpm_fmt)
  CMakeLists.txt:57 (include)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Module support is disabled.
-- Version: 9.1.0
-- Build type: Release
-- CXX_STANDARD: 17
-- Required features: cxx_variadic_templates
-- rapids-cmake [fmt]: applied diff no_debug_warnings.diff to fix issue: 'No warnings during debug builds [https://github.com/fmtlib/fmt/issues/3351]'
-- CPM: Adding package [email protected] (3.11.3)
CMake Warning (dev) at /usr/local/share/cmake-3.30/Modules/FetchContent.cmake:1953 (message):
  Calling FetchContent_Populate(nlohmann_json) is deprecated, call
  FetchContent_MakeAvailable(nlohmann_json) instead.  Policy CMP0169 can be
  set to OLD to allow FetchContent_Populate(nlohmann_json) to be called
  directly for now, but the ability to call it with declared details will be
  removed completely in a future version.
Call Stack (most recent call first):
  build/cmake/CPM_0.38.5.cmake:1004 (FetchContent_Populate)
  build/cmake/CPM_0.38.5.cmake:798 (cpm_fetch_package)
  build/cmake/CPM_0.38.5.cmake:306 (CPMAddPackage)
  build/_deps/rapids-cmake-src/rapids-cmake/cpm/find.cmake:176 (CPMFindPackage)
  cmake/NVBenchDependencies.cmake:27 (rapids_cpm_find)
  CMakeLists.txt:57 (include)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- NVBench CUDA architectures: 60-real;70-real;75-real;80-real;86
-- Configuring done (1.5s)
-- Generating done (0.0s)
-- Build files have been written to: /home/bryan/Projects/NVBENCH/nvbench/build

I believe it may be relevant that I see this line near the top (though it's not expressed clearly as a requirement:

-- Testing CUDA20 Support: FALSE

Upon building, I get a lot of variadic tempate errors that seem to stem from attempting to build json_printer.cu:

bryan@bryan-linux-desktop:~/Projects/NVBENCH/nvbench/build$ make -j
[  5%] Building CXX object _deps/fmt-build/CMakeFiles/fmt.dir/src/format.cc.o
[  5%] Building CXX object _deps/fmt-build/CMakeFiles/fmt.dir/src/os.cc.o
[  7%] Generate git revision file for nvbench_git_revision
[  7%] Built target nvbench_git_revision_compute_git_info
[ 10%] Linking CXX static library libfmt.a
[ 10%] Built target fmt
[ 12%] Building CXX object nvbench/CMakeFiles/nvbench.dir/benchmark_manager.cxx.o
[ 15%] Building CXX object nvbench/CMakeFiles/nvbench.dir/benchmark_base.cxx.o
[ 17%] Building CXX object nvbench/CMakeFiles/nvbench.dir/criterion_manager.cxx.o
[ 20%] Building CXX object nvbench/CMakeFiles/nvbench.dir/axes_metadata.cxx.o
[ 22%] Building CXX object nvbench/CMakeFiles/nvbench.dir/axis_base.cxx.o
[ 25%] Building CUDA object nvbench/CMakeFiles/nvbench.dir/blocking_kernel.cu.o
[ 27%] Building CUDA object nvbench/CMakeFiles/nvbench.dir/csv_printer.cu.o
[ 30%] Building CUDA object nvbench/CMakeFiles/nvbench.dir/device_info.cu.o
[ 32%] Building CUDA object nvbench/CMakeFiles/nvbench.dir/cuda_call.cu.o
[ 35%] Building CUDA object nvbench/CMakeFiles/nvbench.dir/device_manager.cu.o
[ 37%] Building CXX object nvbench/CMakeFiles/nvbench.dir/float64_axis.cxx.o
[ 40%] Building CXX object nvbench/CMakeFiles/nvbench.dir/int64_axis.cxx.o
[ 42%] Building CUDA object nvbench/CMakeFiles/nvbench.dir/markdown_printer.cu.o
[ 45%] Building CXX object nvbench/CMakeFiles/nvbench.dir/printer_base.cxx.o
[ 47%] Building CXX object nvbench/CMakeFiles/nvbench.dir/named_values.cxx.o
[ 50%] Building CXX object nvbench/CMakeFiles/nvbench.dir/printer_multiplex.cxx.o
[ 52%] Building CXX object nvbench/CMakeFiles/nvbench.dir/runner.cxx.o
[ 55%] Building CXX object nvbench/CMakeFiles/nvbench.dir/state.cxx.o
[ 57%] Building CUDA object nvbench/CMakeFiles/nvbench.dir/option_parser.cu.o
[ 60%] Building CXX object nvbench/CMakeFiles/nvbench.dir/stopping_criterion.cxx.o
[ 62%] Building CXX object nvbench/CMakeFiles/nvbench.dir/string_axis.cxx.o
[ 65%] Building CXX object nvbench/CMakeFiles/nvbench.dir/type_axis.cxx.o
[ 67%] Building CXX object nvbench/CMakeFiles/nvbench.dir/type_strings.cxx.o
[ 70%] Building CXX object nvbench/CMakeFiles/nvbench.dir/detail/entropy_criterion.cxx.o
[ 72%] Building CUDA object nvbench/CMakeFiles/nvbench.dir/detail/measure_cold.cu.o
[ 75%] Building CUDA object nvbench/CMakeFiles/nvbench.dir/detail/measure_hot.cu.o
[ 77%] Building CXX object nvbench/CMakeFiles/nvbench.dir/detail/stdrel_criterion.cxx.o
[ 80%] Building CXX object nvbench/CMakeFiles/nvbench.dir/detail/state_generator.cxx.o
[ 82%] Building CXX object nvbench/CMakeFiles/nvbench.dir/internal/nvml.cxx.o
[ 85%] Building CUDA object nvbench/CMakeFiles/nvbench.dir/detail/measure_cupti.cu.o
[ 87%] Building CXX object nvbench/CMakeFiles/nvbench.dir/cupti_profiler.cxx.o
[ 90%] Building CUDA object nvbench/CMakeFiles/nvbench.dir/json_printer.cu.o
/usr/include/c++/11/bits/std_function.h:435:145: error: parameter packs not expanded with ‘...’:
  435 |         function(_Functor&& __f)
      |                                                                                                                                                 ^ 
/usr/include/c++/11/bits/std_function.h:435:145: note:         ‘_ArgTypes’
/usr/include/c++/11/bits/std_function.h:530:146: error: parameter packs not expanded with ‘...’:
  530 |         operator=(_Functor&& __f)
      |                                                                                                                                                  ^ 
/usr/include/c++/11/bits/std_function.h:530:146: note:         ‘_ArgTypes’
cc1plus: note: unrecognized command-line option ‘-Wno-gnu-line-marker’ may have been intended to silence earlier diagnostics
make[2]: *** [nvbench/CMakeFiles/nvbench.dir/build.make:420: nvbench/CMakeFiles/nvbench.dir/detail/measure_cold.cu.o] Error 1
make[2]: *** Waiting for unfinished jobs....
/usr/include/c++/11/bits/std_function.h:435:145: error: parameter packs not expanded with ‘...’:
  435 |         function(_Functor&& __f)
      |                                                                                                                                                 ^ 
/usr/include/c++/11/bits/std_function.h:435:145: note:         ‘_ArgTypes’
/usr/include/c++/11/bits/std_function.h:530:146: error: parameter packs not expanded with ‘...’:
  530 |         operator=(_Functor&& __f)
      |                                                                                                                                                  ^ 
/usr/include/c++/11/bits/std_function.h:530:146: note:         ‘_ArgTypes’
/usr/include/c++/11/bits/std_function.h:435:145: error: parameter packs not expanded with ‘...’:
  435 |         function(_Functor&& __f)
      |                                                                                                                                                 ^ 
/usr/include/c++/11/bits/std_function.h:435:145: note:         ‘_ArgTypes’
/usr/include/c++/11/bits/std_function.h:530:146: error: parameter packs not expanded with ‘...’:
  530 |         operator=(_Functor&& __f)
      |                                                                                                                                                  ^ 
/usr/include/c++/11/bits/std_function.h:530:146: note:         ‘_ArgTypes’
cc1plus: note: unrecognized command-line option ‘-Wno-gnu-line-marker’ may have been intended to silence earlier diagnostics
make[2]: *** [nvbench/CMakeFiles/nvbench.dir/build.make:492: nvbench/CMakeFiles/nvbench.dir/detail/measure_cupti.cu.o] Error 1
cc1plus: note: unrecognized command-line option ‘-Wno-gnu-line-marker’ may have been intended to silence earlier diagnostics
make[2]: *** [nvbench/CMakeFiles/nvbench.dir/build.make:435: nvbench/CMakeFiles/nvbench.dir/detail/measure_hot.cu.o] Error 1
/usr/include/c++/11/bits/std_function.h:435:145: error: parameter packs not expanded with ‘...’:
  435 |         function(_Functor&& __f)
      |                                                                                                                                                 ^ 
/usr/include/c++/11/bits/std_function.h:435:145: note:         ‘_ArgTypes’
/usr/include/c++/11/bits/std_function.h:530:146: error: parameter packs not expanded with ‘...’:
  530 |         operator=(_Functor&& __f)
      |                                                                                                                                                  ^ 
/usr/include/c++/11/bits/std_function.h:530:146: note:         ‘_ArgTypes’
cc1plus: note: unrecognized command-line option ‘-Wno-gnu-line-marker’ may have been intended to silence earlier diagnostics
make[2]: *** [nvbench/CMakeFiles/nvbench.dir/build.make:162: nvbench/CMakeFiles/nvbench.dir/csv_printer.cu.o] Error 1
/usr/include/c++/11/bits/std_function.h:435:145: error: parameter packs not expanded with ‘...’:
  435 |         function(_Functor&& __f)
      |                                                                                                                                                 ^ 
/usr/include/c++/11/bits/std_function.h:435:145: note:         ‘_ArgTypes’
/usr/include/c++/11/bits/std_function.h:530:146: error: parameter packs not expanded with ‘...’:
  530 |         operator=(_Functor&& __f)
      |                                                                                                                                                  ^ 
/usr/include/c++/11/bits/std_function.h:530:146: note:         ‘_ArgTypes’
cc1plus: note: unrecognized command-line option ‘-Wno-gnu-line-marker’ may have been intended to silence earlier diagnostics
make[2]: *** [nvbench/CMakeFiles/nvbench.dir/build.make:250: nvbench/CMakeFiles/nvbench.dir/markdown_printer.cu.o] Error 1
/usr/include/c++/11/bits/std_function.h:435:145: error: parameter packs not expanded with ‘...’:
  435 |         function(_Functor&& __f)
      |                                                                                                                                                 ^ 
/usr/include/c++/11/bits/std_function.h:435:145: note:         ‘_ArgTypes’
/usr/include/c++/11/bits/std_function.h:530:146: error: parameter packs not expanded with ‘...’:
  530 |         operator=(_Functor&& __f)
      |                                                                                                                                                  ^ 
/usr/include/c++/11/bits/std_function.h:530:146: note:         ‘_ArgTypes’
cc1plus: note: unrecognized command-line option ‘-Wno-gnu-line-marker’ may have been intended to silence earlier diagnostics
make[2]: *** [nvbench/CMakeFiles/nvbench.dir/build.make:279: nvbench/CMakeFiles/nvbench.dir/option_parser.cu.o] Error 1
/usr/include/c++/11/bits/std_function.h:435:145: error: parameter packs not expanded with ‘...’:
  435 |         function(_Functor&& __f)
      |                                                                                                                                                 ^ 
/usr/include/c++/11/bits/std_function.h:435:145: note:         ‘_ArgTypes’
/usr/include/c++/11/bits/std_function.h:530:146: error: parameter packs not expanded with ‘...’:
  530 |         operator=(_Functor&& __f)
      |                                                                                                                                                  ^ 
/usr/include/c++/11/bits/std_function.h:530:146: note:         ‘_ArgTypes’
cc1plus: note: unrecognized command-line option ‘-Wno-gnu-line-marker’ may have been intended to silence earlier diagnostics
make[2]: *** [nvbench/CMakeFiles/nvbench.dir/build.make:521: nvbench/CMakeFiles/nvbench.dir/json_printer.cu.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:212: nvbench/CMakeFiles/nvbench.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

At this point I switched to CUDA 12.6, specifically: Cuda compilation tools, release 12.6, V12.6.20. With the updated version of CUDA toolkit, I am able to build all of the examples and run the few that I spot checked.

It may be relevant that when I run cmake with the newer nvcc version, the output now includes -- Testing CUDA20 Support: TRUE as opposed to the previous FALSE with nvcc 11.5.

I am not sure if this a bug, or expected compatibility issue with older CUDA versions.