Ship the CUDA delegate in a CUDA-enabled ExecuTorch wheel

## Why this is needed

The previous change ships a default Linux wheel with the core runtime as a
linkable libexecutorch.so, but no GPU delegate. To actually run a model on
an NVIDIA GPU (including a coalesced TensorRT + CUDA .pte), a C++ program
also needs the CUDA delegate as a loadable library.

This follows the way PyTorch distributes CUDA: the default wheel has the CPU
runtime, and a CUDA-specific wheel (built on a CUDA machine / installed from
a CUDA index) adds the GPU binaries. Here the CUDA-enabled ExecuTorch wheel
ships the CUDA delegate shared libraries alongside the same core runtime, so
a C++ consumer can link and load the CUDA backend with no source build.

Today the CUDA backend only exists as a static archive that is baked into
the Python module (_portable_lib.so). There is no standalone loadable
backend a C++ program can use, the way QNN already ships
libqnn_executorch_backend.so. This change adds that.

## What is inside

Added only when the wheel is built with CUDA (Linux):

- executorch/lib/libexecutorch_cuda_backend.so: a loadable CUDA delegate.
  It wraps the existing static CUDA backend so that loading the library runs
  its "CudaBackend" registration and registers into the one process-wide
  runtime in libexecutorch.so. It is libtorch-free.
- executorch/lib/libextension_cuda.so: the CUDA caller-stream extension. It
  is a single shared library on purpose so its per-thread caller-stream state
  has exactly one instance across the whole process (needed so a TensorRT
  delegate and the CUDA backend share one stream).
- executorch/include/executorch/extension/cuda/*.h: the caller-stream headers.
- CMake targets executorch::cuda_backend and executorch::extension_cuda in the
  shipped executorch-config.cmake, defined only when the libraries are present.

The default (CPU) wheel is unchanged: none of these are added unless the
wheel is built with CUDA. Windows and macOS wheels are unchanged.

## How to use it

```bash
pip install executorch   # from a CUDA-enabled index/build
```

```cmake
find_package(executorch CONFIG REQUIRED)

add_executable(my_runner main.cpp)
# Linking the CUDA backend force-loads it so it registers on startup.
target_link_libraries(my_runner PRIVATE
    executorch::runtime
    executorch::cuda_backend)
```

```bash
cmake -S . -B build \
  -DCMAKE_PREFIX_PATH="$(python -c 'import executorch.utils as u; print(u.cmake_prefix_path)')"
cmake --build build
```

At runtime, get_backend_class("CudaBackend") returns the registered backend.

## Test plan

Verified on Linux x86_64 with an NVIDIA H100 and CUDA 12.8:

- Built the wheel with CUDA enabled and confirmed it contains
  libexecutorch_cuda_backend.so and libextension_cuda.so under executorch/lib/,
  the CUDA headers, and the CMake config.
- Installed the wheel into a clean virtual environment and built a standalone
  C++ program via find_package(executorch) linking executorch::cuda_backend.
  The program compiled, linked, ran, and get_backend_class("CudaBackend")
  returned non-null, both when the backend was linked directly and when it was
  dlopen'd.
- Confirmed it runs with no LD_LIBRARY_PATH: the shipped libraries resolve each
  other through an $ORIGIN rpath because they are co-located in executorch/lib/.
- Confirmed the program and the CUDA backend link no libtorch/libc10 (via ldd).
- Confirmed there is a single libextension_cuda.so that both the backend and
  the consumer resolve to (the one-instance caller-stream requirement).

Root-caused and fixed a load-time abort during development: the CUDA backend
static archive already carries a whole-archive link option, so also wrapping it
in an explicit whole-archive linked it twice and ran its static registration
twice, hitting a duplicate-registration check. The fix links it once.

Known follow-ups: a CUDA wheel CI job to exercise this automatically, and
running auditwheel on release so the shipped rpath is limited to $ORIGIN.


ghstack-source-id: 2098e23
ghstack-comment-id: 5124122149
Pull-Request: #21478

Author: shoumikhin

backends/cuda/CMakeLists.txt

@@ -236,3 +236,63 @@ install(
   EXPORT ExecuTorchTargets
   DESTINATION lib
 )
+
+# Loadable CUDA backend shared library for the C++ SDK / pip wheel.
+#
+# aoti_cuda_backend above is a static archive whose static initializer registers
+# the "CudaBackend". Today that archive is only force-linked into
+# _portable_lib.so for Python. To let a standalone C++ program (or a coalesced
+# TensorRT+CUDA .pte) load the CUDA delegate the same way QNN ships
+# libqnn_executorch_backend.so, wrap it in a shared library that whole-archives
+# aoti_cuda_backend so the registration constructor is retained and runs when
+# the .so is loaded. The backend resolves executorch runtime / extension_cuda
+# symbols from the co-shipped libexecutorch.so and libextension_cuda.so at load
+# time, so this stays libtorch-free.
+#
+# Requires EXECUTORCH_BUILD_SHARED: this target links the shared runtime
+# (executorch_shared -> libexecutorch.so), which only exists in a shared build.
+# Gating here (not just in setup.py) keeps ordinary CUDA source/editable builds
+# with the default EXECUTORCH_BUILD_SHARED=OFF working. Not built on MSVC: the
+# wheel C++ SDK is Linux only.
+if(NOT _cuda_is_msvc_toolchain AND EXECUTORCH_BUILD_SHARED)
+  add_library(
+    executorch_cuda_backend SHARED runtime/executorch_cuda_backend_lib.cpp
+  )
+  # All dependencies are PRIVATE: this wrapper exists only to bundle the
+  # whole-archived static backend into one loadable .so. aoti_cuda_backend
+  # already carries a whole-archive INTERFACE link option
+  # (executorch_target_link_options_shared_lib above), so linking it plainly
+  # force-loads its "CudaBackend" registration into THIS .so once. Keeping it
+  # PRIVATE means the static archive is not re-exported to consumers of
+  # executorch_cuda_backend: otherwise a consumer would link the archive again
+  # and run the registration a second time, hitting the duplicate-registration
+  # ET_CHECK. Linking executorch_shared (not static executorch_core) makes the
+  # runtime symbols resolve from the one process-wide libexecutorch.so at load
+  # (DT_NEEDED) rather than embedding a second registry.
+  target_link_libraries(
+    executorch_cuda_backend PRIVATE aoti_cuda_backend executorch_shared
+                                    extension_cuda
+  )
+  # Runtime search path for the SHIPPED library. In the wheel this .so lives in
+  # executorch/lib/ alongside libexecutorch.so and libextension_cuda.so
+  # ($ORIGIN), while its DT_NEEDED libaoti_cuda_shims.so is shipped in
+  # executorch/backends/cuda/ ($ORIGIN/../backends/cuda). CUDA runtime libraries
+  # (libcudart, libcurand) are expected from the environment (system CUDA or the
+  # torch CUDA wheel), so no absolute toolkit path is baked in.
+  #
+  # BUILD_WITH_INSTALL_RPATH makes the built artifact use INSTALL_RPATH directly
+  # and suppresses CMake's automatic absolute build-tree RPATH entries. This is
+  # required because the wheel packages the build-tree .so as-is (it does not
+  # run cmake --install), so without this the shipped .so would carry absolute
+  # /.../pip-out/... paths and fail to relocate to a user machine.
+  set_target_properties(
+    executorch_cuda_backend
+    PROPERTIES BUILD_WITH_INSTALL_RPATH TRUE INSTALL_RPATH
+                                             "$ORIGIN:$ORIGIN/../backends/cuda"
+  )
+  install(
+    TARGETS executorch_cuda_backend
+    EXPORT ExecuTorchTargets
+    DESTINATION lib
+  )
+endif()

backends/cuda/runtime/executorch_cuda_backend_lib.cpp

@@ -0,0 +1,13 @@
+/*
+ * Copyright (c) Meta Platforms, Inc. and affiliates.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of this source tree.
+ */
+
+// This translation unit intentionally contains no symbols. The
+// executorch_cuda_backend shared library exists solely to bundle the
+// whole-archived aoti_cuda_backend static library so its "CudaBackend"
+// registration constructor is retained and runs when the .so is loaded. CMake
+// requires a SHARED target to have at least one source file.

setup.py

@@ -884,6 +884,12 @@ def run(self):
                     "extension/named_data_map/merged_data_map.h",
                     "extension/memory_allocator/malloc_memory_allocator.h",
                     "extension/memory_allocator/memory_allocator_utils.h",
+                    # CUDA caller-stream extension headers. Harmless on a CPU
+                    # wheel (headers only); the matching libextension_cuda.so and
+                    # the executorch::extension_cuda / executorch::cuda_backend
+                    # CMake targets are shipped/defined only in a CUDA wheel.
+                    "extension/cuda/caller_stream.h",
+                    "extension/cuda/export.h",
                 ]
                 if sys.platform == "linux"
                 else []
@@ -1130,6 +1136,11 @@ def run(self):  # noqa C901
             if cmake_cache.is_enabled("EXECUTORCH_BUILD_CUDA"):
                 cmake_build_args += ["--target", "aoti_cuda_backend"]
                 cmake_build_args += ["--target", "aoti_common_shims_slim"]
+                # Loadable CUDA delegate + caller-stream shared libs for the C++
+                # SDK, built only for a wheel that also builds the shared runtime.
+                if cmake_cache.is_enabled("EXECUTORCH_BUILD_SHARED"):
+                    cmake_build_args += ["--target", "executorch_cuda_backend"]
+                    cmake_build_args += ["--target", "extension_cuda"]
 
             if cmake_cache.is_enabled("EXECUTORCH_BUILD_EXTENSION_MODULE"):
                 cmake_build_args += ["--target", "extension_module"]
@@ -1313,6 +1324,45 @@ def run(self):  # noqa C901
                     if sys.platform == "linux"
                     else []
                 ),
+                # CUDA delegate binaries for the C++ SDK, shipped only in a
+                # CUDA-enabled wheel (PyTorch-style: the default wheel has the core
+                # runtime, the CUDA-index wheel adds these). Co-located with
+                # libexecutorch.so in executorch/lib/ so a C++ consumer, or a
+                # coalesced TensorRT+CUDA .pte, can load and register the CUDA
+                # delegate. executorch_cuda_backend whole-archives the backend so
+                # its "CudaBackend" registration runs on load; extension_cuda
+                # carries the single process-wide caller-stream TLS. Both are
+                # unversioned .so, so a plain dynamic-lib copy is enough. Gated on
+                # BUILD_SHARED as well as BUILD_CUDA: executorch_cuda_backend links
+                # the shared runtime and is only defined/built when SHARED is on,
+                # so packaging must require the same to avoid referencing an
+                # unbuilt artifact.
+                *(
+                    [
+                        BuiltFile(
+                            src_dir="%CMAKE_CACHE_DIR%/backends/cuda/",
+                            src_name="executorch_cuda_backend",
+                            dst="executorch/lib/",
+                            is_dynamic_lib=True,
+                            dependent_cmake_flags=[
+                                "EXECUTORCH_BUILD_CUDA",
+                                "EXECUTORCH_BUILD_SHARED",
+                            ],
+                        ),
+                        BuiltFile(
+                            src_dir="%CMAKE_CACHE_DIR%/extension/cuda/",
+                            src_name="extension_cuda",
+                            dst="executorch/lib/",
+                            is_dynamic_lib=True,
+                            dependent_cmake_flags=[
+                                "EXECUTORCH_BUILD_CUDA",
+                                "EXECUTORCH_BUILD_SHARED",
+                            ],
+                        ),
+                    ]
+                    if sys.platform == "linux"
+                    else []
+                ),
             ]
         ),
     ],

tools/cmake/executorch-wheel-config.cmake

@@ -175,4 +175,81 @@ if(_executorch_shared_LIBRARY)
       )
     endif()
   endforeach()
+
+  # CUDA delegate targets. Present only in a CUDA-enabled wheel, so each target
+  # is defined only when its shared library is shipped in executorch/lib/. These
+  # are real separate shared libraries (not aliases of libexecutorch.so):
+  # extension_cuda holds the one process-wide caller-stream TLS, and
+  # cuda_backend whole-archives the CUDA delegate so loading it registers
+  # "CudaBackend" into the runtime.
+  find_library(
+    _executorch_extension_cuda_LIBRARY
+    NAMES extension_cuda
+    PATHS "${_executorch_sdk_libdir}"
+    NO_DEFAULT_PATH
+  )
+  if(_executorch_extension_cuda_LIBRARY AND NOT TARGET
+                                            executorch::extension_cuda
+  )
+    # caller_stream.h includes <cuda_runtime.h> and the real target links
+    # CUDA::cudart PUBLIC, so reproduce that usage requirement. Use a QUIET,
+    # non-REQUIRED lookup: a consumer that only wants executorch::runtime from a
+    # CUDA-built wheel must still be able to find_package(executorch) on a
+    # machine that has the CUDA runtime but no development toolkit. If the
+    # toolkit is absent we simply skip defining the optional CUDA targets rather
+    # than failing the whole package.
+    find_package(CUDAToolkit QUIET)
+    if(CUDAToolkit_FOUND)
+      add_library(executorch::extension_cuda SHARED IMPORTED)
+      set_target_properties(
+        executorch::extension_cuda
+        PROPERTIES IMPORTED_LOCATION "${_executorch_extension_cuda_LIBRARY}"
+                   INTERFACE_INCLUDE_DIRECTORIES "${EXECUTORCH_INCLUDE_DIRS}"
+                   INTERFACE_COMPILE_FEATURES cxx_std_17
+                   INTERFACE_LINK_LIBRARIES CUDA::cudart
+      )
+    endif()
+  endif()
+
+  find_library(
+    _executorch_cuda_backend_LIBRARY
+    NAMES executorch_cuda_backend
+    PATHS "${_executorch_sdk_libdir}"
+    NO_DEFAULT_PATH
+  )
+  if(_executorch_cuda_backend_LIBRARY
+     AND TARGET executorch::extension_cuda
+     AND NOT TARGET executorch::cuda_backend
+  )
+    # Requires executorch::extension_cuda (only defined when the CUDA toolkit
+    # was found above), since the backend links it. If the toolkit is
+    # unavailable the optional CUDA targets are simply not defined.
+    add_library(executorch::cuda_backend SHARED IMPORTED)
+    set_target_properties(
+      executorch::cuda_backend
+      PROPERTIES IMPORTED_LOCATION "${_executorch_cuda_backend_LIBRARY}"
+                 INTERFACE_INCLUDE_DIRECTORIES "${EXECUTORCH_INCLUDE_DIRS}"
+                 INTERFACE_COMPILE_FEATURES cxx_std_17
+    )
+    set_property(
+      TARGET executorch::cuda_backend
+      APPEND
+      PROPERTY INTERFACE_LINK_LIBRARIES executorch::runtime
+               executorch::extension_cuda
+    )
+    # This library is loaded purely for its side effect: its static initializer
+    # registers "CudaBackend". A consumer references no symbol from it, so under
+    # -Wl,--as-needed (or -lexecutorch_cuda_backend) the linker would drop it
+    # from DT_NEEDED and the registration would never run. Force it to stay
+    # linked with --no-as-needed around this one library on ELF toolchains.
+    if(NOT APPLE AND NOT WIN32)
+      set_property(
+        TARGET executorch::cuda_backend
+        APPEND
+        PROPERTY
+          INTERFACE_LINK_OPTIONS
+          "SHELL:-Wl,--no-as-needed,${_executorch_cuda_backend_LIBRARY},--as-needed"
+      )
+    endif()
+  endif()
 endif()