Ship a C++ SDK in the wheel

The wheel ships the runtime, kernels, delegate, thread pool and profiler as separate shared
libraries, but nothing outside Python can use them, because the installed CMake package names
none of them. A C++ application would have to hard-code paths into
the wheel's private layout.

The headers have the same gap. The wheel installs only the subset a custom-operator build needs,
which leaves out `extension/module`, the entry point the documentation tells C++ callers to use. So
the wheel ships the libraries to run a model and no way to call them.

Name each shipped library as a CMake component, so `find_package` locates them, and ship the
headers a caller needs. A component is just a name a consumer can ask for, and CMake reports a
missing one while configuring rather than at link time.

```cmake
find_package(executorch 1.5 REQUIRED COMPONENTS kernels_optimized)
target_link_libraries(my_app PRIVATE executorch::runtime
                                     executorch::kernels_optimized)
```

| component | library it resolves to |
| --- | --- |
| `executorch::runtime` | `libexecutorch.so` |
| `executorch::kernels_optimized` | `libexecutorch_kernels_optimized.so` |
| `executorch::backend_xnnpack` | `libexecutorch_backend_xnnpack.so` |
| `executorch::threadpool` | `libexecutorch_threadpool.so` |
| `executorch::etdump` | `libexecutorch_etdump.so` |

Each component records where the wheel keeps its libraries, so an application built against it
finds them without the caller setting a library search path.

Headers include the module and tensor entry points, the CPU kernel helpers, the allocator and data
loader concrete classes Module's constructors take, the profiler entry points, and the
FlatTensorDataMap and MergedDataMap types plus the .ptd file header a caller writing a .ptd needs.

CMake 3.28 or newer gets these targets. Older versions do not, because they write the `$ORIGIN`
marker (the "look next to me" token in a library search path) incorrectly:

```
3.24.3, 3.27.9   Makefiles double the dollar sign, Ninja drops the name
3.28.4, 3.31.8   both write the token correctly
```

That would produce a target that runs where it was built and fails once the application is copied
elsewhere, so no target is defined below 3.28. Those versions get plain variables instead:
`EXECUTORCH_LIBRARIES` with the runtime and every shipped library by path, plus
`EXECUTORCH_INCLUDE_DIRS`, `EXECUTORCH_COMPILE_DEFINITIONS` and `EXECUTORCH_CXX_STANDARD`. All four
are needed, because an imported target carries the definitions and the C++ standard along with the
library and a plain path carries neither. Linking the libraries alone stops at
`#error "You need C++17 to compile ExecuTorch"`.

`ET_USE_THREADPOOL` is added to `EXECUTORCH_COMPILE_DEFINITIONS` on the pre-3.28 route when the
thread pool library ships. Without it the runtime header supplies a local inline serial fallback
for `parallel_for`, so a consumer following the documented recipe linked the thread pool library
and still ran serial code with no diagnostic.

Built the wheel, installed it into a clean environment, and built a C++ application against the
installed wheel alone:

- the application links the runtime, runs a model, and matches eager PyTorch, and still runs after
  being copied away from the wheel.
- asking for a component the wheel does not ship fails while configuring, naming the component.
- a version request is honoured, including ranges.
- shipped headers can be included on their own, and one entry point per shipped component also
  links against the shipped libraries. A small number are exempt because they need something outside
  the package: a Windows shim, a test framework, or a header that says in its own text not to
  include it directly. The exempt list is compiled too, so an entry that starts working is reported
  rather than left in place.
- the thread pool probe compiles with `ET_USE_THREADPOOL`, on both the modern-CMake route (from the
  runtime target) and the pre-3.28 route (from `EXECUTORCH_COMPILE_DEFINITIONS`). Without it the
  header supplies a local inline definition and the probe linked identically whether or not the
  library was on the link line, so it could not detect the component being dropped. Measured both
  ways.
- an application's runtime search path is recorded as `DT_RUNPATH`, not the older `DT_RPATH`. That
  matters because `DT_RPATH` is searched ahead of `LD_LIBRARY_PATH` and is inherited by
  dependencies, so a consumer could not point a locally built or instrumented runtime at their
  application. Verified by shadowing the runtime through `LD_LIBRARY_PATH` and watching the loader
  pick it up, which `DT_RPATH` ignores.
- on real CMake 3.24 and 3.27, an application configures, builds and runs through the variables.
  Measured what each one contributes, with the consumer pinned to C++14 so its own standard does not
  hide the package's requirement: linking `EXECUTORCH_LIBRARIES` alone fails on a missing header,
  adding the include directories and definitions then fails on the C++ standard, and applying
  `EXECUTORCH_CXX_STANDARD` builds and loads a model. The kernels also need scoped retention there,
  because a registration-only library exports nothing the application references and the linker
  drops it, which showed up as "Missing operator" at run time rather than as a link error. The
  smoke test now runs the same shape automatically when `EXECUTORCH_PRE_328_CMAKE` points at an
  older cmake binary, so a future change on the fallback path fails a check rather than only
  showing up on the first user with older cmake.
- `find_package` succeeds when the interpreter on PATH is not the one the wheel was built for. The
  extension's own file name carries its suffix, so asking a different interpreter for it reported a
  complete install as not found.

Ran on Linux x86_64 and aarch64. The macOS wheel keeps the fused extension and ships no separate
libraries, so these checks do not apply there and its smoke test does not run them.

ghstack-source-id: 490d066
ghstack-comment-id: 5215967468
Pull-Request: #21639

Author: shoumikhin

.ci/scripts/wheel/test_cpp_sdk.py

@@ -11,6 +11,7 @@
 from pathlib import Path
 
 import test_base
+import test_cpp_sdk
 import test_shared_libraries
 from examples.models import Backend, Model
 
@@ -50,6 +51,13 @@
         with tempfile.TemporaryDirectory() as work_dir:
             test_shared_libraries.run_tests(Path(work_dir))
 
+        # And that a C++ application outside the wheel can actually use them.
+        # Nothing above covers this: the Python extension links those libraries
+        # itself, so it passes whether or not the package config names them or the
+        # shipped headers are complete.
+        with tempfile.TemporaryDirectory() as work_dir:
+            test_cpp_sdk.run_tests(Path(work_dir))
+
     test_base.run_tests(
         model_tests=[
             test_base.ModelTest(

.ci/scripts/wheel/test_linux.py

@@ -9,6 +9,7 @@
 from pathlib import Path
 
 import test_base
+import test_cpp_sdk
 import test_shared_libraries
 from examples.models import Backend, Model
 
@@ -36,6 +37,11 @@
     with tempfile.TemporaryDirectory() as work_dir:
         test_shared_libraries.run_tests(Path(work_dir))
 
+    # And that a C++ application outside the wheel can actually use those
+    # libraries, which nothing above covers.
+    with tempfile.TemporaryDirectory() as work_dir:
+        test_cpp_sdk.run_tests(Path(work_dir))
+
     test_base.run_tests(
         model_tests=[
             test_base.ModelTest(

.ci/scripts/wheel/test_linux_aarch64.py

@@ -22,8 +22,8 @@ The prebuilt `executorch.runtime` module included in this package provides a way
 to run ExecuTorch `.pte` files, with some restrictions:
 * Only [core ATen operators](docs/source/ir-ops-set-definition.md) are linked into the prebuilt module
 * Only the [XNNPACK backend delegate](docs/source/backends/xnnpack/xnnpack-overview.md) is linked into the prebuilt module.
-* \[macOS only] [Core ML](docs/source/backends/coreml/coreml-overview.md) and [MPS](docs/source/backends/mps/mps-overview.md) backend
-  are also linked into the prebuilt module.
+* \[macOS only] [Core ML](docs/source/backends/coreml/coreml-overview.md) backend is
+  also linked into the prebuilt module.
 * \[Linux x86_64] [QNN](docs/source/backends-qualcomm.md) backend is linked into the prebuilt module.
 * \[Linux] [OpenVINO](docs/source/build-run-openvino.md) backend is also linked into the
   prebuilt module. OpenVINO requires the runtime to be installed separately:

README-wheel.md

@@ -74,7 +74,6 @@ class ETDumpGen : public ::executorch::runtime::EventTracer {
  public:
   ETDumpGen(::executorch::runtime::Span<uint8_t> buffer = {nullptr, (size_t)0});
   ~ETDumpGen() override;
-  void clear_builder();
 
   void create_event_block(const char* name) override;
   virtual ::executorch::runtime::EventTracerEntry start_profiling(

devtools/etdump/etdump_flatcc.h

@@ -40,6 +40,151 @@ Running a model using the low-level runtime APIs allows for a high-degree of con
 
 ## Building with CMake
 
+There are two ways to get the C++ runtime. Linking the prebuilt libraries from the pip
+package needs no source checkout and is the quicker option. Building from source gives
+you every option the project has, and is what you need for a platform the wheel does not
+cover.
+
+### Using the prebuilt libraries from the pip package
+
+On Linux, `pip install executorch` includes prebuilt shared libraries, the public
+headers, and a CMake package, so a C++ application can link the runtime without building
+ExecuTorch itself:
+
+```cmake
+# CMakeLists.txt
+cmake_minimum_required(VERSION 3.28)
+project(my_app CXX)
+
+find_package(executorch REQUIRED COMPONENTS kernels_optimized)
+
+add_executable(my_app main.cpp)
+target_link_libraries(my_app PRIVATE executorch::runtime
+                                     executorch::kernels_optimized)
+```
+
+Point CMake at the installed package when you configure:
+
+```
+cmake -S . -B build \
+  -DCMAKE_PREFIX_PATH="$(python -c 'import executorch, pathlib; print(pathlib.Path(executorch.__path__[0]) / "share" / "cmake")')"
+cmake --build build
+```
+
+The application uses the same `Module` and `TensorPtr` APIs described above:
+
+```cpp
+// main.cpp
+#include <executorch/extension/module/module.h>
+#include <executorch/extension/tensor/tensor.h>
+
+#include <cstdio>
+#include <vector>
+
+using namespace executorch::extension;
+
+int main() {
+  Module module("model.pte");
+
+  std::vector<float> data(2 * 8, 1.0f);
+  auto input = make_tensor_ptr({2, 8}, data.data());
+
+  const auto result = module.forward(input);
+  if (!result.ok()) {
+    std::printf("forward failed: 0x%x\n", (unsigned)result.error());
+    return 1;
+  }
+  std::printf("ok, %zu outputs\n", result->size());
+  return 0;
+}
+```
+
+#### What each component provides
+
+Ask for the components your model needs. A component the wheel was not built with is
+reported while CMake configures, rather than failing later at link time.
+
+| Component | What it provides |
+| --- | --- |
+| `executorch::runtime` | the program loader and executor. Always present. |
+| `executorch::kernels_optimized` | CPU operator kernels. Needed for any operator a delegate does not claim. |
+| `executorch::backend_xnnpack` | the XNNPACK delegate. |
+| `executorch::threadpool` | the shared thread pool. |
+| `executorch::etdump` | the profiler. |
+
+The runtime on its own loads a program but registers only primitive operators, not the
+kernels a model computes with, so a model that is not fully delegated needs a kernel
+component too. Linking a delegate is what registers it: a program delegated to XNNPACK
+fails to load in an application that did not link `executorch::backend_xnnpack`.
+
+To require a minimum version, pass it to `find_package`:
+
+```cmake
+find_package(executorch 1.0 REQUIRED)
+```
+
+#### On CMake older than 3.28
+
+The example above needs CMake 3.28. Older versions write the `$ORIGIN` marker (the
+"look next to me" token in a library search path) incorrectly, which would leave you with
+a target that runs where it was built and fails once the application is copied
+elsewhere. Rather than hand you a target that behaves that way, the package defines no
+imported targets below 3.28 and exports plain variables instead.
+
+An imported target carries more than a library path, so on this route you have to apply
+the rest yourself. Linking the libraries alone does not compile:
+
+```cmake
+cmake_minimum_required(VERSION 3.19)
+project(my_app CXX)
+
+find_package(executorch REQUIRED)
+
+add_executable(my_app main.cpp)
+target_include_directories(my_app PRIVATE ${EXECUTORCH_INCLUDE_DIRS})
+target_compile_definitions(my_app PRIVATE ${EXECUTORCH_COMPILE_DEFINITIONS})
+target_link_libraries(my_app PRIVATE ${EXECUTORCH_LIBRARIES})
+set_property(TARGET my_app PROPERTY CXX_STANDARD ${EXECUTORCH_CXX_STANDARD})
+set_property(TARGET my_app PROPERTY CXX_STANDARD_REQUIRED ON)
+```
+
+On this route the application also has to record where the libraries live, or it runs
+from its build directory and then fails to start once installed with a message like
+`libexecutorch.so: cannot open shared object file`. CMake records the wheel's library
+directory while building, because the libraries are named by absolute path, but it removes
+that entry on install. Ask for it to be kept:
+
+```cmake
+set_property(TARGET my_app PROPERTY INSTALL_RPATH "${EXECUTORCH_RUNTIME_LIBRARY_DIR}")
+target_link_options(my_app PRIVATE "LINKER:--enable-new-dtags")
+```
+
+The second line matters on Linux. Without it this linker records the older `DT_RPATH` tag, which is
+searched before `LD_LIBRARY_PATH` and also applies to your dependencies' own dependencies, so you
+could not point the application at a different build of the runtime. With it you get `DT_RUNPATH`,
+which only affects your application and stays overridable.
+
+The imported target route does not need this on Linux: the package sets its search paths as
+explicit link options, and those survive installation.
+
+On macOS it does need one line. CMake removes an entry that points at a directory holding a
+library the application linked, so the entry naming the wheel's own directory is deleted from
+the installed binary and it stops finding the runtime:
+
+```cmake
+set_property(TARGET my_app PROPERTY INSTALL_RPATH_USE_LINK_PATH TRUE)
+```
+
+An application deployed beside the libraries is unaffected either way, because the
+`@loader_path` and `$ORIGIN` entries are kept.
+
+`EXECUTORCH_LIBRARIES` names the runtime and every component the wheel shipped, so you
+cannot choose components on this route. Upgrade to CMake 3.28 and link the specific
+targets you need instead.
+
+### Building from source
+
+
 ExecuTorch uses CMake as the primary build system. Inclusion of the module and tensor APIs are controlled by the `EXECUTORCH_BUILD_EXTENSION_MODULE` and `EXECUTORCH_BUILD_EXTENSION_TENSOR` CMake options. As these APIs may not be supported on embedded systems, they are disabled by default when building from source. The low-level API surface is always included. To link, add the `executorch` target as a CMake dependency, along with `executorch_backends`, `executorch_extensions`, and `extension_kernels`, to link all configured backends, extensions, and kernels.
 
 ```

docs/source/using-executorch-cpp.md

@@ -16,12 +16,10 @@
 #include <executorch/runtime/core/result.h>
 #include <executorch/runtime/platform/compiler.h>
 
-using executorch::runtime::Error;
-using executorch::runtime::Result;
 namespace executorch::extension::utils {
 
 // Util to get alighment adjusted allocation size
-inline Result<size_t> get_aligned_size(size_t size, size_t alignment) {
+inline runtime::Result<size_t> get_aligned_size(size_t size, size_t alignment) {
   // The minimum alignment that malloc() is guaranteed to provide.
   static constexpr size_t kMallocAlignment = alignof(std::max_align_t);
   if (alignment > kMallocAlignment) {
@@ -31,7 +29,7 @@ inline Result<size_t> get_aligned_size(size_t size, size_t alignment) {
     const size_t extra = alignment - 1;
     if ET_UNLIKELY (extra >= SIZE_MAX - size) {
       ET_LOG(Error, "Malloc size overflow: size=%zu + extra=%zu", size, extra);
-      return Result<size_t>(Error::InvalidArgument);
+      return runtime::Result<size_t>(runtime::Error::InvalidArgument);
     }
     size += extra;
   }

extension/memory_allocator/memory_allocator_utils.h

@@ -13,6 +13,7 @@
 #include <cstdint>
 
 #include <c10/util/safe_numerics.h>
+#include <executorch/runtime/core/exec_aten/exec_aten.h>
 #include <executorch/runtime/core/memory_allocator.h>
 #include <executorch/runtime/platform/log.h>
 #include <executorch/runtime/platform/platform.h>