Skip to content

Ship the XNNPACK delegate as its own library - #21526

Closed
shoumikhin wants to merge 92 commits into
gh/shoumikhin/89/headfrom
gh/shoumikhin/81/head
Closed

Ship the XNNPACK delegate as its own library#21526
shoumikhin wants to merge 92 commits into
gh/shoumikhin/89/headfrom
gh/shoumikhin/81/head

Conversation

@shoumikhin

@shoumikhin shoumikhin commented Jul 31, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

The XNNPACK delegate is what makes many models run fast on CPU. Today
it is compiled into whichever component links it, so the wheel carries two copies and a C++
application cannot use it at all without building from source.

before:
  _portable_lib.so     contains its own copy of the XNNPACK delegate
  training extension   contains another copy
  your C++ app         cannot use the delegate without building from source

after:
  executorch/lib/libexecutorch_xnnpack_backend.so.1     one delegate
    both extensions and your app link the same one

How you use it

find_package(executorch REQUIRED)
target_link_libraries(my_app PRIVATE executorch::runtime executorch::xnnpack_backend)

One detail worth knowing: a delegate registers itself when its library loads, and nothing in
your program calls into it directly. A normal link would therefore drop it as unused, so the
target carries the linker options that keep it. You do not have to do anything for that to
work, but it is why linking the target is enough and no explicit registration call is needed.

Tested

Built the wheel on Linux x86_64 and aarch64, installed it into a clean environment, and
confirmed by symbol inspection that exactly one shipped library defines the delegate. Linked a
C++ application against the component, confirmed the delegate survives the link and registers
itself, and ran a delegated model. Existing Python tests pass unchanged.

@shoumikhin
Update
f932b9c 
[ghstack-poisoned]
@shoumikhin

shoumikhin commented Jul 31, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Stack from ghstack (oldest at bottom):

@pytorch-bot

pytorch-bot Bot commented Jul 31, 2026 •
edited
Loading

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21526

Note: Links to docs will display an error until the docs builds have been completed.

❌ 189 New Failures, 1 Cancelled Job, 1 Unrelated Failure, 49 Unclassified Failures

As of commit 637d5d3 with merge base efd6b55 (image):

NEW FAILURES - The following jobs have failed:

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

CANCELLED JOB - The following job was cancelled. Please retry:

FLAKY - The following job failed but was likely due to flakiness present on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 31, 2026
This was referenced Jul 31, 2026
Closed
Closed
Closed
This was referenced Jul 31, 2026
Closed
Closed
@shoumikhin
Update
a231092 
[ghstack-poisoned]
shoumikhin added a commit that referenced this pull request Jul 31, 2026
@shoumikhin
Ship the XNNPACK delegate as its own library
a65f869 
The XNNPACK delegate runs many models on CPU, but it is compiled into whichever
component links it. The wheel ships two copies today, one inside the Python
bindings extension and another inside the training extension, and a C++
application cannot get the delegate at all without building it from source.

Build the delegate as a shared library and ship it in the wheel, so a process
has one copy and both the Python bindings and a C++ application can link the
same one.

XNNPACK and its microkernels are built as static libraries, so they are bundled
inside this library rather than left for each consumer to supply. The runtime
and the thread pool are resolved from their shared libraries instead of
embedding another copy of either. Because the delegate now carries XNNPACK
itself, the places that used to name those static libraries explicitly no
longer do so when building shared, which would otherwise ship the same code
twice.

Registration still happens through a static initializer, and the delegate is
retained on the link line so that initializer runs even though no symbol from
it is referenced directly.

This is gated on the existing `EXECUTORCH_BUILD_SHARED` option, so only the
Linux wheel changes; iOS, Android, and embedded builds keep linking the static
library exactly as before.

Test plan:

The wheel smoke test now asserts that exactly one shipped library defines the
XNNPACK delegate, alongside the existing backend-registry, thread-pool, and CPU
kernel assertions. The symbol it checks was confirmed to exist in the shipped
libraries first, and the assertion was confirmed to fail against a wheel built
before this change, where it correctly reports both copies by name.

Built the wheel from a clean checkout and verified against a fresh virtual
environment with a normal dependency-resolving install:

- The wheel ships the delegate as its own versioned library next to the
  runtime, the thread pool, and the CPU kernels.
- `nm -DC` across every shipped shared object shows exactly one definition of a
  representative delegate symbol, in the new library rather than in the two
  extensions that used to carry it.
- The delegate still appears in the registered backend list at runtime, which
  confirms its static initializer still runs from the shared library.
- `.pte` execution through the Python bindings is unchanged, with outputs
  matching eager PyTorch.
- The backend-registry, thread-pool, and CPU kernel assertions all still hold
  with the new library loaded.
- With `EXECUTORCH_BUILD_SHARED` off, the delegate remains a static library and
  no new shared object is produced, so every build that does not opt in is
  unaffected.

ghstack-source-id: f241eb6
ghstack-comment-id: 5147561128
Pull-Request: #21526
@shoumikhin
Update
4b44a20 
[ghstack-poisoned]
@shoumikhin
Update
0fcd4f5 
[ghstack-poisoned]
shoumikhin added 25 commits August 4, 2026 13:20
@shoumikhin
Update
de92937 
[ghstack-poisoned]
@shoumikhin
Update
b54b04b 
[ghstack-poisoned]
@shoumikhin
Update
289cf00 
[ghstack-poisoned]
@shoumikhin
Update
d7ecfa9 
[ghstack-poisoned]
@shoumikhin
Update
6eb7c6d 
[ghstack-poisoned]
@shoumikhin
Update
662bef7 
[ghstack-poisoned]
@shoumikhin
Update
18f9448 
[ghstack-poisoned]
@shoumikhin
Update
bba6d85 
[ghstack-poisoned]
@shoumikhin
Update
c489bc4 
[ghstack-poisoned]
@shoumikhin
Update
8c892d8 
[ghstack-poisoned]
@shoumikhin
Update
cd98316 
[ghstack-poisoned]
@shoumikhin
Update
74bcef4 
[ghstack-poisoned]
@shoumikhin
Update
9fb9a46 
[ghstack-poisoned]
@shoumikhin
Update
f1e842a 
[ghstack-poisoned]
@shoumikhin
Update
e9198fe 
[ghstack-poisoned]
@shoumikhin
Update
00509a8 
[ghstack-poisoned]
@shoumikhin
Update
4e029cd 
[ghstack-poisoned]
@shoumikhin
Update
873a60a 
[ghstack-poisoned]
@shoumikhin
Update
1152545 
[ghstack-poisoned]
@shoumikhin
Update
e92e3d1 
[ghstack-poisoned]
@shoumikhin
Update
bf30292 
[ghstack-poisoned]
@shoumikhin
Update
c8bd5fd 
[ghstack-poisoned]
@shoumikhin
Update
20e3424 
[ghstack-poisoned]
@shoumikhin
Update
19ea4cf 
[ghstack-poisoned]
@shoumikhin
Update
aa1a2d8 
[ghstack-poisoned]
@shoumikhin shoumikhin mentioned this pull request Aug 5, 2026
Closed
@shoumikhin
Update
476163d 
[ghstack-poisoned]
@shoumikhin shoumikhin mentioned this pull request Aug 5, 2026
Closed
@shoumikhin
Update
637d5d3 
[ghstack-poisoned]
@shoumikhin

shoumikhin commented Aug 8, 2026

Copy link
Copy Markdown
Contributor Author

Closing this in favour of the reworked version.

This was a prototype that proved the approach. The work continues in a rewritten series, which covers
everything here plus the parts this one was missing. Nothing is lost, and two ideas from this prototype
were carried across directly:

  • the table of GPU architectures per release row, so a wheel carries device code for the hardware its row
    claims rather than for whichever machine happened to build it
  • publishing the architecture list through the variable PyTorch's own build reads. The obvious variable is
    rejected and overridden, which silently reduces the build to a single detected architecture. A wheel
    built that way installs everywhere the row claims and then fails at the first kernel launch on any other
    GPU generation, so this detail was the most valuable thing here.

What the reworked series adds:

  • a quantized kernel library and a profiler library, so a C++ application can link them instead of
    discovering at run time that an operator is missing
  • one ownership check driven by a table of components rather than one function per component, so adding a
    component adds a row and cannot be forgotten
  • asking for a component a wheel does not carry fails while CMake is still configuring, naming the
    component, instead of failing later at link time
  • a check that reads the built libraries and confirms the device code matches what the row promised
  • the documented C++ example compiled by a test, so it cannot drift from the code

Thanks to everyone who looked at this one. The review comments here shaped the rewrite.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@larryliu0820 larryliu0820 Awaiting requested review from larryliu0820
@kirklandsign kirklandsign Awaiting requested review from kirklandsign
@mergennachin mergennachin Awaiting requested review from mergennachin

Assignees

No one assigned

Labels

ciflow/binaries/all Release PRs with this label will build wheels for all python versions ciflow/binaries ciflow/cuda ciflow/nightly ciflow/trunk CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@shoumikhin