Build and publish CUDA wheels by shoumikhin · Pull Request #21668 · pytorch/executorch

shoumikhin · 2026-08-07T17:55:54Z

The problem

The wheel can carry the CUDA delegate, but nothing builds one: there is no CUDA row in any
workflow, so a GPU user still has to build from source.

The change

Add the workflows that build and publish CUDA wheels for Linux x86_64 and aarch64, and a smoke test
that checks each wheel from the artifact itself. The build machines for these rows have no GPU, so
the smoke test does not execute a model; it verifies the CUDA libraries are present, that the
declared runtime matches the wheel's CUDA version, that nothing resolves through the build
machine's toolkit, and that the shipped device code covers every GPU architecture the row claims.

executorch-1.5.0-cp312-cp312-manylinux_2_28_x86_64.whl    +cu130

A release publishes CUDA 12.6, 13.0 and 13.2, for Python 3.10 through 3.13. A pull request builds a
single row instead of all twelve, because a full matrix costs hours for little extra signal.

Which GPU architectures each row compiles for is chosen per row rather than detected on the builder.
Detecting it would produce a wheel carrying device code for whatever machine happened to build it,
which installs fine and then fails at the first GPU call.

Two guards keep a release honest:

if the shared matrix generator stops offering a combination this policy advertises, the step fails
instead of quietly publishing fewer wheels. A missing job is otherwise a green check for a wheel
that was never built.
if a row reaches the architecture list with no CUDA version, the build refuses rather than falling
back to the builder's GPU.

Windows CUDA is deliberately absent. The separate shared libraries this wheel exists to ship are
Linux only today, so a Windows CUDA wheel would carry a delegate a C++ application still could not
link.

Test plan

built the full release matrix, twelve wheels, and confirmed each one's contents match the row it
claims: the CUDA libraries present, the CUDA runtime declared, and device code for every GPU
architecture the row advertises.
ran a GPU model end to end from a CI-built wheel on three NVIDIA GPUs covering three device
architectures, with output identical to eager PyTorch on each (largest absolute difference 0),
and inspected the wheel for a fourth device it cannot execute on.
ran the matrix filter over generated inputs, including incomplete and malformed ones, and
confirmed it refuses rather than publishing a partial release: a missing CUDA version, a missing
python, or a python present on rows this policy does not build are each reported by name.
confirmed a CPU row still produces a CPU wheel on a builder that happens to have a CUDA toolkit
installed.

shoumikhin · 2026-08-07T17:55:55Z

Stack from ghstack (oldest at bottom):

pytorch-bot · 2026-08-07T17:55:58Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21668

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

⏳ No Failures, 319 Pending

As of commit c5978e9 with merge base fb5eedc ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

## Why The wheel now ships a CUDA delegate, but no release builds one. Every wheel workflow says: with-cuda: disabled So a user who wants GPU support still clones the repository and builds from source, which is the thing the shipped libraries were supposed to remove. Nothing publishes them because nothing builds them. ## What this change does Adds two workflows, one per architecture, that build CUDA wheels the same way the CPU ones build the default wheel: build-wheels-cuda-linux.yml x86_64 build-wheels-cuda-aarch64-linux.yml aarch64 Both call the same shared matrix generator the CPU rows already use, with CUDA turned on instead of off, then narrow the result. ## Which rows get built, and why not all of them The generator emits every CUDA version it knows about. Publishing all of them would ship wheels for combinations nothing can verify, and a GPU wheel that installs and then cannot run is worse than one that does not exist: the failure appears when a model runs, and it looks like a model problem rather than a packaging one. So a row is kept only when all three of these hold: | | | | --- | --- | | a GPU exists that the row's device code covers | otherwise the wheel installs and dies at the first kernel launch | | a PyTorch build is published for that CUDA version and architecture | otherwise the dependency cannot be satisfied | | a machine is available to run a model before release | otherwise nothing checks it | That leaves: | architecture | CUDA | Python | | --- | --- | --- | | x86_64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 | | aarch64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 | The CUDA versions are chosen so every accelerator consumer can find a wheel to depend on, rather than by what is convenient to verify here. A delegate built against one CUDA version needs an ExecuTorch wheel for that same version, and a missing version leaves that consumer with nothing to pair with, which fails for whoever installs the pair rather than for the row that omitted it. 13.2 is published for that reason even though no machine on hand can execute it: the packaging properties are checked on every row, and running a model is a release-gate step on hardware with the matching GPU. Python 3.14 is excluded because the current CPU rows already fail on it for an unrelated reason in the example requirements, so a GPU row would inherit a known-broken build. The free-threaded builds are excluded because the CUDA dependencies are not published for them. A pull request builds one representative row rather than the whole matrix, because a full matrix on every push costs hours for little added signal. ## Jetson devices Jetson needs its own row: a JetPack container, one Python version, one CUDA version. It cannot take a generic aarch64 wheel, because the generic builds carry no device code for its GPU architecture and no portable fallback either. That row is present in the filter but deliberately empty. Published PyTorch stopped shipping device code for those GPUs after 2.8.0, so a Jetson row today would produce a wheel whose PyTorch dependency cannot execute on the device. The lists are there to be filled in when that changes. ## What to expect Nothing changes for a CPU user. These are additional rows, and the existing workflows are untouched. | | before | after | | --- | --- | --- | | GPU support from an install | build from source | a published wheel | | CUDA runtime | not shipped | declared as a dependency | The build asks for the delegate explicitly rather than letting the build detect a toolkit. A detected build is fine locally, but a release row states what it is producing, and a row that silently produced a CPU wheel because the toolkit was missing would publish under a CUDA name. The environment script fails early for the same reason: without it, packaging looks for CUDA libraries that were never built and reports a confusing missing-file error minutes later. Test plan: A smoke test for the CUDA rows, checking what an artifact can be held to on a builder that has no GPU: - the CUDA libraries are actually in the wheel, so a row named for CUDA cannot ship without a delegate - the CUDA runtime is declared, so a user has something to resolve it from - each CUDA library reaches its runtime through a relative path, not through the builder's toolkit directory, which resolves on the builder and nowhere else Then everything a CPU wheel is already held to: one owner per component, no build-tree search paths, and a C++ application outside the wheel able to link what it ships. Each CUDA check was run against a wheel that should fail it as well as one that should pass, because a check that cannot fail is worse than no check. All three correctly reject a CPU wheel and all three accept a CUDA wheel built from source, which ships both libraries, declares the runtime, and carries a relative path to it. One of them did not fail on a CPU wheel at first: it looped over libraries that were not there and reported a pass, having inspected nothing. It now requires at least one to be present. The filter was exercised against a matrix shaped like the generator's output: 18 rows narrowed to 8, a pull request narrowed to 1, the aarch64 rows given the newer builder image, and an empty result treated as a failure rather than passed through, since a workflow with no build job reads as a green check for a build that never happened. ghstack-source-id: 10f5862 ghstack-comment-id: 5220374521 Pull-Request: #21668

## The problem The wheel can now ship a CUDA delegate, but no release builds one. Every wheel workflow says: ``` with-cuda: disabled ``` So a user who wants GPU support still clones the repository and builds from source, which is the thing the shipped libraries were supposed to remove. Nothing publishes them because nothing builds them. Anything that wants to depend on an ExecuTorch GPU wheel has nothing to depend on. ## The change Adds two workflows that build CUDA wheels the same way the existing ones build the default wheel, calling the same shared matrix generator with CUDA turned on, then narrowing the result. ``` .github/workflows/build-wheels-cuda-linux.yml x86_64 .github/workflows/build-wheels-cuda-aarch64-linux.yml aarch64 ``` | architecture | CUDA | Python | | --- | --- | --- | | x86_64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 | | aarch64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 | The CUDA versions are chosen so every accelerator consumer can find a wheel to depend on, rather than by what is convenient to verify. A delegate built against one CUDA version needs an ExecuTorch wheel for that same version, and a missing version leaves that consumer with nothing to pair with, which fails for whoever installs the pair rather than for the row that omitted it. Python 3.14 is excluded because the current CPU rows already fail on it for an unrelated reason, so a GPU row would inherit a known-broken build. A pull request builds one representative row rather than the whole matrix. Jetson devices need their own row, with a different container and a single Python and CUDA version, because they cannot take a generic aarch64 wheel. That row is in the filter but empty: published PyTorch stopped shipping device code for those GPUs, so a Jetson row today would build a wheel whose PyTorch dependency cannot execute on the device. ## Before and after ``` BEFORE AFTER pip install executorch pip install executorch CPU only, always CPU by default GPU wheels published per CUDA version GPU support means cloning the a published wheel carries the delegate repository and building a delegate built for CUDA 13.2 every consumer CUDA version has a has no ExecuTorch wheel to pair with matching ExecuTorch wheel ``` ## Test plan A smoke test for the CUDA rows, checking what an artifact can be held to on a builder with no GPU: - the CUDA libraries are in the wheel, so a row named for CUDA cannot ship without a delegate - the CUDA runtime is declared, so a user has something to resolve it from - each CUDA library reaches its runtime through a relative path, not the builder's toolkit directory, which resolves on the builder and nowhere else Then everything a CPU wheel is already held to: one owner per component, no build-tree search paths, and a C++ application outside the wheel able to link what it ships. Each check was run against a wheel that should fail it as well as one that should pass, because a check that cannot fail is worse than no check. All three reject a CPU wheel and accept a CUDA wheel built from source. One did not fail at first: it looped over libraries that were not there and reported a pass having inspected nothing, so it now requires at least one to be present. The filter was exercised against a matrix shaped like the generator's output: 36 rows narrowed to 24, a pull request narrowed to 1, the aarch64 rows given the correct builder image, and an empty result treated as a failure rather than passed through, since a workflow with no build job reads as a green check for a build that never happened. ghstack-source-id: bdf89eb ghstack-comment-id: 5220374521 Pull-Request: #21668

## The problem The wheel can now ship a CUDA delegate, but no release builds one. Every wheel workflow says: ``` with-cuda: disabled ``` So a user who wants GPU support still clones the repository and builds from source, which is the thing the shipped libraries were supposed to remove. Nothing publishes them because nothing builds them. Anything that wants to depend on an ExecuTorch GPU wheel has nothing to depend on. ## The change Adds two workflows that build CUDA wheels the same way the existing ones build the default wheel, calling the same shared matrix generator with CUDA turned on, then narrowing the result. ``` .github/workflows/build-wheels-cuda-linux.yml x86_64 .github/workflows/build-wheels-cuda-aarch64-linux.yml aarch64 ``` | architecture | CUDA | Python | | --- | --- | --- | | x86_64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 | | aarch64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 | The CUDA versions are chosen so every accelerator consumer can find a wheel to depend on, rather than by what is convenient to verify. A delegate built against one CUDA version needs an ExecuTorch wheel for that same version, and a missing version leaves that consumer with nothing to pair with, which fails for whoever installs the pair rather than for the row that omitted it. Python 3.14 is excluded because the current CPU rows already fail on it for an unrelated reason, so a GPU row would inherit a known-broken build. A pull request builds one representative row rather than the whole matrix. Jetson devices need their own row, with a different container and a single Python and CUDA version, because they cannot take a generic aarch64 wheel. That row is in the filter but empty: published PyTorch stopped shipping device code for those GPUs, so a Jetson row today would build a wheel whose PyTorch dependency cannot execute on the device. ## Before and after ``` BEFORE AFTER pip install executorch pip install executorch CPU only, always CPU by default GPU wheels published per CUDA version GPU support means cloning the a published wheel carries the delegate repository and building a delegate built for CUDA 13.2 every consumer CUDA version has a has no ExecuTorch wheel to pair with matching ExecuTorch wheel ``` ## Test plan A smoke test for the CUDA rows, checking what an artifact can be held to on a builder with no GPU: - the CUDA libraries are in the wheel, so a row named for CUDA cannot ship without a delegate - the CUDA runtime is declared, so a user has something to resolve it from - each CUDA library reaches its runtime through a relative path, not the builder's toolkit directory, which resolves on the builder and nowhere else Then everything a CPU wheel is already held to: one owner per component, no build-tree search paths, and a C++ application outside the wheel able to link what it ships. Each check was run against a wheel that should fail it as well as one that should pass, because a check that cannot fail is worse than no check. All three reject a CPU wheel and accept a CUDA wheel built from source. One did not fail at first: it looped over libraries that were not there and reported a pass having inspected nothing, so it now requires at least one to be present. The matrix generator accepts "enable" and "disable" for its build-type inputs. Anything else is treated as not enabled, so a plausible-looking "enabled" produced an empty matrix and a failed run rather than a clear error. The checks now assert the accepted spelling. The filter was exercised against a matrix shaped like the generator's output: 36 rows narrowed to 24, a pull request narrowed to 1, the aarch64 rows given the correct builder image, and an empty result treated as a failure rather than passed through, since a workflow with no build job reads as a green check for a build that never happened. ghstack-source-id: df65c37 ghstack-comment-id: 5220374521 Pull-Request: #21668

## The problem The wheel can now ship a CUDA delegate, but no release builds one. Every wheel workflow says: ``` with-cuda: disabled ``` So a user who wants GPU support still clones the repository and builds from source, which is the thing the shipped libraries were supposed to remove. Nothing publishes them because nothing builds them. Anything that wants to depend on an ExecuTorch GPU wheel has nothing to depend on. ## The change Adds two workflows that build CUDA wheels the same way the existing ones build the default wheel, calling the same shared matrix generator with CUDA turned on, then narrowing the result. ``` .github/workflows/build-wheels-cuda-linux.yml x86_64 .github/workflows/build-wheels-cuda-aarch64-linux.yml aarch64 ``` | architecture | CUDA | Python | | --- | --- | --- | | x86_64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 | | aarch64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 | The CUDA versions are chosen so every accelerator consumer can find a wheel to depend on, rather than by what is convenient to verify. A delegate built against one CUDA version needs an ExecuTorch wheel for that same version, and a missing version leaves that consumer with nothing to pair with, which fails for whoever installs the pair rather than for the row that omitted it. Python 3.14 is excluded because the current CPU rows already fail on it for an unrelated reason, so a GPU row would inherit a known-broken build. A pull request builds one representative row rather than the whole matrix. Jetson devices need their own row, with a different container and a single Python and CUDA version, because they cannot take a generic aarch64 wheel. That row is in the filter but empty: published PyTorch stopped shipping device code for those GPUs, so a Jetson row today would build a wheel whose PyTorch dependency cannot execute on the device. ## Before and after ``` BEFORE AFTER pip install executorch pip install executorch CPU only, always CPU by default GPU wheels published per CUDA version GPU support means cloning the a published wheel carries the delegate repository and building a delegate built for CUDA 13.2 every consumer CUDA version has a has no ExecuTorch wheel to pair with matching ExecuTorch wheel ``` ## Test plan A smoke test for the CUDA rows, checking what an artifact can be held to on a builder with no GPU: - the CUDA libraries are in the wheel, so a row named for CUDA cannot ship without a delegate - the CUDA runtime is declared, so a user has something to resolve it from - each CUDA library reaches its runtime through a relative path, not the builder's toolkit directory, which resolves on the builder and nowhere else Then everything a CPU wheel is already held to: one owner per component, no build-tree search paths, and a C++ application outside the wheel able to link what it ships. Each check was run against a wheel that should fail it as well as one that should pass, because a check that cannot fail is worse than no check. All three reject a CPU wheel and accept a CUDA wheel built from source. One did not fail at first: it looped over libraries that were not there and reported a pass having inspected nothing, so it now requires at least one to be present. The rows keep whichever builder image the release provides. Substituting a plain manylinux image broke the shared build setup, which expects conda to be present in that image. The matrix generator accepts "enable" and "disable" for its build-type inputs. Anything else is treated as not enabled, so a plausible-looking "enabled" produced an empty matrix and a failed run rather than a clear error. The checks now assert the accepted spelling. The filter was exercised against a matrix shaped like the generator's output: 36 rows narrowed to 24, a pull request narrowed to 1, the aarch64 rows given the correct builder image, and an empty result treated as a failure rather than passed through, since a workflow with no build job reads as a green check for a build that never happened. ghstack-source-id: 0f17009 ghstack-comment-id: 5220374521 Pull-Request: #21668

## The problem The wheel can now ship a CUDA delegate, but no release builds one. Every wheel workflow says: ``` with-cuda: disabled ``` So a user who wants GPU support still clones the repository and builds from source, which is the thing the shipped libraries were supposed to remove. Nothing publishes them because nothing builds them. Anything that wants to depend on an ExecuTorch GPU wheel has nothing to depend on. ## The change Adds two workflows that build CUDA wheels the same way the existing ones build the default wheel, calling the same shared matrix generator with CUDA turned on, then narrowing the result. ``` .github/workflows/build-wheels-cuda-linux.yml x86_64 .github/workflows/build-wheels-cuda-aarch64-linux.yml aarch64 ``` | architecture | CUDA | Python | | --- | --- | --- | | x86_64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 | | aarch64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 | The CUDA versions are chosen so every accelerator consumer can find a wheel to depend on, rather than by what is convenient to verify. A delegate built against one CUDA version needs an ExecuTorch wheel for that same version, and a missing version leaves that consumer with nothing to pair with, which fails for whoever installs the pair rather than for the row that omitted it. Python 3.14 is excluded because the current CPU rows already fail on it for an unrelated reason, so a GPU row would inherit a known-broken build. A pull request builds one representative row rather than the whole matrix. Jetson devices need their own row, with a different container and a single Python and CUDA version, because they cannot take a generic aarch64 wheel. That row is in the filter but empty: published PyTorch stopped shipping device code for those GPUs, so a Jetson row today would build a wheel whose PyTorch dependency cannot execute on the device. ## Which GPUs the wheels carry code for GPU code is compiled per hardware generation, so a wheel built by detection alone carries code for whichever GPU the builder happened to have. It then installs on every machine the row claims and fails the moment a model runs on a different generation, with an error that looks like a model problem rather than a packaging one. So each row states its architectures instead of detecting them: | row | architectures | | --- | --- | | x86_64 CUDA 13.0 and 13.2 | 8.0, 9.0, 10.0, 12.0 | | aarch64 CUDA 13.0 and 13.2 | 9.0, 10.0, 11.0 | | either, CUDA 12.6 | 8.0, 9.0 | The newest architecture in each row also gets a portable form, so a GPU newer than any in the list can still run the wheel by compiling that form when it loads. A CUDA version with no list makes the build fail rather than fall back to detection, because falling back is the failure this prevents. The CUDA build reads that list and converts it, because it previously enabled the CUDA language without naming any architectures and so used whatever CMake defaults to. On one device that default is older than the intrinsics these sources use, and the compile failed with an undefined identifier that looks like a source problem. The value is published as `TORCH_CUDA_ARCH_LIST`. Setting only `CMAKE_CUDA_ARCHITECTURES` does not work here: PyTorch's CMake overrides it, which silently reduces the build to one detected architecture. ## Before and after ``` BEFORE AFTER pip install executorch pip install executorch CPU only, always CPU by default GPU wheels published per CUDA version GPU support means cloning the a published wheel carries the delegate repository and building a delegate built for CUDA 13.2 every consumer CUDA version has a has no ExecuTorch wheel to pair with matching ExecuTorch wheel ``` ## Test plan A smoke test for the CUDA rows, checking what an artifact can be held to on a builder with no GPU: - the CUDA libraries are in the wheel, so a row named for CUDA cannot ship without a delegate - the CUDA runtime is declared, so a user has something to resolve it from - each CUDA library reaches its runtime through a relative path, not the builder's toolkit directory, which resolves on the builder and nowhere else - the wheel carries device code for every architecture its row claims, read from the built libraries rather than assumed. Searched across every shipped library, because the kernels are compiled into their own library rather than into the delegate, and which one holds them is an internal detail. This caught a real case: a wheel built before the architecture list existed carried code for nothing the row promised, and it installed and loaded cleanly, so every other check passed on it. Then everything a CPU wheel is already held to: one owner per component, no build-tree search paths, and a C++ application outside the wheel able to link what it ships. Each check was run against a wheel that should fail it as well as one that should pass, because a check that cannot fail is worse than no check. All three reject a CPU wheel and accept a CUDA wheel built from source. One did not fail at first: it looped over libraries that were not there and reported a pass having inspected nothing, so it now requires at least one to be present. The aarch64 workflow passes the architecture the shared build workflow needs. Without it that workflow prepares an x86_64 job and skips the aarch64 conda install, so the first build step fails on a missing conda. The rows keep whichever builder image the release provides. Substituting a plain manylinux image broke the shared build setup, which expects conda to be present in that image. The matrix generator accepts "enable" and "disable" for its build-type inputs. Anything else is treated as not enabled, so a plausible-looking "enabled" produced an empty matrix and a failed run rather than a clear error. The checks now assert the accepted spelling. The filter was exercised against a matrix shaped like the generator's output: 36 rows narrowed to 24, a pull request narrowed to 1, the aarch64 rows given the correct builder image, and an empty result treated as a failure rather than passed through, since a workflow with no build job reads as a green check for a build that never happened. ghstack-source-id: 6c8cd05 ghstack-comment-id: 5220374521 Pull-Request: #21668

## The problem The wheel can now ship a CUDA delegate, but no release builds one. Every wheel workflow says: ``` with-cuda: disabled ``` So a user who wants GPU support still clones the repository and builds from source, which is the thing the shipped libraries were supposed to remove. Nothing publishes them because nothing builds them. Anything that wants to depend on an ExecuTorch GPU wheel has nothing to depend on. ## The change Adds two workflows that build CUDA wheels the same way the existing ones build the default wheel, calling the same shared matrix generator with CUDA turned on, then narrowing the result. ``` .github/workflows/build-wheels-cuda-linux.yml x86_64 .github/workflows/build-wheels-cuda-aarch64-linux.yml aarch64 ``` | architecture | CUDA | Python | | --- | --- | --- | | x86_64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 | | aarch64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 | The CUDA versions are chosen so every accelerator consumer can find a wheel to depend on, rather than by what is convenient to verify. A delegate built against one CUDA version needs an ExecuTorch wheel for that same version, and a missing version leaves that consumer with nothing to pair with, which fails for whoever installs the pair rather than for the row that omitted it. Python 3.14 is excluded because the current CPU rows already fail on it for an unrelated reason, so a GPU row would inherit a known-broken build. A pull request builds one representative row rather than the whole matrix. Jetson devices need their own row, with a different container and a single Python and CUDA version, because they cannot take a generic aarch64 wheel. That row is in the filter but empty: published PyTorch stopped shipping device code for those GPUs, so a Jetson row today would build a wheel whose PyTorch dependency cannot execute on the device. ## Which GPUs the wheels carry code for GPU code is compiled per hardware generation, so a wheel built by detection alone carries code for whichever GPU the builder happened to have. It then installs on every machine the row claims and fails the moment a model runs on a different generation, with an error that looks like a model problem rather than a packaging one. So each row states its architectures instead of detecting them: | row | architectures | | --- | --- | | x86_64 CUDA 13.0 and 13.2 | 8.0, 9.0, 10.0, 12.0 | | aarch64 CUDA 13.0 and 13.2 | 9.0, 10.0, 11.0 | | either, CUDA 12.6 | 8.0, 9.0 | The newest architecture in each row also gets a portable form, so a GPU newer than any in the list can still run the wheel by compiling that form when it loads. A CUDA version with no list makes the build fail rather than fall back to detection, because falling back is the failure this prevents. The CUDA build reads that list and converts it, because it previously enabled the CUDA language without naming any architectures and so used whatever CMake defaults to. On one device that default is older than the intrinsics these sources use, and the compile failed with an undefined identifier that looks like a source problem. The value is published as `TORCH_CUDA_ARCH_LIST`. Setting only `CMAKE_CUDA_ARCHITECTURES` does not work here: PyTorch's CMake overrides it, which silently reduces the build to one detected architecture. ## One dependency had to stop following the CUDA channel The requirements installer points both PyTorch and torchao at a CUDA-specific package channel when it detects a CUDA toolkit. For PyTorch that is the whole point. For torchao it makes the build impossible on aarch64, because that channel publishes no aarch64 build. Measured for the pinned version: | channel | files published | usable on aarch64 | | --- | --- | --- | | CUDA-suffixed | 2 | 0 | | plain nightly | 16 | 6 | So torchao now resolves from the plain channel, which serves every architecture, while PyTorch keeps its CUDA channel. Nothing in the wheel links or bundles torchao, which is a quantization-workflow dependency of the examples and tests, and a CUDA wheel built by hand on an aarch64 GPU machine used the non-CUDA torchao build without issue. ## Before and after ``` BEFORE AFTER pip install executorch pip install executorch CPU only, always CPU by default GPU wheels published per CUDA version GPU support means cloning the a published wheel carries the delegate repository and building a delegate built for CUDA 13.2 every consumer CUDA version has a has no ExecuTorch wheel to pair with matching ExecuTorch wheel ``` ## Test plan A smoke test for the CUDA rows, checking what an artifact can be held to on a builder with no GPU: - the CUDA libraries are in the wheel, so a row named for CUDA cannot ship without a delegate - the CUDA runtime is declared, so a user has something to resolve it from - each CUDA library reaches its runtime through a relative path, not the builder's toolkit directory, which resolves on the builder and nowhere else - the wheel carries device code for every architecture its row claims, read from the built libraries rather than assumed. Searched across every shipped library, because the kernels are compiled into their own library rather than into the delegate, and which one holds them is an internal detail. This caught a real case: a wheel built before the architecture list existed carried code for nothing the row promised, and it installed and loaded cleanly, so every other check passed on it. Then everything a CPU wheel is already held to: one owner per component, no build-tree search paths, and a C++ application outside the wheel able to link what it ships. Each check was run against a wheel that should fail it as well as one that should pass, because a check that cannot fail is worse than no check. All three reject a CPU wheel and accept a CUDA wheel built from source. One did not fail at first: it looped over libraries that were not there and reported a pass having inspected nothing, so it now requires at least one to be present. The aarch64 workflow passes the architecture the shared build workflow needs. Without it that workflow prepares an x86_64 job and skips the aarch64 conda install, so the first build step fails on a missing conda. The rows keep whichever builder image the release provides. Substituting a plain manylinux image broke the shared build setup, which expects conda to be present in that image. The matrix generator accepts "enable" and "disable" for its build-type inputs. Anything else is treated as not enabled, so a plausible-looking "enabled" produced an empty matrix and a failed run rather than a clear error. The checks now assert the accepted spelling. The filter was exercised against a matrix shaped like the generator's output: 36 rows narrowed to 24, a pull request narrowed to 1, the aarch64 rows given the correct builder image, and an empty result treated as a failure rather than passed through, since a workflow with no build job reads as a green check for a build that never happened. ghstack-source-id: d05089b ghstack-comment-id: 5220374521 Pull-Request: #21668

## The problem The wheel can now ship a CUDA delegate, but no release builds one. Every wheel workflow says: ``` with-cuda: disabled ``` So a user who wants GPU support still clones the repository and builds from source, which is the thing the shipped libraries were supposed to remove. Nothing publishes them because nothing builds them. Anything that wants to depend on an ExecuTorch GPU wheel has nothing to depend on. ## The change Adds two workflows that build CUDA wheels the same way the existing ones build the default wheel, calling the same shared matrix generator with CUDA turned on, then narrowing the result. ``` .github/workflows/build-wheels-cuda-linux.yml x86_64 .github/workflows/build-wheels-cuda-aarch64-linux.yml aarch64 ``` | architecture | CUDA | Python | | --- | --- | --- | | x86_64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 | | aarch64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 | The CUDA versions are chosen so every accelerator consumer can find a wheel to depend on, rather than by what is convenient to verify. A delegate built against one CUDA version needs an ExecuTorch wheel for that same version, and a missing version leaves that consumer with nothing to pair with, which fails for whoever installs the pair rather than for the row that omitted it. Python 3.14 is excluded because the current CPU rows already fail on it for an unrelated reason, so a GPU row would inherit a known-broken build. A pull request builds one representative row rather than the whole matrix. Jetson devices need their own row, with a different container and a single Python and CUDA version, because they cannot take a generic aarch64 wheel. That row is in the filter but empty: published PyTorch stopped shipping device code for those GPUs, so a Jetson row today would build a wheel whose PyTorch dependency cannot execute on the device. ## Which GPUs the wheels carry code for GPU code is compiled per hardware generation, so a wheel built by detection alone carries code for whichever GPU the builder happened to have. It then installs on every machine the row claims and fails the moment a model runs on a different generation, with an error that looks like a model problem rather than a packaging one. So each row states its architectures instead of detecting them: | row | architectures | | --- | --- | | x86_64 CUDA 13.0 and 13.2 | 8.0, 9.0, 10.0, 12.0 | | aarch64 CUDA 13.0 and 13.2 | 9.0, 10.0, 11.0 | | either, CUDA 12.6 | 8.0, 9.0 | The newest architecture in each row also gets a portable form, so a GPU newer than any in the list can still run the wheel by compiling that form when it loads. A CUDA version with no list makes the build fail rather than fall back to detection, because falling back is the failure this prevents. The CUDA build reads that list and converts it, because it previously enabled the CUDA language without naming any architectures and so used whatever CMake defaults to. On one device that default is older than the intrinsics these sources use, and the compile failed with an undefined identifier that looks like a source problem. The value is published as `TORCH_CUDA_ARCH_LIST`. Setting only `CMAKE_CUDA_ARCHITECTURES` does not work here: PyTorch's CMake overrides it, which silently reduces the build to one detected architecture. ## One dependency had to stop following the CUDA channel The requirements installer points both PyTorch and torchao at a CUDA-specific package channel when it detects a CUDA toolkit. For PyTorch that is the whole point. For torchao it makes the build impossible on aarch64, because that channel publishes no aarch64 build. Measured for the pinned version: | channel | files published | usable on aarch64 | | --- | --- | --- | | CUDA-suffixed | 2 | 0 | | plain nightly | 16 | 6 | So torchao now resolves from the plain channel, which serves every architecture, while PyTorch keeps its CUDA channel. Nothing in the wheel links or bundles torchao, which is a quantization-workflow dependency of the examples and tests, and a CUDA wheel built by hand on an aarch64 GPU machine used the non-CUDA torchao build without issue. ## Before and after ``` BEFORE AFTER pip install executorch pip install executorch CPU only, always CPU by default GPU wheels published per CUDA version GPU support means cloning the a published wheel carries the delegate repository and building a delegate built for CUDA 13.2 every consumer CUDA version has a has no ExecuTorch wheel to pair with matching ExecuTorch wheel ``` ## Test plan A smoke test for the CUDA rows, checking what an artifact can be held to on a builder with no GPU: - the CUDA libraries are in the wheel, so a row named for CUDA cannot ship without a delegate - the CUDA runtime is declared, so a user has something to resolve it from - each CUDA library reaches its runtime through a relative path, not the builder's toolkit directory, which resolves on the builder and nowhere else - the wheel carries device code for every architecture its row claims, read from the built libraries rather than assumed. Searched across every shipped library, because the kernels are compiled into their own library rather than into the delegate, and which one holds them is an internal detail. This caught a real case: a wheel built before the architecture list existed carried code for nothing the row promised, and it installed and loaded cleanly, so every other check passed on it. Then everything a CPU wheel is already held to: one owner per component, no build-tree search paths, and a C++ application outside the wheel able to link what it ships. Each check was run against a wheel that should fail it as well as one that should pass, because a check that cannot fail is worse than no check. All three reject a CPU wheel and accept a CUDA wheel built from source. One did not fail at first: it looped over libraries that were not there and reported a pass having inspected nothing, so it now requires at least one to be present. The aarch64 workflow passes the architecture the shared build workflow needs. Without it that workflow prepares an x86_64 job and skips the aarch64 conda install, so the first build step fails on a missing conda. The rows keep whichever builder image the release provides. Substituting a plain manylinux image broke the shared build setup, which expects conda to be present in that image. The matrix generator accepts "enable" and "disable" for its build-type inputs. Anything else is treated as not enabled, so a plausible-looking "enabled" produced an empty matrix and a failed run rather than a clear error. The checks now assert the accepted spelling. The filter was exercised against a matrix shaped like the generator's output: 36 rows narrowed to 24, a pull request narrowed to 1, the aarch64 rows given the correct builder image, and an empty result treated as a failure rather than passed through, since a workflow with no build job reads as a green check for a build that never happened. ghstack-source-id: 51078d0 ghstack-comment-id: 5220374521 Pull-Request: #21668

## The problem The wheel can now ship a CUDA delegate, but no release builds one. Every wheel workflow says: ``` with-cuda: disabled ``` So a user who wants GPU support still clones the repository and builds from source, which is the thing the shipped libraries were supposed to remove. Nothing publishes them because nothing builds them. Anything that wants to depend on an ExecuTorch GPU wheel has nothing to depend on. ## The change Adds two workflows that build CUDA wheels the same way the existing ones build the default wheel, calling the same shared matrix generator with CUDA turned on, then narrowing the result. ``` .github/workflows/build-wheels-cuda-linux.yml x86_64 .github/workflows/build-wheels-cuda-aarch64-linux.yml aarch64 ``` | architecture | CUDA | Python | | --- | --- | --- | | x86_64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 | | aarch64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 | The CUDA versions are chosen so every accelerator consumer can find a wheel to depend on, rather than by what is convenient to verify. A delegate built against one CUDA version needs an ExecuTorch wheel for that same version, and a missing version leaves that consumer with nothing to pair with, which fails for whoever installs the pair rather than for the row that omitted it. Python 3.14 is excluded because the current CPU rows already fail on it for an unrelated reason, so a GPU row would inherit a known-broken build. A pull request builds one representative row rather than the whole matrix. Jetson devices need their own row, with a different container and a single Python and CUDA version, because they cannot take a generic aarch64 wheel. That row is in the filter but empty: published PyTorch stopped shipping device code for those GPUs, so a Jetson row today would build a wheel whose PyTorch dependency cannot execute on the device. ## Which GPUs the wheels carry code for GPU code is compiled per hardware generation, so a wheel built by detection alone carries code for whichever GPU the builder happened to have. It then installs on every machine the row claims and fails the moment a model runs on a different generation, with an error that looks like a model problem rather than a packaging one. So each row states its architectures instead of detecting them: | row | architectures | | --- | --- | | x86_64 CUDA 13.0 and 13.2 | 8.0, 9.0, 10.0, 12.0 | | aarch64 CUDA 13.0 and 13.2 | 9.0, 10.0, 11.0 | | either, CUDA 12.6 | 8.0, 9.0 | The newest architecture in each row also gets a portable form, so a GPU newer than any in the list can still run the wheel by compiling that form when it loads. A CUDA version with no list makes the build fail rather than fall back to detection, because falling back is the failure this prevents. The CUDA build reads that list and converts it, because it previously enabled the CUDA language without naming any architectures and so used whatever CMake defaults to. On one device that default is older than the intrinsics these sources use, and the compile failed with an undefined identifier that looks like a source problem. The value is published as `TORCH_CUDA_ARCH_LIST`. Setting only `CMAKE_CUDA_ARCHITECTURES` does not work here: PyTorch's CMake overrides it, which silently reduces the build to one detected architecture. ## One dependency had to stop following the CUDA channel The requirements installer points both PyTorch and torchao at a CUDA-specific package channel when it detects a CUDA toolkit. For PyTorch that is the whole point. For torchao it makes the build impossible on aarch64, because that channel publishes no aarch64 build. Measured for the pinned version: | channel | files published | usable on aarch64 | | --- | --- | --- | | CUDA-suffixed | 2 | 0 | | plain nightly | 16 | 6 | So torchao now resolves from the plain channel, which serves every architecture, while PyTorch keeps its CUDA channel. Nothing in the wheel links or bundles torchao, which is a quantization-workflow dependency of the examples and tests, and a CUDA wheel built by hand on an aarch64 GPU machine used the non-CUDA torchao build without issue. Windows is recorded as an explicit gap rather than left unmentioned. The matrix could express a Windows CUDA row and PyTorch publishes Windows CUDA wheels, but the separate libraries this wheel exists to ship are Linux only today, so a Windows CUDA wheel would carry a delegate a C++ application still could not link. Splitting the libraries there is the prerequisite, and the filter says so. ## Before and after ``` BEFORE AFTER pip install executorch pip install executorch CPU only, always CPU by default GPU wheels published per CUDA version GPU support means cloning the a published wheel carries the delegate repository and building a delegate built for CUDA 13.2 every consumer CUDA version has a has no ExecuTorch wheel to pair with matching ExecuTorch wheel ``` ## Test plan A smoke test for the CUDA rows, checking what an artifact can be held to on a builder with no GPU: - the CUDA libraries are in the wheel, so a row named for CUDA cannot ship without a delegate - the CUDA runtime is declared, so a user has something to resolve it from - each CUDA library reaches its runtime through a relative path, not the builder's toolkit directory, which resolves on the builder and nowhere else - the wheel carries device code for every architecture its row claims, read from the built libraries rather than assumed. Searched across every shipped library, because the kernels are compiled into their own library rather than into the delegate, and which one holds them is an internal detail. This caught a real case: a wheel built before the architecture list existed carried code for nothing the row promised, and it installed and loaded cleanly, so every other check passed on it. Then everything a CPU wheel is already held to: one owner per component, no build-tree search paths, and a C++ application outside the wheel able to link what it ships. Each check was run against a wheel that should fail it as well as one that should pass, because a check that cannot fail is worse than no check. All three reject a CPU wheel and accept a CUDA wheel built from source. One did not fail at first: it looped over libraries that were not there and reported a pass having inspected nothing, so it now requires at least one to be present. The aarch64 workflow passes the architecture the shared build workflow needs. Without it that workflow prepares an x86_64 job and skips the aarch64 conda install, so the first build step fails on a missing conda. The rows keep whichever builder image the release provides. Substituting a plain manylinux image broke the shared build setup, which expects conda to be present in that image. The matrix generator accepts "enable" and "disable" for its build-type inputs. Anything else is treated as not enabled, so a plausible-looking "enabled" produced an empty matrix and a failed run rather than a clear error. The checks now assert the accepted spelling. The filter was exercised against a matrix shaped like the generator's output: 36 rows narrowed to 24, a pull request narrowed to 1, the aarch64 rows given the correct builder image, and an empty result treated as a failure rather than passed through, since a workflow with no build job reads as a green check for a build that never happened. ghstack-source-id: 9e3b175 ghstack-comment-id: 5220374521 Pull-Request: #21668

## The problem The wheel can now ship a CUDA delegate, but no release builds one. Every wheel workflow says: ``` with-cuda: disabled ``` So a user who wants GPU support still clones the repository and builds from source, which is the thing the shipped libraries were supposed to remove. Nothing publishes them because nothing builds them. Anything that wants to depend on an ExecuTorch GPU wheel has nothing to depend on. ## The change Adds two workflows that build CUDA wheels the same way the existing ones build the default wheel, calling the same shared matrix generator with CUDA turned on, then narrowing the result. ``` .github/workflows/build-wheels-cuda-linux.yml x86_64 .github/workflows/build-wheels-cuda-aarch64-linux.yml aarch64 ``` | architecture | CUDA | Python | | --- | --- | --- | | x86_64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 | | aarch64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 | The CUDA versions are chosen so every accelerator consumer can find a wheel to depend on, rather than by what is convenient to verify. A delegate built against one CUDA version needs an ExecuTorch wheel for that same version, and a missing version leaves that consumer with nothing to pair with, which fails for whoever installs the pair rather than for the row that omitted it. Python 3.14 is excluded because the current CPU rows already fail on it for an unrelated reason, so a GPU row would inherit a known-broken build. A pull request builds one representative row rather than the whole matrix. Jetson devices need their own row, with a different container and a single Python and CUDA version, because they cannot take a generic aarch64 wheel. That row is in the filter but empty: published PyTorch stopped shipping device code for those GPUs, so a Jetson row today would build a wheel whose PyTorch dependency cannot execute on the device. ## Which GPUs the wheels carry code for GPU code is compiled per hardware generation, so a wheel built by detection alone carries code for whichever GPU the builder happened to have. It then installs on every machine the row claims and fails the moment a model runs on a different generation, with an error that looks like a model problem rather than a packaging one. So each row states its architectures instead of detecting them: | row | architectures | | --- | --- | | x86_64 CUDA 13.0 and 13.2 | 8.0, 9.0, 10.0, 12.0 | | aarch64 CUDA 13.0 and 13.2 | 9.0, 10.0, 11.0 | | either, CUDA 12.6 | 8.0, 9.0 | The newest architecture in each row also gets a portable form, so a GPU newer than any in the list can still run the wheel by compiling that form when it loads. A CUDA version with no list makes the build fail rather than fall back to detection, because falling back is the failure this prevents. The CUDA build reads that list and converts it, because it previously enabled the CUDA language without naming any architectures and so used whatever CMake defaults to. On one device that default is older than the intrinsics these sources use, and the compile failed with an undefined identifier that looks like a source problem. The value is published as `TORCH_CUDA_ARCH_LIST`. Setting only `CMAKE_CUDA_ARCHITECTURES` does not work here: PyTorch's CMake overrides it, which silently reduces the build to one detected architecture. ## One dependency had to stop following the CUDA channel The requirements installer points both PyTorch and torchao at a CUDA-specific package channel when it detects a CUDA toolkit. For PyTorch that is the whole point. For torchao it makes the build impossible on aarch64, because that channel publishes no aarch64 build. Measured for the pinned version: | channel | files published | usable on aarch64 | | --- | --- | --- | | CUDA-suffixed | 2 | 0 | | plain nightly | 16 | 6 | So torchao now resolves from the plain channel, which serves every architecture, while PyTorch keeps its CUDA channel. Nothing in the wheel links or bundles torchao, which is a quantization-workflow dependency of the examples and tests, and a CUDA wheel built by hand on an aarch64 GPU machine used the non-CUDA torchao build without issue. Windows is recorded as an explicit gap rather than left unmentioned. The matrix could express a Windows CUDA row and PyTorch publishes Windows CUDA wheels, but the separate libraries this wheel exists to ship are Linux only today, so a Windows CUDA wheel would carry a delegate a C++ application still could not link. Splitting the libraries there is the prerequisite, and the filter says so. ## Before and after ``` BEFORE AFTER pip install executorch pip install executorch CPU only, always CPU by default GPU wheels published per CUDA version GPU support means cloning the a published wheel carries the delegate repository and building a delegate built for CUDA 13.2 every consumer CUDA version has a has no ExecuTorch wheel to pair with matching ExecuTorch wheel ``` ## Test plan A smoke test for the CUDA rows, checking what an artifact can be held to on a builder with no GPU: - the CUDA libraries are in the wheel, so a row named for CUDA cannot ship without a delegate - the CUDA runtime is declared, so a user has something to resolve it from - each CUDA library reaches its runtime through a relative path, not the builder's toolkit directory, which resolves on the builder and nowhere else - the wheel carries device code for every architecture its row claims, read from the built libraries rather than assumed. Searched across every shipped library, because the kernels are compiled into their own library rather than into the delegate, and which one holds them is an internal detail. This caught a real case: a wheel built before the architecture list existed carried code for nothing the row promised, and it installed and loaded cleanly, so every other check passed on it. Then everything a CPU wheel is already held to: one owner per component, no build-tree search paths, and a C++ application outside the wheel able to link what it ships. Each check was run against a wheel that should fail it as well as one that should pass, because a check that cannot fail is worse than no check. All three reject a CPU wheel and accept a CUDA wheel built from source. One did not fail at first: it looped over libraries that were not there and reported a pass having inspected nothing, so it now requires at least one to be present. The aarch64 workflow passes the architecture the shared build workflow needs. Without it that workflow prepares an x86_64 job and skips the aarch64 conda install, so the first build step fails on a missing conda. The rows keep whichever builder image the release provides. Substituting a plain manylinux image broke the shared build setup, which expects conda to be present in that image. The matrix generator accepts "enable" and "disable" for its build-type inputs. Anything else is treated as not enabled, so a plausible-looking "enabled" produced an empty matrix and a failed run rather than a clear error. The checks now assert the accepted spelling. The filter was exercised against a matrix shaped like the generator's output: 36 rows narrowed to 24, a pull request narrowed to 1, the aarch64 rows given the correct builder image, and an empty result treated as a failure rather than passed through, since a workflow with no build job reads as a green check for a build that never happened. ghstack-source-id: 00b4a80 ghstack-comment-id: 5220374521 Pull-Request: #21668

## The problem The wheel can now ship a CUDA delegate, but no release builds one. Every wheel workflow says: ``` with-cuda: disabled ``` So a user who wants GPU support still clones the repository and builds from source, which is the thing the shipped libraries were supposed to remove. Nothing publishes them because nothing builds them. Anything that wants to depend on an ExecuTorch GPU wheel has nothing to depend on. ## The change Adds two workflows that build CUDA wheels the same way the existing ones build the default wheel, calling the same shared matrix generator with CUDA turned on, then narrowing the result. ``` .github/workflows/build-wheels-cuda-linux.yml x86_64 .github/workflows/build-wheels-cuda-aarch64-linux.yml aarch64 ``` | architecture | CUDA | Python | | --- | --- | --- | | x86_64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 | | aarch64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 | The CUDA versions are chosen so every accelerator consumer can find a wheel to depend on, rather than by what is convenient to verify. A delegate built against one CUDA version needs an ExecuTorch wheel for that same version, and a missing version leaves that consumer with nothing to pair with, which fails for whoever installs the pair rather than for the row that omitted it. A pull request builds the newest CUDA version among the rows on offer, chosen by preference rather than by naming an exact combination. An exact request degrades quietly: the generator currently emits one python version, so asking for another left a pull request building the oldest CUDA version while still reporting success, which tested a different wheel than intended. Python 3.14 is excluded because the current CPU rows already fail on it for an unrelated reason, so a GPU row would inherit a known-broken build. A pull request builds one representative row rather than the whole matrix. Jetson devices need their own row, with a different container and a single Python and CUDA version, because they cannot take a generic aarch64 wheel. That row is in the filter but empty: published PyTorch stopped shipping device code for those GPUs, so a Jetson row today would build a wheel whose PyTorch dependency cannot execute on the device. ## Which GPUs the wheels carry code for GPU code is compiled per hardware generation, so a wheel built by detection alone carries code for whichever GPU the builder happened to have. It then installs on every machine the row claims and fails the moment a model runs on a different generation, with an error that looks like a model problem rather than a packaging one. So each row states its architectures instead of detecting them: | row | architectures | | --- | --- | | x86_64 CUDA 13.0 and 13.2 | 8.0, 9.0, 10.0, 12.0 | | aarch64 CUDA 13.0 and 13.2 | 9.0, 10.0, 11.0 | | either, CUDA 12.6 | 8.0, 9.0 | The newest architecture in each row also gets a portable form, so a GPU newer than any in the list can still run the wheel by compiling that form when it loads. A CUDA version with no list makes the build fail rather than fall back to detection, because falling back is the failure this prevents. The CUDA build reads that list and converts it, because it previously enabled the CUDA language without naming any architectures and so used whatever CMake defaults to. On one device that default is older than the intrinsics these sources use, and the compile failed with an undefined identifier that looks like a source problem. The value is published as `TORCH_CUDA_ARCH_LIST`. Setting only `CMAKE_CUDA_ARCHITECTURES` does not work here: PyTorch's CMake overrides it, which silently reduces the build to one detected architecture. ## One dependency had to stop following the CUDA channel The requirements installer points both PyTorch and torchao at a CUDA-specific package channel when it detects a CUDA toolkit. For PyTorch that is the whole point. For torchao it makes the build impossible on aarch64, because that channel publishes no aarch64 build. Measured for the pinned version: | channel | files published | usable on aarch64 | | --- | --- | --- | | CUDA-suffixed | 2 | 0 | | plain nightly | 16 | 6 | So torchao now resolves from the plain channel, which serves every architecture, while PyTorch keeps its CUDA channel. Nothing in the wheel links or bundles torchao, which is a quantization-workflow dependency of the examples and tests, and a CUDA wheel built by hand on an aarch64 GPU machine used the non-CUDA torchao build without issue. Windows is recorded as an explicit gap rather than left unmentioned. The matrix could express a Windows CUDA row and PyTorch publishes Windows CUDA wheels, but the separate libraries this wheel exists to ship are Linux only today, so a Windows CUDA wheel would carry a delegate a C++ application still could not link. Splitting the libraries there is the prerequisite, and the filter says so. ## Before and after ``` BEFORE AFTER pip install executorch pip install executorch CPU only, always CPU by default GPU wheels published per CUDA version GPU support means cloning the a published wheel carries the delegate repository and building a delegate built for CUDA 13.2 every consumer CUDA version has a has no ExecuTorch wheel to pair with matching ExecuTorch wheel ``` ## Test plan A smoke test for the CUDA rows, checking what an artifact can be held to on a builder with no GPU: - the CUDA libraries are in the wheel, so a row named for CUDA cannot ship without a delegate - the CUDA runtime is declared, so a user has something to resolve it from - each CUDA library reaches its runtime through a relative path, not the builder's toolkit directory, which resolves on the builder and nowhere else - the wheel carries device code for every architecture its row claims, read from the built libraries rather than assumed. Searched across every shipped library, because the kernels are compiled into their own library rather than into the delegate, and which one holds them is an internal detail. This caught a real case: a wheel built before the architecture list existed carried code for nothing the row promised, and it installed and loaded cleanly, so every other check passed on it. Then everything a CPU wheel is already held to: one owner per component, no build-tree search paths, and a C++ application outside the wheel able to link what it ships. Each check was run against a wheel that should fail it as well as one that should pass, because a check that cannot fail is worse than no check. All three reject a CPU wheel and accept a CUDA wheel built from source. One did not fail at first: it looped over libraries that were not there and reported a pass having inspected nothing, so it now requires at least one to be present. The aarch64 workflow passes the architecture the shared build workflow needs. Without it that workflow prepares an x86_64 job and skips the aarch64 conda install, so the first build step fails on a missing conda. The rows keep whichever builder image the release provides. Substituting a plain manylinux image broke the shared build setup, which expects conda to be present in that image. The matrix generator accepts "enable" and "disable" for its build-type inputs. Anything else is treated as not enabled, so a plausible-looking "enabled" produced an empty matrix and a failed run rather than a clear error. The checks now assert the accepted spelling. The filter was exercised against a matrix shaped like the generator's output: 36 rows narrowed to 24, a pull request narrowed to 1, the aarch64 rows given the correct builder image, and an empty result treated as a failure rather than passed through, since a workflow with no build job reads as a green check for a build that never happened. ghstack-source-id: 32f0284 ghstack-comment-id: 5220374521 Pull-Request: #21668

## The problem The wheel can now ship a CUDA delegate, but no release builds one. Every wheel workflow says: ``` with-cuda: disabled ``` So a user who wants GPU support still clones the repository and builds from source, which is the thing the shipped libraries were supposed to remove. Nothing publishes them because nothing builds them. Anything that wants to depend on an ExecuTorch GPU wheel has nothing to depend on. ## The change Adds two workflows that build CUDA wheels the same way the existing ones build the default wheel, calling the same shared matrix generator with CUDA turned on, then narrowing the result. ``` .github/workflows/build-wheels-cuda-linux.yml x86_64 .github/workflows/build-wheels-cuda-aarch64-linux.yml aarch64 ``` | architecture | CUDA | Python | | --- | --- | --- | | x86_64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 | | aarch64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 | The CUDA versions are chosen so every accelerator consumer can find a wheel to depend on, rather than by what is convenient to verify. A delegate built against one CUDA version needs an ExecuTorch wheel for that same version, and a missing version leaves that consumer with nothing to pair with, which fails for whoever installs the pair rather than for the row that omitted it. A pull request builds the newest CUDA version among the rows on offer, chosen by preference rather than by naming an exact combination. An exact request degrades quietly: the generator currently emits one python version, so asking for another left a pull request building the oldest CUDA version while still reporting success, which tested a different wheel than intended. Python 3.14 is excluded because the current CPU rows already fail on it for an unrelated reason, so a GPU row would inherit a known-broken build. A pull request builds one representative row rather than the whole matrix. Jetson devices need their own row, with a different container and a single Python and CUDA version, because they cannot take a generic aarch64 wheel. That row is in the filter but empty: published PyTorch stopped shipping device code for those GPUs, so a Jetson row today would build a wheel whose PyTorch dependency cannot execute on the device. ## Which GPUs the wheels carry code for GPU code is compiled per hardware generation, so a wheel built by detection alone carries code for whichever GPU the builder happened to have. It then installs on every machine the row claims and fails the moment a model runs on a different generation, with an error that looks like a model problem rather than a packaging one. So each row states its architectures instead of detecting them: | row | architectures | | --- | --- | | x86_64 CUDA 13.0 and 13.2 | 8.0, 9.0, 10.0, 12.0 | | aarch64 CUDA 13.0 and 13.2 | 9.0, 10.0, 11.0 | | either, CUDA 12.6 | 8.0, 9.0 | The newest architecture in each row also gets a portable form, so a GPU newer than any in the list can still run the wheel by compiling that form when it loads. A CUDA version with no list makes the build fail rather than fall back to detection, because falling back is the failure this prevents. The CUDA build reads that list and converts it, because it previously enabled the CUDA language without naming any architectures and so used whatever CMake defaults to. On one device that default is older than the intrinsics these sources use, and the compile failed with an undefined identifier that looks like a source problem. The value is published as `TORCH_CUDA_ARCH_LIST`. Setting only `CMAKE_CUDA_ARCHITECTURES` does not work here: PyTorch's CMake overrides it, which silently reduces the build to one detected architecture. ## One dependency had to stop following the CUDA channel The requirements installer points both PyTorch and torchao at a CUDA-specific package channel when it detects a CUDA toolkit. For PyTorch that is the whole point. For torchao it makes the build impossible on aarch64, because that channel publishes no aarch64 build. Measured for the pinned version: | channel | files published | usable on aarch64 | | --- | --- | --- | | CUDA-suffixed | 2 | 0 | | plain nightly | 16 | 6 | So torchao now resolves from the plain channel, which serves every architecture, while PyTorch keeps its CUDA channel. Nothing in the wheel links or bundles torchao, which is a quantization-workflow dependency of the examples and tests, and a CUDA wheel built by hand on an aarch64 GPU machine used the non-CUDA torchao build without issue. Windows is recorded as an explicit gap rather than left unmentioned. The matrix could express a Windows CUDA row and PyTorch publishes Windows CUDA wheels, but the separate libraries this wheel exists to ship are Linux only today, so a Windows CUDA wheel would carry a delegate a C++ application still could not link. Splitting the libraries there is the prerequisite, and the filter says so. ## Before and after ``` BEFORE AFTER pip install executorch pip install executorch CPU only, always CPU by default GPU wheels published per CUDA version GPU support means cloning the a published wheel carries the delegate repository and building a delegate built for CUDA 13.2 every consumer CUDA version has a has no ExecuTorch wheel to pair with matching ExecuTorch wheel ``` ## Test plan A smoke test for the CUDA rows, checking what an artifact can be held to on a builder with no GPU: - the CUDA libraries are in the wheel, so a row named for CUDA cannot ship without a delegate - the CUDA runtime is declared, so a user has something to resolve it from - each CUDA library reaches its runtime through a relative path, not the builder's toolkit directory, which resolves on the builder and nowhere else - the wheel carries device code for every architecture its row claims, read from the built libraries rather than assumed. Searched across every shipped library, because the kernels are compiled into their own library rather than into the delegate, and which one holds them is an internal detail. This caught a real case: a wheel built before the architecture list existed carried code for nothing the row promised, and it installed and loaded cleanly, so every other check passed on it. Then everything a CPU wheel is already held to: one owner per component, no build-tree search paths, and a C++ application outside the wheel able to link what it ships. Each check was run against a wheel that should fail it as well as one that should pass, because a check that cannot fail is worse than no check. All three reject a CPU wheel and accept a CUDA wheel built from source. One did not fail at first: it looped over libraries that were not there and reported a pass having inspected nothing, so it now requires at least one to be present. The aarch64 workflow passes the architecture the shared build workflow needs. Without it that workflow prepares an x86_64 job and skips the aarch64 conda install, so the first build step fails on a missing conda. The rows keep whichever builder image the release provides. Substituting a plain manylinux image broke the shared build setup, which expects conda to be present in that image. The matrix generator accepts "enable" and "disable" for its build-type inputs. Anything else is treated as not enabled, so a plausible-looking "enabled" produced an empty matrix and a failed run rather than a clear error. The checks now assert the accepted spelling. The filter was exercised against a matrix shaped like the generator's output: 36 rows narrowed to 24, a pull request narrowed to 1, the aarch64 rows given the correct builder image, and an empty result treated as a failure rather than passed through, since a workflow with no build job reads as a green check for a build that never happened. ghstack-source-id: 07a025a ghstack-comment-id: 5220374521 Pull-Request: #21668

## The problem The wheel can now ship a CUDA delegate, but no release builds one. Every wheel workflow says: ``` with-cuda: disabled ``` So a user who wants GPU support still clones the repository and builds from source, which is the thing the shipped libraries were supposed to remove. Nothing publishes them because nothing builds them. Anything that wants to depend on an ExecuTorch GPU wheel has nothing to depend on. ## The change Adds two workflows that build CUDA wheels the same way the existing ones build the default wheel, calling the same shared matrix generator with CUDA turned on, then narrowing the result. ``` .github/workflows/build-wheels-cuda-linux.yml x86_64 .github/workflows/build-wheels-cuda-aarch64-linux.yml aarch64 ``` | architecture | CUDA | Python | | --- | --- | --- | | x86_64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 | | aarch64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 | The CUDA versions are chosen so every accelerator consumer can find a wheel to depend on, rather than by what is convenient to verify. A delegate built against one CUDA version needs an ExecuTorch wheel for that same version, and a missing version leaves that consumer with nothing to pair with, which fails for whoever installs the pair rather than for the row that omitted it. A pull request builds the newest CUDA version among the rows on offer, chosen by preference rather than by naming an exact combination. An exact request degrades quietly: the generator currently emits one python version, so asking for another left a pull request building the oldest CUDA version while still reporting success, which tested a different wheel than intended. Python 3.14 is excluded because the current CPU rows already fail on it for an unrelated reason, so a GPU row would inherit a known-broken build. A pull request builds one representative row rather than the whole matrix. Jetson devices need their own row, with a different container and a single Python and CUDA version, because they cannot take a generic aarch64 wheel. That row is in the filter but empty: published PyTorch stopped shipping device code for those GPUs, so a Jetson row today would build a wheel whose PyTorch dependency cannot execute on the device. ## Which GPUs the wheels carry code for GPU code is compiled per hardware generation, so a wheel built by detection alone carries code for whichever GPU the builder happened to have. It then installs on every machine the row claims and fails the moment a model runs on a different generation, with an error that looks like a model problem rather than a packaging one. So each row states its architectures instead of detecting them: | row | architectures | | --- | --- | | x86_64 CUDA 13.0 and 13.2 | 8.0, 9.0, 10.0, 12.0 | | aarch64 CUDA 13.0 and 13.2 | 9.0, 10.0, 11.0 | | either, CUDA 12.6 | 8.0, 9.0 | The newest architecture in each row also gets a portable form, so a GPU newer than any in the list can still run the wheel by compiling that form when it loads. A CUDA version with no list makes the build fail rather than fall back to detection, because falling back is the failure this prevents. The CUDA build reads that list and converts it, because it previously enabled the CUDA language without naming any architectures and so used whatever CMake defaults to. On one device that default is older than the intrinsics these sources use, and the compile failed with an undefined identifier that looks like a source problem. The value is published as `TORCH_CUDA_ARCH_LIST`. Setting only `CMAKE_CUDA_ARCHITECTURES` does not work here: PyTorch's CMake overrides it, which silently reduces the build to one detected architecture. ## One dependency had to stop following the CUDA channel The requirements installer points both PyTorch and torchao at a CUDA-specific package channel when it detects a CUDA toolkit. For PyTorch that is the whole point. For torchao it makes the build impossible on aarch64, because that channel publishes no aarch64 build. Measured for the pinned version: | channel | files published | usable on aarch64 | | --- | --- | --- | | CUDA-suffixed | 2 | 0 | | plain nightly | 16 | 6 | So torchao now resolves from the plain channel, which serves every architecture, while PyTorch keeps its CUDA channel. Nothing in the wheel links or bundles torchao, which is a quantization-workflow dependency of the examples and tests, and a CUDA wheel built by hand on an aarch64 GPU machine used the non-CUDA torchao build without issue. Windows is recorded as an explicit gap rather than left unmentioned. The matrix could express a Windows CUDA row and PyTorch publishes Windows CUDA wheels, but the separate libraries this wheel exists to ship are Linux only today, so a Windows CUDA wheel would carry a delegate a C++ application still could not link. Splitting the libraries there is the prerequisite, and the filter says so. ## Before and after ``` BEFORE AFTER pip install executorch pip install executorch CPU only, always CPU by default GPU wheels published per CUDA version GPU support means cloning the a published wheel carries the delegate repository and building a delegate built for CUDA 13.2 every consumer CUDA version has a has no ExecuTorch wheel to pair with matching ExecuTorch wheel ``` ## Test plan A smoke test for the CUDA rows, checking what an artifact can be held to on a builder with no GPU: - the CUDA libraries are in the wheel, so a row named for CUDA cannot ship without a delegate - the CUDA runtime is declared, so a user has something to resolve it from - each CUDA library reaches its runtime through a relative path, not the builder's toolkit directory, which resolves on the builder and nowhere else - the wheel carries device code for every architecture its row claims, read from the built libraries rather than assumed. Searched across every shipped library, because the kernels are compiled into their own library rather than into the delegate, and which one holds them is an internal detail. This caught a real case: a wheel built before the architecture list existed carried code for nothing the row promised, and it installed and loaded cleanly, so every other check passed on it. Then everything a CPU wheel is already held to: one owner per component, no build-tree search paths, and a C++ application outside the wheel able to link what it ships. Each check was run against a wheel that should fail it as well as one that should pass, because a check that cannot fail is worse than no check. All three reject a CPU wheel and accept a CUDA wheel built from source. One did not fail at first: it looped over libraries that were not there and reported a pass having inspected nothing, so it now requires at least one to be present. The aarch64 workflow passes the architecture the shared build workflow needs. Without it that workflow prepares an x86_64 job and skips the aarch64 conda install, so the first build step fails on a missing conda. The rows keep whichever builder image the release provides. Substituting a plain manylinux image broke the shared build setup, which expects conda to be present in that image. The matrix generator accepts "enable" and "disable" for its build-type inputs. Anything else is treated as not enabled, so a plausible-looking "enabled" produced an empty matrix and a failed run rather than a clear error. The checks now assert the accepted spelling. The filter was exercised against a matrix shaped like the generator's output: 36 rows narrowed to 24, a pull request narrowed to 1, the aarch64 rows given the correct builder image, and an empty result treated as a failure rather than passed through, since a workflow with no build job reads as a green check for a build that never happened. ghstack-source-id: 7877e9f ghstack-comment-id: 5220374521 Pull-Request: #21668

## The problem The wheel can now ship a CUDA delegate, but no release builds one. Every wheel workflow says: ``` with-cuda: disabled ``` So a user who wants GPU support still clones the repository and builds from source, which is the thing the shipped libraries were supposed to remove. Nothing publishes them because nothing builds them. Anything that wants to depend on an ExecuTorch GPU wheel has nothing to depend on. ## The change Adds two workflows that build CUDA wheels the same way the existing ones build the default wheel, calling the same shared matrix generator with CUDA turned on, then narrowing the result. ``` .github/workflows/build-wheels-cuda-linux.yml x86_64 .github/workflows/build-wheels-cuda-aarch64-linux.yml aarch64 ``` | architecture | CUDA | Python | | --- | --- | --- | | x86_64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 | | aarch64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 | The CUDA versions are chosen so every accelerator consumer can find a wheel to depend on, rather than by what is convenient to verify. A delegate built against one CUDA version needs an ExecuTorch wheel for that same version, and a missing version leaves that consumer with nothing to pair with, which fails for whoever installs the pair rather than for the row that omitted it. A pull request builds the newest CUDA version among the rows on offer, chosen by preference rather than by naming an exact combination. An exact request degrades quietly: the generator currently emits one python version, so asking for another left a pull request building the oldest CUDA version while still reporting success, which tested a different wheel than intended. Python 3.14 is excluded because the current CPU rows already fail on it for an unrelated reason, so a GPU row would inherit a known-broken build. A pull request builds one representative row rather than the whole matrix. Jetson devices need their own row, with a different container and a single Python and CUDA version, because they cannot take a generic aarch64 wheel. That row is in the filter but empty: published PyTorch stopped shipping device code for those GPUs, so a Jetson row today would build a wheel whose PyTorch dependency cannot execute on the device. ## Which GPUs the wheels carry code for GPU code is compiled per hardware generation, so a wheel built by detection alone carries code for whichever GPU the builder happened to have. It then installs on every machine the row claims and fails the moment a model runs on a different generation, with an error that looks like a model problem rather than a packaging one. So each row states its architectures instead of detecting them: | row | architectures | | --- | --- | | x86_64 CUDA 13.0 and 13.2 | 7.5, 8.0, 8.6, 8.9, 9.0, 10.0, 12.0 | | x86_64 CUDA 12.6 | 7.5, 8.0, 8.6, 8.9, 9.0 | | aarch64 CUDA 13.0 and 13.2 | 9.0, 10.0, 11.0 | | aarch64 CUDA 12.6 | 8.0, 9.0 | The x86_64 lists are read from the published PyTorch CUDA library rather than chosen by judgement. A delegate is only useful where PyTorch already runs, so an architecture PyTorch supports and this wheel omits gives a user a wheel that installs and then fails at the first kernel launch. Checking that way found two omissions worth having: the GPU on the runner that tests these wheels, and a common desktop card. The newest architecture in each row also gets a portable form, so a GPU newer than any in the list can still run the wheel by compiling that form when it loads. A CUDA version with no list makes the build fail rather than fall back to detection, because falling back is the failure this prevents. The CUDA build reads that list and converts it, because it previously enabled the CUDA language without naming any architectures and so used whatever CMake defaults to. On one device that default is older than the intrinsics these sources use, and the compile failed with an undefined identifier that looks like a source problem. The value is published as `TORCH_CUDA_ARCH_LIST`. Setting only `CMAKE_CUDA_ARCHITECTURES` does not work here: PyTorch's CMake overrides it, which silently reduces the build to one detected architecture. ## One dependency had to stop following the CUDA channel The requirements installer points both PyTorch and torchao at a CUDA-specific package channel when it detects a CUDA toolkit. For PyTorch that is the whole point. For torchao it makes the build impossible on aarch64, because that channel publishes no aarch64 build. Measured for the pinned version: | channel | files published | usable on aarch64 | | --- | --- | --- | | CUDA-suffixed | 2 | 0 | | plain nightly | 16 | 6 | So torchao now resolves from the plain channel, which serves every architecture, while PyTorch keeps its CUDA channel. Nothing in the wheel links or bundles torchao, which is a quantization-workflow dependency of the examples and tests, and a CUDA wheel built by hand on an aarch64 GPU machine used the non-CUDA torchao build without issue. Windows is recorded as an explicit gap rather than left unmentioned. The matrix could express a Windows CUDA row and PyTorch publishes Windows CUDA wheels, but the separate libraries this wheel exists to ship are Linux only today, so a Windows CUDA wheel would carry a delegate a C++ application still could not link. Splitting the libraries there is the prerequisite, and the filter says so. ## Before and after ``` BEFORE AFTER pip install executorch pip install executorch CPU only, always CPU by default GPU wheels published per CUDA version GPU support means cloning the a published wheel carries the delegate repository and building a delegate built for CUDA 13.2 every consumer CUDA version has a has no ExecuTorch wheel to pair with matching ExecuTorch wheel ``` ## Test plan A smoke test for the CUDA rows, checking what an artifact can be held to on a builder with no GPU: - the CUDA libraries are in the wheel, so a row named for CUDA cannot ship without a delegate - the CUDA runtime is declared, so a user has something to resolve it from - each CUDA library reaches its runtime through a relative path, not the builder's toolkit directory, which resolves on the builder and nowhere else - the wheel carries device code for every architecture its row claims, read from the built libraries rather than assumed. Searched across every shipped library, because the kernels are compiled into their own library rather than into the delegate, and which one holds them is an internal detail. This caught a real case: a wheel built before the architecture list existed carried code for nothing the row promised, and it installed and loaded cleanly, so every other check passed on it. Then everything a CPU wheel is already held to: one owner per component, no build-tree search paths, and a C++ application outside the wheel able to link what it ships. Each check was run against a wheel that should fail it as well as one that should pass, because a check that cannot fail is worse than no check. All three reject a CPU wheel and accept a CUDA wheel built from source. One did not fail at first: it looped over libraries that were not there and reported a pass having inspected nothing, so it now requires at least one to be present. The aarch64 workflow passes the architecture the shared build workflow needs. Without it that workflow prepares an x86_64 job and skips the aarch64 conda install, so the first build step fails on a missing conda. The rows keep whichever builder image the release provides. Substituting a plain manylinux image broke the shared build setup, which expects conda to be present in that image. The matrix generator accepts "enable" and "disable" for its build-type inputs. Anything else is treated as not enabled, so a plausible-looking "enabled" produced an empty matrix and a failed run rather than a clear error. The checks now assert the accepted spelling. The filter was exercised against a matrix shaped like the generator's output: 36 rows narrowed to 24, a pull request narrowed to 1, the aarch64 rows given the correct builder image, and an empty result treated as a failure rather than passed through, since a workflow with no build job reads as a green check for a build that never happened. ghstack-source-id: 32d7a2a ghstack-comment-id: 5220374521 Pull-Request: #21668

## The problem The wheel can carry the CUDA delegate, but nothing builds one: there is no CUDA row in any workflow, so a GPU user still has to build from source. ## The change Add the workflows that build and publish CUDA wheels for Linux x86_64 and aarch64, and a smoke test that checks each wheel from the artifact itself. The build machines for these rows have no GPU, so the smoke test does not execute a model; it verifies the CUDA libraries are present, that the declared runtime matches the wheel's CUDA version, that nothing resolves through the build machine's toolkit, and that the shipped device code covers every GPU architecture the row claims. ``` executorch-1.5.0-cp312-cp312-manylinux_2_28_x86_64.whl +cu130 ``` A release publishes CUDA 12.6, 13.0 and 13.2, for Python 3.10 through 3.13. A pull request builds a single row instead of all twelve, because a full matrix costs hours for little extra signal. Which GPU architectures each row compiles for is chosen per row rather than detected on the builder. Detecting it would produce a wheel carrying device code for whatever machine happened to build it, which installs fine and then fails at the first GPU call. Two guards keep a release honest: - if the shared matrix generator stops offering a combination this policy advertises, the step fails instead of quietly publishing fewer wheels. A missing job is otherwise a green check for a wheel that was never built. - if a row reaches the architecture list with no CUDA version, the build refuses rather than falling back to the builder's GPU. Windows CUDA is deliberately absent. The separate shared libraries this wheel exists to ship are Linux only today, so a Windows CUDA wheel would carry a delegate a C++ application still could not link. ## Test plan - built the full release matrix, twelve wheels, and confirmed each one's contents match the row it claims: the CUDA libraries present, the CUDA runtime declared, and device code for every GPU architecture the row advertises. - ran a GPU model end to end from a CI-built wheel on three NVIDIA GPUs covering three device architectures, with output identical to eager PyTorch on each (largest absolute difference 0), and inspected the wheel for a fourth device it cannot execute on. - ran the matrix filter over generated inputs, including incomplete and malformed ones, and confirmed it refuses rather than publishing a partial release: a missing CUDA version, a missing python, or a python present on rows this policy does not build are each reported by name. - confirmed a CPU row still produces a CPU wheel on a builder that happens to have a CUDA toolkit installed. - the newest architecture also ships in its portable form, so a GPU newer than any in the row can still run by having the driver compile it at load time. Checked with `cuobjdump --list-ptx`: measured on two real libraries built with and without that form, `--list-elf` prints identical output either way, so the previous check could not see this at all and a lowercase `+ptx` typo would have shipped green. - every library that carries GPU device code covers the whole row on its own. The coverage was unioned across libraries, so one library with kernels could cover part of the row while an unrelated object supplied the rest, leaving no executable kernel on such a GPU. - the declared CUDA packages are compared against the exact set for the wheel's train rather than against a name suffix. For CUDA 13 the expected suffix is empty and every name ends with that, so a `cu130` wheel declaring `nvidia-cuda-runtime-cu11` was accepted; a version pin also made a correct name fail. Checked 10 combinations of train, pin and environment marker. - the python axis is an allowlist, matching the CUDA axis. Testing only the disabled list let any python not on it through: a 3.9 row was emitted successfully. - `install_utils.py` is in both CUDA workflows' path filters. It owns the supported CUDA train list and the toolkit detection, so a change there previously ran no CUDA wheel job. - requesting the JetPack rows fails with its own reason instead of the generic empty-matrix message, since both of its lists are deliberately empty and no workflow asks for them. - torchao keeps its CUDA channel where that channel exists. Falling back to the plain nightly index was needed only on aarch64, where the CUDA channel publishes nothing, and doing it everywhere changed which torchao an x86_64 install resolves. ghstack-source-id: aced89e ghstack-comment-id: 5220374521 Pull-Request: #21668

shoumikhin requested review from abhinaykukkadapu, digantdesai, kirklandsign, larryliu0820, psiddh and rascani as code owners August 7, 2026 17:55

shoumikhin had a problem deploying to cadence August 7, 2026 17:56 — with GitHub Actions Error

This was referenced Aug 7, 2026

Split the runtime out of the Python bindings extension #21610

Open

Ship a C++ SDK in the wheel #21639

Open

Ship the quantized kernels as their own library #21642

Open

Ship the CUDA delegate in the wheel #21645

Open

meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 7, 2026

shoumikhin had a problem deploying to cadence August 7, 2026 17:59 — with GitHub Actions Error

shoumikhin had a problem deploying to cadence August 7, 2026 18:22 — with GitHub Actions Error

shoumikhin temporarily deployed to cadence August 7, 2026 18:22 — with GitHub Actions Inactive

shoumikhin had a problem deploying to cadence August 7, 2026 18:58 — with GitHub Actions Error

shoumikhin had a problem deploying to cadence August 7, 2026 19:01 — with GitHub Actions Error

shoumikhin had a problem deploying to cadence August 7, 2026 19:16 — with GitHub Actions Error

shoumikhin had a problem deploying to cadence August 7, 2026 21:24 — with GitHub Actions Error

shoumikhin had a problem deploying to cadence August 7, 2026 21:41 — with GitHub Actions Error

shoumikhin had a problem deploying to cadence August 7, 2026 21:45 — with GitHub Actions Error

shoumikhin temporarily deployed to cadence August 7, 2026 21:45 — with GitHub Actions Inactive

shoumikhin temporarily deployed to cadence August 7, 2026 22:12 — with GitHub Actions Inactive

shoumikhin had a problem deploying to cadence August 7, 2026 22:41 — with GitHub Actions Error

shoumikhin had a problem deploying to cadence August 7, 2026 22:52 — with GitHub Actions Error

shoumikhin had a problem deploying to cadence August 7, 2026 23:05 — with GitHub Actions Error

shoumikhin temporarily deployed to cadence August 7, 2026 23:27 — with GitHub Actions Inactive

shoumikhin temporarily deployed to cadence August 7, 2026 23:36 — with GitHub Actions Inactive

shoumikhin had a problem deploying to cadence August 7, 2026 23:44 — with GitHub Actions Error

shoumikhin temporarily deployed to cadence August 7, 2026 23:44 — with GitHub Actions Inactive

Navigation Menu

Uh oh!

Build and publish CUDA wheels - #21668

Build and publish CUDA wheels#21668
shoumikhin wants to merge 1 commit into
gh/shoumikhin/95/headfrom
gh/shoumikhin/96/head

shoumikhin commented Aug 7, 2026 •

edited

Loading

Uh oh!

shoumikhin commented Aug 7, 2026 •

edited

Loading

Uh oh!

pytorch-bot Bot commented Aug 7, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

shoumikhin commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The problem

The change

Test plan

Uh oh!

shoumikhin commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot Bot commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21668

⏳ No Failures, 319 Pending

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

shoumikhin commented Aug 7, 2026 •

edited

Loading

shoumikhin commented Aug 7, 2026 •

edited

Loading

pytorch-bot Bot commented Aug 7, 2026 •

edited

Loading