Arm backend: Add CMSIS Pack all-ops test and fix pack build infrastructure (#21709)

Reworks the CMSIS Pack build infrastructure for the 1.4 operator surface
and adds a consumer-side all-ops test that builds and runs every
operator the pack ships.

### Review order

Start with `scripts/`. `generate_register_all_kernels.py` is rewritten
to emit registrations from the NativeFunctions yaml rather than the
previous hand-maintained mapping, and operator discovery moves into the
new `op_guards.py` shared by the registration and component generators.
`copy_sources.sh` and `test/validate_pack.py` then hard-fail on a pack
missing the build-generated headers rather than producing a quietly
incomplete archive. `templates/PyTorch.ExecuTorch.pdsc.tpl` gains the
Extension Tensor component and per-op Cortex-M components.

Then `test/all_ops/`, which is the bulk of the diff and independent of
the above. `op_recipes.py` defines one recipe per operator;
`generate_test_models.py` turns each into a self-contained `.pte`
carrying its inputs, expected outputs and tolerances as constant
methods; `gen_cproject.py` generates a consumer project selecting every
operator component; `main.cpp` runs each model on target and prints a
per-op cycle table, a memory high-water report and a `SUMMARY
<pass>/<total>` line. `run.sh` drives the flow. Host-side unit tests
accompany each generator.

Coverage is two-tier, documented in `SKIPPED_OPS.md`: every operator is
build/link covered with no exemptions, and those not value-checked on
device each carry a machine-enforced reason (nondeterministic RNG,
uninitialized `empty`, data-dependent output shape), so a silent
coverage gap fails the test suite.

The last commit is a one-line, independent fix: the generated PDSC
`<url>` omitted the tag's leading `v`, so `releases/download/1.4.0/`
404s where `releases/download/v1.4.0/` serves the asset.

### Validation

Built from the 1.4.0 release sources with these scripts, the resulting
pack validates structurally, and a consumer project selecting all 198
operator components compiles and links clean (368 compile units, `1
succeeded, 0 failed`) for Corstone-315 with arm-none-eabi-gcc 14.3.1 and
CMSIS-Toolbox 2.14.1. Booting that image on `FVP_Corstone_SSE-315`
gives:

```
Test_result: SUMMARY 167/167 PASS
MemReport: method=84292 temp=0 meta=88 metatmp=0
```

The test targets two Cortex-M85 MPS4 subsystems: Corstone-315 (SSE-315)
for the CPU path and Corstone-320 (SSE-320) with an Ethos-U85 for a
delegated variant.


cc @digantdesai @freddan80 @per @zingo @oscarandersson8218 @mansnils
@Sebastian-Larsson @robell @rascani

---------

Signed-off-by: Matthias Hertel <matthias.hertel@arm.com>

Author: MatthiasHertelArm

.github/workflows/build-cmsis-pack.yml

@@ -22,7 +22,6 @@ on:
     paths:
       - .github/workflows/build-cmsis-pack.yml
       - backends/arm/cmsis_pack/**
-      - backends/arm/cmsis_pack/scripts/**
       - backends/arm/runtime/**
       - backends/cortex_m/**
       - kernels/portable/**
@@ -33,7 +32,6 @@ on:
     paths:
       - .github/workflows/build-cmsis-pack.yml
       - backends/arm/cmsis_pack/**
-      - backends/arm/cmsis_pack/scripts/**
   workflow_dispatch:
     inputs:
       version_override:
@@ -80,7 +78,9 @@ jobs:
         # Stage 1: Build core ExecuTorch with arm-none-eabi-gcc
         # This generates required headers (flatbuffers, schema)
         backends/arm/scripts/build_executorch.sh
-        CMAKE_BUILD_DIR="$(pwd)/cmake-out-arm"
+        # Must match build_executorch.sh's output dir (et_build_root/cmake-out);
+        # copy_sources.sh hard-fails if the generated schema headers are absent.
+        CMAKE_BUILD_DIR="$(pwd)/arm_test/cmake-out"
         echo "::endgroup::"
 
         echo "::group::Determine pack version"

backends/arm/cmsis_pack/contributions/add/Documentation/README.md

@@ -28,10 +28,12 @@ This pack provides:
      - component: Machine Learning:ExecuTorch:Runtime
      - component: Machine Learning:ExecuTorch:Kernel Utils
 
-     # Backend (choose one or more)
-     - component: Machine Learning:ExecuTorch:Backend::EthosU
+     # Backend (choose one or more). CMSIS component IDs use a single
+     # colon between Cclass / Cgroup / Csub, with a space (not "::")
+     # inside the Csub name.
+     - component: Machine Learning:ExecuTorch:Backend EthosU
      # or
-     - component: Machine Learning:ExecuTorch:Backend::CortexM
+     - component: Machine Learning:ExecuTorch:Backend CortexM
    ```
 
 3. Include ExecuTorch headers in your code:

backends/arm/cmsis_pack/scripts/build_pack.sh

@@ -107,18 +107,21 @@ cd "$PACK_BUILD"
 if command -v zip &> /dev/null; then
     zip -r "$PACK_FILE" . -x "*.DS_Store" -x ".git*" -x "*/generated/*" -x "*.py"
 else
-    python3 -c "
-import zipfile, os
-with zipfile.ZipFile('$PACK_FILE', 'w', zipfile.ZIP_DEFLATED) as zf:
+    # Pass $PACK_FILE as sys.argv[1] so the path is safe against
+    # shell-special characters (single quotes in particular).
+    python3 - "$PACK_FILE" <<'PYEOF'
+import sys, zipfile, os
+pack_file = sys.argv[1]
+with zipfile.ZipFile(pack_file, 'w', zipfile.ZIP_DEFLATED) as zf:
     for root, dirs, files in os.walk('.'):
         dirs[:] = [d for d in dirs if not d.startswith('.') and d != 'generated']
         for f in files:
             if f.startswith('.') or f == '.DS_Store' or f.endswith('.py'):
                 continue
             p = os.path.join(root, f)
             zf.write(p, os.path.relpath(p, '.'))
-print(f'Created: $PACK_FILE')
-"
+print(f'Created: {pack_file}')
+PYEOF
 fi
 
 echo ""

backends/arm/cmsis_pack/scripts/copy_sources.sh

@@ -14,7 +14,7 @@
 # Usage:
 #   ./copy_sources.sh --executorch-root <path> --build-dir <path> \
 #                     --pack-staging <path>
-set -e
+set -euo pipefail
 
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 PACK_DIR="$(cd "$SCRIPT_DIR/../../cmsis_pack" && pwd)"
@@ -115,7 +115,7 @@ cp -r "${PACK_SRC}/src/kernels" "${PACK_SRC}/include/executorch/"
 
 echo "  Copying extension sources..."
 mkdir -p "${PACK_SRC}/src/extension"
-for d in data_loader memory_allocator runner_util; do
+for d in data_loader memory_allocator runner_util tensor; do
     if [[ -d "${EXECUTORCH_ROOT}/extension/$d" ]]; then
         cp -r "${EXECUTORCH_ROOT}/extension/$d" "${PACK_SRC}/src/extension/"
     fi
@@ -138,6 +138,18 @@ for candidate in \
         break
     fi
 done
+# A pack without the generated flatbuffers schema headers compiles nothing:
+# every consumer fails at executorch/schema/program_generated.h. Fail here
+# rather than shipping a broken archive.
+for required in program_generated.h scalar_type_generated.h; do
+    if [[ ! -f "${PACK_SRC}/src/schema/${required}" ]]; then
+        echo "ERROR: ${required} not found in the CMake build tree." >&2
+        echo "  Looked in: ${BUILD_DIR}/schema/include/executorch/schema" >&2
+        echo "  Run the ExecuTorch CMake configure+build first so BUILD_DIR" >&2
+        echo "  contains the generated schema headers (see build_pack.sh -h)." >&2
+        exit 1
+    fi
+done
 cp -r "${PACK_SRC}/src/schema" "${PACK_SRC}/include/executorch/"
 
 echo "  Copying generated flatbuffers headers..."
@@ -150,14 +162,27 @@ for candidate in \
         break
     fi
 done
+# Same rule as the schema headers: the pack bundles flatbuffers/, and the
+# runtime does not compile without it. Refuse to produce a pack missing it.
+if [[ ! -f "${PACK_SRC}/include/flatbuffers/flatbuffers.h" ]]; then
+    echo "ERROR: flatbuffers headers not found in the CMake build tree." >&2
+    echo "  Looked in: ${BUILD_DIR}/third-party/{flatbuffers,flatc_ep}/include" >&2
+    echo "  Run the ExecuTorch CMake configure+build first so BUILD_DIR" >&2
+    echo "  contains the flatc external-project headers." >&2
+    exit 1
+fi
 
-# Flatbuffers headers are redistributed inside the pack; ship the
-# upstream Apache-2.0 LICENSE alongside per its terms.
-if [[ -f "${EXECUTORCH_ROOT}/third-party/flatbuffers/LICENSE" ]] \
-   && [[ -d "${PACK_SRC}/include/flatbuffers" ]]; then
-    cp "${EXECUTORCH_ROOT}/third-party/flatbuffers/LICENSE" \
-       "${PACK_SRC}/include/flatbuffers/LICENSE"
+# Flatbuffers headers are redistributed inside the pack, so shipping the
+# upstream Apache-2.0 LICENSE alongside them is a condition of that
+# redistribution -- a pack without it must not be published.
+if [[ ! -f "${EXECUTORCH_ROOT}/third-party/flatbuffers/LICENSE" ]]; then
+    echo "ERROR: third-party/flatbuffers/LICENSE not found in the source tree." >&2
+    echo "  The pack redistributes the flatbuffers headers and must ship" >&2
+    echo "  their licence. Run: git submodule update --init third-party/flatbuffers" >&2
+    exit 1
 fi
+cp "${EXECUTORCH_ROOT}/third-party/flatbuffers/LICENSE" \
+   "${PACK_SRC}/include/flatbuffers/LICENSE"
 
 echo "  Creating c10 and torch include paths..."
 C10_DIR="${PACK_SRC}/src/runtime/core/portable_type/c10/c10"