Arm backend: Add MobileSAM Ethos-U example (#21397)

Add a MobileSAM prompt segmentation example for Ethos-U85-256.

Prepare the pinned official source in an external managed checkout.

Apply a small configurable-input patch instead of copying model code.

Use int8 globally and A16W8 for TinyViT attention.

Lower the complete graph to Ethos-U and compare output with FP32.

Emit delegation summaries, validation metrics, and debug artifacts.

Build a bare-metal Corstone-320 app using exporter metadata.

Validate inference on FVP through the Arm OOTB test.


cc @digantdesai @freddan80 @per @zingo @oscarandersson8218 @mansnils
@Sebastian-Larsson @robell @rascani

Signed-off-by: Usamah Zaheer <usamah.zaheer@arm.com>

Author: usamahz

backends/arm/scripts/docgen/ethos-u/backends-arm-ethos-u-overview.md.in

@@ -67,6 +67,11 @@ how to run `model_export/export_deit.py`, build the sample firmware, and convert
 test images into C arrays so the workflow described in this guide can be tried
 end to end.
 
+[`examples/arm/mobilesam_prompt_segmentation_example_ethos_u`](https://github.com/pytorch/executorch/tree/main/examples/arm/mobilesam_prompt_segmentation_example_ethos_u)
+contains a complete MobileSAM prompt segmentation workflow for Ethos-U85,
+including fixed-prompt export, PT2E quantization, transformer lowering, debug
+masks and overlays, a bare-metal Corstone-320 runtime app, and FVP execution.
+
 ### Ethos-U memory modes
 
 The Ethos-U NPU provides two distinct memory interfaces:

backends/arm/scripts/run_fvp.sh

@@ -24,6 +24,8 @@ target="ethos-u55-128"
 timeout="600"
 etrecord_file=""
 trace_file=""
+semihosting_cwd=""
+ethosu_fast=0
 
 help() {
     echo "Usage: $(basename $0) [options]"
@@ -35,6 +37,8 @@ help() {
     echo "  --timeout=<TIME_IN_SEC>  Maximum target runtime, used to detect hanging, might need to be higer on large models Default: ${timeout}"
     echo "  --etrecord=<FILE>        If ETDump is used you can supply a ETRecord file matching the PTE"
     echo "  --trace_file=<FILE>      File to write PMU trace output to"
+    echo "  --semihosting-cwd=<DIR>  Enable target semihosting with this host working directory"
+    echo "  --fast                   Use fast Ethos-U model simulation for Ethos-U targets"
     exit 0
 }
 
@@ -48,6 +52,8 @@ for arg in "$@"; do
       --timeout=*) timeout="${arg#*=}";;
       --etrecord=*) etrecord_file="${arg#*=}";;
       --trace_file=*) trace_file="${arg#*=}";;
+      --semihosting-cwd=*) semihosting_cwd="${arg#*=}";;
+      --fast) ethosu_fast=1;;
       *)
       ;;
     esac
@@ -104,9 +110,34 @@ log_file=$(mktemp)
 extra_args_u55=()
 extra_args_u85=()
 
+ethosu_extra_args=()
+if [[ "${ethosu_fast}" == 1 ]]; then
+    ethosu_extra_args+=("--fast")
+fi
 if [[ -n "${trace_file}" ]]; then
-    extra_args_u55+=(-C "ethosu.extra_args=--pmu-trace ${trace_file}")
-    extra_args_u85+=(-C "mps4_board.subsystem.ethosu.extra_args=--pmu-trace ${trace_file}")
+    ethosu_extra_args+=("--pmu-trace" "${trace_file}")
+fi
+if [[ ${#ethosu_extra_args[@]} -gt 0 ]]; then
+    extra_args_u55+=(-C "ethosu.extra_args=${ethosu_extra_args[*]}")
+    extra_args_u85+=(-C "mps4_board.subsystem.ethosu.extra_args=${ethosu_extra_args[*]}")
+fi
+
+semihosting_args_u55=()
+semihosting_args_u85=()
+if [[ -n "${semihosting_cwd}" ]]; then
+    semihosting_cwd=$(realpath "${semihosting_cwd}")
+    semihosting_args_u55+=(
+        -C cpu0.semihosting-enable=1
+        -C cpu0.semihosting-stack_base=0
+        -C cpu0.semihosting-heap_limit=0
+        -C "cpu0.semihosting-cwd=${semihosting_cwd}"
+    )
+    semihosting_args_u85+=(
+        -C mps4_board.subsystem.cpu0.semihosting-enable=1
+        -C mps4_board.subsystem.cpu0.semihosting-stack_base=0
+        -C mps4_board.subsystem.cpu0.semihosting-heap_limit=0
+        -C "mps4_board.subsystem.cpu0.semihosting-cwd=${semihosting_cwd}"
+    )
 fi
 
 if [[ ${target} == cortex-m* ]]; then
@@ -154,6 +185,7 @@ elif [[ ${target} == *"ethos-u55"* || ${target} == *"ethos-u65"* ]]; then
         -C mps3_board.uart0.out_file='-'                    \
         -C mps3_board.uart0.shutdown_on_eot=1               \
         ${extra_args_u55[@]+"${extra_args_u55[@]}"}         \
+        ${semihosting_args_u55[@]+"${semihosting_args_u55[@]}"} \
         -a "${elf_file}"                                    \
         ${data_file}                                        \
         --timelimit ${timeout} 2>&1 | sed 's/\r$//' | tee ${log_file} || true # seconds
@@ -165,8 +197,10 @@ elif [[ ${target} == *"ethos-u85"*  ]]; then
         -C vis_hdlcd.disable_visualisation=1                \
         -C mps4_board.telnetterminal0.start_telnet=0        \
         -C mps4_board.uart0.out_file='-'                    \
+        -C mps4_board.uart0.unbuffered_output=1             \
         -C mps4_board.uart0.shutdown_on_eot=1               \
         ${extra_args_u85[@]+"${extra_args_u85[@]}"}         \
+        ${semihosting_args_u85[@]+"${semihosting_args_u85[@]}"} \
         -a "${elf_file}"                                    \
         ${data_file}                                        \
         --timelimit ${timeout} 2>&1 | sed 's/\r$//' | tee ${log_file} || true # seconds
@@ -200,11 +234,21 @@ if [ $? != 0 ]; then
 fi
 
 echo "Checking for problems in log:"
-! grep -E "^(F|E|\\[critical\\]|Hard fault.|Info: Simulation is stopping. Reason: CPU time has been exceeded.).*$" ${log_file}
-if [ $? != 0 ]; then
+problem_log=$(mktemp)
+grep -E "^(F|E|\\[critical\\]|Hard fault.|Info: Simulation is stopping. Reason: CPU time has been exceeded.).*$" "${log_file}" > "${problem_log}" || true
+if [[ "${ethosu_fast}" == 1 ]]; then
+    filtered_problem_log=$(mktemp)
+    timing_adapter_fast_mode_regex="Failed to initialize timing-adapter #[0-9]+|Timing adapter has no effect if option --fast/-F is enabled"
+    grep -Ev "${timing_adapter_fast_mode_regex}" "${problem_log}" > "${filtered_problem_log}" || true
+    mv "${filtered_problem_log}" "${problem_log}"
+fi
+if [[ -s "${problem_log}" ]]; then
+    cat "${problem_log}"
     echo "Found ERROR"
+    rm "${problem_log}"
     rm "${log_file}"
     exit 1
 fi
 echo "No problems found!"
+rm "${problem_log}"
 rm "${log_file}"

backends/arm/test/test_arm_ootb.sh

@@ -28,7 +28,7 @@ if [[ "$1" == "-h" || "$1" == "--help" ]]; then
 fi
 
 if [[ $# -eq 0 ]]; then
-    TEST_SUITES=(run_ootb_tests_ethos_u run_ootb_tests_tosa run_ootb_tests_vgf run_deit_e2e_ethos_u run_swin2sr_e2e_vgf)
+    TEST_SUITES=(run_ootb_tests_ethos_u run_ootb_tests_tosa run_ootb_tests_vgf run_deit_e2e_ethos_u run_mobilesam_e2e_ethos_u run_swin2sr_e2e_vgf)
 else
     TEST_SUITES=("$1")
 fi
@@ -164,6 +164,143 @@ run_deit_e2e_ethos_u() {
     echo "${FUNCNAME}: PASS"
 }
 
+run_mobilesam_e2e_ethos_u() {
+    echo "$FUNCNAME: Export, build, and run the MobileSAM e2e test"
+
+    local example_dir="${et_root_dir}/examples/arm/mobilesam_prompt_segmentation_example_ethos_u"
+    local work_root="${et_root_dir}/arm_test/mobilesam_ootb_smoke"
+    local export_dir="${work_root}/export"
+    local artifact_dir="${work_root}/artifacts"
+    local debug_dir="${work_root}/debug"
+    local et_build_dir="${work_root}/cmake-out-arm"
+    local quantized_aot_build_dir="${work_root}/quantized_ops_aot"
+    local build_dir="${work_root}/runtime"
+    local mobile_sam_source="${work_root}/mobile_sam/source"
+    local image_path="${et_root_dir}/examples/models/dinov2/dog.jpg"
+    local pte_path="${export_dir}/mobilesam_prompt_smoke.pte"
+    local metadata_path="${export_dir}/mobilesam_prompt_smoke.json"
+    local fvp_log="${work_root}/fvp.log"
+    local toolchain_file="${et_root_dir}/examples/arm/ethos-u-setup/arm-none-eabi-gcc.cmake"
+    local input_size=448
+    local fvp_timelimit="${FVP_TIMELIMIT:-300}"
+    echo "${FUNCNAME}: Work directory: ${work_root}; existing artifacts will be reused if present"
+
+    mkdir -p "${export_dir}" "${artifact_dir}" "${debug_dir}" "${build_dir}"
+
+    setup_path_script=${et_root_dir}/examples/arm/arm-scratch/setup_path.sh
+    source ${setup_path_script}
+
+    source ${et_root_dir}/backends/arm/scripts/utils.sh
+    local n_proc="$(get_parallel_jobs)"
+
+    echo "${FUNCNAME}: Building ExecuTorch (if needed)"
+    cmake --preset arm-baremetal -B "${et_build_dir}"
+    cmake --build "${et_build_dir}" --target install -j"$n_proc"
+
+    echo "${FUNCNAME}: Building host quantized AOT library"
+    local python_executable
+    python_executable="$(python3 -c 'import sys; print(sys.executable)')"
+    cmake \
+        -S "${et_root_dir}" \
+        -B "${quantized_aot_build_dir}" \
+        -DCMAKE_BUILD_TYPE=Release \
+        -DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
+        -DEXECUTORCH_BUILD_KERNELS_QUANTIZED_AOT=ON \
+        -DEXECUTORCH_BUILD_XNNPACK=OFF \
+        -DPYTHON_EXECUTABLE="${python_executable}"
+    cmake --build "${quantized_aot_build_dir}" --target quantized_ops_aot_lib -j"$n_proc"
+
+    local quantized_ops_library
+    quantized_ops_library="$(
+        find "${quantized_aot_build_dir}/kernels/quantized" \
+            -name 'libquantized_ops_aot_lib.*' \
+            -type f \
+            -print \
+            -quit
+    )"
+    [[ -n "${quantized_ops_library}" ]] || {
+        echo "${FUNCNAME}: Missing quantized AOT library under ${quantized_aot_build_dir}"
+        return 1
+    }
+
+    echo "${FUNCNAME}: Installing example requirements"
+    pip install -r "${example_dir}/requirements.txt"
+
+    echo "${FUNCNAME}: Preparing pinned MobileSAM source"
+    python3 "${example_dir}/model_export/prepare_mobilesam.py" \
+        --source-dir "${mobile_sam_source}"
+
+    echo "${FUNCNAME}: Exporting quantized MobileSAM PTE"
+    env EXECUTORCH_QUANTIZED_OPS_AOT_LIBRARY="${quantized_ops_library}" \
+        python3 "${example_dir}/model_export/export_mobilesam.py" \
+        --output-path "${pte_path}" \
+        --calibration-image "${image_path}" \
+        --eval-image "${image_path}" \
+        --mobile-sam-source "${mobile_sam_source}" \
+        --input-size "${input_size}" \
+        --point 219 193 \
+        --num-calibration-samples 1 \
+        --num-eval-samples 1 \
+        --num-debug-samples 1 \
+        --minimum-fp32-quantized-iou 0.9 \
+        --artifact-dir "${artifact_dir}" \
+        --debug-output-dir "${debug_dir}"
+
+    for artifact in \
+        "${pte_path}" \
+        "${metadata_path}" \
+        "${export_dir}/mobilesam_prompt_smoke_delegation.txt" \
+        "${export_dir}/mobilesam_prompt_smoke_metrics.json"; do
+        [[ -f "${artifact}" ]] || {
+            echo "${FUNCNAME}: Missing export artifact ${artifact}"
+            return 1
+        }
+    done
+
+    echo "${FUNCNAME}: Configuring the MobileSAM application"
+    cmake \
+        -U "LIB_*" \
+        -U executorch_DIR \
+        -S "${example_dir}/runtime" \
+        -B "${build_dir}" \
+        -DCMAKE_TOOLCHAIN_FILE="${toolchain_file}" \
+        -DET_PTE_FILE_PATH="${pte_path}" \
+        -DMODEL_METADATA_PATH="${metadata_path}" \
+        -DIMAGE_PATH="${image_path}" \
+        -DMASK_THRESHOLD=0.0 \
+        -DET_SEGMENTATION_DUMP_MASK=ON \
+        -DPYTHON_EXECUTABLE="${python_executable}" \
+        -DET_BUILD_DIR_PATH="${et_build_dir}"
+
+    echo "${FUNCNAME}: Building mobilesam_prompt_segmentation_example"
+    cmake --build "${build_dir}" -j"$n_proc" --target mobilesam_prompt_segmentation_example
+
+    local elf="${build_dir}/mobilesam_prompt_segmentation_example"
+
+    echo "${FUNCNAME}: Running on FVP"
+    backends/arm/scripts/run_fvp.sh \
+        --elf="${elf}" \
+        --target=ethos-u85-256 \
+        --timeout="${fvp_timelimit}" \
+        --semihosting-cwd="${build_dir}" \
+        --fast | tee "${fvp_log}"
+
+    grep -q "Model executed successfully." "${fvp_log}" || {
+        echo "${FUNCNAME}: FVP run did not report successful execution"
+        return 1
+    }
+
+    python3 "${example_dir}/runtime/visualize_fvp_output.py" \
+        --fvp-log "${fvp_log}" \
+        --input-image "${image_path}" \
+        --metadata "${metadata_path}" \
+        --reference-mask "${debug_dir}/dog/quantized_mask.png" \
+        --minimum-iou 0.9 \
+        --output-dir "${work_root}/fvp_visual"
+
+    echo "${FUNCNAME}: PASS"
+}
+
 run_swin2sr_e2e_vgf() {
     echo "$FUNCNAME: Prepare demo assets, export FP/INT8, build, and run the Swin2SR VGF e2e test"
 

docs/source/backends/arm-ethos-u/arm-ethos-u-overview.md

@@ -121,6 +121,11 @@ how to run `model_export/export_deit.py`, build the sample firmware, and convert
 test images into C arrays so the workflow described in this guide can be tried
 end to end.
 
+[`examples/arm/mobilesam_prompt_segmentation_example_ethos_u`](https://github.com/pytorch/executorch/tree/main/examples/arm/mobilesam_prompt_segmentation_example_ethos_u)
+contains a complete MobileSAM prompt segmentation workflow for Ethos-U85,
+including fixed-prompt export, PT2E quantization, transformer lowering, debug
+masks and overlays, a bare-metal Corstone-320 runtime app, and FVP execution.
+
 ### Ethos-U memory modes
 
 The Ethos-U NPU provides two distinct memory interfaces:

examples/arm/README.md

@@ -77,7 +77,11 @@ For Cortex-M testing, use a Cortex-M target and bundled I/O:
   - End-to-end DEiT-Tiny image classification flow for Ethos-U, including
   model fine-tuning, export, bare-metal runtime build, and Corstone-320 FVP
   execution.
-- [image_classification_example_vgf](image_classification_example_vgf/) -
+ - [mobilesam_prompt_segmentation_example_ethos_u](mobilesam_prompt_segmentation_example_ethos_u/)
+   - End-to-end MobileSAM prompt segmentation flow for Ethos-U, including
+   export, quantization, debug mask generation, bare-metal runtime build, and
+   Corstone-320 FVP execution.
+ - [image_classification_example_vgf](image_classification_example_vgf/) -
   DEiT-Tiny image classification flow for VGF host execution.
 - [super_resolution_example_vgf](super_resolution_example_vgf) - Swin2SR image
   super-resolution.

examples/arm/mobilesam_prompt_segmentation_example_ethos_u/README.md

@@ -0,0 +1,43 @@
+# MobileSAM Prompt Segmentation Example Application
+
+This end-to-end example shows how to use the Arm Ethos-U backend in
+ExecuTorch for transformer-based prompt segmentation. MobileSAM predicts a
+binary mask for fixed positive point prompts rather than semantic class IDs.
+The host debug flow validates quantization by comparing FP32 and quantized
+masks, with an optional binary reference mask when one is available.
+
+It covers:
+
+- Loading the MobileSAM `vit_t` checkpoint.
+- Freezing one or more positive point prompts into the exported graph.
+- Applying post-training quantization with the Ethos-U quantizer.
+- Lowering the quantized model to an Ethos-U85-256 ExecuTorch program.
+- Producing validation and debugging artifacts such as masks, overlays,
+  mismatch heatmaps, metrics, and delegation summaries.
+- Building a bare-metal Corstone-320 runtime app and running it on FVP.
+
+The default export uses a reduced `448x448` image input and returns one
+low-resolution `[1, 1, 112, 112]` mask-logit tensor. The example prepares the
+official MobileSAM GitHub source at a pinned revision in an external checkout
+and applies a small configurable-input patch there. Neither the MobileSAM
+source nor checkpoint is redistributed in ExecuTorch.
+
+The export uses int8 activations and int8 weights globally, and A16W8
+quantization for TinyViT attention modules. This keeps the transformer
+attention numerically stable while still producing one Ethos-U delegate.
+
+The exported graph intentionally uses `multimask_output=False` and leaves
+mask thresholding outside the model. SAM-style candidate-mask selection can be
+numerically sensitive after export and quantization, so this example keeps the
+target graph focused on the fixed-prompt image encoder and mask decoder.
+
+## Layout
+
+- `model_export/prepare_mobilesam.py` - Prepares the pinned external MobileSAM
+  checkout and applies the configurable-input patch.
+- `model_export/README.md` - Model loading, quantization, lowering,
+  validation, and debug artifact generation.
+- `runtime/README.md` - Bare-metal runtime build, image header generation, and
+  Corstone-320 FVP execution.
+- `runtime/visualize_fvp_output.py` - Decodes the target mask dump, creates an
+  overlay, and compares FVP output with the host quantized mask.