-
Notifications
You must be signed in to change notification settings - Fork 3.6k
[QNN EP] Adjust tolerance for Clip and Transpose tests due to FP16 default in QNN HTP #26499
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
|
@microsoft-github-policy-service agree company="Qualcomm" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR adjusts tolerance thresholds for Clip and Transpose operator tests in the QNN HTP backend to accommodate accuracy differences introduced by the default FP16 precision in QNN HTP 2.35+. The precision change affects FP32 models where internal computations now use FP16 (FP32→FP16→FP32 conversions), introducing expected minor accuracy loss.
Key Changes:
- Removes deprecated
enable_fp16_precisionprovider option handling from test helper functions - Increases tolerance from 1e-5f to 5e-3f for affected float32 tests
- Re-enables previously disabled tests that were failing due to the precision change
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| onnxruntime/test/providers/qnn/transpose_htp_test.cc | Refactored RunTransposeNonQDQOnHTP to accept tolerance parameter; re-enabled and renamed TransposeFloat32OnHTP test with adjusted 5e-3f tolerance |
| onnxruntime/test/providers/qnn/clip_op_test.cc | Refactored RunClipTest to accept tolerance parameter; re-enabled Clip_f32 test with 5e-3f tolerance; cleaned up GPU test call to use default parameters |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
/azp run Linux QNN CI Pipeline, Win_TRT_Minimal_CUDA_Test_CI, Windows ARM64 QNN CI Pipeline, Windows GPU CUDA CI Pipeline, Windows GPU DML CI Pipeline, Windows GPU Doc Gen CI Pipeline, Windows GPU TensorRT CI Pipeline, Windows OpenVINO CI Pipeline, Windows x64 QNN CI Pipeline |
|
Azure Pipelines successfully started running 4 pipeline(s). |
|
/azp run Linux QNN CI Pipeline, Win_TRT_Minimal_CUDA_Test_CI, Windows ARM64 QNN CI Pipeline, Windows GPU CUDA CI Pipeline, Windows GPU DML CI Pipeline, Windows GPU Doc Gen CI Pipeline, Windows GPU TensorRT CI Pipeline, Windows OpenVINO CI Pipeline, Windows x64 QNN CI Pipeline |
|
Azure Pipelines successfully started running 4 pipeline(s). |
|
closing and reopening to restart CI pipelines |
|
Please, address and/or comment on the Copilot review comments so we can get this moving. |
* Updated tolerance thresholds to align with FP16 becoming the default precision in QNN HTP.
bc52f0a to
6e01c0e
Compare
|
Hi @yuslepukhin |
|
/azp run Windows CPU CI Pipeline, Windows GPU CI Pipeline, Windows GPU TensorRT CI Pipeline, ONNX Runtime Web CI Pipeline, Windows ARM64 QNN CI Pipeline, Windows x64 QNN CI Pipeline, onnxruntime-python-checks-ci-pipeline |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run Linux QNN CI Pipeline, Win_TRT_Minimal_CUDA_Test_CI, Windows ARM64 QNN CI Pipeline, Windows GPU CUDA CI Pipeline, Windows GPU DML CI Pipeline, Windows GPU Doc Gen CI Pipeline, Windows GPU TensorRT CI Pipeline, Windows OpenVINO CI Pipeline, Windows x64 QNN CI Pipeline |
|
Azure Pipelines successfully started running 4 pipeline(s). |
|
/azp run Linux QNN CI Pipeline, Win_TRT_Minimal_CUDA_Test_CI,Windows ARM64 QNN CI Pipeline,Windows GPU Doc Gen CI Pipeline |
|
Azure Pipelines successfully started running 4 pipeline(s). |
|
You are force pushing which makes it impossible to focus on your recent changes. Which in turn makes it hard to find out why CI pipelines are failing. Any particular reason why you do not want to preserve the history at review time? |
|
Hi @yuslepukhin, |
Description
This PR updates the tolerance thresholds for the Clip and Transpose tests in QnnHTPBackendTests. The adjustment accounts for minor accuracy differences introduced by the change in default floating-point precision in QNN HTP starting from version 2.35.
Motivation and Context
Since QNN 2.35, the default floating-point precision in QNN HTP has changed from FP32 to FP16. Additionally, the configuration option
QNN_HTP_GRAPH_CONFIG_OPTION_PRECISIONhas been deprecated.This change in precision can lead to expected accuracy loss, especially in scenarios where graph inputs and outputs are defined as float 32, but internal computations are performed in FP16 (e.g., FP32 → FP16 → FP32 conversions). To accommodate this, the tolerance thresholds for the affected tests have been increased to prevent false negatives due to precision differences.
@microsoft-github-policy-service agree company="Qualcomm"