-
Notifications
You must be signed in to change notification settings - Fork 17
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
ci: Modify test jobs to improve reliability
This commit is meant to do the following: * Modify test-application-output job to test blinky application only as it can be used as a smoke test. * Test ML inference output for both keyword-detection and speech-recognition applications as part of the OTA test job which is now renamed to test_applications as it is testing OTA update and ML inference outputs as well. Note: * The mentioned changes would result in a better reliability and better CI performance. Signed-off-by: Ahmed Ismail <[email protected]>
- Loading branch information
1 parent
1a2992b
commit 4052abe
Showing
14 changed files
with
39 additions
and
81 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
File renamed without changes.
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
Failed to provision device private key | ||
Failed job document content check | ||
Failed to execute state transition handler |
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
Failed to provision device private key | ||
Failed job document content check | ||
Failed to execute state transition handler |
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,8 +1,10 @@ | ||
Starting bootloader | ||
Waiting for provisioning bundle | ||
Running provisioning bundle | ||
Application version from appFirmwareVersion 0.0.10 | ||
Starting bootloader | ||
Booting TF-M v2.0.0 | ||
PSA Framework version is: 257 | ||
Application version from appFirmwareVersion 0.0.20 | ||
Init speex | ||
ML interface initialised | ||
Complete recognition: turn down the temperature in the bedroom |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,4 @@ | ||
# Copyright 2023 Arm Limited and/or its affiliates | ||
# Copyright 2023-2024 Arm Limited and/or its affiliates | ||
# <[email protected]> | ||
# SPDX-License-Identifier: MIT | ||
|
||
|
@@ -27,30 +27,30 @@ def setup_resources(build_artefacts_path, credentials_path, signed_update_bin_na | |
cleanup_aws_resources(flags) | ||
|
||
|
||
def test_ota( | ||
def test_applications( | ||
setup_resources, | ||
fvp_process: subprocess.Popen, | ||
pass_ota_output_file: str, | ||
fail_ota_output_file: str, | ||
pass_output_file: str, | ||
fail_output_file: str, | ||
timeout_seconds: int, | ||
) -> None: | ||
""" | ||
Compare the actual output on the FVP with the expectations in | ||
pass and fail output files for the OTA update test. | ||
pass and fail output files for the applications. | ||
setup_resources: Input coming out as a result of executing of setup_resources | ||
function defined above. | ||
fvp_process (subprocess.Popen): FVP execution process | ||
pass_ota_output_file (str): Path to the file containing the output when application | ||
pass_output_file (str): Path to the file containing the output when application | ||
runs without errors. | ||
fail_ota_output_file (str): Path to the file containing the output when application | ||
fail_output_file (str): Path to the file containing the output when application | ||
runs with errors. | ||
timeout_seconds (int): Timeout in seconds to wait before terminating the test. | ||
""" | ||
with open(pass_ota_output_file, "r") as file: | ||
with open(pass_output_file, "r") as file: | ||
pass_output = file.readlines() | ||
pass_output = [line.replace("\n", "") for line in pass_output] | ||
with open(fail_ota_output_file, "r") as file: | ||
with open(fail_output_file, "r") as file: | ||
fail_output = file.readlines() | ||
fail_output = [line.replace("\n", "") for line in fail_output] | ||
|
||
|
8 changes: 4 additions & 4 deletions
8
tools/tests/test_application_output.py → tools/tests/test_blinky_output.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,20 +1,20 @@ | ||
# Copyright 2023 Arm Limited and/or its affiliates | ||
# Copyright 2023-2024 Arm Limited and/or its affiliates | ||
# <[email protected]> | ||
# SPDX-License-Identifier: MIT | ||
|
||
import subprocess | ||
from timeit import default_timer as timer | ||
|
||
|
||
def test_application_output( | ||
def test_blinky_output( | ||
fvp_process: subprocess.Popen, | ||
pass_output_file: str, | ||
fail_output_file: str, | ||
timeout_seconds: int, | ||
) -> None: | ||
""" | ||
Compare the actual output on the FVP with the expectations in | ||
pass and fail output files. | ||
Compare the actual output of running blinky application on | ||
the FVP with the expectations in pass and fail output files. | ||
fvp_process (subprocess.Popen): FVP execution process | ||
pass_output_file (str): Path to the file containing the output when application | ||
|