Skip to content

Commit

Permalink
ci: Modify test jobs to improve reliability
Browse files Browse the repository at this point in the history
This commit is meant to do the following:

* Modify test-application-output job to test blinky
application only as it can be used as a smoke test.

* Test ML inference output for both keyword-detection and
speech-recognition applications as part of the OTA test job
which is now renamed to test_applications as it is testing
OTA update and ML inference outputs as well.

Note:
* The mentioned changes would result in a better reliability
and better CI performance.

Signed-off-by: Ahmed Ismail <[email protected]>
  • Loading branch information
AhmedIsmail02 committed Feb 6, 2024
1 parent 1a2992b commit 4052abe
Show file tree
Hide file tree
Showing 14 changed files with 39 additions and 81 deletions.
58 changes: 14 additions & 44 deletions .gitlab-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -123,87 +123,57 @@ build-applications:
variables:
GIT_SUBMODULE_STRATEGY: recursive

test-applications-output:
test-blinky-output:
extends: .test_job
script:
- tar xf ${TARGET}_${APP}_${TOOLCHAIN}_${INFERENCE}_${AUDIO}_build.tar.gz
- export APP_UNDERSCORED=$(echo ${APP} | tr '-' '_')
- |
if [[ $AUDIO == "VSI" ]]; then
pytest -s tools/tests/test_application_output.py \
--audio-file-path "applications/${APP_UNDERSCORED}/resources/test.wav" \
--build-artefacts-path "build" \
--fvp $FVP_BIN \
--merged-elf-name "${APP}_merged.elf" \
--timeout-seconds 900 \
--pass-output-file "applications/${APP_UNDERSCORED}/tests/pass_output.log" \
--fail-output-file "applications/${APP_UNDERSCORED}/tests/fail_output.log" \
--pythonhome-path "/opt/python/3.9.18"
else
pytest -s tools/tests/test_application_output.py \
pytest -s tools/tests/test_blinky_output.py \
--build-artefacts-path "build" \
--fvp $FVP_BIN \
--merged-elf-name "${APP}_merged.elf" \
--timeout-seconds 900 \
--pass-output-file "applications/${APP_UNDERSCORED}/tests/pass_output.log" \
--fail-output-file "applications/${APP_UNDERSCORED}/tests/fail_output.log"
fi
--pass-output-file "applications/${APP}/tests/pass_output.log" \
--fail-output-file "applications/${APP}/tests/fail_output.log"
parallel:
matrix:
-
<< : *pipeline_config_non_ml_applications
APP: [blinky]
-
<< : *pipeline_config_ml_applications
APP: [keyword-detection]
INFERENCE: [ETHOS, SOFTWARE]
-
<< : *pipeline_config_ml_applications
APP: [speech-recognition]
INFERENCE: [ETHOS]

artifacts:
paths:
- ${TARGET}_${APP}_${TOOLCHAIN}_${INFERENCE}_${AUDIO}_build.tar.gz
- ${TARGET}_aws-iot-example_${TOOLCHAIN}_${INFERENCE}_${AUDIO}_build.tar.gz
expire_in: 1 week

# The test-ota job should wait for build-applications job to finish as
# test-ota job uses the output build artifacts from build-applications job.
# The test-ota job should also wait for test-applications-output job to finish
# to prevent conflicts when doing both jobs at the same time as they both
# use the same credentials for the same target platform and application.
test-ota:
# The test-applications job should wait for build-applications job to finish as
# test-applications job uses the output build artifacts from build-applications job.
test-applications:
extends: .test_job
needs:
- job: test-applications-output
- job: build-applications
artifacts: true
script:
- tar xf ${TARGET}_${APP}_${TOOLCHAIN}_${INFERENCE}_${AUDIO}_build.tar.gz
- export APP_UNDERSCORED=$(echo ${APP} | tr '-' '_')
- |
if [[ $AUDIO == "VSI" ]]; then
pytest -s tools/tests/test_ota.py \
pytest -s tools/tests/test_applications.py \
--audio-file-path "applications/${APP_UNDERSCORED}/resources/test.wav" \
--build-artefacts-path "build" \
--fvp $FVP_BIN \
--credentials-path "applications/${APP_UNDERSCORED}/configs/aws_configs" \
--merged-elf-name "${APP}_merged.elf" \
--signed-update-bin-name "${APP}-update_signed.bin" \
--timeout-seconds 1800 \
--pass-ota-output-file "applications/${APP_UNDERSCORED}/tests/pass_ota_output.log" \
--fail-ota-output-file "applications/${APP_UNDERSCORED}/tests/fail_ota_output.log" \
--pass-output-file "applications/${APP_UNDERSCORED}/tests/pass_output.log" \
--fail-output-file "applications/${APP_UNDERSCORED}/tests/fail_output.log" \
--pythonhome-path "/opt/python/3.9.18"
else
pytest -s tools/tests/test_ota.py \
pytest -s tools/tests/test_applications.py \
--build-artefacts-path "build" \
--fvp $FVP_BIN \
--credentials-path "applications/${APP_UNDERSCORED}/configs/aws_configs" \
--merged-elf-name "${APP}_merged.elf" \
--signed-update-bin-name "${APP}-update_signed.bin" \
--timeout-seconds 1800 \
--pass-ota-output-file "applications/${APP_UNDERSCORED}/tests/pass_ota_output.log" \
--fail-ota-output-file "applications/${APP_UNDERSCORED}/tests/fail_ota_output.log"
--pass-output-file "applications/${APP_UNDERSCORED}/tests/pass_output.log" \
--fail-output-file "applications/${APP_UNDERSCORED}/tests/fail_output.log"
fi
parallel:
matrix:
Expand Down
3 changes: 0 additions & 3 deletions applications/keyword_detection/tests/fail_ota_output.log

This file was deleted.

3 changes: 3 additions & 0 deletions applications/keyword_detection/tests/fail_output.log
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Failed to provision device private key
Failed job document content check
Failed to execute state transition handler
2 changes: 0 additions & 2 deletions applications/keyword_detection/tests/pass_ota_output.log

This file was deleted.

4 changes: 3 additions & 1 deletion applications/keyword_detection/tests/pass_output.log
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
Starting bootloader
Waiting for provisioning bundle
Running provisioning bundle
Application version from appFirmwareVersion 0.0.10
Starting bootloader
Booting TF-M v2.0.0
PSA Framework version is: 257
Application version from appFirmwareVersion 0.0.20
ML interface initialised
ML_HEARD_ON
ML UNKNOWN
Expand Down
3 changes: 0 additions & 3 deletions applications/speech_recognition/tests/fail_ota_output.log

This file was deleted.

3 changes: 3 additions & 0 deletions applications/speech_recognition/tests/fail_output.log
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Failed to provision device private key
Failed job document content check
Failed to execute state transition handler
2 changes: 0 additions & 2 deletions applications/speech_recognition/tests/pass_ota_output.log

This file was deleted.

4 changes: 3 additions & 1 deletion applications/speech_recognition/tests/pass_output.log
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
Starting bootloader
Waiting for provisioning bundle
Running provisioning bundle
Application version from appFirmwareVersion 0.0.10
Starting bootloader
Booting TF-M v2.0.0
PSA Framework version is: 257
Application version from appFirmwareVersion 0.0.20
Init speex
ML interface initialised
Complete recognition: turn down the temperature in the bedroom
12 changes: 0 additions & 12 deletions tools/tests/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,6 @@ def pytest_addoption(parser):
parser.addoption("--timeout-seconds", type=int, action="store", default=1800)
parser.addoption("--pass-output-file", action="store", default="")
parser.addoption("--fail-output-file", action="store", default="")
parser.addoption("--pass-ota-output-file", action="store", default="")
parser.addoption("--fail-ota-output-file", action="store", default="")
parser.addoption(
"--pythonhome-path",
type=str,
Expand Down Expand Up @@ -111,16 +109,6 @@ def fail_output_file(pytestconfig):
yield get_absolute_path(pytestconfig.getoption("--fail-output-file"))


@pytest.fixture
def pass_ota_output_file(pytestconfig):
yield get_absolute_path(pytestconfig.getoption("--pass-ota-output-file"))


@pytest.fixture
def fail_ota_output_file(pytestconfig):
yield get_absolute_path(pytestconfig.getoption("--fail-ota-output-file"))


@pytest.fixture
def pythonhome_path(pytestconfig):
yield pytestconfig.getoption("--pythonhome-path")
Expand Down
18 changes: 9 additions & 9 deletions tools/tests/test_ota.py → tools/tests/test_applications.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright 2023 Arm Limited and/or its affiliates
# Copyright 2023-2024 Arm Limited and/or its affiliates
# <[email protected]>
# SPDX-License-Identifier: MIT

Expand Down Expand Up @@ -27,30 +27,30 @@ def setup_resources(build_artefacts_path, credentials_path, signed_update_bin_na
cleanup_aws_resources(flags)


def test_ota(
def test_applications(
setup_resources,
fvp_process: subprocess.Popen,
pass_ota_output_file: str,
fail_ota_output_file: str,
pass_output_file: str,
fail_output_file: str,
timeout_seconds: int,
) -> None:
"""
Compare the actual output on the FVP with the expectations in
pass and fail output files for the OTA update test.
pass and fail output files for the applications.
setup_resources: Input coming out as a result of executing of setup_resources
function defined above.
fvp_process (subprocess.Popen): FVP execution process
pass_ota_output_file (str): Path to the file containing the output when application
pass_output_file (str): Path to the file containing the output when application
runs without errors.
fail_ota_output_file (str): Path to the file containing the output when application
fail_output_file (str): Path to the file containing the output when application
runs with errors.
timeout_seconds (int): Timeout in seconds to wait before terminating the test.
"""
with open(pass_ota_output_file, "r") as file:
with open(pass_output_file, "r") as file:
pass_output = file.readlines()
pass_output = [line.replace("\n", "") for line in pass_output]
with open(fail_ota_output_file, "r") as file:
with open(fail_output_file, "r") as file:
fail_output = file.readlines()
fail_output = [line.replace("\n", "") for line in fail_output]

Expand Down
Original file line number Diff line number Diff line change
@@ -1,20 +1,20 @@
# Copyright 2023 Arm Limited and/or its affiliates
# Copyright 2023-2024 Arm Limited and/or its affiliates
# <[email protected]>
# SPDX-License-Identifier: MIT

import subprocess
from timeit import default_timer as timer


def test_application_output(
def test_blinky_output(
fvp_process: subprocess.Popen,
pass_output_file: str,
fail_output_file: str,
timeout_seconds: int,
) -> None:
"""
Compare the actual output on the FVP with the expectations in
pass and fail output files.
Compare the actual output of running blinky application on
the FVP with the expectations in pass and fail output files.
fvp_process (subprocess.Popen): FVP execution process
pass_output_file (str): Path to the file containing the output when application
Expand Down

0 comments on commit 4052abe

Please sign in to comment.