Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sharing my local test runner script for CI-like test app build and execution #35625

Merged
merged 13 commits into from
Sep 17, 2024

Conversation

andy31415
Copy link
Contributor

@andy31415 andy31415 commented Sep 17, 2024

This script generally makes it convenient to build and python-tests for local execution and debugging.

image

image

Copy link

Review changes with SemanticDiff.

@github-actions github-actions bot added documentation Improvements or additions to documentation scripts labels Sep 17, 2024
Copy link

github-actions bot commented Sep 17, 2024

PR #35625: Size comparison from 32200d1 to 815c255

Full report (82 builds for bl602, bl702, bl702l, cyw30739, efr32, esp32, linux, nrfconnect, nxp, psoc6, qpg, stm32, telink, tizen)
platform target config section 32200d1 815c255 change % change
bl602 lighting-app bl602 FLASH 1278168 1278168 0 0.0
RAM 95824 95824 0 0.0
bl602+mfd FLASH 1292278 1292278 0 0.0
RAM 95968 95968 0 0.0
bl602+rpc FLASH 1317132 1317132 0 0.0
RAM 104248 104248 0 0.0
bl702 lighting-app bl702 FLASH 943874 943874 0 0.0
RAM 15145 15145 0 0.0
bl702+mfd FLASH 946274 946274 0 0.0
RAM 15305 15305 0 0.0
bl702+rpc FLASH 1039194 1039194 0 0.0
RAM 24173 24173 0 0.0
bl706-eth FLASH 646070 646070 0 0.0
RAM 25233 25233 0 0.0
bl706-wifi FLASH 894082 894082 0 0.0
RAM 14477 14477 0 0.0
bl702l lighting-app bl702l FLASH 960534 960534 0 0.0
RAM 16804 16804 0 0.0
bl702l+mfd FLASH 963156 963156 0 0.0
RAM 16964 16964 0 0.0
cyw30739 light CYW30739B2-P5-EVK-01 unknown 2040 2040 0 0.0
FLASH 677261 677261 0 0.0
RAM 78492 78492 0 0.0
CYW30739B2-P5-EVK-02 unknown 2040 2040 0 0.0
FLASH 697113 697113 0 0.0
RAM 81124 81124 0 0.0
CYW30739B2-P5-EVK-03 unknown 2040 2040 0 0.0
FLASH 697113 697113 0 0.0
RAM 81124 81124 0 0.0
CYW930739M2EVB-02 unknown 2040 2040 0 0.0
FLASH 654041 654041 0 0.0
RAM 73560 73560 0 0.0
light-switch CYW30739B2-P5-EVK-01 unknown 2040 2040 0 0.0
FLASH 613453 613453 0 0.0
RAM 71452 71452 0 0.0
CYW30739B2-P5-EVK-02 unknown 2040 2040 0 0.0
FLASH 633081 633081 0 0.0
RAM 74004 74004 0 0.0
CYW30739B2-P5-EVK-03 unknown 2040 2040 0 0.0
FLASH 633081 633081 0 0.0
RAM 74004 74004 0 0.0
lock CYW30739B2-P5-EVK-01 unknown 2040 2040 0 0.0
FLASH 632573 632573 0 0.0
RAM 74500 74500 0 0.0
CYW30739B2-P5-EVK-02 unknown 2040 2040 0 0.0
FLASH 652289 652289 0 0.0
RAM 77052 77052 0 0.0
CYW30739B2-P5-EVK-03 unknown 2040 2040 0 0.0
FLASH 652289 652289 0 0.0
RAM 77052 77052 0 0.0
thermostat CYW30739B2-P5-EVK-01 unknown 2040 2040 0 0.0
FLASH 607989 607989 0 0.0
RAM 68588 68588 0 0.0
CYW30739B2-P5-EVK-02 unknown 2040 2040 0 0.0
FLASH 627841 627841 0 0.0
RAM 71220 71220 0 0.0
CYW30739B2-P5-EVK-03 unknown 2040 2040 0 0.0
FLASH 627841 627841 0 0.0
RAM 71220 71220 0 0.0
efr32 lighting-app BRD4187C FLASH 932468 932468 0 0.0
RAM 135056 135056 0 0.0
lock-app BRD2605a FLASH 739960 739952 -8 -0.0
RAM 231212 231212 0 0.0
BRD4338a FLASH 740608 740600 -8 -0.0
RAM 231244 231244 0 0.0
window-app BRD4187C FLASH 1015544 1015536 -8 -0.0
RAM 126996 126996 0 0.0
esp32 all-clusters-app c3devkit DRAM 94152 94152 0 0.0
FLASH 1539526 1539526 0 0.0
IRAM 82538 82538 0 0.0
m5stack DRAM 115096 115096 0 0.0
FLASH 1549846 1549846 0 0.0
IRAM 117039 117039 0 0.0
linux air-purifier-app debug unknown 4688 4688 0 0.0
FLASH 2771233 2771233 0 0.0
RAM 128944 128944 0 0.0
all-clusters-app debug unknown 5456 5456 0 0.0
FLASH 6082726 6082726 0 0.0
RAM 514288 514288 0 0.0
all-clusters-minimal-app debug unknown 5352 5352 0 0.0
FLASH 5406782 5406782 0 0.0
RAM 240640 240640 0 0.0
bridge-app debug unknown 5336 5336 0 0.0
FLASH 4740294 4740294 0 0.0
RAM 217840 217840 0 0.0
chip-tool debug unknown 5888 5888 0 0.0
FLASH 1267635 1267635 0 0.0
RAM 562282 562282 0 0.0
chip-tool-ipv6only arm64 unknown 20384 20384 0 0.0
FLASH 11341604 11341604 0 0.0
RAM 612384 612384 0 0.0
fabric-admin debug unknown 5720 5720 0 0.0
FLASH 10979239 10979239 0 0.0
RAM 561322 561322 0 0.0
fabric-bridge-app debug unknown 4568 4568 0 0.0
FLASH 4563032 4563032 0 0.0
RAM 204456 204456 0 0.0
lighting-app debug+rpc+ui unknown 5984 5984 0 0.0
FLASH 5676609 5676609 0 0.0
RAM 228128 228128 0 0.0
lock-app debug unknown 5272 5272 0 0.0
FLASH 4789574 4789574 0 0.0
RAM 203928 203928 0 0.0
ota-provider-app debug unknown 4648 4648 0 0.0
FLASH 4422762 4422762 0 0.0
RAM 197632 197632 0 0.0
ota-requestor-app debug unknown 4584 4584 0 0.0
FLASH 4561508 4561508 0 0.0
RAM 202200 202200 0 0.0
shell debug unknown 4216 4216 0 0.0
FLASH 3107901 3107901 0 0.0
RAM 159448 159448 0 0.0
thermostat-no-ble arm64 unknown 9336 9336 0 0.0
FLASH 4333260 4333260 0 0.0
RAM 242256 242256 0 0.0
tv-app debug unknown 5552 5552 0 0.0
FLASH 6022261 6022261 0 0.0
RAM 587088 587088 0 0.0
tv-casting-app debug unknown 5208 5208 0 0.0
FLASH 10883805 10883805 0 0.0
RAM 650256 650256 0 0.0
nrfconnect all-clusters-app nrf52840dk_nrf52840 FLASH 915124 915124 0 0.0
RAM 142259 142259 0 0.0
nrf7002dk_nrf5340_cpuapp FLASH 886440 886440 0 0.0
RAM 140398 140398 0 0.0
all-clusters-minimal-app nrf52840dk_nrf52840 FLASH 846468 846468 0 0.0
RAM 141077 141077 0 0.0
nxp contact k32w0+release FLASH 580520 580520 0 0.0
RAM 70720 70720 0 0.0
k32w1+release FLASH 593344 593344 0 0.0
RAM 62896 62896 0 0.0
mcxw71+release FLASH 593096 593096 0 0.0
RAM 62896 62896 0 0.0
light k32w0+release FLASH 616316 616316 0 0.0
RAM 70184 70184 0 0.0
k32w1+release FLASH 679584 679584 0 0.0
RAM 48512 48512 0 0.0
mcxw71+release FLASH 679600 679600 0 0.0
RAM 48512 48512 0 0.0
lock k32w1+release FLASH 701896 701896 0 0.0
RAM 67036 67036 0 0.0
mcxw71+release FLASH 701904 701904 0 0.0
RAM 67036 67036 0 0.0
psoc6 all-clusters cy8ckit_062s2_43012 FLASH 1646916 1646916 0 0.0
RAM 210952 210952 0 0.0
all-clusters-minimal cy8ckit_062s2_43012 FLASH 1550556 1550556 0 0.0
RAM 207664 207664 0 0.0
light cy8ckit_062s2_43012 FLASH 1466468 1466468 0 0.0
RAM 200672 200672 0 0.0
lock cy8ckit_062s2_43012 FLASH 1462988 1462988 0 0.0
RAM 225024 225024 0 0.0
qpg lighting-app qpg6105+debug FLASH 659096 659096 0 0.0
RAM 105212 105212 0 0.0
lock-app qpg6105+debug FLASH 616988 616988 0 0.0
RAM 99688 99688 0 0.0
stm32 light STM32WB5MM-DK FLASH 480568 480568 0 0.0
RAM 144668 144668 0 0.0
telink air-quality-sensor-app tlsr9528a_retention FLASH 621708 621708 0 0.0
RAM 50964 50964 0 0.0
all-clusters-app tlsr9118bdk40d FLASH 688210 688210 0 0.0
RAM 148388 148388 0 0.0
all-clusters-minimal-app tlsr9528a FLASH 779892 779892 0 0.0
RAM 110756 110756 0 0.0
bridge-app tlsr9258a FLASH 680728 680728 0 0.0
RAM 91620 91620 0 0.0
contact-sensor-app tlsr9528a_retention FLASH 621472 621472 0 0.0
RAM 50916 50916 0 0.0
light-switch-app-ota-shell-factory-data tlsr9528a FLASH 707404 707404 0 0.0
RAM 74256 74256 0 0.0
lighting-app-ota-factory-data tlsr9118bdk40d FLASH 624626 624626 0 0.0
RAM 144292 144292 0 0.0
lighting-app-ota-rpc-factory-data-4mb tlsr9518adk80d FLASH 810034 810034 0 0.0
RAM 99424 99424 0 0.0
lock-app-dfu tlsr9528a FLASH 655224 655224 0 0.0
RAM 66976 66976 0 0.0
ota-requestor-app tlsr9258a FLASH 696704 696704 0 0.0
RAM 91212 91212 0 0.0
pump-app-usb tlsr9518adk80d FLASH 633060 633060 0 0.0
RAM 55792 55792 0 0.0
pump-controller-app tlsr9518adk80d FLASH 610250 610250 0 0.0
RAM 53036 53036 0 0.0
shell tlsr9518adk80d FLASH 467176 467176 0 0.0
RAM 68668 68668 0 0.0
smoke_co_alarm-app tlsr9528a_retention FLASH 628436 628436 0 0.0
RAM 52636 52636 0 0.0
temperature-measurement-app-mars-ota tlsr9518adk80d FLASH 652386 652386 0 0.0
RAM 56584 56584 0 0.0
thermostat tlsr9518adk80d FLASH 636980 636980 0 0.0
RAM 53428 53428 0 0.0
window-covering tlsr9118bdk40d FLASH 523156 523156 0 0.0
RAM 97268 97268 0 0.0
tizen all-clusters-app arm unknown 4852 4852 0 0.0
FLASH 1724840 1724840 0 0.0
RAM 89524 89524 0 0.0
chip-tool-ubsan arm unknown 10284 10284 0 0.0
FLASH 17532050 17532050 0 0.0
RAM 7609748 7609748 0 0.0

Copy link

github-actions bot commented Sep 17, 2024

PR #35625: Size comparison from 32200d1 to 4a0f922

Full report (82 builds for bl602, bl702, bl702l, cyw30739, efr32, esp32, linux, nrfconnect, nxp, psoc6, qpg, stm32, telink, tizen)
platform target config section 32200d1 4a0f922 change % change
bl602 lighting-app bl602 FLASH 1278168 1278192 24 0.0
RAM 95824 95824 0 0.0
bl602+mfd FLASH 1292278 1292302 24 0.0
RAM 95968 95968 0 0.0
bl602+rpc FLASH 1317132 1317156 24 0.0
RAM 104248 104248 0 0.0
bl702 lighting-app bl702 FLASH 943874 943868 -6 -0.0
RAM 15145 15145 0 0.0
bl702+mfd FLASH 946274 946268 -6 -0.0
RAM 15305 15305 0 0.0
bl702+rpc FLASH 1039194 1039188 -6 -0.0
RAM 24173 24173 0 0.0
bl706-eth FLASH 646070 646064 -6 -0.0
RAM 25233 25233 0 0.0
bl706-wifi FLASH 894082 894076 -6 -0.0
RAM 14477 14477 0 0.0
bl702l lighting-app bl702l FLASH 960534 960528 -6 -0.0
RAM 16804 16804 0 0.0
bl702l+mfd FLASH 963156 963150 -6 -0.0
RAM 16964 16964 0 0.0
cyw30739 light CYW30739B2-P5-EVK-01 unknown 2040 2040 0 0.0
FLASH 677261 677269 8 0.0
RAM 78492 78492 0 0.0
CYW30739B2-P5-EVK-02 unknown 2040 2040 0 0.0
FLASH 697113 697121 8 0.0
RAM 81124 81124 0 0.0
CYW30739B2-P5-EVK-03 unknown 2040 2040 0 0.0
FLASH 697113 697121 8 0.0
RAM 81124 81124 0 0.0
CYW930739M2EVB-02 unknown 2040 2040 0 0.0
FLASH 654041 654049 8 0.0
RAM 73560 73560 0 0.0
light-switch CYW30739B2-P5-EVK-01 unknown 2040 2040 0 0.0
FLASH 613453 613461 8 0.0
RAM 71452 71452 0 0.0
CYW30739B2-P5-EVK-02 unknown 2040 2040 0 0.0
FLASH 633081 633089 8 0.0
RAM 74004 74004 0 0.0
CYW30739B2-P5-EVK-03 unknown 2040 2040 0 0.0
FLASH 633081 633089 8 0.0
RAM 74004 74004 0 0.0
lock CYW30739B2-P5-EVK-01 unknown 2040 2040 0 0.0
FLASH 632573 632589 16 0.0
RAM 74500 74500 0 0.0
CYW30739B2-P5-EVK-02 unknown 2040 2040 0 0.0
FLASH 652289 652297 8 0.0
RAM 77052 77052 0 0.0
CYW30739B2-P5-EVK-03 unknown 2040 2040 0 0.0
FLASH 652289 652297 8 0.0
RAM 77052 77052 0 0.0
thermostat CYW30739B2-P5-EVK-01 unknown 2040 2040 0 0.0
FLASH 607989 608005 16 0.0
RAM 68588 68588 0 0.0
CYW30739B2-P5-EVK-02 unknown 2040 2040 0 0.0
FLASH 627841 627857 16 0.0
RAM 71220 71220 0 0.0
CYW30739B2-P5-EVK-03 unknown 2040 2040 0 0.0
FLASH 627841 627857 16 0.0
RAM 71220 71220 0 0.0
efr32 lighting-app BRD4187C FLASH 932468 932500 32 0.0
RAM 135056 135056 0 0.0
lock-app BRD2605a FLASH 739960 739952 -8 -0.0
RAM 231212 231212 0 0.0
BRD4338a FLASH 740608 740600 -8 -0.0
RAM 231244 231244 0 0.0
window-app BRD4187C FLASH 1015544 1015536 -8 -0.0
RAM 126996 126996 0 0.0
esp32 all-clusters-app c3devkit DRAM 94152 94152 0 0.0
FLASH 1539526 1539520 -6 -0.0
IRAM 82538 82538 0 0.0
m5stack DRAM 115096 115096 0 0.0
FLASH 1549846 1549862 16 0.0
IRAM 117039 117039 0 0.0
linux air-purifier-app debug unknown 4688 4688 0 0.0
FLASH 2771233 2771259 26 0.0
RAM 128944 128944 0 0.0
all-clusters-app debug unknown 5456 5456 0 0.0
FLASH 6082726 6082752 26 0.0
RAM 514288 514288 0 0.0
all-clusters-minimal-app debug unknown 5352 5352 0 0.0
FLASH 5406782 5406808 26 0.0
RAM 240640 240640 0 0.0
bridge-app debug unknown 5336 5336 0 0.0
FLASH 4740294 4740320 26 0.0
RAM 217840 217840 0 0.0
chip-tool debug unknown 5888 5888 0 0.0
FLASH 1267635 12676380 26 0.0
RAM 562282 562282 0 0.0
chip-tool-ipv6only arm64 unknown 20384 20384 0 0.0
FLASH 11341604 11341636 32 0.0
RAM 612384 612384 0 0.0
fabric-admin debug unknown 5720 5720 0 0.0
FLASH 10979239 10979263 24 0.0
RAM 561322 561322 0 0.0
fabric-bridge-app debug unknown 4568 4568 0 0.0
FLASH 4563032 4563058 26 0.0
RAM 204456 204456 0 0.0
lighting-app debug+rpc+ui unknown 5984 5984 0 0.0
FLASH 5676609 5676641 32 0.0
RAM 228128 228128 0 0.0
lock-app debug unknown 5272 5272 0 0.0
FLASH 4789574 4789600 26 0.0
RAM 203928 203928 0 0.0
ota-provider-app debug unknown 4648 4648 0 0.0
FLASH 4422762 4422786 24 0.0
RAM 197632 197632 0 0.0
ota-requestor-app debug unknown 4584 4584 0 0.0
FLASH 4561508 4561532 24 0.0
RAM 202200 202200 0 0.0
shell debug unknown 4216 4216 0 0.0
FLASH 3107901 3107917 16 0.0
RAM 159448 159448 0 0.0
thermostat-no-ble arm64 unknown 9336 9336 0 0.0
FLASH 4333260 4333292 32 0.0
RAM 242256 242256 0 0.0
tv-app debug unknown 5552 5552 0 0.0
FLASH 6022261 6022277 16 0.0
RAM 587088 587088 0 0.0
tv-casting-app debug unknown 5208 5208 0 0.0
FLASH 10883805 10883837 32 0.0
RAM 650256 650256 0 0.0
nrfconnect all-clusters-app nrf52840dk_nrf52840 FLASH 915124 915128 4 0.0
RAM 142259 142259 0 0.0
nrf7002dk_nrf5340_cpuapp FLASH 886440 886444 4 0.0
RAM 140398 140398 0 0.0
all-clusters-minimal-app nrf52840dk_nrf52840 FLASH 846468 846476 8 0.0
RAM 141077 141077 0 0.0
nxp contact k32w0+release FLASH 580520 580520 0 0.0
RAM 70720 70720 0 0.0
k32w1+release FLASH 593344 593352 8 0.0
RAM 62896 62896 0 0.0
mcxw71+release FLASH 593096 593104 8 0.0
RAM 62896 62896 0 0.0
light k32w0+release FLASH 616316 616332 16 0.0
RAM 70184 70184 0 0.0
k32w1+release FLASH 679584 679608 24 0.0
RAM 48512 48512 0 0.0
mcxw71+release FLASH 679600 679608 8 0.0
RAM 48512 48512 0 0.0
lock k32w1+release FLASH 701896 701904 8 0.0
RAM 67036 67036 0 0.0
mcxw71+release FLASH 701904 701928 24 0.0
RAM 67036 67036 0 0.0
psoc6 all-clusters cy8ckit_062s2_43012 FLASH 1646916 1646932 16 0.0
RAM 210952 210952 0 0.0
all-clusters-minimal cy8ckit_062s2_43012 FLASH 1550556 1550556 0 0.0
RAM 207664 207664 0 0.0
light cy8ckit_062s2_43012 FLASH 1466468 1466484 16 0.0
RAM 200672 200672 0 0.0
lock cy8ckit_062s2_43012 FLASH 1462988 1462988 0 0.0
RAM 225024 225024 0 0.0
qpg lighting-app qpg6105+debug FLASH 659096 659104 8 0.0
RAM 105212 105212 0 0.0
lock-app qpg6105+debug FLASH 616988 616996 8 0.0
RAM 99688 99688 0 0.0
stm32 light STM32WB5MM-DK FLASH 480568 480576 8 0.0
RAM 144668 144668 0 0.0
telink air-quality-sensor-app tlsr9528a_retention FLASH 621708 621702 -6 -0.0
RAM 50964 50964 0 0.0
all-clusters-app tlsr9118bdk40d FLASH 688210 688204 -6 -0.0
RAM 148388 148388 0 0.0
all-clusters-minimal-app tlsr9528a FLASH 779892 779886 -6 -0.0
RAM 110756 110756 0 0.0
bridge-app tlsr9258a FLASH 680728 680722 -6 -0.0
RAM 91620 91620 0 0.0
contact-sensor-app tlsr9528a_retention FLASH 621472 621466 -6 -0.0
RAM 50916 50916 0 0.0
light-switch-app-ota-shell-factory-data tlsr9528a FLASH 707404 707398 -6 -0.0
RAM 74256 74256 0 0.0
lighting-app-ota-factory-data tlsr9118bdk40d FLASH 624626 624620 -6 -0.0
RAM 144292 144292 0 0.0
lighting-app-ota-rpc-factory-data-4mb tlsr9518adk80d FLASH 810034 810028 -6 -0.0
RAM 99424 99424 0 0.0
lock-app-dfu tlsr9528a FLASH 655224 655218 -6 -0.0
RAM 66976 66976 0 0.0
ota-requestor-app tlsr9258a FLASH 696704 696698 -6 -0.0
RAM 91212 91212 0 0.0
pump-app-usb tlsr9518adk80d FLASH 633060 633054 -6 -0.0
RAM 55792 55792 0 0.0
pump-controller-app tlsr9518adk80d FLASH 610250 610244 -6 -0.0
RAM 53036 53036 0 0.0
shell tlsr9518adk80d FLASH 467176 467170 -6 -0.0
RAM 68668 68668 0 0.0
smoke_co_alarm-app tlsr9528a_retention FLASH 628436 628430 -6 -0.0
RAM 52636 52636 0 0.0
temperature-measurement-app-mars-ota tlsr9518adk80d FLASH 652386 652380 -6 -0.0
RAM 56584 56584 0 0.0
thermostat tlsr9518adk80d FLASH 636980 636974 -6 -0.0
RAM 53428 53428 0 0.0
window-covering tlsr9118bdk40d FLASH 523156 523150 -6 -0.0
RAM 97268 97268 0 0.0
tizen all-clusters-app arm unknown 4852 4852 0 0.0
FLASH 1724840 1724856 16 0.0
RAM 89524 89524 0 0.0
chip-tool-ubsan arm unknown 10284 10284 0 0.0
FLASH 17532050 17532082 32 0.0
RAM 7609748 7609764 16 0.0

@mergify mergify bot merged commit 85b2fd3 into project-chip:master Sep 17, 2024
71 checks passed
if not os.path.exists("out/trace_data"):
os.mkdir("out/trace_data")

# IGNORES are taken out of `src/python_testing/execute_python_tests.py` in the SDK
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we have a single source of truth for this? Otherwise, these lists will diverge, which will cause problems.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will work towards it with @vatsalghelani-csa - we probably want something loadable and then we can use it from both places (one our official runner and one being this local runner)

yyzhong-g pushed a commit to yyzhong-g/connectedhomeip that referenced this pull request Dec 12, 2024
…xecution` (project-chip#35625)

* Add a test runner for convenient python runs

* Add some documentation

* Restyle

* Update requirements

* Use local build platforms, to maybe be able to run under darwin

* Use local build platforms, to maybe be able to run under darwin

* Fix typos

* More typo fixes

* Restyle

* Make platform name actually work

* Restyled by prettier-markdown

* Fix linter

* Add license

---------

Co-authored-by: Andrei Litvin <[email protected]>
Co-authored-by: Restyled.io <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation review - approved scripts
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants