[BUG] ESP32: esptool below 4.8 silently produces broken firmware. #13824

cederom · 2024-10-04T23:22:04Z

Description / Steps to reproduce the issue

Compiler.

esp32-elf-cc --version
xtensa-esp32-elf-cc (crosstool-NG esp-2021r2-patch5) 8.4.0
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

With esptool 4.3 - cannot build firmware image - visible error:

% /usr/bin/time -h ./tools/configure.sh -B esp32-devkitc:nsh
        3,92s real              1,98s user              2,23s sys

% /usr/bin/time -h gmake -j8
 /usr/bin/time -h gmake -j8
Create version.h
Cloning Espressif HAL for 3rd Party Platforms
Clone: chip/esp-hal-3rdparty LN: platform/board to /XXX/nuttx-apps.git/platform/dummy
Register: nsh
Register: sh
Espressif HAL for 3rd Party Platforms: 20690e67695f0a8170a19ec99e2e9a13b620e94d
CPP:  /XXX/nuttx.git/arch/xtensa/src/chip/esp-hal-3rdparty/components/esp_rom/esp32/ld/esp32.rom.newlib-funcs.ld-> /XXX/nuttx.git/arch/xtensa/src/chip/esp-hal-3rdparty/components/esp_rom/esp32/ld/esp32.rom.newlib-funcs.ld.tmpLD: nuttx
CP: nuttx.hex
MKIMAGE: ESP32 binary
esptool.py -c esp32 elf2image --ram-only-header -fs 4MB -fm dio -ff 40m -o nuttx.bin nuttx
usage: esptool [-h] [--chip {auto,esp8266,esp32,esp32s2,esp32s3beta2,esp32s3,esp32c3,esp32c6beta,esp32h2beta1,esp32h2beta2,esp32c2,esp32c6}] [--port PORT] [--baud BAUD] [--before {default_reset,usb_reset,no_reset,no_reset_no_sync}] [--after {hard_reset,soft_reset,no_reset,no_reset_stub}] [--no-stub] [--trace]
               [--override-vddsdio [{1.8V,1.9V,OFF}]] [--connect-attempts CONNECT_ATTEMPTS]
               {load_ram,dump_mem,read_mem,write_mem,write_flash,run,image_info,make_image,elf2image,read_mac,chip_id,flash_id,read_flash_status,write_flash_status,read_flash,verify_flash,erase_flash,erase_region,merge_bin,get_security_info,version} ...
esptool: error: unrecognized arguments: --ram-only-header
gmake: *** [tools/Unix.mk:558: nuttx] Error 2
        40,96s real             1m32,23s user           1m1,82s sys

% pip show esptool
Name: esptool
Version: 4.3
Summary: A serial utility to communicate & flash code to Espressif chips.
Home-page: https://github.com/espressif/esptool/
Author: Fredrik Ahlberg (themadinventor) & Angus Gratton (projectgus) & Espressif Systems
Author-email:
License: GPLv2+
Location: /XXX/venv3.9embedded/lib/python3.9/site-packages
Requires: bitstring, cryptography, ecdsa, pyserial, reedsolo
Required-by: #N/A

With latest esptool 4.8.1 - all works fine.

(venv3.9embedded) pip install -U esptool

(venv3.9embedded) /usr/bin/time -h gmake -j8
CPP:  /XXX/nuttx.git/arch/xtensa/src/chip/esp-hal-3rdparty/components/esp_rom/esp32/ld/esp32.rom.newlib-funcs.ld-> /XXX/nuttx.git/arch/xtensa/src/chip/esp-hal-3rdparty/components/esp_rom/esp32/ld/esp32.rom.newlib-funcs.ld.tmpLD: nuttx
CP: nuttx.hex
MKIMAGE: ESP32 binary
esptool.py -c esp32 elf2image --ram-only-header -fs 4MB -fm dio -ff 40m -o nuttx.bin nuttx
esptool.py v4.8.1
Creating esp32 image...
Image has only RAM segments visible. ROM segments are hidden and SHA256 digest is not appended.
Merged 1 ELF section
Successfully created esp32 image.
Generated: nuttx.bin
        4,19s real              2,96s user              3,44s sys

(venv3.9embedded) /usr/bin/time -h gmake flash
        15,26s real             3,03s user              3,24s sys

(venv3.9embedded) cu -l /dev/cuaU0 -s 115200
Stale lock on cuaU0 PID=4016... overriding.
Connected
ts Jul 29 2019 12:21:46

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3ffb1f40,len:1388
ho 0 tail 12 room 4
load:0x40080000,len:20320
entry 0x40082650
*** Booting NuttX ***
dram: lma 0x00001020 vma 0x3ffb1f40 len 0x56c    (1388)
iram: lma 0x00001594 vma 0x40080000 len 0x4f60   (20320)
padd: lma 0x00006508 vma 0x00000000 len 0x9af0   (39664)
imap: lma 0x00010000 vma 0x400e0000 len 0x10d78  (68984)
padd: lma 0x00020d80 vma 0x00000000 len 0xf298   (62104)
dmap: lma 0x00030020 vma 0x3f400020 len 0x2848   (10312)
total segments stored 6

NuttShell (NSH) NuttX-12.7.0
nsh> uname -a
NuttX 12.7.0 10e44f8915-dirty Oct  4 2024 22:22:13 xtensa esp32-devkitc
nsh> free
                 total       used       free    maxused    maxfree  nused  nfree
      Umem:     323568       6424     317144       6800     186696     22      3

With esptool 4.7.0 (default system package on FreeBSD) - all seems fine, no errors, but firmware is broken.

% gmake clean distclean

% /usr/bin/time -h ./tools/configure.sh -B esp32-devkitc:nsh
        3,89s real              1,77s user              2,41s sys

% /usr/bin/time -h gmake -j8
Create version.h
Cloning Espressif HAL for 3rd Party Platforms
Clone: chip/esp-hal-3rdparty LN: platform/board to /XXX/nuttx-apps.git/platform/dummy
Register: nsh
Register: sh
Espressif HAL for 3rd Party Platforms: 20690e67695f0a8170a19ec99e2e9a13b620e94d
CPP:  /XXX/nuttx.git/arch/xtensa/src/chip/esp-hal-3rdparty/components/esp_rom/esp32/ld/esp32.rom.newlib-funcs.ld-> /XXX/nuttx.git/arch/xtensa/src/chip/esp-hal-3rdparty/components/esp_rom/esp32/ld/esp32.rom.newlib-funcs.ld.tmp
LD: nuttx
CP: nuttx.hex
MKIMAGE: ESP32 binary
esptool.py -c esp32 elf2image --ram-only-header -fs 4MB -fm dio -ff 40m -o nuttx.bin nuttx
esptool.py v4.7.0
Creating esp32 image...
ROM segments hidden - only RAM segments are visible to the ROM loader!
Merged 1 ELF section
Successfully created esp32 image.
Generated: nuttx.bin
        39,74s real             1m31,54s user           1m2,00s sys

%  /usr/bin/time -h gmake flash
CPP:  /XXX/nuttx.git/arch/xtensa/src/chip/esp-hal-3rdparty/components/esp_rom/esp32/ld/esp32.rom.newlib-funcs.ld-> /XXX/nuttx.git/arch/xtensa/src/chip/esp-hal-3rdparty/components/esp_rom/esp32/ld/esp32.rom.newlib-funcs.ld.tmpLD: nuttx
CP: nuttx.hex
MKIMAGE: ESP32 binary
esptool.py -c esp32 elf2image --ram-only-header -fs 4MB -fm dio -ff 40m -o nuttx.bin nuttx
esptool.py v4.7.0
Creating esp32 image...
ROM segments hidden - only RAM segments are visible to the ROM loader!
Merged 1 ELF section
Successfully created esp32 image.
Generated: nuttx.bin
esptool.py -c esp32 -p /dev/cuaU0 -b 115200  write_flash -fs detect -fm dio -ff 40m 0x1000 nuttx.bin
esptool.py v4.7.0
Serial port /dev/cuaU0
Connecting.........
Chip is ESP32-D0WD-V3 (revision v3.0)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: XXX
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Auto-detected Flash size: 4MB
Flash will be erased from 0x00001000 to 0x00033fff...
Compressed 206984 bytes to 72429...
Wrote 206984 bytes (72429 compressed) at 0x00001000 in 6.7 seconds (effective 247.0 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...
        15,25s real             3,07s user              3,17s sys


% cu -l /dev/cuaU0 -s 115200
Connected
ets Jul 29 2019 12:21:46

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3ffb1f40,len:1388
ho 0 tail 12 room 4
load:0x40080000,len:20320
entry 0x40082650
*** Booting NuttX ***
dram: lma 0x00001020 vma 0x3ffb1f40 len 0x56c    (1388)
iram: lma 0x00001594 vma 0x40080000 len 0x4f60   (20320)
padd: lma 0x00006508 vma 0x00000000 len 0xaaf0   (43760)
imap: lma 0x00011000 vma 0x400e0000 len 0x10d78  (68984)
padd: lma 0x00021d80 vma 0x00000000 len 0xf298   (62104)
dmap: lma 0x00031020 vma 0x3f400020 len 0x2848   (10312)
total segments stored 6

>>> NO NSH HERE DEVICE HANGS <<<

Lets try with the CoreMark:

% gmake clean distclean

% /usr/bin/time -h ./tools/configure.sh -B esp32-devkitc:coremark
        3,93s real              1,73s user              2,46s sys

% /usr/bin/time -h gmake -j8
        42,46s real             1m35,59s user           1m0,58s sys

%  /usr/bin/time -h gmake flash
        13,78s real             2,69s user              2,63s sys

% cu -l /dev/cuaU0 -s 115200
Connected
ets Jul 29 2019 12:21:46

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3ffb30a0,len:2016
load:0x40080000,len:30100
entry 0x40083940
*** Booting NuttX ***
dram: lma 0x00001020 vma 0x3ffb30a0 len 0x7e0    (2016)
iram: lma 0x00001808 vma 0x40080000 len 0x7594   (30100)
padd: lma 0x00008da8 vma 0x00000000 len 0x8250   (33360)
imap: lma 0x00011000 vma 0x400e0000 len 0xde14   (56852)
padd: lma 0x0001ee1c vma 0x00000000 len 0x21fc   (8700)
dmap: lma 0x00021020 vma 0x3f400020 len 0x14fc   (5372)
total segments stored 6

>>> COREMARK DOES NOT RUN <<<

I found this commit 457f9d3 to update the CI in order to build for ESP32 but manual builds should return error on older esptool versions or the older bootloader should be used? :-)

On which OS does this issue occur?

[OS: BSD]

What is the version of your OS?

13.3-RELEASE-p7 GENERIC amd64

NuttX Version

12.7.0-RC0

Issue Architecture

[Arch: xtensa]

Issue Area

[Area: Build System]

Verification

I have verified before submitting the report.

The text was updated successfully, but these errors were encountered:

acassis · 2024-10-06T09:37:54Z

@fdcavalcanti please take a look ^

cederom · 2024-10-06T22:28:47Z

Thanks @acassis, @fdcavalcanti this is quite important issue as it needs to be fixed in 12.7.0 release (it was found during 12.7.0-RC0 testing).

fdcavalcanti · 2024-10-07T11:17:45Z

I believe the proper way to proceed is using the most up-to-date esptool.
Let me check if there's any nuance we're missing.

cederom · 2024-10-07T19:03:43Z

If version required is not present (i.e. 4.8 is required but 4.7 present) then error must be reported with build abort. So far no error is reported and silently the faulty image is built.
We should be able to build the firmware image also with older version of esptool (use older bootloader?).

fdcavalcanti · 2024-10-07T19:34:20Z

@almir-okato

jerpelea · 2024-10-10T14:21:26Z

@cederom any update ?

cederom · 2024-10-10T16:18:29Z

Waiting for a fix proposition from @fdcavalcanti and @almir-okato of Espressif.

cederom · 2024-10-10T16:31:33Z

For sure we may leave current state as-is "using the most up-to-date esptool" then people with esptool <4.8 will get broken firmware without warning. Two impacts possible:

NuttX users that want to play with ESP32 will probably never come back to ESP32.
ESP32 users that want to play with NuttX will probably never come back to NuttX.

The simplest fix is just to enforce latest esptool and error build when older tool is in use. Not elegant nor compatible.

The best fix would be to adapt ESP32 bootloader to what esptool version is available (i.e up to -3 minor, or build error).

https://github.com/apache/nuttx/blob/master/INVIOLABLES.md

jerpelea · 2024-10-10T17:36:56Z

@cederom what do you think about adding a config option which is set to 4.8 but can be changed by the user?

cederom · 2024-10-10T17:55:49Z

@cederom what do you think about adding a config option which is set to 4.8 but can be changed by the user?

This part should be done automatically based on the available tool version with some log message on what happened (something like: esptool version X.Y implies bootloader version P.Q). Adding version to config will not change anything because that version needs to be checked anyway which is not done right now what causes the problem.

Do we know why exactly 4.8 esptool needs to be used for current NuttX master? Are there any specific features in the new bootloader that were not there before and we need them for some reason? Why older esptool creates invalid firmware image silently? @fdcavalcanti @almir-okato

I think we should bump this issue on esptool repo too maybe we get more details that way :-)

fdcavalcanti · 2024-10-10T19:17:38Z

Hi @cederom sorry about the delay. We'll set it to latest version and later add a warning for unsupported esptool versions during build, to avoid a broken firwmare. This way we can proceed with the release and force the latest esptool.

cederom · 2024-10-11T00:50:06Z

As reported in espressif/esptool#1018 by @radimkarnis:

Taking a brief look at this - I believe this is a result of some of the recent fixes (espressif/esptool@4c5874a, espressif/esptool@4394a65, espressif/esptool@e87cc3e) in the --ram-only-header option in v4.8.

These changes have been implemented by the Zephyr team. Hopefully, they can shed some more light on this - cc @almir-okato, @sylvioalves PTAL.

what would be the way to approach and fix

Currently, I can only recommend that you use the latest esptool. We don't plan to backport any of these changes and they are still somewhat "experimental".

Looking at the provided information --ram-only-header is used by esptool command elf2image that converts ELF to firmware image flashed into the chip to trigger "simple boot" mode that redirects BootROM to hardware initialization functions bundled within the image:

The --ram-only-header configuration is mainly applicable for use within the Espressif’s SIMPLE_BOOT option from 3rd party OSes such as ZephyrOS and NuttX OS. This option makes only the RAM segments visible to the ROM bootloader placing them at the beginning of the file and altering the segment count from the image header with the quantity of these segments, and also writing only their checksum. This segment placement may result in a more fragmented binary because of flash alignment constraints. It is strongly recommended to use this configuration with care, because the image built must then handle the basic hardware initialization and the flash mapping for code execution after ROM bootloader boot it.

Previous releases of esptool did not have this functionality implemented (i.e. version 4.3) or produced invalid image with overlapping regions (i.e. version 4.7). Fixes will not be backported to previous releases. For now we are tied to esptool version >= 4.8.

I have asked about possible alternative solutions for older esptool versions, but most feasible solution for now is to verify version and throw build error if version older than 4.8.

Can you please prepare this kind of patch @fdcavalcanti @almir-okato ? :-)

This should be placed right before the firmware image is generated.
Verify and report available esptool version.
Throw build error if esptool version < 4.8 with a message that v4.8 contains fixes required to build valid firmware image and tool needs an upgrade.

Thank you :-)

fdcavalcanti · 2024-10-11T11:11:21Z

@cederom thanks for posting the team's response here on this thread. Its good to keep it all documented.
And sure, like I said, I'll be updating the build system with those checks for version soon.

jerpelea · 2024-10-16T08:48:59Z

backport #14268

jerpelea · 2024-10-16T10:43:31Z

@cederom please test releases/12.7 and close the bug

cederom · 2024-10-16T13:27:02Z

WORKS LIKE A CHARM! THANK YOU @fdcavalcanti @acassis @jerpelea @radimkarnis :-)

% uname -a
FreeBSD octagon 13.3-RELEASE-p7 FreeBSD 13.3-RELEASE-p7 GENERIC amd64

% git checkout releases/12.7

% ./tools/configure.sh -B esp32-devkitc:nsh

% gmake -j8
Create version.h
LN: platform/board to /XXX/nuttx-apps.git/platform/dummy
Register: nsh
Register: sh
Cloning Espressif HAL for 3rd Party Platforms
Espressif HAL for 3rd Party Platforms: 20690e67695f0a8170a19ec99e2e9a13b620e94d
LD: nuttx
CP: nuttx.hex
MKIMAGE: ESP32 binary
esptool.py -c esp32 elf2image --ram-only-header -fs 4MB -fm dio -ff 40m -o nuttx.bin nuttx
esptool.py v4.8.1
Creating esp32 image...
Image has only RAM segments visible. ROM segments are hidden and SHA256 digest is not appended.
Merged 1 ELF section
Successfully created esp32 image.
Generated: nuttx.bin

% gmake flash
LD: nuttx
CP: nuttx.hex
MKIMAGE: ESP32 binary
esptool.py -c esp32 elf2image --ram-only-header -fs 4MB -fm dio -ff 40m -o nuttx.bin nuttx
esptool.py v4.8.1
Creating esp32 image...
Image has only RAM segments visible. ROM segments are hidden and SHA256 digest is not appended.
Merged 1 ELF section
Successfully created esp32 image.
Generated: nuttx.bin
esptool.py -c esp32 -p /dev/cuaU0 -b 115200  write_flash -fs detect -fm dio -ff 40m 0x1000 nuttx.bin
esptool.py v4.8.1
Serial port /dev/cuaU0
Connecting............
Chip is ESP32-D0WD-V3 (revision v3.0)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: XXX
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Auto-detected Flash size: 4MB
Flash will be erased from 0x00001000 to 0x00032fff...
Compressed 202860 bytes to 72355...
Wrote 202860 bytes (72355 compressed) at 0x00001000 in 6.8 seconds (effective 238.9 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...

% cu -l /dev/cuaU0 -s 115200
Connected

nsh> uname -a
NuttX 12.7.0 71d57aac25 Oct 16 2024 15:22:18 xtensa esp32-devkitc
~.

% evenvsdk.sh venv

(venv3.9embedded) pip install esptool==4.7.0

(venv3.9embedded) gmake -j8
LD: nuttx
CP: nuttx.hex
MKIMAGE: ESP32 binary
Unsupported esptool version: 4.7.0 expects >= 4.8.0
Upgrade using: 'pip install --upgrade esptool' and run 'make' again
gmake: *** [tools/Unix.mk:558: nuttx] Error 1

Also we have updated esptool system package to 4.8.1 on FreeBSD :-)

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=282000

github-actions bot added Arch: xtensa Issues related to the Xtensa architecture Area: Build system OS: BSD Issues related to *BSD OSes (building system, etc) labels Oct 4, 2024

xiaoxiang781216 added the blocker Release Blocker label Oct 7, 2024

cederom mentioned this issue Oct 10, 2024

esptool <4.8 builds invalid ESP32 NuttX firmware image silently. (ESPTOOL-939) espressif/esptool#1018

Closed

1 task

fdcavalcanti mentioned this issue Oct 11, 2024

Update esptool version on docs and build system #14120

Merged

xiaoxiang781216 linked a pull request Oct 11, 2024 that will close this issue

Update esptool version on docs and build system #14120

Merged

fdcavalcanti mentioned this issue Oct 14, 2024

tools/espressif: add esptool version check to Espressif build system #14267

Merged

jerpelea mentioned this issue Oct 14, 2024

{Bp-14267} tools/espressif: add esptool version check to Espressif build system #14268

Merged

cederom closed this as completed Oct 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] ESP32: esptool below 4.8 silently produces broken firmware. #13824

[BUG] ESP32: esptool below 4.8 silently produces broken firmware. #13824

cederom commented Oct 4, 2024

acassis commented Oct 6, 2024

cederom commented Oct 6, 2024

fdcavalcanti commented Oct 7, 2024

cederom commented Oct 7, 2024 •

edited

Loading

fdcavalcanti commented Oct 7, 2024

jerpelea commented Oct 10, 2024

cederom commented Oct 10, 2024

cederom commented Oct 10, 2024

jerpelea commented Oct 10, 2024

cederom commented Oct 10, 2024

fdcavalcanti commented Oct 10, 2024

cederom commented Oct 11, 2024

fdcavalcanti commented Oct 11, 2024

jerpelea commented Oct 16, 2024

jerpelea commented Oct 16, 2024

cederom commented Oct 16, 2024

[BUG] ESP32: esptool below 4.8 silently produces broken firmware. #13824

[BUG] ESP32: esptool below 4.8 silently produces broken firmware. #13824

Comments

cederom commented Oct 4, 2024

Description / Steps to reproduce the issue

On which OS does this issue occur?

What is the version of your OS?

NuttX Version

Issue Architecture

Issue Area

Verification

acassis commented Oct 6, 2024

cederom commented Oct 6, 2024

fdcavalcanti commented Oct 7, 2024

cederom commented Oct 7, 2024 • edited Loading

fdcavalcanti commented Oct 7, 2024

jerpelea commented Oct 10, 2024

cederom commented Oct 10, 2024

cederom commented Oct 10, 2024

jerpelea commented Oct 10, 2024

cederom commented Oct 10, 2024

fdcavalcanti commented Oct 10, 2024

cederom commented Oct 11, 2024

fdcavalcanti commented Oct 11, 2024

jerpelea commented Oct 16, 2024

jerpelea commented Oct 16, 2024

cederom commented Oct 16, 2024

cederom commented Oct 7, 2024 •

edited

Loading