Boot fails with Cannot find any partition table for 00030000 after flashing

[apbr]

We are flashing devices with Jetson Orin NX 8GB modules using initrd-flash.
On some devices we observe that they do not boot properly after flashing, even though the flashing process is successful.
Booting fails with the following message in the MB2 step:

I> Task: Partition Manager Init
I> strt_pg_num(1) num_of_pgs(1) read_buf(0x273801000)
I> strt_pg_num(131071) num_of_pgs(1) read_buf(0x273801000)
W> Cannot find any partition table for 00030000
W> PARTITION_MANAGER: Failed to publish partition.
... (some lines left out) ...
I> Partition name: A_spe-fw
E> Cannot find partition A_spe-fw
E> Partition A_spe-fw not found
E> MB2_FW_LOAD: Failed to load FW.
C> Task 0x0 failed (err: 0x224d090d)
E> Top caller module: LOADER, error module: PARTITION_MANAGER, reason: 0x0d, aux_info: 0x09

Those modules were brand new and never flashed before by us.
Does anyone has an idea what could cause this issue?
When flashing with initrd-flash, do we flash the partition table onto the QSPI or is that left untouched?

Another difference we noticed:
Devices that boot successfully after flashing have the following section in the flash log:

[   1.1341 ] Dumping customer Info
[   1.1344 ] tegrarcm_v2 --instance 1-4 --chip 0x23 0 --oem dump bct tmp.bct
[   1.1346 ] MB2 Applet version 01.00.0000
[   1.1407 ] 0000000054220302: E> NV3P_SERVER: Failed to get address for br bct from nv3p helper.
[   1.2198 ] 
[   1.2199 ] 
[   1.2199 ] Error: try getting custinfo fail. Moving on

Devices that don't boot properly after flashing instead contain this lines for the same step:

[   1.1259 ] Dumping customer Info
[   1.1262 ] tegrarcm_v2 --instance 1-4 --chip 0x23 0 --oem dump bct tmp.bct
[   1.1265 ] MB2 Applet version 01.00.0000
[   1.1325 ] Saved bct in tmp.bct
[   1.1710 ] tegrabct_v2 --brbct tmp.bct --chip 0x23 0 --custinfo /root/temp/image-nx-factory_548_1533_1943.tegraflash/custinfo_out.bin
[   1.1712 ] C[   1.1714 ] ustomer data saved in /root/temp/image-nx-factory_548_1533_1943.tegraflash/custinfo_out.bin successfully

[madisongh]

When flashing with initrd-flash, do we flash the partition table onto the QSPI or is that left untouched?

By default it does - everything the QSPI flash should get updated. The script has an option --skip-bootloader to omit this step.

Which branch are you using, and is your machine definition based off the p3768-0000-p3767-0001 or something else? My guess is that there are some settings for the machine that are wrong for that particular SKU.

[apbr]

We are using scarthgap at 5d965c3.
Yes, our machine config is based on p3768-0000-p3767-0001.
So you think that some of our Jetson modules are a different SKU than others? How could we find that out?

Thank you for your help.
(Sorry, I should have given a bit more context from the start on.)

[madisongh]

They should all have the same board SKU, if they're all Orin NX 8GBs. It's possible there's some other variation between them that isn't being handled, or that one of the boot firmware blobs getting programmed into the SPI flash is coincidentally working in some cases but not others.

Doing some side-by-side, line-by-line comparison of the entire set of flashing logs between the working and non-working case might reveal something useful.

[apbr]

I already did a diff between flashing logs of modules that worked and ones that didn't (Though it was probably not the first flashing attempt for them):
This is a diff from the log of Step 1 (working vs non-working):

 [timestamps] Dumping customer Info
 [timestamps] tegrarcm_v2 --instance 1-4 --chip 0x23 0 --oem dump bct tmp.bct
 [timestamps] MB2 Applet version 01.00.0000
-[timestamps] 0000000054220302: E> NV3P_SERVER: Failed to get address for br bct from nv3p helper.
-[timestamps] 
-[timestamps] 
-[timestamps] Error: try getting custinfo fail. Moving on
+[timestamps] Saved bct in tmp.bct
+[timestamps] tegrabct_v2 --brbct tmp.bct --chip 0x23 0 --custinfo /root/temp/our-image-devel-nx-factory_548_1533_1943.tegraflash/custinfo_out.bin
+[timestamps] C[   1.1714 ] ustomer data saved in /root/temp/our-image-devel-nx-factory_548_1533_1943.tegraflash/custinfo_out.bin successfully
 [timestamps] Rebooting to recovery mode
 [timestamps] tegrarcm_v2 --instance 1-4 --chip 0x23 0 --ismb2
 [timestamps] tegrarcm_v2 --instance 1-4 --chip 0x23 0 --ismb2applet
 [timestamps] MB2 Applet version 01.00.0000
 [timestamps] Booting to recovery mode
 [timestamps] tegrarcm_v2 --instance 1-4 --chip 0x23 0 --reboot recovery
 [timestamps] MB2 Applet version 01.00.0000

There is also an interesting part in the diff from the serial console. This is the first boot when flashing, even before booting the initrd Linux (working vs non-working):

 I> QSPI Flash: Macronix 64MB
 I> QSPI-0l initialized successfully
 I> Secondary storage device: SDMMC_BOOT instance: 3
-E> Error in command_complete 18000 int_status
+E> Error in command_complete 18001 int_status
 E> OCR failed, error = 39390706
 E> STORAGE: Failed to open SDMMC: 3.
 W> Ignoring init failure for device 0-3
....
 E> LOADER: Invalid value Header magic.
 E> Validation failed for 1 copy of BRBCT @ 0
 I> strt_pg_num(512) num_of_pgs(16) read_buf(0x40071738)
-E> LOADER: Invalid value Header magic.
-E> Validation failed for 2 copy of BRBCT @ 262144
-I> strt_pg_num(1024) num_of_pgs(16) read_buf(0x40071738)
-E> LOADER: Invalid value Header magic.
-E> Validation failed for 3 copy of BRBCT @ 524288
-I> strt_pg_num(1536) num_of_pgs(16) read_buf(0x40071738)
-E> LOADER: Invalid value Header magic.
-E> Validation failed for 4 copy of BRBCT @ 786432
-E> NV3P_SERVER: Failed to get address for br bct from nv3p helper.
+I> BRBCT integrity check is success
+I> Validation passed for 2 copy of BRBCT @ 262144
 I> Rebooting : reboot-recovery

Apart from that I did not notice any relevant differences.
Anything specific to look out for?

[madisongh]

Look for any difference at all in the names of the files used by the scripts to put together the boot firmware blobs (in the log.initrd-flash.<datetime> file and/or the device side logs in the device-logs-<datetime> subdirectory.

It might be easier to check for differences by using ./doflash --spi-only to do the boot firmware flashing. And you'd get even more detail by adding a set -x to the top of the tegra-flash.sh script before doing that. That's the usual approach I take when trying to troubleshoot this kind of thing.