OTA update on Jetson NX with NVME

[aurelien-enchanted-tools]

Hello,

I have never implemented an OTA symmetric update system. I am researching on it. I saw there are several main solutions: mender, swupdate and rauc. The first one requires a mender server. The two last can use an hawkbit server.
I saw there are some demo implementations in tegra-test-distro. But I did not see an integration documentation. tegra-test-distro does not seem to offer an implement for xavier NX which works with nvme.

Do you have any advice or good documentation on this topic for a beginner in OTA update system?

Thank you in advance.

[dwalkes]

Hi @aurelien-enchanted-tools and welcome.

I think the best way to start is to use the Jetpack 4 implementation of mender in tegrademo-mender at https://github.com/OE4T/tegra-demo-distro/tree/kirkstone-l4t-r32.7.x - this one represents the latest available turn-key update demo image for Xavier NX. It doesn't have turn key support for NVME but it should be possible to support it with minimal changes and using the initrd flashing backported to kirkstone.

Another option is to go through the discussions at https://matrix.to/#/#OE4T_jetpack-5-device-update:gitter.im for the latest in Jetpack 5 device update on different platforms. Several people have working implementations but there is no public turn-key demo yet.

[DaveBubenik]

Can you elaborate a bit more on the backporting? When I try to install the tegra-test-distro with "TNSPEC_BOOTDEV = "nvme0n1p1"", the device doesn't boot.

Are there certain scripts or directories I should be attempting to merge into kirkstone-l4t-r32.7.x from kirkstone? Or is there a certain commit/change set I can look at to understand what I need to do?

I appreciate any guidance you can give!

[dwalkes]

When I try to install the tegra-test-distro with "TNSPEC_BOOTDEV = "nvme0n1p1"", the device doesn't boot.

Are you following https://github.com/OE4T/meta-tegra/wiki/initrd-flashing-support to flash?

Are there certain scripts or directories I should be attempting to merge into kirkstone-l4t-r32.7.x from kirkstone?

You shouldn't need to do anything to just get initrd-flash working, sorry I wasn't more clear about this, the kirkstone-l4t-r32.7.x branch already contains a backport (see notes at the top of https://github.com/OE4T/meta-tegra/wiki/initrd-flashing-support). Someone on gitter has recently successfully completed this with tegra-demo-distro just last month so I think this was working. Maybe share some logs from initrd-flash or connect a serial terminal so you can see what's failing during boot.

You might try with tegrademo rather than tegrademo-mender just in case there's a relevant difference there.

[dwalkes]

Also are you using jetson-xavier-nx-devkit-emmc or jetson-xavier-nx-devkit?

[DaveBubenik]

This is what I'm building (branch is kirstone-l4t-r32.7.x on tegra-test-distro):

MACHINE ?= "jetson-xavier-nx-devkit-emmc"
DISTRO ?= "testdistro"
TNSPEC_BOOTDEV = "nvme0n1p1"

I ended up having to modify the ROOTFSPART_SIZE in testdistro.conf to make the image fit the partition:

ROOTFSPART_SIZE:tegra:semi-stateless = "4861943808"

I run the following command to build:

bitbake demo-swupdate-full

initrd-flash logs:

Starting at 2023-11-10T10:36:18-06:00
Machine:       jetson-xavier-nx-devkit-emmc
Rootfs device: nvme0n1p1
Found Jetson device in recovery mode at USB 1-1
== Step 1: Signing binaries at 2023-11-10T10:36:18-06:00 ==
== Step 2: Boot Jetson via RCM at 2023-11-10T10:36:34-06:00 ==
Found Jetson device in recovery mode at USB 1-1
== Step 3: Sending flash sequence commands at 2023-11-10T10:36:44-06:00 ==
Waiting for USB storage device flashpkg from 901bf606.......[/dev/sda]
Device size in blocks: 262144
Unmounted /dev/sda.
== Step 4: Writing partitions on external storage device at 2023-11-10T10:37:05-06:00 ==
Waiting for USB storage device nvme0n1 from 901bf606...[/dev/sda]
Creating partitions
  [01] name=APP start=0 size=9495984 sectors
  [02] name=kernel start=0 size=131072 sectors
  [03] name=kernel_b start=0 size=131072 sectors
  [04] name=kernel-dtb start=0 size=896 sectors
  [05] name=kernel-dtb_b start=0 size=896 sectors
  [06] name=recovery start=0 size=129024 sectors
  [07] name=recovery-dtb start=0 size=1024 sectors
  [08] name=kernel-bootctrl start=0 size=512 sectors
  [09] name=kernel-bootctrl_b start=0 size=512 sectors
  [10] name=RECROOTFS start=0 size=614400 sectors
  [11] name=UDA (fills to end)
Writing partitions
  Writing demo-image-full.ext4 (size=4861943808) to /dev/sda1 (size=4861943808)...
  Writing signed/boot_sigheader.img.encrypt (size=29888512) to /dev/sda2 (size=67108864)...
  Writing signed/boot_sigheader.img.encrypt (size=29888512) to /dev/sda3 (size=67108864)...
  Writing signed/kernel_tegra194-p3668-all-p3509-0000_sigheader.dtb.encrypt (size=208624) to /dev/sda4 (size=458752)...
  Writing signed/kernel_tegra194-p3668-all-p3509-0000_sigheader.dtb.encrypt (size=208624) to /dev/sda5 (size=458752)...
[OK: /dev/sda]
== Step 5: Waiting for final status from device at 2023-11-10T10:44:01-06:00 ==
Waiting for USB storage device flashpkg from 901bf606...[/dev/sda]
Final status: SUCCESS
Unmounted /dev/sda.
Successfully finished at 2023-11-10T10:44:05-06:00
Host-side log:              log.initrd-flash.2023-11-10-10.36.18
Device-side logs stored in: device-logs-2023-11-10-10.36.18

I believe the image I'm building uses U-Boot. Maybe it requires further configuration? Unfortunately I'm unfamiliar with how to see logs during boot. If they can help diagnose the issue, I can try to get them, I just need a bit of direction on how to set this up.

[dwalkes]

(branch is kirstone-l4t-r32.7.x on tegra-test-distro):

I can reproduce on the test distro at https://github.com/madisongh/tegra-test-distro/tree/kirkstone-l4t-r32.7.x

However, I don't see any issues with nvme boot at tegra-demo-distro and branch https://github.com/OE4T/tegra-demo-distro/tree/kirkstone-l4t-r32.7.x I linked earlier, so there must be some difference between the two causing this. I'd probably start by bisecting differences between the two.

Unfortunately I'm unfamiliar with how to see logs during boot

See https://jetsonhacks.com/2019/04/19/jetson-nano-serial-console/

Here's a capture of the failing case - https://pastebin.com/g3HVLTrd

[DaveBubenik]

Thanks @dwalkes!

I went ahead and hooked up serial to see console, it made troubleshooting way easier.

I'm leaving this message for anyone else reading this page. I was attempting to use tegra-test-disto because I was trying to get swupdate running. I got swupdate working on tegra-test-distro when installed on the mmc, but the distro has problems when you use it on a nvme.

I ended up copying the swupdate pieces from tegra-test-disto to tegra-demo-distro instead. After copying, I had the exact same issue, the software would not boot off the nvme. After tons of trial and error, I found out the issue was being caused by the following code:
https://github.com/madisongh/tegra-test-distro/tree/dunfell/layers/meta-tegra-support/dynamic-layers/meta-rauc/recipes-bsp/cboot

It looks like those cboot updates are only needed by rauc, so I just deleted them and everything ran great afterwards (including swupdate). I initially included those files because there is some overlap between rauc and swupdate in tegra-test-distro and I wasn't sure what I needed.

I tried deleting the cboot directory from the tegra-test-distro, but then it failed to boot for a different reason. I’m going to move forward with my working code and ditch the tegra-test-demo.

[dwalkes]

Thanks @DaveBubenik, that's great you were able to get it working.

If you can share a fork of tegra-demo-distro with the swupdate bits working with NVMe that would be helpful for others who may have this issue later. I also might use this myself to test swupdate deployment on NVMe.

[DaveBubenik]

I'm actually integrating these changes in a different repo that I can't share publicly. I went ahead and created a fork with a quick and dirty copy of a working swupdate on a NVMe. I'm sure I've done a few things completely wrong but this is definitely a good place to start for others who want to use swupdate. I put instructions for using/verifying swupdate in the readme, so please read:

https://github.com/DaveBubenik/tegra-demo-distro/tree/kirkstone-l4t-r32.7.x

[dwalkes]

Awesome! Thanks for sharing.