Multi-booting for the Jetson AGX Xavier with NVMe SSD

This project provides tools to add boot entry to extlinux.conf for a partition with JetPack installed. This allows multi-booting via the serial console.

Overview

Why NVMe SSD

• SSD provides much faster IO speed

/dev/mmcblk0p1 (internal EMMC): Bufferd reads = 292 MB/s
/dev/mmcblk1p1 (Sandisk Ultra U1 SD card): Bufferd reads = 81 MB/s
/dev/nvme0n1p1 (Kingston NVME M.2): Bufferd reads = 1316 MB/s
/dev/nvme0n1p1 (970 EVO Plus M.2): Bufferd reads = 1670 MB/s

• SSD has larger storage capacity (>1TB) than eMMC (32GB)
• There is no way to upgrade/replace eMMC (although NVIDIA claims it can last up to 5 years)

Why Multi-booting

• Try different JetPack versions without changing SSD or reflash. (For example, JetPack 4.4 on /dev/nvme0n1p1, JetPack 4.3 on /dev/nvme0n1p2)
• Run different projects with same JetPack version without changing SSD or reflash (Say, jp44_human_pose on /dev/nvme0n1p1, jp44_jetbot on /dev/nvme0n1p2)
• Replace a SSD without reflash (Maybe john_jetpack_44 on /dev/nvme0n1p1, jane_jetpack_45 on /dev/nvme0n1p1)
• Just for fun

What JetPack Versions are Supported:

The project has been tested with the following JetPack versions:

• JetPack 4.5.1
• JetPack 4.4.1
• JetPack 4.3
• JetPack 4.2.3

Preparation

This section will help you to prepare the bootable partition(s) with desired JetPack version. We assume that you know how to format disk with GPT partitioning scheme and create/delete partition(s) using GNOME Disk Utility

How to Flash

Method 1: Use SDK Manager GUI

1. Connect Jetson Xavier (in Recovery Mode) to the host with SDK Manager installed
2. Select target operating system (JetPack version)
3. Complete the Ubuntu system configuration

Method 2: Use flash.sh

This method is more precise but only available when Jetson OS image has been created before (The path varies from version to version. By default it should be under user's home folder). For example, to flash JetPack 4.4.1:

$ cd ~/nvidia/nvidia_sdk/
$ cd JetPack_4.4.1_Linux_JETSON_AGX_XAVIER/Linux_for_Tegra
$ sudo ./flash.sh jetson-xavier mmcblk0p1

After flashing, complete the Ubuntu system configuration.

Clone rootfs to the target partition

My method here is simple but not a beautiful/clean cloning. You could find other way from Internet or use it as-is. Now, Jetson Xavier should be booted with JetPack version installed in previous step from eMMC. If we would like to clone rootfs to /dev/nvme0n1p1, please follow the steps below:

$ mkdir /tmp/rootfs
$ sudo mount /dev/nvme0n1p1 /tmp/rootfs
$ cd /
$ sudo cp -ax ./ /tmp/rootfs

Prepare a Backup entry on eMMC

This allow you to boot Jetson Xavier even there is no SSD installed (or something wrong with the SSD, point to wrong partition or incorrect boot entry). We should install the latest JetPack version (to support booting older version). Follows section How to Flash to flash JetPack 4.5.1

After that, JetPack 4.5.1 should be installed on eMMC and JetPack 4.4.1 on /dev/nvme0n1p1.

Add Boot Entry

To continue the previous example, to enable multi-booting (JetPack 4.5.1 on eMMC and JetPack 4.4.1 on /dev/nvme0n1p1), we could run the following steps:

$ sudo -i
# /path/to/multi-booting-xavier/add_jetpack.sh /dev/nvme0n1p1 human_pose

Once completed successfully and reboot, you could find something like the following via the serial console:

[0005.815] I> L4T boot options
[0005.815] I> [1]: "primary kernel"
[0005.815] I> [2]: "JetPack 4.4 human_pose"
[0005.816] I> Enter choice:

At that point, you can choose the kernel before expiration of the TIMEOUT period (3 seconds).

A Complete Example

Let's say the NVMe SSD has four partitions and are assigned as the following (You can check the final multi-booting configuration):

Partition	JetPack Version	Project
/dev/nvme0n1p1	JetPack 4.5.1	human_pose
/dev/nvme0n1p2	JetPack 4.4.1	jp44_jetbot
/dev/nvme0n1p3	JetPack 4.3	DeepSpeech
/dev/nvme0n1p4	JetPack 4.2.3	smart_detector

To enable multi-booting with different JetPack versions like that. Just follow the steps below:

1. Flash JetPack 4.2.3 to eMMC
2. Clone rootfs to /dev/nvme0n1p4
3. Flash JetPack 4.3 to eMMC
4. Clone rootfs to /dev/nvme0n1p3
5. Flash JetPack 4.4.1 to eMMC
6. Clone rootfs to /dev/nvme0n1p2
7. Flash JetPack 4.5.1 to eMMC
8. Clone rootfs to /dev/nvme0n1p1
9. Run add_jetpack.sh /dev/nvme0n1p1 human_pose
10. Run add_jetpack.sh /dev/nvme0n1p2 jp44_jetbot
11. Run add_jetpack.sh /dev/nvme0n1p3 DeepSpeech
12. Run add_jetpack.sh /dev/nvme0n1p4 smart_detector
13. Done

When the system boots, you could find something like the following via the serial console:

[0005.815] I> L4T boot options
[0005.815] I> [1]: "primary kernel"
[0005.815] I> [2]: "JetPack 4.5 human_pose"
[0005.815] I> [3]: "JetPack 4.4 jp44_jetbot"
[0005.815] I> [4]: "JetPack 4.3 DeepSpeech"
[0005.816] I> [5]: "JetPack 4.2 smart_detector"
[0005.816] I> Enter choice:

You can select the kernel everytime or you could update the default by editing extlinux.conf

Notes

Maybe you will find that it is not that useful as you need to flash the eMMC and clone the rootfs everytime for a new partition.

To avoid the redundant work, it is recommended to backup the partition(s) you need. You can use dd, cpio or a spare storage to keep the partition(s).

Next time, you can simply restore/clone the backup to the new partition, run the command add_jetpack.sh on Jetson Xavier and done.

Miscellaneous

Force Recovery Mode (when Power Off)

1. Press and hold down the Force Recovery button
2. Press and hold down the Power button
3. Release both buttons

Force Recovery Mode (when Power On)

1. Press and hold down the Force Recovery button
2. Press and hold down the Reset button
3. Release both buttons

Edit extlinux.conf

Be Cautious. Invalid extlinux.conf will cause the system to become unbootable. Backup before you edit, think twice before you act, double check before you save.

To ensure we are editing the correct extlinux.conf, do the following:

$ mkdir /tmp/emmc
$ sudo mount /dev/mmcblk0p1 /tmp/emmc
$ sudo vi /tmp/emmc/boot/extlinux/extlinux.conf

Reference

• NVIDIA Jetson Xavier - CBoot
• Disk Utility
• How to Boot from NVMe SSD
• NVIDIA Jetson Xavier - Using the serial console
• Quad-RS232 Driver for Windows