Trying to boot TalosOS on Volterra using kernel jg/ubuntu-qcom-x1e-6.17

[SUB0PT1MAL]

Hello, me again

I'm trying to make a custom TalosOS image for Volterra following the instructions here

I compiled the branch "jg/ubuntu-qcom-x1e-6.17" on Volterra itself to not need to deal with cross compiling following mostly the steps here.

I've been able to make the image and flash it on a thumbdrive, it boots fine to grub but fails to boot the OS with:

'''
EFI stub: Decompressing Linux Kernel...
EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA_GUID device path
EFI stub: Measured initrd data into PCR 9
EFI stub: Generating empty DTB
EFI stub: Exiting boot services...
'''

I have very little experience thinkering with kernels so i dont know how to troubleshoot this further.

TalosOS is a bit weird in what to OSes come so i dont even know if something like this is possible, but i cant find anyone that has tried this before.

[jglathe]

The output says "no dtb found", meaning it hasn't loaded a device tree - which would be sc8280xp-microsoft-blackrock.dtb Depending on the grub version (and some really odd circumstances I haven't quite understood) not all add the required devicetree line in the grub menu entry. To make it work you will need to:

• copy over the dtb from the /lib/firmware//devicetree/qcom/ directory to /boot/
• add devicetree /boot/sc8280xp-microsoft-blackrock.dtb in the boot menu entry in in /boot/grub/grub.cfg.

Please be aware that you also need some mandatory kernel command line parameters, or boot will fail. These are pd_ignore_unused clk_ignore_unused regulator_ignore_unused arm64.nopauth efi=noruntime on the WDK.

[SUB0PT1MAL]

Yeah... my dreams of a nice arm64 k8s node are cooked.

Sorry for throwing random stuff at you and thanks for your time.

[jglathe]

How about Turing RK1? I mean, thank you for pointing on Talos. I have an assembled Turing Pi box, and RK1 modules.

[SUB0PT1MAL]

I have several RPi4 and 2 jetson nano to play with arm stuff, but i had the volterra getting dusty and wanted to join it to my home lab/prod cluster.

It would be cool to get a RK1 but its very expensive and like 50% of the performance of the Voterra.

I cant justify getting one now.

i might run Talos DinD and join it to the cluster, its ugly and has limitations but i have computing power to spare if it fails and /or i want to get rid of it at any point.

I will probably open a thread on one of their forums to see if someone knows a way to customize a Talos image this granularly, if i manage to make a bootable Volterra image for Talos i will get in contact with you to see if we can make it available for everyone.

[jglathe]

Hmm if VM is a requirement, don't :) I use one Volterra as a desktop, and its nice. Another one down here in the compute cellar running on EL2 with /dev/kvm and zfs for tests and stuff. The RK1 has ~75% of compile compute of the Volterra, and yes itd probably expensive for what it is. I was on the kickstarter for the Turing Pi 2, and that was befor Volterra was available. :)

[SUB0PT1MAL]

Docker seems to work fine on that ubuntu distro you provide, no need for VM stuff.

Those tiny cluster boards are really cool, but the price kills them, for what it costs to get and populate one i can get an itx am4 board with a R9 and 64gb of ddr5 (maybe not anymore)

my cluster has currently 3 nodes:

• Master: MeLE Quieter 4C N100 16gb
• Storage Node: E5-2696 v3 64 GiB with 8 nvme of storage 16tb total
• Compute Node: R9 5900X 64GiB RTX 3090 and Quadro P2000

I have many workloads that are simple webpages, webhooks or APIs and i thought it would be fun to have an efficient node to run them on, but it was only pain and no gain hahaha

[jglathe]

Hmm looks like you could use Talos using docker, so this would go without /dev/kvm. Not as elegant, but whatever works 🤷 https://docs.siderolabs.com/talos/v1.11/platform-specific-installations/local-platforms/docker

[SUB0PT1MAL]

Hey, yeah, thats what i meant with DinD, Docker in Docker

[SUB0PT1MAL]

I MADE PROGRESS!!

I found how the systemd-boot is used by talos and i added the files necessary.

Now the device tree is loaded and the kernel starts, but i get a kernel panic:

'''
1970/01/01 00:00:00 recovered from: early boot failed: error mounting efivars: 1 error(s) occurred:
operation not supported
'''

and then it reboots.

I tried adding "systemd.efi_vars=0" and "efi=noruntime" but nothing changed.

[jglathe]

Hmm that one sounds like the qcom_scm driver isn't loaded, or better built-in. In my config it is built-in and it has to be IMO. The compatibility needs to be explicitly stated (which it is). Do you build the kernel yourself?

I would recommend to use the .config that you can create from my repo with ./devbuildconfig.sh as base for whatever TalosOs wants to have as config.

[SUB0PT1MAL]

Yeah I compiled it myself, but i only compiled kernels a couple of times so im not sure i did it right or i selected the right stuff on the configuration.

I dont remember changing anything tho, does this come enabled by default?

I might have to compile it a couple more times before i can make it work

[jglathe]

there is a ./rebuild.sh in the repo, unfortunately it has 2 dependencies:

• my signing key (easy to replace or reconfigure for non-signing)
• static dependency to the stubble repository with my username, in the debian makefiles.

I need to clean that up eventually.

But as I mentioned, using ./devbuildconfig.sh creates the normal .config file from the ubuntu annotations I am using. After executing this (and saving the config) just execute the following:

make -j8 bindeb-pkg

Wait 60 mins or so, voila.

if there are missing dependencies, after executing ./rebuild.sh at least once (even though it fails):

sudo apt-get build-dep .

Should list and install all missing dependencies.

[SUB0PT1MAL]

Finally got around back to this.

yeah, when i compiled it the first time i did run "./devbuildconfig.sh", but i have no clue what options if any i had to modify, so i just looked for the initramfs settings as Talos needs (i think) and thats it, i compiled it like that.

I just have been exploring the settings on that script but cant find anything efivar related.

I'm pretty lost atm.

[SUB0PT1MAL]

I recompiled the kernel, dpkg'ed it, and used the vmlinuz file there to boot Talos, but got the same error.