Problem with nvidia extension - Modules not found

[AndersVikenes]

Hi,

I'm following the guide for talos with nvida gpu. Created an iso from factory.talos.dev with Nvidia OSS extension: nvidia-open-gpu-kernel-modules and nvidia-container-toolkit. Did install on a clean disk, then removing the iso and doing multiple reboot. After enabling the kernel-modules, talos cannot find any of the nvidia modules.

Error in the logs is:
[talos] controller failed {"component": "controller-runtime", "controller": "runtime.KernelModuleSpecController", "error": "error loading module "nvidia": module not found error loading module "nvidia_drm": module not found error loading module "nvidia_modeset": module not found error loading module "nvidia_uvm": module not found"}

Where does talos put kernel modules (nvidia.ko, nvidia-modeset.ko, nvidia-uvm.ko, nvidia-drm.ko, and nvidia-peermem.ko), so i can check if they actually are present?

[smira]

When installing from the ISO, make sure you use a proper installer image which contains extensions as well.

https://www.talos.dev/v1.8/talos-guides/install/boot-assets/#example-bare-metal-with-image-factory

Or follow https://factory.talos.dev/ wizard.

[AndersVikenes]

Thank you for your response. I did use the factory wizard to create the iso. Selecting both the nvidia extensions together with siderolabs/vmtoolsd-guest-agent, and the iso-image ended up around 306MB, compared to default iso-image which is around 108mb. However I still have the same error. I tried to delete/recreate the virtual disk to be 100% sure i didn't have a old installation there.

Should I be able to list a folder on the node with the modules to verify?

[smira]

When you boot from an ISO, the first step is to apply a machine configuration to the machine. That machine configuration contains install image in .machine.install.image - it should point to the one from the Image Factory page, not a generic one.

You can verify that extensions are installed with talosctl get extensions

[AndersVikenes]

I'm trying that now. I'm booting with the factory-generated iso with extensions, and providing url to the factory-image in the machine-configuration, however it is then trying to contact factory.talos.dev/v2/installer/......../manifests/v1.8.3 during install.

What is the correct way to apply image in .machine.install.image in air-gapped environment?

Doesn't the generated iso with extension contain the installer image?

[smira]

In air-gapped environment you can mirror that image to your internal registry and use it, like any other external image the cluster might need.

ISO doesn't contain the installer image.

[AndersVikenes]

Thank you very much @smira for your help. I thought generating an iso with the extensions would install the extension as well. Uploading the image with extensions from factory to a local registry and adding mirror config for factory.talos.dev in the machine config solved it.