IoT Provisioner infinitely loops during install to /dev/mmcblk devices

[Sebmaster]

Describe the bug

During install of IoT Provisioner to a mmcblk device, coreos-installer-growfs.service will fail (without log output). I believe this applies equally to nvme devices. I think the root cause is an issue in coreos-installer-dracut (coreos/coreos-installer-dracut#35) which got fixed upstream, but never packaged for Fedora.

To Reproduce

1. Start the Fedora IoT via UEFI HTTP boot with a grub entry like linuxefi /images/pxeboot/vmlinuz rd.neednet=1 coreos.inst.crypt_root=1 coreos.inst.install_dev=/dev/mmcblk0 coreos.inst.image_url=http://<SERVER_HOST>/image.raw.xz coreos.inst.insecure quiet coreos.inst.append=ignition.config.url=http://<SERVER_HOST>/dell.ign coreos.inst.append=rd.neednet=1
2. Wait for install to finish
3. Install never finishes.

I've tried figuring out a repro for libvirt, but mmcblk seems to be unsupported and nvme proved annoying to set up.

Expected behavior

Install should finish successfully with a shell/powered off/rebooted depending on cmdline.

Screenshots

Log output (best-effort transcription)

[TIMESTAMP] coreos-installer-service[PID]: Modifying kernel arguments
[TIMESTAMP] coreos-installer-service[PID]: Install complete.
Failed to start coreos-installer-growfs.service - Grow root filesystem.
Dependency failed for initrd.target - Initrd Default Target.
[TIMESTAMP] coreos-installer-service[PID]: coreos-installer install <...>
[TIMESTAMP] coreos-installer-service[PID]: Downloading image from [SERVER_HOST]
...
[TIMESTAMP] coreos-installer-service[PID]: Partitions in use on /dev/mmcblk0
[TIMESTAMP] coreos-installer-service[PID]:    /dev/mmcblk0p3 in use by /dev/dm-0

this continues to loop (forever?).

OS version:

Fedora-IoT-provisioner-41-20241027.0.x86_64

Additional context

Originally this bug was reported in RH #2232406 for RHEL Edge and got fixed (in the RH package repo?). Apparently this fix didn't make it into the Fedora packages. I created https://bugzilla.redhat.com/show_bug.cgi?id=2328530 for Fedora specifically.

[miabbott]

Nice analysis and conclusion!

It's a sad state of affairs that the last change to coreos-installer-dracut that landed in Fedora is from April 2023 (coreos/coreos-installer-dracut@a4be31d)

We should propose an update to the rust-coreos-installer package (https://src.fedoraproject.org/rpms/rust-coreos-installer) to include the latest changes from coreos-installer-dracut

[Sebmaster]

@miabbott Already sent out a PR in https://src.fedoraproject.org/rpms/rust-coreos-installer/pull-request/72, but not sure where to poke to get it reviewed.

[miabbott]

@Sebmaster Excellent! I'll reach out to some folks who can assist.