[Silent Error] transactional-update fails to rebuild kdump initrd

[pavinjosdev]

Overview:

transactional-update (TU) silently fails to rebuild kdump initrd because /var/lib/kdump is not writable in the snapshot.
I'm running TU in a normal read-write Tumbleweed installation to perform dup.
System and packages are full updated as of Feb 22 2024. No failed systemd units or priority 3 errors in journal.

• Tumbleweed version: 20240220
• transactional-update version: 4.5.0-2.3
• tukit version: 4.5.0-2.3

Steps to Reproduce:

1. sudo transactional-update <kdump | dup>

Logs below are from calling dup where there was new kernel installed, thus necessitating rebuilding of initrd (success), bootloader (sucess), and kdump initrd (silent fail).

Actual results:

TU fails to regenerate kdump initrd as /var/lib/kdump is not mounted in the snapshot.

Feb 22 03:14:07 suse-pc transactional-update[12262]: Trying to rebuild kdump initrd
Feb 22 03:14:07 suse-pc transactional-update[24392]: 2024-02-22 03:14:07 tukit 4.5.0 started
Feb 22 03:14:07 suse-pc transactional-update[24392]: 2024-02-22 03:14:07 Options: --discard call 274 /sbin/mkdumprd
Feb 22 03:14:07 suse-pc transactional-update[24392]: 2024-02-22 03:14:07 Executing `/sbin/mkdumprd`:
Feb 22 03:14:07 suse-pc transactional-update[24392]: /var/lib/kdump not writable, not regenerating initrd.
Feb 22 03:14:07 suse-pc transactional-update[24392]: 2024-02-22 03:14:07 Application returned with exit status 0.
Feb 22 03:14:07 suse-pc transactional-update[24392]: 2024-02-22 03:14:07 Transaction completed.

There are also other warnings due to /var/lib/... contents being changed in the snapshot but which are not propagated to the new system.

Feb 22 03:14:18 suse-pc transactional-update[12262]: Warning: The following files were changed in the snapshot, but are shadowed by
Feb 22 03:14:18 suse-pc transactional-update[12262]: other mounts and will not be visible to the system:
Feb 22 03:14:18 suse-pc transactional-update[12262]: /.snapshots/274/snapshot/var/lib/YaST2/hooks/README.md
Feb 22 03:14:18 suse-pc transactional-update[12262]: /.snapshots/274/snapshot/var/lib/systemd/catalog/database
Feb 22 03:14:18 suse-pc transactional-update[12262]: /.snapshots/274/snapshot/var/lib/systemd/rpm/container-machines_subvol

Expected results:

TU successfully regenerates kdump initrd by mounting /var/lib/kdump in the snapshot.

[thkukuk]

TU successfully regenerates kdump initrd by mounting /var/lib/kdump in the snapshot.

Nothing should ever write something into /var during update. This breaks the whole concept of "atomic" and "transactional", because /var is not part of the snapshot, often even on an own partition. You can count everything, which writes into /var, as bug of that package.

That's why it is better to report bugs for openSUSE in the openSUSE bugzilla, so that the bug reports could be assigned to the right people and projects. That's not possible with github issues reported against the wrong project.

[pavinjosdev]

@thkukuk Ah okay, thanks! I will close this issue and create a bugzilla report.

[laenion]

I should add: This is just an informational message. The kdump initrd will be generated during boot instead, so it's working as designed.

[pavinjosdev]

Yes, i confirm this. But it's very annoying because i have to reboot twice on a kernel upgrade. First reboot after TU dup boots into the old kernel, some systemd service updates the kdump initrd, then on another reboot the system boots with the new kernel.

…

On Thu, Feb 22, 2024, 2:01 PM Ignaz Forster ***@***.***> wrote: I should add: This is just an informational message. The kdump initrd will be generated during boot instead, so it's working as designed. — Reply to this email directly, view it on GitHub <#119 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACXGU5FAI5URDU5JBVG2IQTYU367RAVCNFSM6AAAAABDUL4F46VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJYHE2DIMBTGA> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

[laenion]

That's not how it's supposed to work: The kdump initrd should be regenerated during boot right before it is loaded, so you only need to reboot once. If that wouldn't be working that would indeed be a bug...

[pavinjosdev]

The kdump initrd should be regenerated during boot right before it is loaded, so you only need to reboot once. If that wouldn't be working that would indeed be a bug...

Would using kexec reboot cause any issues with this process? I set it up as per the Suse docs.

[laenion]

Yes, that would indeed result in the behavior you see.

What you may want to use in this case is transactional-update reboot with REBOOT_METHOD set to kexec in the transactional-update configuration file: https://manpages.opensuse.org/Tumbleweed/transactional-update/transactional-update.conf.5.en.html

However the implementation in transactional-update also wasn't adopted accordingly when the kdump-initrd mechanism changed to the mechanism as it is today: The initrd will have to be recreated here, too.

Reopening.

[pavinjosdev]

@laenion Thanks, I changed my systemd timer's service file to get kdump and kexec to work correctly.
Took a lot of trial and error 😅

[Unit]
Description=Backup and transactional update/reboot

[Service]
Type=oneshot
ExecStartPre=/usr/local/bin/check-device
ExecStartPre=/usr/bin/borgmatic --verbosity 1 --syslog-verbosity 1
ExecStartPre=/usr/bin/snapper -c root create -c number -d 'RO snap before dup'
ExecStartPre=-/usr/bin/zypperoni ref
ExecStartPre=-/usr/bin/zypperoni --no-confirm dup-download
ExecStart=/usr/sbin/transactional-update --drop-if-no-change apply cleanup dup
ExecStartPost=/usr/bin/systemctl restart kdump.service
ExecStartPost=/usr/bin/systemctl restart kexec-load.service
ExecStartPost=/usr/bin/systemctl reboot