I/O error mounting volume created while userspace tools and kernel module mismatch

[dscottboggs]

What happened: after a system update (NixOS), but before a reboot was performed, a volume was created and data was written to it. After a reboot was eventually performed, the volume would not mount due to an I/O error.

More details: I discovered the version mismatch while investigating an error being produced by zfs send ("warning: cannot send [volume@snapshot]: Invalid argument"). I found that zfs --version output two different versions, ran another system update and rebooted.

$ # BEFORE reboot
# zfs --version
zfs-2.2.6-1
zfs-kmod-2.2.4-1
$ # AFTER reboot
$ uname -a
Linux revivajxo 6.6.58 #1-NixOS SMP PREEMPT_DYNAMIC Tue Oct 22 13:46:36 UTC 2024 x86_64 GNU/Linux

Relevant link: https://discourse.practicalzfs.com/t/error-syncing-some-datasets-invalid-argument-from-zfs-send/1896

Edit: I do still have a copy of the corrupted dataset so if there is more information about it I can give you please do let me know.

[dscottboggs]

Interesting, the copy of that dataset that I have is now refusing to send via zfs send

sudo zfs send revivajxo2_dup/mastodon-postgres-corrupted | pv -pteras 4664520704 | sudo zfs recv -o keylocation=file:///etc/revivajxo2.key revivajxo/mastodon-postgres-corrupted
warning: cannot send 'revivajxo2_dup/mastodon-postgres-corrupted': Invalid argument
0:00:00 [0.00  B/s] [0.00  B/s] [>                                                               ]   0%            
cannot receive: failed to read from stream