zfs-2.2.6 patchset #16472

tonyhutter · 2024-08-22T22:53:37Z

Motivation and Context

Proposed patchset for zfs-2.2.6.

Description

Support 6.10 kernel, misc bugfixes.

How Has This Been Tested?

Buildbot will test

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Performance enhancement (non-breaking change which improves efficiency)
Code cleanup (non-breaking change which makes code smaller or more readable)
Breaking change (fix or feature that would cause existing functionality to change)
Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
Documentation (a change to man pages or other documentation)

Checklist:

My code follows the OpenZFS code style requirements.
I have updated the documentation accordingly.
I have read the contributing document.
I have added tests to cover my changes.
I have run the ZFS Test Suite with this change applied.
All commit messages are properly formatted and contain Signed-off-by.

Simplify the test, by using the variable "$PLATFORM_ID" in favor of "$REDHAT_SUPPORT_PRODUCT_VERSION". Signed-off-by: Tino Reichardt <[email protected]> Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: George Melikov <[email protected]>

SET_ERROR is our facility for tracking errors internally. The negation is to match the what the kernel expects from us. Thus, the negation should happen outside of the SET_ERROR. Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Sponsored-by: Klara, Inc. Sponsored-by: Wasabi Technology, Inc. Signed-off-by: Rob Norris <[email protected]> Closes openzfs#16364

Make sure log record don't stray beyond valid memory region. There is a lack of verification of the space occupied by fixed members of lr_t in the zil_parse. We can create a crafted image to trigger an out of bounds read by following these steps: 1) Do some file operations and reboot to simulate abnormal exit without umount 2) zil_chain.zc_nused: 0x1000 3) First lr_t lr_t.lrc_txtype: 0x0 lr_t.lrc_reclen: 0x1000-0xb8-0x1 lr_t.lrc_txg: 0x0 lr_t.lrc_seq: 0x1 4) Update checksum in zil_chain.zc_eck Fix: Add some checks to make sure the remaining bytes are large enough to hold an log record. Signed-off-by: XDTG <[email protected]> Reviewed-by: Alexander Motin <[email protected]> Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]>

ZFS implements copy_file_range(2) using block cloning when possible. This implementation must respect the RLIMIT_FSIZE limit. zfs_clone_range() already checks the limit, so it is safe to remove this check in zfs_freebsd_copy_file_range(). Moreover, the removed check produces false positives: the length passed to copy_file_range(2) may be larger than the input file size; as the man page notes, "for best performance, call copy_file_range() with the largest len value possible." In particular, some existing code passes SSIZE_MAX there. The check in zfs_clone_range() clamps the length to the input file's size before checking, but the removed check uses the caller supplied length, so something like $ echo a > /tmp/foo $ limits -f 1024 cat /tmp/foo > /tmp/bar fails because FreeBSD's cat(1) uses copy_file_range(2) in the manner described above. Reported-by: Philip Paeps <[email protected]> Signed-off-by: Mark Johnston <[email protected]> Reviewed-by: Alexander Motin <[email protected]> Reviewed-by: Allan Jude <[email protected]> Reviewed-by: Tino Reichardt <[email protected]> Reviewed-by: Tony Hutter <[email protected]>

Signed-off-by: Mark Johnston <[email protected]> Reviewed-by: Alexander Motin <[email protected]> Reviewed-by: Allan Jude <[email protected]> Reviewed-by: Tino Reichardt <[email protected]> Reviewed-by: Tony Hutter <[email protected]>

In 6.11 struct queue_limits gains a 'features' field, where, among other things, flush and write-cache are enabled. Detect it and use it. Along the way, the blk_queue_set_write_cache() compat wrapper gets a little cleanup. Since both flags are alway set together, its now a single bool. Also the very very ancient version that sets q->flush_flags directly couldn't actually turn it off, so I've fixed that. Not that we use it, but still. Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Rob Norris <[email protected]> Sponsored-by: https://despairlabs.com/sponsor/ Closes openzfs#16400

It's no longer available directly on the request queue, but its easy to get from the attached disk. Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Rob Norris <[email protected]> Sponsored-by: https://despairlabs.com/sponsor/ Closes openzfs#16400

Detect it, and use a macro to make sure we always match the prototype. Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Rob Norris <[email protected]> Sponsored-by: https://despairlabs.com/sponsor/ Closes openzfs#16400

Apply them with with the rest of the settings. Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Rob Norris <[email protected]> Sponsored-by: https://despairlabs.com/sponsor/ Closes openzfs#16400

These fields are very old, so no detection necessary; we just move them into the limit setup functions. Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Rob Norris <[email protected]> Sponsored-by: https://despairlabs.com/sponsor/ Closes openzfs#16400

Since the change to folios it has just been a wrapper anyway. Linux has removed their wrapper, so we add one. Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Rob Norris <[email protected]> Sponsored-by: https://despairlabs.com/sponsor/ Closes openzfs#16400

Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Rob Norris <[email protected]> Sponsored-by: https://despairlabs.com/sponsor/ Closes openzfs#16400

Use /dev/urandom so we never have to wait on entropy. Reviewed-by: George Melikov <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Tony Hutter <[email protected]> Closes openzfs#16442

The 6.10 kernel broke our rpm-kmod builds. The 6.10 kernel really wants the source files in the same directory as the object files. This workaround makes rpm-kmod work again. It also updates the builtin kernel codepath to work correctly with 6.10. See kernel commits: b1992c3772e6 kbuild: use $(src) instead of $(srctree)/$(src) for source directory 9a0ebe5011f4 kbuild: use $(obj)/ instead of $(src)/ for common pattern rules Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Tony Hutter <[email protected]> Closes openzfs#16439 Closes openzfs#16450

zvol_alloc_non_blk_mq()->blk_queue_set_write_cache() needs the disk queue setup to prevent a NULL pointer deference. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Tony Hutter <[email protected]> Closes openzfs#16453

openzfs#16316) If a zvol is renamed, and it has one or more snapshots, and snapdev=visible is true for the zvol, then the rename causes a kernel null pointer dereference error. This has the effect (on Linux, anyway) of killing the z_zvol taskq kthread, with locks still held; which in turn causes a variety of zvol-related operations afterward to hang indefinitely (such as udev workers, among other things). The problem occurs because of an oversight in openzfs#15486 (e36ff84). As documented in dataset_kstats_create, some datasets may not actually have kstats allocated for them; and at least at the present time, this is true for snapshots. In practical terms, this means that for snapshots, dk->dk_kstats will be NULL. The dataset_kstats_rename function introduced in the patch above does not first check whether dk->dk_kstats is NULL before proceeding, unlike e.g. the nearby dataset_kstats_update_* functions. In the very particular circumstance in which a zvol is renamed, AND that zvol has one or more snapshots, AND that zvol also has snapdev=visible, zvol_rename_minors_impl will loop over not just the zvol dataset itself, but each of the zvol's snapshots as well, so that their device nodes will be renamed as well. This results in dataset_kstats_create being called for snapshots, where, as we've established, dk->dk_kstats is NULL. Fix this by simply adding a NULL check before doing anything in dataset_kstats_rename. This still allows the dataset_name kstat value for the zvol to be updated (as was the intent of the original patch), and merely blocks attempts by the code to act upon the zvol's non-kstat-having snapshots. If at some future time, kstats are added for snapshots, then things should work as intended in that case as well. Signed-off-by: Justin Gottula <[email protected]> Reviewed-by: Rob Norris <[email protected]> Reviewed-by: Alexander Motin <[email protected]> Reviewed-by: Alan Somers <[email protected]> Reviewed-by: Allan Jude <[email protected]> Reviewed-by: Tony Hutter <[email protected]>

In kernels 6.8 and later, the zvol block device is allocated with qlimits passed during initialization. However, the zvol driver does not set `max_hw_discard_sectors`, which is necessary to properly initialize `max_discard_sectors`. This causes the `zvol_misc_trim` test to fail on 6.8+ kernels when invoking the `blkdiscard` command. Setting `max_hw_discard_sectors` in the `HAVE_BLK_ALLOC_DISK_2ARG` case resolve the issue. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Rob Norris <[email protected]> Signed-off-by: Ameer Hamza <[email protected]> Closes openzfs#16462

Rob Noris suggested that we could clean up redundant limits for the case of non-blk mq scenario. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Rob Norris <[email protected]> Signed-off-by: Ameer Hamza <[email protected]> Closes openzfs#16462

Update the META file to reflect compatibility with the 6.10 kernel. Reviewed-by: Rob Norris <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Tony Hutter <[email protected]> Closes openzfs#16466

robn

This one is tight, I like it. Good stuff!

tonyhutter · 2024-08-22T23:53:12Z

I also need 767b370 bdf4d6b - let me fix that.

jumbi77 · 2024-08-23T18:22:39Z

Maybe too early (?) but what about to include this recent fix: 2420ee6 (not sure if the original PR #16171 is in 2.2.x... if not just ignore my message).

Harry-Chen · 2024-08-23T18:30:22Z

I looked back at #16359 and noticed some suggestions that were not in 2.2.5: #16376, #16409, #16410, #16411.

Also there are two of my early fixes that could be quickly pulled: #15557, #15623.

jumbi77 · 2024-08-23T18:33:44Z

I looked back at #16359 and noticed some suggestions that were not in 2.2.5: #16376, #16409, #16410, #16411.

Also there are two of my early fixes that could be quickly pulled: #15557, #15623.

Beside that, I just got a general question: I tried using GitHub compare master->ifs-2.2.6-staging. It shows a lot of commits, but I know at least some of them are included in 2.2.x, just as another commit-hash I think... is there a better/nice way to really check which underlying changes/commits are not included from master?

robn · 2024-08-24T03:07:15Z

@jumbi77 git cherry zfs-2.2.6-staging master -v is good for this (show commits that aren't in zfs-2.2.6-staging but not in master, by content).

This patch uses __ARM_ARCH set by compiler (both GCC and Clang have this) whenever possible instead of hardcoding it to 7. This change allows code to compile on earlier ARM architectures such as armv5te. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Shengqi Chen <[email protected]> Closes openzfs#15557

My merged pull request openzfs#15557 fixes compilation of sha2 kernels on arm v5/6. However, the compiler guards only allows sha256/512_armv7_impl to be used when __ARM_ARCH > 6. This patch enables these ASM kernels on all arm architectures. Some compiler guards are adjusted accordingly to avoid the unnecessary compilation of SIMD (e.g., neon, armv8ce) kernels on old architectures. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Shengqi Chen <[email protected]> Closes openzfs#15623

Currently user won't have completion of `zpool` command until they trigger completion of `zfs` first. This patch adds a link to `zfs`, thus user can use both to initialize the completion. Fixes: openzfs#16320 Signed-off-by: Shengqi Chen <[email protected]> Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Rob Norris <[email protected]> Reviewed-by: Tino Reichardt <[email protected]>

The timezone "US/Mountain" isn't supported on newer linux versions. Using the correct timezone "America/Denver" like it's done in FreeBSD will fix this. Older Linux distros should behave also okay with this. Signed-off-by: Tino Reichardt <[email protected]> Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: George Melikov <[email protected]>

This test was failing before: - FAIL cli_root/zfs_copies/zfs_copies_006_pos (expected PASS) Signed-off-by: Tino Reichardt <[email protected]> Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: George Melikov <[email protected]>

It gets hairier again in Linux 6.11, so I want some actual theory of operation laid out for next time. Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Rob Norris <[email protected]> Sponsored-by: https://despairlabs.com/sponsor/ Closes openzfs#16400

zvol queue limits initialization depends on `zv_volblocksize`, but it is initialized later, leading to several limits being initialized with incorrect values, including `max_discard_*` limits. This also causes `blkdiscard` command to consistently fail, as `blk_ioctl_discard` reads `bdev_max_discard_sectors()` limits as 0, leading to failure. The fix is straightforward: initialize `zv->zv_volblocksize` early, before setting the queue limits. This PR should fix `zvol/zvol_misc/zvol_misc_trim` failure on recent PRs, as the test case issues `blkdiscard` for a zvol. Additionally, `zvol_misc_trim` was recently enabled in `6c7d41a`, which is why the issue wasn't identified earlier. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Tony Hutter <[email protected]> Signed-off-by: Ameer Hamza <[email protected]> Closes openzfs#16454

jaen · 2024-08-27T18:10:39Z

Any chance to get #16402 in for this, or building own modules still the only choice?

rincebrain · 2024-08-27T20:44:28Z

I looked back at #16359 and noticed some suggestions that were not in 2.2.5: #16376, #16409, #16410, #16411.
Also there are two of my early fixes that could be quickly pulled: #15557, #15623.

Beside that, I just got a general question: I tried using GitHub compare master->ifs-2.2.6-staging. It shows a lot of commits, but I know at least some of them are included in 2.2.x, just as another commit-hash I think... is there a better/nice way to really check which underlying changes/commits are not included from master?

github lets you select a branch, you could diff against zfs-2.2.5 or zfs-2.2-release.

`l2arc_mfuonly` was added to avoid wasting L2 ARC on read-once MRU data and metadata. However it can be useful to cache as much metadata as possible while, at the same time, restricting data cache to MFU buffers only. This patch allow for such behavior by setting `l2arc_mfuonly` to 2 (or higher). The list of possible values is the following: 0: cache both MRU and MFU for both data and metadata; 1: cache only MFU for both data and metadata; 2: cache both MRU and MFU for metadata, but only MFU for data. Reviewed-by: Alexander Motin <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Gionatan Danti <[email protected]> Closes openzfs#16343 Closes openzfs#16402

META file and changelog updated. Signed-off-by: Tony Hutter <[email protected]>

tonyhutter · 2024-08-27T21:55:46Z

@jumbi77 taskq stats aren't in 2.2.x, so I skipped 2420ee6
@Harry-Chen I pulled all those in except #16411.
@jaen I pulled in #16402

mcmilk and others added 19 commits August 22, 2024 15:06

Linux 6.11: avoid passing "end" sentinel to register_sysctl()

8479a45

Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Rob Norris <[email protected]> Sponsored-by: https://despairlabs.com/sponsor/ Closes openzfs#16400

ZTS: Use /dev/urandom instead of /dev/random

ce22dc2

Use /dev/urandom so we never have to wait on entropy. Reviewed-by: George Melikov <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Tony Hutter <[email protected]> Closes openzfs#16442

Linux 6.10 compat: Fix zvol NULL pointer deference

84a9861

zvol_alloc_non_blk_mq()->blk_queue_set_write_cache() needs the disk queue setup to prevent a NULL pointer deference. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Tony Hutter <[email protected]> Closes openzfs#16453

Linux 6.10 compat: META

86492e3

Update the META file to reflect compatibility with the 6.10 kernel. Reviewed-by: Rob Norris <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Tony Hutter <[email protected]> Closes openzfs#16466

tonyhutter changed the title ~~Zfs 2.2.6 hutter~~ zfs-2.2.6 patchset Aug 22, 2024

tonyhutter mentioned this pull request Aug 22, 2024

Linux 6.10 compatibility and new promptly release #16425

Closed

robn approved these changes Aug 22, 2024

View reviewed changes

behlendorf added the Status: Code Review Needed Ready for review and testing label Aug 23, 2024

Harry-Chen added 2 commits August 26, 2024 15:10

Harry-Chen and others added 5 commits August 26, 2024 15:10

tonyhutter force-pushed the zfs-2.2.6-hutter branch from 018b9cb to a81918b Compare August 26, 2024 22:12

shodanshok and others added 2 commits August 27, 2024 14:53

Tag zfs-2.2.6

baa5031

META file and changelog updated. Signed-off-by: Tony Hutter <[email protected]>

tonyhutter force-pushed the zfs-2.2.6-hutter branch from a81918b to baa5031 Compare August 27, 2024 21:53

bsherman mentioned this pull request Aug 28, 2024

feat: gated fsync kernel (matches coreos-stable) ublue-os/akmods#234

Merged

tonyhutter merged commit baa5031 into openzfs:zfs-2.2-release Sep 4, 2024
22 of 23 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

zfs-2.2.6 patchset #16472

zfs-2.2.6 patchset #16472

tonyhutter commented Aug 22, 2024

robn left a comment

tonyhutter commented Aug 22, 2024

jumbi77 commented Aug 23, 2024 •

edited

Loading

Harry-Chen commented Aug 23, 2024

jumbi77 commented Aug 23, 2024 •

edited

Loading

robn commented Aug 24, 2024

jaen commented Aug 27, 2024

rincebrain commented Aug 27, 2024

tonyhutter commented Aug 27, 2024

zfs-2.2.6 patchset #16472

zfs-2.2.6 patchset #16472

Conversation

tonyhutter commented Aug 22, 2024

Motivation and Context

Description

How Has This Been Tested?

Types of changes

Checklist:

robn left a comment

Choose a reason for hiding this comment

tonyhutter commented Aug 22, 2024

jumbi77 commented Aug 23, 2024 • edited Loading

Harry-Chen commented Aug 23, 2024

jumbi77 commented Aug 23, 2024 • edited Loading

robn commented Aug 24, 2024

jaen commented Aug 27, 2024

rincebrain commented Aug 27, 2024

tonyhutter commented Aug 27, 2024

jumbi77 commented Aug 23, 2024 •

edited

Loading

jumbi77 commented Aug 23, 2024 •

edited

Loading