Physical rewrite

[amotin]

Motivation and Context

The earlier implemented zfs rewrite functionality for simplicity updated logical birth times of all rewritten blocks. It makes them look modified from perspective of replication, snapshot diffs, etc, even though the actual user data remain the same. While some people found it useful to recover corrupted remote backups, for majority replication of large extra amounts of logically unchanged blocks can be a huge waste of time and resources.

Description

This PR implements a new variation of rewrite, called "physical rewrite", controlled by the new -P argument to the zfs rewrite subcommand. When possible, it tries to keep logical birth times unchanged. It allows to distinguish blocks that were just relocated within a pool from blocks that were actually modified by users. While the first may occupy additional disk space due to snapshots, block cloning, etc, that should be accounted as such, they should be ignored by replication, etc.

Previously we've had block pointers with physical birth times bigger than logical birth times only as result of device removal remap process. But in that case space usage accounting was still based on block's logical birth times. Since physical rewrites require space reallocation accounted based on the physical birth times, to differentiate those two cases this PR introduces new "R"/"rewrite" flag in the block pointer structure. When set, it means the block's space accounting should use physical birth time instead of traditional logical birth time. Since read-only pool imports do not really care about space accounting, the new per-dataset pool feature "physical_rewrite" gating this is declared as read-compatible. The feature will be activated on first use and deactivated when last of affected datasets is deleted.

There are two exceptions when logical birth time might still be modified around physical rewrite:

• In case of dedup hit, producing a different block pointer due to change of checksum algorithm or number of copies. Since the physical birth time must come from the DDT record, we can not put the current TXG there for space accounting, and have to update logical birth time instead, as done for logical rewrite. Aside of this quite rare case attempts to do physical rewrite on dedup'ed blocks should be NOP, but it can be used to enable/disable dedup.
• In case of device is removal after physical rewrite. Since pointer remapping after device removal must set physical birth times to the removal time, it has to remove the rewrite flag and copy the physical birth times of the blocks into logical birth times to maintain correct space accounting.

Now that we have different birth times in block pointers, traversal code got new TRAVERSE_LOGICAL flag, allowing to choose between traversing only logical changes (replication, diff, etc), or physical changes (scrub/resilver, dataset destroy, etc).

How Has This Been Tested?

Several successful CI runs. Manual testing with zfs rewrite and zfs rewrite -P vs zfs send -i.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Quality assurance (non-breaking change which makes the code more robust against bugs)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[gamanakis]

@amotin, thank you for this! on a first pass it looks good to me.

[amotin]

Just a rebase and conflict resolution.

[robn]

This all looks pretty straightforward to me, good job. That one comment describing the macros made the whole thing very understandable.

[robn]

Question: it seems like the only time you wouldn't want the physical rewrite option is if you don't want the pool feature? If true, could we make it the default somehow? I know its tricky if you don't have the feature enabled, and I don't think zfs rewrite with no options should choose or anything (it should still be predictable). I just don't like having the best/common case require a flag.

[amotin]

@robn As I mentioned, somebody was actually happy to use logical rewrite to force replication, but that is a minor case. The requirement of the feature was indeed the main factor for me though. It was not feeling right to me also to change the behavior based on feature status. Plus logical rewrite was implemented first, and we've already merged it to our branch of ZFS 2.3 in upcoming TrueNAS 25.10, so it would be a change in behavior, while we can't merge the physical rewrite there too due to the feature. I don't object it being default, but I am open to hear how.

[stuartthebruce]

Are either of the rewrite options mature enough to consider for 2.3.4?

[amotin]

@stuartthebruce The logical rewrite is quite simple, that's why we've merged them into our TrueNAS ZFS sources. I could include it into #17595, if people desire. Physical rewrite though extends the on-disk format and so require a new pool feature, that's why it will not go below 3.4.

[stuartthebruce]

@stuartthebruce The logical rewrite is quite simple, that's why we've merged them into our TrueNAS ZFS sources. I could include it into #17595, if people desire.

+1

[vedranmiletic]

Physical rewrite though extends the on-disk format and so require a new pool feature, that's why it will not go below 3.4.

As a user, I would like to have it, but is there a precedent for extending on-disk format in a patch release? It is not so easy to deduce this from the changelog.

[Momi-V]

I believe getting logical rewrite out sooner rather than later is a good thing, especially if it keeps the different OpenZFS Forks mostly feature aligned. It's just strictly superior to the way things have to be done without it.

On disk format changes obviously are a different thing so they can wait, but it's not like running a zsh script checking mtime for consistency is a robust solution.

[amotin]

but is there a precedent for extending on-disk format in a patch release?

Not that I can recall, at least since 2.0. It would be a bad practice to have patch releases incompatible. It should be a decision of distributions, and through them users, when to switch to the next stable branch, accepting risks of possible incompatibilities. Patch release updates should remain a safe routine.