Several questions about "zfs rewrite"

[isZYKerman]

My questions are:

1. I have a pool with a single raidz2 vdev. I expanded it from 5 disks to 6 disks several months ago using the zfs attach command. However the existing data are not rebalanced and thus there is a space loss. I learnt that the new rewrite subcommand can solve this, but how? My pool, named Data, is mounted at /mnt/Data. One of its sub-datasets, Backups, contains replication data (snapshots) from other pools that was sent via zfs send ... | zfs receive ...; it is not mounted since the data inside should remain untouched by anyone else. Meanwhile, all other (sub-)datasets are mounted recursively and have got snapshots recursively; I use the tool zrepl to create new snapshots every hour for data safety concerns. If the rewrite command is subject to files, what should I do to this Backup dataset and all other snapshots? Shall I stop zrepl from generating new snapshots during the rewrite process?

2. The root file system of my server is also in a pool named "System" and it has also got snapshots recursively, auto-generated by zrepl. It is too fragmented now (the value of the fragmentation attribute is over 40%). Can I use the rewrite subcommand to solve this? If yes, how? How to deal with the snapshots and should I stop zrepl during the rewrite process? Is it possible to set up a cron job, etc. to defragment automatically from time to time?

ps. The zfs rewrite subcommand was introduced just 20 days ago, and all the articles I can find on the Internet are talking about what can it do instead of how to achieve a goal using it; even the official wiki has no examples at all. :(

[amotin]

Since you have used raidz expansion, it might be reasonable to rewrite all your data at some point in some way to get better space efficiency. But it has to be done in coordination with your snapshot retention policy, since until your oldest snapshots rotate you may have a double space usage on the pool as result of rewrite.

Rewrite in its current implementation works on mounted datasets. It is not a strict requirement in general, but that is what is is now. If you need it for unmounted datasets, then send/receive/destroy might be a better alternative.

Rewrite theoretically might reduce fragmentation, but it might be not very predictable. Fragmentation metrics for the pool is a free space fragmentation. There is no metrics for data fragmentation, so attempt to rewrite everything blindly might make is either better or worse.

[isZYKerman]

Anyway -P doesn't seem to fully fit my situation, so

1. I can leave the Backup dataset untouched for the snapshots inside are received after the expansion process. As it is not mounted and there is nothing aside of snapshots inside, running zfs rewrite -Prvx /mnt/Data will not affect this dataset.
2. I should delete all the snapshots for other datasets and perform the rewrite process. Meanwhile I should stop the zrepl service for a while to prevent new snapshots from generating.
3. There is currently no defragment tool and I should just leave my system pool as-is.

Am I right?

[amotin]

I should delete all the snapshots for other datasets and perform the rewrite process.

I don't like deleting all snapshots. Instead I prefer some reasonable rotation set up, that would delete old snapshots after some days/weeks, still keeping protection from deleting some data by mistakes, but not influencing space usage too much. But for pure rebalance/defragmentation process more space free would be better.

There is currently no defragment tool and I should just leave my system pool as-is.

Right. Rewrite at this stage is not a defragmentation tool, but it can be used as a base for one. From my look on several other Linux file systems, their defragmentation calls are quite similar to the rewrite, but the question is to find objects that really worth defragmenting. For example, if you downloaded some file with torrent -- it definitely worth defragmenting it, may be even always downloading it on a different pool and copying after completion. But if you just written something sequentially, then what a chance that second write attempt will give better result than the first?

[isZYKerman]

1. I can leave the Backup dataset untouched for the snapshots inside are received after the expansion process. As it is not mounted and there is nothing aside of snapshots inside, running zfs rewrite -Prvx /mnt/Data will not affect this dataset.

Well, what about this part?

[amotin]

1. I can leave the Backup dataset untouched for the snapshots inside are received after the expansion process. As it is not mounted and there is nothing aside of snapshots inside, running zfs rewrite -Prvx /mnt/Data will not affect this dataset.

Well, what about this part?

zfs rewrite only works with mounted datasets, so your unmounted backups won't be affected.

[isZYKerman]

I got it. Thank you all!