Repository navigation
Performance
These tips can make zfs-autobackup much faster.
Use persistent SSH connections. On the server that starts the backup, add this to ~/.ssh/config:
Host *
ControlPath ~/.ssh/control-master-%r@%h:%p
ControlMaster auto
ControlPersist 3600
Thanks @mariusvw :)
If SSH encryption is too slow, you can send the data without encryption via TCP. For example, with netcat:
zfs-autobackup ... --send-pipe "nc server_name 8023" --recv-pipe "nc -l -p 8023"You can use any transfer program in this way. Only the zfs send/zfs recv data goes through this pipe. zfs-autobackup still uses SSH for all other commands.
See also issue #15.
-
--buffer: Adds an IO buffer. This makes the bursts ofzfs sendandzfs recvsmoother. -
--compress: Use this for a slow link and data that compresses well.
See Piping.
These options decrease the number of commands and the IO for each snapshot:
-
--no-holds: No hold and release commands. -
--allow-empty: No commands to find out if a snapshot is empty. -
--no-guid-check: No GUID check of the common snapshots. -
--clear-mountpoint: zfs-autobackup does not mount the target datasets.
A bug in ZFS 2 adds a delay of 1 second to each transfer. This is a problem when you have many small transfers. Use --no-progress to prevent this.
Test setup: a laptop with an SKHynix_HFS512GD9TNI-L2B0B disk, ZFS 2.0.2, --no-holds, --allow-empty and SSH ControlMaster. The test measures the total time for 100 empty datasets:
| Setup | Time | Speed |
|---|---|---|
| No SSH | 15 s | more than 6 datasets/s |
--ssh-target or --ssh-source to localhost |
20 s | 5 datasets/s |
--ssh-target and --ssh-source to localhost |
24 s | 4 datasets/s |
A test with 2500 datasets also had no problems. Thus, zfs-autobackup can work with thousands of datasets.
If you need more performance, tell me.
By default, Linux uses cubic for TCP congestion control. This is good for almost all situations. But for large transfers over the internet, short congestion (packet loss) can make the transfer slow for a long time.
scalable recovers much faster from packet loss. To use it, make the file /etc/sysctl.d/60-send-bulk-tcp.conf:
# Recovers rapidly from congestion
net.ipv4.tcp_congestion_control = scalable
More information: Scalable TCP. (Thanks @xrobau)
zfs-autobackup:
- Introduction (README.md)
- Getting started
- Full manual
- Mounting backup datasets
- Performance tips (recommended)
- Common problems and errors
- Thinning out obsolete snapshots
- Common snapshots, holds and bookmarks
- Handling ZFS encryption
- Transfer buffering, compression and rate limiting.
- Custom Pre- and post-snapshot commands
- Monitoring
Examples: