Skip to content

Commit

Permalink
Merge pull request #637 from jimklimov/autoCreation-dstN
Browse files Browse the repository at this point in the history 
Support `dst_N_autocreation` ZFS properties
  • Loading branch information
@oetiker
oetiker authored May 2, 2024
2 parents 1fe9271 + af05425 commit b02cfff
Show file tree
Hide file tree
Showing 6 changed files with 294 additions and 20 deletions.
1 change: 1 addition & 0 deletions .github/workflows/spelling/expect.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ attr
austingroupbugs
autocommit
autoconf
autocreation
autom
automake
automounting
Expand Down
1 change: 1 addition & 0 deletions CHANGES
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ znapzend (0.21.3) unstable; urgency=medium
* Added rc-script and integration documentation for FreeBSD and similar platforms
* Converted configure.ac and numerous Makefile.am to avoid GNU Make syntax in favor of portability: tested with Solaris/illumos Sun make and with FreeBSD make
* Extended `--autoCreation` effect (or lack thereof) to newly appearing sub-datasets; added a `--noautoCreation` option to help override configuration file settings (where used)
* Introduced `dst_N_autocreation` setting via ZFS properties (per-destination, inheritable)

-- Jim Klimov <[email protected]> Tue, 12 Mar 2024 13:42:28 +0100

Expand Down
3 changes: 3 additions & 0 deletions bin/znapzend
View file
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,9 @@ sub main {
$opts->{forbidDestRollback} = 0;
}

# Note: default is "undef" to use a ZFS property dst_N_autocreation
# (lower-case "c" in the name) if present; finally assumes 0 (false)
# if not set in any configuration source for a particular dataset.
if (defined($opts->{noautoCreation})) {
$opts->{autoCreation} = 0;
delete $opts->{noautoCreation};
Expand Down
54 changes: 54 additions & 0 deletions bin/znapzendzetup
View file
Original file line number Diff line number Diff line change
Expand Up @@ -402,6 +402,54 @@ sub main {

last;
};
/^enable-dst-autocreation$/ && do {
$opts->{dst} = pop @ARGV;
$opts->{src} = pop @ARGV;
if (!defined $opts->{src}) {
pod2usage(-exitval => 'NOEXIT');
die ("ERROR: source argument for option $mainOpt was not provided\n");
}
if (!defined $opts->{dst}) {
pod2usage(-exitval => 'NOEXIT');
die ("ERROR: destination argument for option $mainOpt was not provided\n");
}
$zConfig->enableBackupSetDstAutoCreation($opts->{src}, $opts->{dst})
or die "ERROR: cannot enable backup config for $opts->{src} destination $opts->{dst}. Did you create this config?\n";

last;
};
/^disable-dst-autocreation$/ && do {
$opts->{dst} = pop @ARGV;
$opts->{src} = pop @ARGV;
if (!defined $opts->{src}) {
pod2usage(-exitval => 'NOEXIT');
die ("ERROR: source argument for option $mainOpt was not provided\n");
}
if (!defined $opts->{dst}) {
pod2usage(-exitval => 'NOEXIT');
die ("ERROR: destination argument for option $mainOpt was not provided\n");
}
$zConfig->disableBackupSetDstAutoCreation($opts->{src}, $opts->{dst})
or die "ERROR: cannot disable backup config for $opts->{src} destination $opts->{dst}. Did you create this config?\n";

last;
};
/^inherit-dst-autocreation$/ && do {
$opts->{dst} = pop @ARGV;
$opts->{src} = pop @ARGV;
if (!defined $opts->{src}) {
pod2usage(-exitval => 'NOEXIT');
die ("ERROR: source argument for option $mainOpt was not provided\n");
}
if (!defined $opts->{dst}) {
pod2usage(-exitval => 'NOEXIT');
die ("ERROR: destination argument for option $mainOpt was not provided\n");
}
$zConfig->inheritBackupSetDstAutoCreation($opts->{src}, $opts->{dst})
or die "ERROR: cannot disable backup config for $opts->{src} destination $opts->{dst}. Did you create this config?\n";

last;
};
/^list$/ && do {
GetOptions($opts, (@ROOT_EXEC_OPTIONS, qw(recursive|r inherited))) or exit 1;

Expand Down Expand Up @@ -591,6 +639,12 @@ and where 'command' and its unique options is one of the following:
disable-dst <src_dataset> <DST_key>
enable-dst-autocreation <src_dataset> <DST_key>
disable-dst-autocreation <src_dataset> <DST_key>
inherit-dst-autocreation <src_dataset> <DST_key>
list [--recursive] [--inherited] [src_dataset...]
export <src_dataset>
Expand Down
82 changes: 68 additions & 14 deletions lib/ZnapZend.pm
View file
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ has pidfile => sub { q{} };
has forcedSnapshotSuffix => sub { q{} };
has defaultPidFile => sub { q{/var/run/znapzend.pid} };
has terminate => sub { 0 };
has autoCreation => sub { 0 };
has autoCreation => sub { undef };
has timeWarp => sub { undef };
has nodelay => sub { 0 };
has skipOnPreSnapCmdFail => sub { 0 };
Expand Down Expand Up @@ -404,14 +404,23 @@ my $refreshBackupPlans = sub {
#create backup hashes for all destinations
for (keys %$backupSet){
my ($key) = /^dst_([^_]+)_plan$/ or next;
my $autoCreation = $self->autoCreation;
if (!defined($autoCreation)) {
# Caller did not require any particular behavior, so
# check the ZFS property name (note lower-case "c"):
$autoCreation = (exists $backupSet->{"dst_$key" . '_autocreation'} ? $backupSet->{"dst_$key" . '_autocreation'} : undef);
}
if (!defined($autoCreation)) {
$autoCreation = 0;
}

#check if destination exists (i.e. is valid) otherwise recheck as dst might be online, now
if (!$backupSet->{"dst_$key" . '_valid'}){

$backupSet->{"dst_$key" . '_valid'} =
$self->zZfs->dataSetExists($backupSet->{"dst_$key"}) or do {

if ($self->autoCreation && !$self->sendRaw) {
if ($autoCreation && !$self->sendRaw) {
my ($zpool) = $backupSet->{"dst_$key"} =~ /(^[^\/]+)\//;

# check if we can access destination zpool, if so create parent dataset
Expand All @@ -429,7 +438,7 @@ my $refreshBackupPlans = sub {
$backupSet->{"dst_$key" . '_valid'} or
$self->zLog->warn("destination '" . $backupSet->{"dst_$key"}
. "' does not exist or is offline. will be rechecked every run..."
. ( $self->autoCreation ? "" : " Consider running znapzend --autoCreation" ) );
. ( $autoCreation ? "" : " Consider running znapzend --autoCreation" ) );
};

$self->zLog->debug('refreshBackupPlans(): detected dst_' . $key . '_valid status for ' . $backupSet->{"dst_$key"} . ': ' . $backupSet->{"dst_$key" . '_valid'}) if ($self->debug);
Expand Down Expand Up @@ -563,10 +572,23 @@ my $sendRecvCleanup = sub {
#recheck non valid dst as it might be online, now
if (!$backupSet->{"dst_$key" . '_valid'}) {

my $autoCreation = $self->autoCreation;
if (!defined($autoCreation)) {
# Caller did not require any particular behavior, so
# check the ZFS property name (note lower-case "c").
# Note we are looking at "root" datasets with a backup
# schedule here (enumerated earlier); children if any
# would be checked below:
$autoCreation = (exists $backupSet->{"dst_$key" . '_autocreation'} ? $backupSet->{"dst_$key" . '_autocreation'} : undef);
}
if (!defined($autoCreation)) {
$autoCreation = 0;
}

$backupSet->{"dst_$key" . '_valid'} =
$self->zZfs->dataSetExists($backupSet->{"dst_$key"}) or do {

if ($self->autoCreation && !$self->sendRaw) {
if ($autoCreation && !$self->sendRaw) {
my ($zpool) = $backupSet->{"dst_$key"} =~ /(^[^\/]+)\//;

# check if we can access destination zpool, if so -
Expand All @@ -584,12 +606,18 @@ my $sendRecvCleanup = sub {
}
};
}
( $backupSet->{"dst_$key" . '_valid'} || ($self->sendRaw && $self->autoCreation) ) or do {
( $backupSet->{"dst_$key" . '_valid'} || ($self->sendRaw && $autoCreation) ) or do {
my $errmsg = "destination '" . $backupSet->{"dst_$key"}
. "' does not exist or is offline; ignoring it for this round...";
$self->zLog->warn($errmsg);
push (@sendFailed, $errmsg);
$thisSendFailed = 1;
# Avoid spamming for every loop cycle, if we do not have

Check failure on line 612 in lib/ZnapZend.pm

View workflow job for this annotation

GitHub Actions / Spell checking 

`spamming` is not a recognized word. (unrecognized-spelling)
# the dataset and know we do not intend to auto-create it
$self->zLog->warn($errmsg) if ($autoCreation or $self->debug);
if (!$autoCreation) {
$self->zLog->warn("Autocreation is disabled for this dataset or whole run, so skipping without error") if ($self->debug);
} else {
push (@sendFailed, $errmsg);
$thisSendFailed = 1;
}
next;
};
};
Expand All @@ -607,6 +635,26 @@ my $sendRecvCleanup = sub {
my $dstDataSet = $srcDataSet;
$dstDataSet =~ s/^\Q$backupSet->{src}\E/$backupSet->{$dst}/;

my $autoCreation = $self->autoCreation;
if (!defined($autoCreation)) {
# Caller did not require any particular behavior, so
# check the ZFS property name (note lower-case "c").
# Look at properties of this dataset, allow inherited
# values. TOTHINK: Get properties once for all tree?
my $properties = $self->zZfs->getDataSetProperties($srcDataSet, 0, 1);
if ($properties->[0]) {
for my $prop (keys %{$properties->[0]}) {
if ($prop eq "dst_$key" . '_autocreation') {
$autoCreation = (%{$properties->[0]}{$prop} eq "on" ? 1 : 0);
last;
}
}
}
}
if (!defined($autoCreation)) {
$autoCreation = 0;
}

my $srcDataSetDisabled = (grep (/^\Q$srcDataSet\E$/, @dataSetsExplicitlyDisabled));
$self->zLog->debug('sending snapshots from ' . $srcDataSet . ' to ' . $dstDataSet .
($srcDataSetDisabled ? ": not enabled, skipped" : ""));
Expand All @@ -616,14 +664,20 @@ my $sendRecvCleanup = sub {

# Time to check if the target sub-dataset exists
# at all (unless we would auto-create one anyway).
if (!$self->autoCreation && !$self->sendRaw && !$self->zZfs->dataSetExists($dstDataSet)) {
if ((!$autoCreation || !$self->sendRaw) && !($self->zZfs->dataSetExists($dstDataSet))) {
my $errmsg = "sub-destination '" . $dstDataSet
. "' does not exist or is offline; ignoring it for this round... Consider "
. ( $self->autoCreation || $self->sendRaw ? "" : "running znapzend --autoCreation or " )
. ( $autoCreation || $self->sendRaw ? "" : "running znapzend --autoCreation or " )
. "disabling this dataset from znapzend handling.";
$self->zLog->warn($errmsg);
push (@sendFailed, $errmsg);
$thisSendFailed = 1;
# Avoid spamming for every loop cycle, if we do not have

Check failure on line 672 in lib/ZnapZend.pm

View workflow job for this annotation

GitHub Actions / Spell checking 

`spamming` is not a recognized word. (unrecognized-spelling)
# the dataset and know we do not intend to auto-create it
$self->zLog->warn($errmsg) if ($autoCreation or $self->debug);
if (!$autoCreation) {
$self->zLog->warn("Autocreation is disabled for this dataset or whole run, so skipping without error") if ($self->debug);
} else {
push (@sendFailed, $errmsg);
$thisSendFailed = 1;
}
next;
}

Expand All @@ -635,7 +689,7 @@ my $sendRecvCleanup = sub {
'since=="' . $self->since . '"'.
', skipIntermediates=="' . $self->skipIntermediates . '"' .
', forbidDestRollback=="' . $self->forbidDestRollback . '"' .
', autoCreation=="' . $self->autoCreation . '"' .
', autoCreation=="' . ( $autoCreation ? "true" : "false" ) . '"' .
', sendRaw=="' . $self->sendRaw . '"' .
', valid=="' . ( $backupSet->{"dst_$key" . '_valid'} ? "true" : "false" ) . '"' .
', justCreated=="' . ( $backupSet->{"dst_$key" . '_justCreated'} ? "true" : "false" ) . '"'
Expand Down
Loading

1 comment on commit b02cfff

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@check-spelling-bot Report

🔴 Please review

See the 📜action log or 📝 job summary for details.

Unrecognized words (1)

spamming

Previously acknowledged words that are now absent Balert Bcreate Bdebug Bdelete Bedit Berr Bexport Bimport Binfo Bnoaction Bnot Bpidfile Bpost Bpre Bsyslog Bwarning Bzfs Bznapzend Bznapzendzetup Bznapzendztatz cpanfile cpanm cpanmin CPANSNAPV crt DBD DESTDIR distdir DTDs endif EXTRADIST forkcall Icommand Icommon Icreate Idataset Idestroy Idocuments Iexport Ifacility Ifeature Ifilepath Ihome Ilimited imandir Inumber Ioptions Ipath Ipictures Irecursive Isend Iskip Isnapshots Isnapsuffix Isources Itank Ithirdparty Itimeout Iusbbackup Iuser Ivalue Iznapzendzetup lpr nobase notest nroff ODBC SUBDIRS svcdir troff unicode utf vroff xargs 🫥
To accept these unrecognized words as correct and remove the previously acknowledged and now absent words, you could run the following commands

... in a clone of the [email protected]:oetiker/znapzend.git repository
on the master branch (ℹ️ how do I use this?):

curl -s -S -L 'https://raw.githubusercontent.com/check-spelling/check-spelling/v0.0.22/apply.pl' |
perl - 'https://github.com/oetiker/znapzend/actions/runs/8926442404/attempts/1'
Available 📚 dictionaries could cover words (expected and unrecognized) not in the 📘 dictionary

This includes both expected items (664) from .github/workflows//spelling/expect.txt and unrecognized words (1)

Dictionary Entries Covers Uniquely
cspell:software-terms/dict/softwareTerms.txt 1288 82 18
cspell:php/dict/php.txt 1689 59 6
cspell:python/src/python/python-lib.txt 2417 54 6
cspell:node/dict/node.txt 891 58 5
cspell:filetypes/filetypes.txt 264 19 4

Consider adding them (in .github/workflows/spelling.yml):

      with:
        extra_dictionaries:
          cspell:software-terms/dict/softwareTerms.txt
          cspell:php/dict/php.txt
          cspell:python/src/python/python-lib.txt
          cspell:node/dict/node.txt
          cspell:filetypes/filetypes.txt

To stop checking additional dictionaries, add (in .github/workflows/spelling.yml):

check_extra_dictionaries: ''

Please sign in to comment.