mysqlctl,vttablet,vtbackup: support restore tablet and vtbackup restore phase with CLONE #19084

maxenglander · 2025-12-24T23:52:29Z

Description

Next step after #19064. Adapted from @nickvanw code in private fork.

Add support to VTTablet and VTBackup for --restore-with-clone.
--restore-with-clone uses tabletmanager.restoreFromClone().
restoreFromClone() and RestoreBackup() use shared functions extracted from RestoreBackup.

Related Issue(s)

Feature Request: MySQL CLONE for vtbackup and vttablet restore #19061

Checklist

"Backport to:" labels have been added if this change should be back-ported to release branches
If this change is to be back-ported to previous releases, a justification is included in the PR description
Tests were added or are not required
Did the new or modified tests pass consistently locally and on CI?
Documentation was added or is not required

Deployment Notes

AI Disclosure

AI was used.

Add core infrastructure for MySQL CLONE REMOTE operations: - CloneExecutor in mysqlctl/clone.go for executing CLONE INSTANCE FROM - Clone user configuration in dbconfigs (--db-clone-user, --db-clone-password, --db-clone-use-ssl) - vt_clone user in init_db.sql with BACKUP_ADMIN privilege (MySQL 8.0.17+ only) - Clone plugin loading in mycnf configs (plugin-load-add = mysql_clone.so) - MySQLClonePluginFlavorCapability for version checking MySQL CLONE copies data at the physical level over the network, providing a faster alternative to logical backup/restore for replica provisioning. Requires MySQL 8.0.17+ and InnoDB-only tables. Signed-off-by: Nick Van Wiggeren <[email protected]>

Signed-off-by: Nick Van Wiggeren <[email protected]>

Signed-off-by: Max Englander <[email protected]>

…o be marked as completed after the CLONE is done Signed-off-by: Nick Van Wiggeren <[email protected]>

Signed-off-by: Max Englander <[email protected]>

Signed-off-by: Nick Van Wiggeren <[email protected]>

Signed-off-by: Max Englander <[email protected]>

vitess-bot · 2025-12-24T23:52:50Z

Signed-off-by: Max Englander <[email protected]>

maxenglander · 2025-12-25T23:53:57Z

go/vt/mysqlctl/backup.go

 // and returns whether a restore action should be performed
-func ShouldRestore(ctx context.Context, params RestoreParams) (bool, error) {
-	if params.DeleteBeforeRestore || RestoreWasInterrupted(params.Cnf) {
+func ShouldRestore(ctx context.Context, logger logutil.Logger, cnf *Mycnf, mysqld MysqlDaemon,


these changes allow this to be called from both tabletmanager.restoreFromClone() and tabletmanager.RestoreBackup() without the former needing to construct RestoreParams, which is otherwise not needed by restoreFromClone().

…9089) Signed-off-by: Max Englander <[email protected]>

Signed-off-by: Max Englander <[email protected]>

mattlord

I like it! ❤️ Just some relatively minor comments/questions/requests. Please let me know what you think.

go/cmd/vtbackup/cli/vtbackup.go

mattlord · 2026-01-15T06:02:45Z

go/vt/vttablet/tabletmanager/tm_init.go

 			// restoreFromBackup will just be a regular action
 			// (same as if it was triggered remotely)
-			if err := tm.RestoreData(ctx, logutil.NewConsoleLogger(), waitForBackupInterval, false /* deleteBeforeRestore */, backupTime, restoreToTimestamp, restoreToPos, restoreFromBackupAllowedEngines, mysqlShutdownTimeout); err != nil {
+			if err := tm.RestoreBackup(ctx, logutil.NewConsoleLogger(), waitForBackupInterval, false /* deleteBeforeRestore */, backupTime, restoreToTimestamp, restoreToPos, restoreFromBackupAllowedEngines, mysqlShutdownTimeout); err != nil {


Why don't we pass in the logger from line 900 here?

i'm not sure..it was already like this. want me to change it?

We didn't have the logger variable before. I think we should use it here too now that we have added it.

oh i see, will do

mattlord · 2026-01-15T06:14:51Z

go/test/endtoend/backup/clone/backup_test.go

+	// Initialize shard primary.
+	err := localCluster.VtctldClientProcess.InitShardPrimary(keyspaceName, shardName, cell, primary.TabletUID)
+	require.NoError(t, err)
+
+	// Now check if MySQL version supports clone (need vttablet running to query).
+	if !mysqlVersionSupportsClone(t, primary) {
+		t.Skip("Skipping clone test: MySQL version does not support CLONE (requires 8.0.17+)")
+	}
+
+	// Check if clone plugin is available.
+	if !clonePluginAvailable(t, primary) {
+		t.Skip("Skipping clone test: clone plugin not available")
+	}
+
+	// Set up clean test data (table may have data from previous tests).
+	_, err = primary.VttabletProcess.QueryTablet(vtInsertTest, keyspaceName, true)
+	require.NoError(t, err)
+	_, err = primary.VttabletProcess.QueryTablet("TRUNCATE TABLE vt_insert_test", keyspaceName, true)
+	require.NoError(t, err)
+	_, err = primary.VttabletProcess.QueryTablet("insert into vt_insert_test (msg) values ('clone_test_1')", keyspaceName, true)
+	require.NoError(t, err)
+	_, err = primary.VttabletProcess.QueryTablet("insert into vt_insert_test (msg) values ('clone_test_2')", keyspaceName, true)
+	require.NoError(t, err)


For this to be reliable, you have to explicitly disable vtorc using DisableVTOrcRecoveries(t). The tests fail locally for me, and I think it's because vtorc is running and competing with InitShardPrimary:

❯ go test -v -count=1 -timeout 30m vitess.io/vitess/go/test/endtoend/backup/clone E0115 05:15:03.786892 68921 vtorc_process.go:113] configuration - { "instance-poll-time": "10h" } === RUN TestCloneBackup Flag --init_db_sql_file has been deprecated, use --init-db-sql-file instead Flag --init_keyspace has been deprecated, use --init-keyspace instead Flag --allow_first_backup has been deprecated, use --allow-first-backup instead Flag --mysql_socket has been deprecated, use --mysql-socket instead --- PASS: TestCloneBackup (29.37s) === RUN TestCloneRestore restore_test.go:65: Error Trace: /Users/matt/git/vitess/go/test/endtoend/backup/clone/restore_test.go:65 Error: Received unexpected error: The MySQL server is running with the --super-read-only option so it cannot execute this statement (errno 1290) (sqlstate HY000) during query: create table if not exists vt_insert_test ( id bigint auto_increment, msg varchar(64), primary key (id) ) Engine=InnoDB Test: TestCloneRestore E0115 05:15:47.389171 68921 vtctldclient_process.go:72] Output: E0115 05:15:47.384561 72793 main.go:62] rpc error: code = Unknown desc = node doesn't exist: /zone1/tablets/zone1-0000006962/Tablet: while deleting 1 tablets; please inspect the topo --- FAIL: TestCloneRestore (14.26s) FAIL FAIL vitess.io/vitess/go/test/endtoend/backup/clone 59.062s FAIL

mattlord · 2026-01-15T06:17:04Z

go/test/endtoend/backup/clone/backup_test.go

+	// Give replica2 time to catch up via replication.
+	time.Sleep(5 * time.Second)


These sorts of sleeps where we assume some state will be in place within a given timeframe have proven to lead to flakiness. Can we instead use require.Eventually using a query to check for the expected number of rows or for the replication state? I think we can just use require.Eventually with the VerifyRowsInTablet check or similar.

making the change. require.Eventually still requires to pass in a duration to wait for. are you thinking i should set that to higher than 5 seconds?

I usually set it well beyond what I would reasonably expect, since sometimes runners are resource starved and have long pauses etc and there's literally nothing we can do but re-run/re-try the workflow when the test fails.

ok cool i set the wait time pretty high

go/test/endtoend/backup/clone/backup_test.go

mattlord · 2026-01-15T14:29:54Z

go/test/endtoend/backup/clone/restore_test.go

+func TestCloneRestore(t *testing.T) {
+	t.Cleanup(func() { removeBackups(t) })
+	t.Cleanup(tearDownRestoreTest)
+


Same note here about disable vtorc global recoveries so that it's not racing/competing with the InitShardPrimary call to set the primary.

mattlord · 2026-01-15T14:33:29Z

go/test/endtoend/backup/clone/restore_test.go

+	require.NoError(t, err)
+
+	// Wait for replica1 to catch up.
+	time.Sleep(2 * time.Second)


Same note here about using require.Eventually to check for the state we are only assuming we have after 2 seconds here.

mattlord · 2026-01-15T14:34:49Z

go/test/endtoend/backup/clone/restore_test.go

+
+	// Now check if MySQL version supports clone (need vttablet running to query).
+	if !mysqlVersionSupportsClone(t, primary) {
+		t.Skip("Skipping clone test: MySQL version does not support CLONE (requires 8.0.17+)")


For both of the new e2e tests... I think that we should return an error if we are running in the CI (CI environment variable is set), and only skip otherwise. This prevents us from unknowingly not testing things in the CI for some odd reason.

mattlord · 2026-01-15T14:36:22Z

go/test/endtoend/backup/clone/restore_test.go

+	time.Sleep(5 * time.Second)
+
+	cluster.VerifyRowsInTablet(t, replica2, keyspaceName, 8)


Same note here about using require.Eventually to check of the rows.

mattlord · 2026-01-15T14:37:03Z

go/test/endtoend/backup/clone/restore_test.go

+// verifyClonedData checks that the specific test data we inserted on primary
+// exists on the cloned replica. This proves data was actually transferred.
+func verifyClonedData(t *testing.T, tablet *cluster.Vttablet) {
+	qr, err := tablet.VttabletProcess.QueryTablet(


We have the testing.T here so we could push the require.Eventually down here. Same thing for the other verify helpers here.

mattlord · 2026-01-15T14:41:02Z

@Promptless can you please create a new suggestion for documenting this PR along with everything else recently added for the MySQL CLONE backup/restore feature?

Signed-off-by: Max Englander <[email protected]>

maxenglander · 2026-01-14T18:55:17Z

go/flags/endtoend/mysqlctl.txt

      --log_dir string                                              If non-empty, write log files in this directory
      --logtostderr                                                 log to standard error instead of files
      --max-stack-size int                                          configure the maximum stack size in bytes (default 67108864)
-      --mysql-clone-enabled                                         Enable MySQL CLONE plugin and user for backup/replica provisioning (requires MySQL 8.0.17+)


this was added in an earlier PR, i don't think it's relevant for this binary

maxenglander · 2026-01-14T18:55:32Z

go/flags/endtoend/mysqlctld.txt

      --log_dir string                                                   If non-empty, write log files in this directory
      --logtostderr                                                      log to standard error instead of files
      --max-stack-size int                                               configure the maximum stack size in bytes (default 67108864)
-      --mysql-clone-enabled                                              Enable MySQL CLONE plugin and user for backup/replica provisioning (requires MySQL 8.0.17+)


same as comment above

maxenglander · 2026-01-14T21:16:40Z

go/vt/mysqlctl/mysqld.go

 	utils.SetFlagStringVar(fs, &mycnfTemplateFile, "mysqlctl-mycnf-template", mycnfTemplateFile, "template file to use for generating the my.cnf file during server init")
 	utils.SetFlagStringVar(fs, &socketFile, "mysqlctl-socket", socketFile, "socket file to use for remote mysqlctl actions (empty for local actions)")
 	utils.SetFlagDurationVar(fs, &replicationConnectRetry, "replication-connect-retry", replicationConnectRetry, "how long to wait in between replica reconnect attempts. Only precise to the second.")
-	utils.SetFlagBoolVar(fs, &mysqlCloneEnabled, "mysql-clone-enabled", mysqlCloneEnabled, "Enable MySQL CLONE plugin and user for backup/replica provisioning (requires MySQL 8.0.17+)")


imo since this isn't used by mysqlctl or mysqlctld it shouldn't be grouped here.

maxenglander · 2026-01-15T20:11:40Z

go/test/endtoend/backup/clone/backup_test.go

+	// Initialize shard primary.
+	err := localCluster.VtctldClientProcess.InitShardPrimary(keyspaceName, shardName, cell, primary.TabletUID)
+	require.NoError(t, err)
+
+	// Now check if MySQL version supports clone (need vttablet running to query).
+	if !mysqlVersionSupportsClone(t, primary) {
+		t.Skip("Skipping clone test: MySQL version does not support CLONE (requires 8.0.17+)")
+	}
+
+	// Check if clone plugin is available.
+	if !clonePluginAvailable(t, primary) {
+		t.Skip("Skipping clone test: clone plugin not available")
+	}
+
+	// Set up clean test data (table may have data from previous tests).
+	_, err = primary.VttabletProcess.QueryTablet(vtInsertTest, keyspaceName, true)
+	require.NoError(t, err)
+	_, err = primary.VttabletProcess.QueryTablet("TRUNCATE TABLE vt_insert_test", keyspaceName, true)
+	require.NoError(t, err)
+	_, err = primary.VttabletProcess.QueryTablet("insert into vt_insert_test (msg) values ('clone_test_1')", keyspaceName, true)
+	require.NoError(t, err)
+	_, err = primary.VttabletProcess.QueryTablet("insert into vt_insert_test (msg) values ('clone_test_2')", keyspaceName, true)
+	require.NoError(t, err)


maxenglander · 2026-01-15T20:19:47Z

go/test/endtoend/backup/clone/backup_test.go

+	// Give replica2 time to catch up via replication.
+	time.Sleep(5 * time.Second)


making the change. require.Eventually still requires to pass in a duration to wait for. are you thinking i should set that to higher than 5 seconds?

go/test/endtoend/backup/clone/backup_test.go

maxenglander · 2026-01-15T20:23:47Z

go/test/endtoend/backup/clone/main_test.go

+func parseVersionFromRow(row []sqltypes.Value) (int, int, int, error) {
+	if len(row) == 0 {
+		return 0, 0, 0, errors.New("empty row")
+	}


this must have been leftover from an earlier iteration, it's unused. removing.

maxenglander · 2026-01-15T20:27:29Z

go/test/endtoend/backup/clone/main_test.go

+	}
+}
+
+// splitLines splits a string by newlines, filtering out empty lines.


i don't know! switched to strings.SplitSeq and it's working fine, removing this.

maxenglander · 2026-01-16T15:02:25Z

go/vt/vttablet/tabletmanager/tm_init.go

 			// restoreFromBackup will just be a regular action
 			// (same as if it was triggered remotely)
-			if err := tm.RestoreData(ctx, logutil.NewConsoleLogger(), waitForBackupInterval, false /* deleteBeforeRestore */, backupTime, restoreToTimestamp, restoreToPos, restoreFromBackupAllowedEngines, mysqlShutdownTimeout); err != nil {
+			if err := tm.RestoreBackup(ctx, logutil.NewConsoleLogger(), waitForBackupInterval, false /* deleteBeforeRestore */, backupTime, restoreToTimestamp, restoreToPos, restoreFromBackupAllowedEngines, mysqlShutdownTimeout); err != nil {


i'm not sure..it was already like this. want me to change it?

Signed-off-by: Max Englander <[email protected]>

nickvanw and others added 19 commits December 19, 2025 15:23

add flags to tests + fix errors from linters

0fe8efd

Signed-off-by: Nick Van Wiggeren <[email protected]>

fix up more tests

8941459

Signed-off-by: Nick Van Wiggeren <[email protected]>

go/vt/mysqlctl: add CloneFromDonor

94624dc

Signed-off-by: Max Englander <[email protected]>

actually validate the source and destination, and poll for the copy t…

0342d74

…o be marked as completed after the CLONE is done Signed-off-by: Nick Van Wiggeren <[email protected]>

TestCloneFromDonor: add a few cases

95dccd1

Signed-off-by: Max Englander <[email protected]>

merge <- clone-infrastructure

ba66344

Signed-off-by: Max Englander <[email protected]>

more tests

e66d1ee

Signed-off-by: Max Englander <[email protected]>

make it nicer

0d57e26

Signed-off-by: Max Englander <[email protected]>

make tests nicer

06a370b

Signed-off-by: Max Englander <[email protected]>

slight reorg add comments

88340e5

Signed-off-by: Max Englander <[email protected]>

fix imports

01fabc0

Signed-off-by: Max Englander <[email protected]>

fix lint

ca83a7a

Signed-off-by: Max Englander <[email protected]>

one more lint

2cdb9c4

Signed-off-by: Max Englander <[email protected]>

tidy

66e4998

Signed-off-by: Max Englander <[email protected]>

add an end to end test and do some cleanup

8633eaf

Signed-off-by: Nick Van Wiggeren <[email protected]>

use time tickers in the for loop

13ace33

Signed-off-by: Nick Van Wiggeren <[email protected]>

tmrpctest: fix flaky TestGRPCTMServer timeout test

43a534f

Signed-off-by: Nick Van Wiggeren <[email protected]>

Merge branch 'clone-infrastructure' into clone-from-donor

0599231

Signed-off-by: Max Englander <[email protected]>

github-actions bot added this to the v24.0.0 milestone Dec 24, 2025

fix test

dbe89f7

Signed-off-by: Max Englander <[email protected]>

maxenglander added 3 commits December 26, 2025 08:39

rm redundant clone sql init

b44ad0f

Signed-off-by: Max Englander <[email protected]>

Merge branch 'clone-infrastructure' into clone-from-donor

d40490b

Signed-off-by: Max Englander <[email protected]>

Merge branch 'clone-from-donor' into clone-restore-init

425c440

Signed-off-by: Max Englander <[email protected]>

maxenglander mentioned this pull request Dec 26, 2025

go/cmd/vtbackup: add --restore-from-clone support to vtbackup #19089

Merged

5 tasks

maxenglander added 3 commits December 26, 2025 09:49

fix imports

a50414a

Signed-off-by: Max Englander <[email protected]>

fix tests

dbbf495

Signed-off-by: Max Englander <[email protected]>

fix tests

cf1a804

Signed-off-by: Max Englander <[email protected]>

maxenglander force-pushed the clone-restore-init branch from 6ee25db to cf1a804 Compare December 26, 2025 16:03

maxenglander marked this pull request as ready for review December 27, 2025 19:07

maxenglander requested review from frouioui, harshit-gangal, mattlord, rohit-nayak-ps, shlomi-noach and timvaillancourt as code owners December 27, 2025 19:07

maxenglander commented Dec 27, 2025

View reviewed changes

Base automatically changed from clone-from-donor to main January 13, 2026 22:38

maxenglander requested a review from systay as a code owner January 13, 2026 22:38

go/cmd/vtbackup: add --restore-from-clone support to vtbackup (#1…

3c58666

…9089) Signed-off-by: Max Englander <[email protected]>

maxenglander changed the title ~~go/vt/{mysqlctl,vttablet}: support restore tablet with CLONE~~ mysqlctl,vttablet,vtbackup: support restore tablet and vtbackup restore phase with CLONE Jan 14, 2026

mattlord removed the NeedsDescriptionUpdate The description is not clear or comprehensive enough, and needs work label Jan 14, 2026

maxenglander added 4 commits January 14, 2026 00:16

merge <- main

327b31a

Signed-off-by: Max Englander <[email protected]>

update flags

05bb719

Signed-off-by: Max Englander <[email protected]>

tidy

3f157ae

Signed-off-by: Max Englander <[email protected]>

flags

4d539b7

Signed-off-by: Max Englander <[email protected]>

mattlord reviewed Jan 15, 2026

View reviewed changes

address cr feedback

fc6b8e7

Signed-off-by: Max Englander <[email protected]>

maxenglander commented Jan 17, 2026

View reviewed changes

cr: use logger var

4f94b9e

Signed-off-by: Max Englander <[email protected]>

		// Give replica2 time to catch up via replication.
		time.Sleep(5 * time.Second)

		time.Sleep(5 * time.Second)

		cluster.VerifyRowsInTablet(t, replica2, keyspaceName, 8)

mysqlctl,vttablet,vtbackup: support restore tablet and vtbackup restore phase with CLONE #19084

Are you sure you want to change the base?

mysqlctl,vttablet,vtbackup: support restore tablet and vtbackup restore phase with CLONE #19084

Uh oh!

Conversation

maxenglander commented Dec 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue(s)

Checklist

Deployment Notes

AI Disclosure

Uh oh!

vitess-bot bot commented Dec 24, 2025

Review Checklist

General

Tests

Documentation

New flags

If a workflow is added or modified:

Backward compatibility

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mattlord left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mattlord commented Jan 15, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

maxenglander commented Dec 24, 2025 •

edited

Loading