Cheksum comparison

[karol-kokoszka]

Fixes #3827

This PR adds explicit stage to the backup process - deduplication.
It follows https://docs.google.com/document/d/1EtGlF6UGNy34D_7QsnCheaukp3UwVObZU56PBdd0CQ8/edit#heading=h.jl2qbpcarwp9

It is done to mitigate the problem with current deduplication/checksum comparison which causes SM to create a checksum of every SSTable file and to compare it with the checksum from remote.

This PR changes restore of versioned snapshot as well to corrupt CRC32 files.

---

Please make sure that:

• Code is split to commits that address a single change
• Commit messages are informative
• Commit titles have module prefix
• Commit titles have issue nr. suffix

[karol-kokoszka]

Integration tests are failing. To be checked.

Done

[karol-kokoszka]

This jenkins build is expected to trigger DTests and SCT.
https://jenkins.scylladb.com/view/scylla-manager/job/manager-master/job/manager-build/718/

Let's check if we are asserting the backup progress.

[karol-kokoszka]

DTest -> https://jenkins.scylladb.com/view/scylla-manager/job/manager-master/job/manager-dtest/1595/ FAILED
SCT Sanity Debian 11 -> https://jenkins.scylladb.com/view/scylla-manager/job/manager-master/job/debian11-sanity-test/

[Michal-Leszczynski]

Nice work!

In the past there was an idea of making it possible to delete files in batches (I don't think that it is currently supported in our rclone fork). This seems like a simple change on rclone side that would allow us to save maaaany API calls (both in backup purger, and now in backup deduplication).

@karol-kokoszka maybe we should include it in the scope of this effort (perhaps in a separate series of rclone/SM PRs)?

[karol-kokoszka]

@Michal-Leszczynski please re-review.

[Michal-Leszczynski]

This PR adds the deduplication stage, but it does not change the rclone options of the upload stage - it still has checksum. Is this expected?

In theory it shouldn't be a big deal, but we could disable the checksum comparison in rclone, so that when we checked that the crc32 doesn't match, we don't compare the hashes for nothing.

[karol-kokoszka]

Some sample logs (after changing from DEBUG to INFO)

Jul 18 09:04:14 manager-regression-master-monitor-node-d5777710-1 scylla-manager[10584]: {"L":"INFO","T":"2024-07-18T09:04:14.845Z","N":"backup.deduplicate","M":"Removing local snapshot file (deduplication based on generation UUID)","host":"10.15.1.78","file":"data:system_distributed_everywhere/cdc_generation_descriptions_v2-234d2227dd633d37ac5fc013e2ea9e6e/snapshots/sm_20240718090409UTC/me-3ghx_0net_4rw9s2bhesqa79ne6z-big-CompressionInfo.db","_trace_id":"LkFf76dqQ-miLcvXFxGsMQ"}
Jul 18 09:04:14 manager-regression-master-monitor-node-d5777710-1 scylla-manager[10584]: {"L":"INFO","T":"2024-07-18T09:04:14.845Z","N":"backup.deduplicate","M":"Removing local snapshot file (deduplication based on generation UUID)","host":"10.15.1.78","file":"data:system_distributed_everywhere/cdc_generation_descriptions_v2-234d2227dd633d37ac5fc013e2ea9e6e/snapshots/sm_20240718090409UTC/me-3ghx_0net_4rw9s2bhesqa79ne6z-big-Data.db","_trace_id":"LkFf76dqQ-miLcvXFxGsMQ"}
Jul 18 09:04:14 manager-regression-master-monitor-node-d5777710-1 scylla-manager[10584]: {"L":"INFO","T":"2024-07-18T09:04:14.846Z","N":"backup.deduplicate","M":"Removing local snapshot file (deduplication based on generation UUID)","host":"10.15.1.78","file":"data:system_distributed_everywhere/cdc_generation_descriptions_v2-234d2227dd633d37ac5fc013e2ea9e6e/snapshots/sm_20240718090409UTC/me-3ghx_0net_4rw9s2bhesqa79ne6z-big-Digest.crc32","_trace_id":"LkFf76dqQ-miLcvXFxGsMQ"}
Jul 18 09:04:14 manager-regression-master-monitor-node-d5777710-1 scylla-manager[10584]: {"L":"INFO","T":"2024-07-18T09:04:14.846Z","N":"backup.deduplicate","M":"Removing local snapshot file (deduplication based on generation UUID)","host":"10.15.1.78","file":"data:system_distributed_everywhere/cdc_generation_descriptions_v2-234d2227dd633d37ac5fc013e2ea9e6e/snapshots/sm_20240718090409UTC/me-3ghx_0net_4rw9s2bhesqa79ne6z-big-Filter.db","_trace_id":"LkFf76dqQ-miLcvXFxGsMQ"}
Jul 18 09:04:14 manager-regression-master-monitor-node-d5777710-1 scylla-manager[10584]: {"L":"INFO","T":"2024-07-18T09:04:14.847Z","N":"backup.deduplicate","M":"Removing local snapshot file (deduplication based on generation UUID)","host":"10.15.1.78","file":"data:system_distributed_everywhere/cdc_generation_descriptions_v2-234d2227dd633d37ac5fc013e2ea9e6e/snapshots/sm_20240718090409UTC/me-3ghx_0net_4rw9s2bhesqa79ne6z-big-Index.db","_trace_id":"LkFf76dqQ-miLcvXFxGsMQ"}
Jul 18 09:04:14 manager-regression-master-monitor-node-d5777710-1 scylla-manager[10584]: {"L":"INFO","T":"2024-07-18T09:04:14.847Z","N":"backup.deduplicate","M":"Removing local snapshot file (deduplication based on generation UUID)","host":"10.15.1.78","file":"data:system_distributed_everywhere/cdc_generation_descriptions_v2-234d2227dd633d37ac5fc013e2ea9e6e/snapshots/sm_20240718090409UTC/me-3ghx_0net_4rw9s2bhesqa79ne6z-big-Scylla.db","_trace_id":"LkFf76dqQ-miLcvXFxGsMQ"}
Jul 18 09:04:14 manager-regression-master-monitor-node-d5777710-1 scylla-manager[10584]: {"L":"INFO","T":"2024-07-18T09:04:14.847Z","N":"backup.deduplicate","M":"Removing local snapshot file (deduplication based on generation UUID)","host":"10.15.1.78","file":"data:system_distributed_everywhere/cdc_generation_descriptions_v2-234d2227dd633d37ac5fc013e2ea9e6e/snapshots/sm_20240718090409UTC/me-3ghx_0net_4rw9s2bhesqa79ne6z-big-Statistics.db","_trace_id":"LkFf76dqQ-miLcvXFxGsMQ"}
Jul 18 09:04:14 manager-regression-master-monitor-node-d5777710-1 scylla-manager[10584]: {"L":"INFO","T":"2024-07-18T09:04:14.848Z","N":"backup.deduplicate","M":"Removing local snapshot file (deduplication based on generation

Overall, it will be logged on debug level as there may be plenty of these entries during the backup.

[Michal-Leszczynski]

Actually, there is a small important thing missing.
AFAIU, the d.Progress.files is initialized only during the INDEX stage. This means that pausing and resuming backup during DEDUPLICATION stage would result in d.Progress.files being empty and the whole process falling to pieces.

This is the reason why some stages allow reindexing on resume:

		// Skip completed stages
		if run.PrevID != uuid.Nil {
			// Allow reindexing if previous state is manifest creation or upload
			if stage == StageIndex && (prevStage == StageManifest || prevStage == StageUpload) { //nolint: revive
				// continue
			} else if prevStage.Index() > stage.Index() {
				return nil
			}
		}

DEDUPLICATE should be one of those stages as well.

What's funny is that it looks like checking for reindexing is bugged (by me), as the moved SCHEMA stage should also be reindexed on resume (as it is before UPLOAD, which supposedly requires reindexation - although I think that it only impacts progress recording/display there).

[Michal-Leszczynski]

Perhaps it would be good to have a test which checks backup pause/resume during each stage.

[karol-kokoszka]

Actually, there is a small important thing missing.
AFAIU, the d.Progress.files is initialized only during the INDEX stage. This means that pausing and resuming backup during DEDUPLICATION stage would result in d.Progress.files being empty and the whole process falling to pieces.

Nice catch !
It should be reindexed indeed ! Added integration test to service_deduplicate_integration_test that proves it.

[karol-kokoszka]

@Michal-Leszczynski please re-review

[Michal-Leszczynski]

@karol-kokoszka kudos for adding dedicated deduplication test. I re-review the changes and left some small comments.