backup.velero.io/backup-volumes but for CSI snapshots

[makarov-roman]

Hi,
currently we are supporting on-premise and cloud environments.

We're using FSB on-premise and switching to CSI snapshots on cloud. Backups were configured using backup.velero.io/backup-volumes annotation.
Is there a way to achieve the same(opt-in) behavior with CSI snapshots?

Thanks in advance.

[kaovilai]

Yes. See https://velero.io/docs/v1.15/resource-filtering/#resource-policies

[kaovilai]

Basically like opt in via pod annotation. But since here csi do not require pod, annotation will be on PVCs themselves.

[kaovilai]

IIUC velero may not be listing every resource type and parsing all things not yet marked as includedResources.
So it may be easier to "snapshot all volumes found in backup" then annotate to opt out.

Alternatively velero will need another feature flag to always list and parse each PVC to detect this "opt-in annotation".

[sseago]

@kaovilai right. But I think this is because there's basic confusion on what opt-in/opt-out pod annotations do for fs-backup. They're not opting in/out of backing up (i.e. fs-backup vs nothing) but opting in/out of fs-backup (i.e. fs-backup vs. snapshot) therefore opting out of fs-backup is opting in to snapshot backup. If they want to exclude certain volumes from the backup and use csi for the others, that is supported via volume policies. Once we implement the label selector bit, that means that they get exactly what is being asked here, but with PVC labels rather than pod annotations.

[kaovilai]

ok I think label selector is most reasonable. cause you can list via labelselector and/or name but definitely not via annotation (require listing all then looking for each object IIUC)

[sseago]

@kaovilai also using label selector with the existing volume policies functionality gives you the flexibility for both opt-in and opt-out. i.e. a policy that says "back up via snapshot with this label" followed by a generic "skip" policy, alternatively, you could reverse it and "skip PVCs with this label" followed by a generic "snapshot" policy.

[sseago]

"Backups were configured using backup.velero.io/backup-volumes annotation.
Is there a way to achieve the same(opt-in) behavior with CSI snapshots?"

Opt-in/opt-out switches between fs-backup and snapshots. If you opt out of fs-backup, that means you're using csi snapshots (if you've configured CSI and your volumes are CSI volumes). In other words, if you're using opt-in, then the volumes not opted in use CSI snapshots. If your use case is swapped and you want to use fs-backup by default but "opt in" to CSI, then you could just use the opt-out approach. Set defaultVolumesToFsBackup to true and use backup.velero.io/backup-volumes-excludes for the volumes you want to use for CSI.

[makarov-roman]

The goal is to be able to tell velero to backup particular PV using annotation. In other words I'm trying to use either CSI or FSB, but never mix them, because It makes no sense in our environment. (e.g. CSI snaphosts disabled when using FSB, node agent disabled when using CSI snapshots)

Opt-in behaviour for our case means, that volumes without annotation are not backed up.

[sseago]

"opt-in" and "opt-out" are defined by velero as "opting into fs-backup" and "opting out of fs-backup". These annotations ("backup-volumes" and "backup-volumes-excludes") don't determine whether or not to back up the volume -- they determine whether or not to backup via fs-backup (vs. snapshot). Therefore, "opt-in to fs-backup ('backup-volumes')" is also already functioning as "opt-out of snapshot backup", and "opt-out of fs-backup ('backup-volumes-excludes')" is also already functioning as "opt-in to snapshot backup".

It sounds like your use case is not swapping between fs-backup and snapshot but rather choosing which volumes to not back up at all. For that use case, the "skip" volume policy would be the way to go, as that is the defined function of this policy. The only problem is that right now you can't define volume policies by label. The proposed label selector volume policy RFE should get you what you want.