Backup/Restore of statically provisioned NFS persistent volumes

[glowing-axolotl]

Hi team,

Following the resolution of these issues (and some relevant quotes added):
#3450
#2721
#4967

Restic has no problem with NFS volumes. But if you don't have NFS PVs pre-created, then the PVCs can't bind to them. It's not really restic related -- you'll have the same error creating new pods with NFS PVCs without available PVs.

#6489

During pod volume restore, Velero creates a new pod with a PVC only and expect the system to provision a PV. As the current behavior, if the PV cannot provisioned for the PVC, the initialContainer will not complete and the pod volume restore will hang there until the timeout.
If static provision is being used, you need to make sure that by the time of the PVC is created, a PV could be bound to it by any means.

#6623

NFS without StorageClass doesn't have dynamic provisioning, so you will generally have to create PVs to be bound. When using fs backup via Kopia or Restic, the PV is discarded on restore since a new PV will be provisioned for the PVC on restore. In static provisioning cases, the PVs must exist prior to running a restore. For cases like NFS, it is common to have a pool of PVs already in the cluster to be bound to PVCs upon creation.

Thank you for the updates. In this case, if I want to take a full backup of a namespace, including its data, what is the best practice?

With NFS, you'll need to use fs backup (restic/kopia). The main thing with NFS vs. using dynamic provisioning is that since you aren't using a storage class with dynamic provisioning, you need to have available PVs in the cluster before restore so that they can be bound to the PVCs being created at restore time. In contrast, if you were using the gp2 storageclass for ebs volumes, you wouldn't need to have PVs available since the storage provisioner would create them for you.

https://docs.openshift.com/container-platform/4.12/backup_and_restore/application_backup_and_restore/troubleshooting.html#podvolumerestore-fails_oadp-troubleshooting
https://docs.okd.io/latest/backup_and_restore/application_backup_and_restore/troubleshooting.html#podvolumerestore-fails_oadp-troubleshooting

The NFS mount path is not unique for the two volumes to restore. As a result, the velero lock files use the same file on the NFS server during the restore, causing the PodVolumeRestore to fail.

You can resolve this issue by setting up a unique pathPattern for each volume, while defining the StorageClass for nfs-subdir-external-provisioner in the deploy/class.yaml file.

Would it be safe to say that statically provisioned NFS PVs are NOT supported in Velero?

There are a few possible solutions so far but none of them seem to work as expected:

• Make a backup and restore without deleting the project to be restored (you still have the PVCs intact) -> Data is not completely reset, only overwritten (tested with Kopia).
• Make a new project and manually create clean PV/PVCs after excluding them from the backup -> Works but then you have no volume backups and the applications start with empty volumes.
• Keep the PVs but make Velero create the PVCs -> The claims are created but they have no ".spec.volumeName" set, thus they need to be manually modified.

I wanted to make this thread to act as some sort of documentation, so if any internauts were to pass by they could find this thread.

Thank you!

• AXoLotl

[sseago]

"Would it be safe to say that statically provisioned NFS PVs are NOT supported in Velero?"

I don't think this is correct. It's probably better to say "Statically provisioned NFS PVs are supported by Velero with certain caveats" and then document that there must be unbound PVs in the cluster for the newly-restored PVCs to bind to. This is just the way static provisioning works in kubernetes. When you create a PVC, the cluster looks for an existing available and unbound PV that is large enough for the PVC request to bind to. If those are available, then Velero restore should work just fine with data restored to the restored PVCs which will be bound to available PVs.

[glowing-axolotl]

Yup, I can confirm that this is the main solution for this usecase.

An interesting thing I found is you can actually create a clone of your NFS PV with a specific claimRef section pointing to the yet-non-existing namespace:

  claimRef:
    name: test-app-nfs-pvc
    namespace: namespace-where-you-want-to-restore

This way, you'll get 2 PVs pointing to the same location, one in Bound state, the other in Available. You can therefore "Trick" velero into thinking this will be a new PV, but it will still point to the original location.

For anyone wanting to try it, there are 2 big downsides (AFAIK) is that you won't be able to use the original name of your PVs, since there seems to be no way to specify volumeName on a Velero restore-created PVC.

Downside, the data on the PV will not get cleaned (maybe one could do a specific init container to delete all the previous data though, depending on the application).
We are also talking about a same-cluster restore inside a different namespace here.

I'll be posting some more YAMLs among the way so that anyone may use them as reference.