CSI plugin cannot work with PVC using selector to filter PV

[blackpiglet]

Describe the problem/challenge you have

When PV and PVC are already dynamically allocated, do the following operation to make the PVC using a selector to filter the PV. Then create a Velero backup with the Velero CSI plugin, and restore from the backup. The restore will success, but the PVC and related Pod are always in a Pending state due to no available PV can be found.

• Change the Reclaim policy to Retain: kubectl patch pv <your-pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
• Tag the PV: kubectl label pv <pv-name> <key=pair>
• Delete the PVC and modify it to use labels to invoke the PV just labeled
• Change the PV to Available state: kubectl patch pv <pv-name> -p '{"spec":{"claimRef": null}}'
• Deploy the PVC modified to invoke PV using labels.

# k -n restore describe pvc my-pvc
Name:          my-pvc
Namespace:     restore
StorageClass:  standard-rwo
Status:        Pending
Volume:        
Labels:        app=hello-app
               backup.velero.io/must-include-additional-items=true
               velero.io/backup-name=3240504-test
               velero.io/restore-name=3240504-test-20230630061156
               velero.io/volume-snapshot-name=velero-my-pvc-6qshv
Annotations:   backup.velero.io/must-include-additional-items: true
               velero.io/backup-name: 3240504-test
               velero.io/volume-snapshot-name: velero-my-pvc-6qshv
               volume.beta.kubernetes.io/storage-provisioner: pd.csi.storage.gke.io
               volume.kubernetes.io/selected-node: <node-name>
               volume.kubernetes.io/storage-provisioner: pd.csi.storage.gke.io
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
DataSource:
  APIGroup:  snapshot.storage.k8s.io
  Kind:      VolumeSnapshot
  Name:      velero-my-pvc-6qshv
Used By:     hello-app-7579d6745b-nzjlx
Events:
  Type     Reason                Age                From                                                                                              Message
  ----     ------                ----               ----                                                                                              -------
  Normal   WaitForFirstConsumer  37s (x2 over 37s)  persistentvolume-controller                                                                       waiting for first consumer to be created before binding
  Normal   Provisioning          6s (x6 over 37s)   External provisioner is provisioning volume for claim "restore/my-pvc"
  Warning  ProvisioningFailed    6s (x6 over 37s)  failed to provision volume with StorageClass "standard-rwo": claim Selector is not supported
  Normal   ExternalProvisioning  3s (x4 over 37s)   persistentvolume-controller                                                                       waiting for a volume to be created, either by external provisioner "pd.csi.storage.gke.io" or manually created by system administrator

Describe the solution you'd like

Although this issue is triggered by a hacking way to create PV and PVC, need to consider whether there is a way to make CSI restore in the scenario.

Anything else you would like to add:

Environment:

• Velero version (use velero version): v1.12
• Kubernetes version (use kubectl version): v1.25
• Kubernetes installer & version: GKE
• Cloud provider or hardware configuration: GCP
• OS (e.g. from /etc/os-release): Debian

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "The project would be better with this feature added"
• 👎 for "This feature will not enhance the project in a meaningful way"

[reasonerjt]

IMO we may want to remove the label selector from pvc when we apply the dynamic provision (podvolume/CSI).
Curious if there are provisioners which indeed support the selector in pvc and what's the behavior.