-
Notifications
You must be signed in to change notification settings - Fork 1.4k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #7610 from reasonerjt/restore-vol-info-design
Add design to introduce restore volume info
- Loading branch information
Showing
2 changed files
with
144 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| Add design to introduce restore volume info |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,143 @@ | ||
| # Volume information for restore design | ||
|
|
||
| ## Background | ||
| Velero has different ways to handle data in the volumes during restore. The users want to have more clarity in terms of how | ||
| the volumes are handled in restore process via either Velero CLI or other downstream product which consumes Velero. | ||
|
|
||
| ## Goals | ||
| - Create new metadata to store the information of the restored volume, which will have the same life-cycle as the restore CR. | ||
| - Consume the metadata in velero CLI to enable it display more details for volumes in the output of `velero restore describe --details` | ||
|
|
||
| ## Non Goals | ||
| - Provide finer grained control of the volume restore process. The focus of the design is to enable displaying more details. | ||
| - Persist additional metadata like podvolume, datadownloads etc to the restore folder in backup-location. | ||
|
|
||
| ## Design | ||
|
|
||
| ### Structure of the restore volume info | ||
| The restore volume info will be stored in a file named like `${restore_name}-vol-info.json`. The content of the file will | ||
| be a list of volume info objects, each of which will map to a volume that is restored, and will contain the information | ||
| like name of the restored PV/PVC, restore method and related objects to provide details depending on the way it's restored, | ||
| it will look like this: | ||
| ``` | ||
| [ | ||
| { | ||
| "pvcName": "nginx-logs-2", | ||
| "pvcNamespace": "nginx-app-restore", | ||
| "pvName": "pvc-e320d75b-a788-41a3-b6ba-267a553efa5e", | ||
| "restoreMethod": "PodVolumeRestore", | ||
| "snapshotDataMoved": false, | ||
| "pvrInfo": { | ||
| "snapshotHandle": "81973157c3a945a5229285c931b02c68", | ||
| "uploaderType": "kopia", | ||
| "volumeName": "nginx-logs", | ||
| "podName": "nginx-deployment-79b56c644b-mjdhp", | ||
| "podNamespace": "nginx-app-restore" | ||
| } | ||
| }, | ||
| { | ||
| "pvcName": "nginx-logs-1", | ||
| "pvcNamespace": "nginx-app-restore", | ||
| "pvName": "pvc-98c151f4-df47-4980-ba6d-470842f652cc", | ||
| "restoreMethod": "CSISnapshot", | ||
| "snapshotDataMoved": false, | ||
| "csiSnapshotInfo": { | ||
| "snapshotHandle": "snap-01a3b21a5e9f85528", | ||
| "size": 2147483648, | ||
| "driver": "ebs.csi.aws.com", | ||
| "vscName": "velero-velero-nginx-logs-1-jxmbg-hx9x5" | ||
| } | ||
| } | ||
| ...... | ||
| ] | ||
| ``` | ||
| Each field will have the same meaning as the corresponding field in the backup volume info. It will not have the fields | ||
| that were introduced to help with the backup process, like `pvInfo`, `dataupload` etc. | ||
|
|
||
| ### How the restore volume info is generated | ||
| Two steps are involved in generating the restore volume info, the first is "collection", which is to gather the information | ||
| for restoration of the volumes, the second is "generation", which is to iterate through the data collected in the first step | ||
| and generate the volume info list as is described above. | ||
|
|
||
| Unlike backup, the CR objects created during the restore process will not be persisted to the backup storage location. | ||
| Therefore, to gather the information needed to generate volume information, we either need to collect the CRs in the middle | ||
| of the restore process, or retrieve the objects based on the `resouce-list.json` of the restore via API server. | ||
| The information to be collected are: | ||
| - **PV/PVC mapping relationship:** It will be collected via the `restore-resource-list.json`, b/c at the time the json is ready, all | ||
| PVCs and PVs are already created. | ||
| - **Native snapshot information:** It will be collected in the restore workflow when each snapshot is restored. | ||
| - **podvolumerestore CRs:** It will be collected in the restore workflow after each pvr is created. | ||
| - **volumesnapshot CRs for CSI snapshot:** It will be collected in the step of collecting PVC info, by reading the `dataSource` | ||
| field in the spec of the PVC. | ||
| - **datadownload CRs** It will be collected in the phase of collecting PVC info, by querying the API-server to list the datadownload | ||
| CRs labeled with the restore name. | ||
|
|
||
| After the collection step, the generation step is relatively straight-forward, as we have all the information needed in | ||
| the data structures. | ||
|
|
||
| The whole collection and generation steps will be done with the "best-effort" manner, i.e. if there are any failures we | ||
| will only log the error in restore log, rather than failing the whole restore process, we will not put these errors or warnings | ||
| into the `result.json`, b/c it won't impact the restored resources. | ||
|
|
||
| Depending on the number of the restored PVCs the "collection" step may involve many API calls, but it's considered acceptable | ||
| b/c at that time the resources are already created, so the actual RTO is not impacted. By using the client of controller runtime | ||
| we can make the collection step more efficient by using the cache of the API server. We may consider to make improvements if | ||
| we observe performance issues, like using multiple go-routines in the collection. | ||
|
|
||
| ### Implementation | ||
| Because the restore volume info shares the same data structures with the backup volume info, we will refactor the code in | ||
| package `internal/volume` to make the sub-components in backup volume info shared by both backup and restore volume info. | ||
|
|
||
| We'll introduce a struct called `RestoreVolumeInfoTracker` which encapsulates the logic of collecting and generating the restore volume info: | ||
| ``` | ||
| // RestoreVolumeInfoTracker is used to track the volume information during restore. | ||
| // It is used to generate the RestoreVolumeInfo array. | ||
| type RestoreVolumeInfoTracker struct { | ||
| *sync.Mutex | ||
| restore *velerov1api.Restore | ||
| log logrus.FieldLogger | ||
| client kbclient.Client | ||
| pvPvc *pvcPvMap | ||
| // map of PV name to the NativeSnapshotInfo from which the PV is restored | ||
| pvNativeSnapshotMap map[string]NativeSnapshotInfo | ||
| // map of PV name to the CSISnapshot object from which the PV is restored | ||
| pvCSISnapshotMap map[string]snapshotv1api.VolumeSnapshot | ||
| datadownloadList *velerov2alpha1.DataDownloadList | ||
| pvrs []*velerov1api.PodVolumeRestore | ||
| } | ||
| ``` | ||
| The `RestoreVolumeInfoTracker` will be created when the restore request is initialized, and it will be passed to the `restoreContext` | ||
| and carried over the whole restore process. | ||
|
|
||
| The `client` in this struct is to be used to query the resources in the restored namespace, and the current client in restore | ||
| reconciler only watches the resources in the namespace where velero is installed. Therefore, we need to introduce the | ||
| `CrClient` which has the same life-cycle of velero server to the restore reconciler, because this is the client that watches all the | ||
| resources on the cluster. | ||
|
|
||
| In addition to that, we will make small changes in the restore workflow to collect the information needed. We'll make the | ||
| changes un-intrusive and make sure not to change the logic of the restore to avoid break change or regression. | ||
| We'll also introduce routine changes in the package `pkg/persistence` to persist the restore volume info to the backup storage location. | ||
|
|
||
| Last but not least, the `velero restore describe --details` will be updated to display the volume info in the output. | ||
|
|
||
| ## Alternatives Considered | ||
| There used to be suggestion that to provide more details about volume, we can query the `backup-vol-info.json` with the resource | ||
| identifier in `restore-resource-list.json`. This will not work when there're resource modifiers involved in the restore process, | ||
| which may change the metadata of PVC/PV. In addition, we may add more detailed restore-specific information about the volumes that is not available | ||
| in the `backup-vol-info.json`. Therefore, the `restore-vol-info.json` is a better approach. | ||
|
|
||
| ## Security Considerations | ||
| There should be no security impact introduced by this design. | ||
|
|
||
| ## Compatibility | ||
| The restore volume info will be consumed by Velero CLI and downstream products for displaying details. So the functionality | ||
| of backup and restore will not be impacted for restores created by older versions of Velero which do not have the restore volume info | ||
| metadata. The client should properly handle the case when the restore volume info does not exist. | ||
|
|
||
| The data structures referenced by volume info is shared between both restore and backup and it's not versioned, so in the future | ||
| we must make sure there will only be incremental changes to the metadata, such that no break change will be introduced to the client. | ||
|
|
||
| ## Open Issues | ||
| https://github.com/vmware-tanzu/velero/issues/7546 | ||
| https://github.com/vmware-tanzu/velero/issues/6478 |