-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Documentation Update for OpenShift IBM Cloud for CSI snapshot data movement #8069
Documentation Update for OpenShift IBM Cloud for CSI snapshot data movement #8069
Conversation
Updates the documentation for CSI snapshot data movement for OpenShift on IBM Cloud. The default hostpath /var/lib/kubelet/pods cannot find PersistentVolumeClaims with volumeMode: Block on host. The correct hostpath for OpenShift on IBM Cloud is /var/data/kubelet/pods. Signed-off-by: Michael Fruchtman <[email protected]>
/kind changelog-not-required |
Signed-off-by: Michael Fruchtman <[email protected]>
nice @msfrucht! I'll make sure we doc this in downstream OADP as well. |
|
||
```yaml | ||
hostPath: | ||
path: /var/data/kubet/pods |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@msfrucht Shouldn't this be kubelet
? (s/kubet/kubelet
)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes. My error. I'll get that fixed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe I've resolved the error and looked over the changes again to check for any other issues.
Signed-off-by: Michael Fruchtman <[email protected]>
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #8069 +/- ##
==========================================
+ Coverage 58.79% 58.80% +0.01%
==========================================
Files 357 358 +1
Lines 29841 30085 +244
==========================================
+ Hits 17545 17692 +147
- Misses 10851 10945 +94
- Partials 1445 1448 +3 ☔ View full report in Codecov by Sentry. |
I have to admit to some significant error and embarrassment regarding the documentation update about Openshift on IBM Cloud pull request vmware-tanzu#8069. I will correct my error before it gets any further. Just exposing /var/data/kubelet/pods is incorrect and host path /var/lib/kubelet/pods should remain unchanged. The errors with the defaults during csi snapshot data movement were: data path backup failed: Failed to run kopia backup: unable to get local block device entry: resolveSymlink: lstat /var/data/: no such file or directory I suspected this was the same as RancherOS and Nutanix. It is not. The original tested changes changed both /var/lib/kubelet/{pods,plugins} to /var/data/kubelet/{pods,plugins}. The published changes only result in the error: ``` status: completionTimestamp: '2024-08-02T17:12:29Z' message: >- data path backup failed: Failed to run kopia backup: unable to get local block device entry: resolveSymlink: lstat /var/data/kubelet/plugins: no such file or directory node: 10.240.0.5 phase: Failed progress: {} startTimestamp: '2024-08-02T17:12:11Z' ``` After making continued modifications to the daemonset the correct configuration was: ``` volumeMounts: - name: host-pods mountPath: /host_pods mountPropagation: HostToContainer - name: host-plugins mountPath: /var/data/kubelet/plugins mountPropagation: HostToContainer ``` ``` volumes: - name: host-pods hostPath: path: /var/lib/kubelet/pods type: '' - name: host-plugins hostPath: path: /var/data/kubelet/plugins type: '' ``` Only the changes to the plugin path were required. The plugin path changes were required to both the mount path and the host path. Regardless of whether /var/lib/kubelet/pods or /var/data/kubelet/pods host path, backups and restore succeeded provided the plugin path was modified. ``` volumeMounts: - name: host-pods mountPath: /host_pods mountPropagation: HostToContainer - name: host-plugins mountPath: /var/data/kubelet/plugins mountPropagation: HostToContainer ``` ``` volumes: - name: host-pods hostPath: path: /var/data/kubelet/pods type: '' - name: host-plugins hostPath: path: /var/data/kubelet/plugins type: '' ``` After getting on-host access was able to confirm. Pods are at /var/lib/kubelet/pods. ``` ls /var/lib/kubelet/pods 07c0be63-335d-4cfb-b39f-816bc2fb32cd 51f31b3e-4710-4ef0-8626-5f1a78a624b2 a4802fd3-3b62-45a4-8f21-974880b6f92a cccb35c9-b4f9-4ca9-a697-736ae64f09ad 0a5d4366-7fa1-4525-9e45-a43a362b8542 558b0643-0661-4d4a-b03e-aac60c6ad710 a4b106fb-5b7b-48e5-828a-ea7b41ba0e59 ce1290e1-4330-4df6-8166-14784bcce930 ``` On host the volumes are in /var/data/kubelet/plugins. ``` ls /var/data/kubelet/plugins/kubernetes.io/csi/openshift-storage.cephfs.csi.ceph.com/231e04896c4f528efb95d23a3c153db9fc4a7206b7320f74443f30de7228dba5/globalmount/velero/backups/backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c/ backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-csi-volumesnapshotclasses.json.gz backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-resource-list.json.gz backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-csi-volumesnapshotcontents.json.gz backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-results.gz backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-csi-volumesnapshots.json.gz backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-volumesnapshots.json.gz backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-itemoperations.json.gz backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c.tar.gz backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-logs.gz velero-backup.json backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-podvolumebackups.json.gz ``` With the volume config changed to expose /var/data/kubelet/plugins for the plugin hostPath, the DataUploads and DataDownloads succeed for both Filesystem and Block mode PVCs. ``` status: completionTimestamp: '2024-08-02T17:23:33Z' node: 10.240.0.5 path: >- /host_pods/7fcb9d56-7885-437c-acd3-67db6b1ee8ae/volumeDevices/kubernetes.io~csi/pvc-47b91f56-db8c-44bf-9ecc-737170561b4b phase: Completed progress: bytesDone: 5368709120 totalBytes: 5368709120 snapshotID: 8faae36b3592fee4efbfad024f26033e startTimestamp: '2024-08-02T17:21:22Z' ``` ``` status: completionTimestamp: '2024-08-02T18:42:19Z' node: 10.240.0.5 phase: Completed progress: bytesDone: 5368709120 totalBytes: 5368709120 startTimestamp: '2024-08-02T18:41:00Z' ``` My apologies for the error. Signed-off-by: Michael Fruchtman <[email protected]>
* Correcting Openshift on IBM Documentation Error I have to admit to some significant error and embarrassment regarding the documentation update about Openshift on IBM Cloud pull request #8069. I will correct my error before it gets any further. Just exposing /var/data/kubelet/pods is incorrect and host path /var/lib/kubelet/pods should remain unchanged. The errors with the defaults during csi snapshot data movement were: data path backup failed: Failed to run kopia backup: unable to get local block device entry: resolveSymlink: lstat /var/data/: no such file or directory I suspected this was the same as RancherOS and Nutanix. It is not. The original tested changes changed both /var/lib/kubelet/{pods,plugins} to /var/data/kubelet/{pods,plugins}. The published changes only result in the error: ``` status: completionTimestamp: '2024-08-02T17:12:29Z' message: >- data path backup failed: Failed to run kopia backup: unable to get local block device entry: resolveSymlink: lstat /var/data/kubelet/plugins: no such file or directory node: 10.240.0.5 phase: Failed progress: {} startTimestamp: '2024-08-02T17:12:11Z' ``` After making continued modifications to the daemonset the correct configuration was: ``` volumeMounts: - name: host-pods mountPath: /host_pods mountPropagation: HostToContainer - name: host-plugins mountPath: /var/data/kubelet/plugins mountPropagation: HostToContainer ``` ``` volumes: - name: host-pods hostPath: path: /var/lib/kubelet/pods type: '' - name: host-plugins hostPath: path: /var/data/kubelet/plugins type: '' ``` Only the changes to the plugin path were required. The plugin path changes were required to both the mount path and the host path. Regardless of whether /var/lib/kubelet/pods or /var/data/kubelet/pods host path, backups and restore succeeded provided the plugin path was modified. ``` volumeMounts: - name: host-pods mountPath: /host_pods mountPropagation: HostToContainer - name: host-plugins mountPath: /var/data/kubelet/plugins mountPropagation: HostToContainer ``` ``` volumes: - name: host-pods hostPath: path: /var/data/kubelet/pods type: '' - name: host-plugins hostPath: path: /var/data/kubelet/plugins type: '' ``` After getting on-host access was able to confirm. Pods are at /var/lib/kubelet/pods. ``` ls /var/lib/kubelet/pods 07c0be63-335d-4cfb-b39f-816bc2fb32cd 51f31b3e-4710-4ef0-8626-5f1a78a624b2 a4802fd3-3b62-45a4-8f21-974880b6f92a cccb35c9-b4f9-4ca9-a697-736ae64f09ad 0a5d4366-7fa1-4525-9e45-a43a362b8542 558b0643-0661-4d4a-b03e-aac60c6ad710 a4b106fb-5b7b-48e5-828a-ea7b41ba0e59 ce1290e1-4330-4df6-8166-14784bcce930 ``` On host the volumes are in /var/data/kubelet/plugins. ``` ls /var/data/kubelet/plugins/kubernetes.io/csi/openshift-storage.cephfs.csi.ceph.com/231e04896c4f528efb95d23a3c153db9fc4a7206b7320f74443f30de7228dba5/globalmount/velero/backups/backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c/ backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-csi-volumesnapshotclasses.json.gz backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-resource-list.json.gz backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-csi-volumesnapshotcontents.json.gz backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-results.gz backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-csi-volumesnapshots.json.gz backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-volumesnapshots.json.gz backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-itemoperations.json.gz backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c.tar.gz backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-logs.gz velero-backup.json backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-podvolumebackups.json.gz ``` With the volume config changed to expose /var/data/kubelet/plugins for the plugin hostPath, the DataUploads and DataDownloads succeed for both Filesystem and Block mode PVCs. ``` status: completionTimestamp: '2024-08-02T17:23:33Z' node: 10.240.0.5 path: >- /host_pods/7fcb9d56-7885-437c-acd3-67db6b1ee8ae/volumeDevices/kubernetes.io~csi/pvc-47b91f56-db8c-44bf-9ecc-737170561b4b phase: Completed progress: bytesDone: 5368709120 totalBytes: 5368709120 snapshotID: 8faae36b3592fee4efbfad024f26033e startTimestamp: '2024-08-02T17:21:22Z' ``` ``` status: completionTimestamp: '2024-08-02T18:42:19Z' node: 10.240.0.5 phase: Completed progress: bytesDone: 5368709120 totalBytes: 5368709120 startTimestamp: '2024-08-02T18:41:00Z' ``` My apologies for the error. Signed-off-by: Michael Fruchtman <[email protected]> * Add context to plugins mountPath Signed-off-by: Michael Fruchtman <[email protected]> --------- Signed-off-by: Michael Fruchtman <[email protected]>
Please add a summary of your change
Updates the documentation for CSI snapshot data movement for OpenShift on IBM Cloud.
The default hostpath /var/lib/kubelet/pods cannot find PersistentVolumeClaims with volumeMode: Block on host.
The correct hostpath for OpenShift on IBM Cloud is /var/data/kubelet/pods
Does your change fix a particular issue?
Fixes #(issue)
Documentation only. Fixes failed CSI snapshot movement for OpenShift on IBM Cloud
Please indicate you've done the following:
/kind changelog-not-required
as a comment on this pull request.site/content/docs/main
.