`test_katib_experiments` fails on Canonical K8s: Cannot provision PVC with `ReadWriteMany` Access Mode

### Bug Description

This issue was encountered in the [PR to move to Canonical K8s for our CI](https://github.com/canonical/katib-operators/pull/346). The [simple-pbt](https://github.com/canonical/katib-operators/blob/095f91892df507ff2d47118c62fee9bfdd6547a8/tests/assets/crs/experiments/simple-pbt.yaml#L2) experiment stays in `Created` and never finishes.

It seems that K8s cannot provision the PVC in order for the experiment to run.

Events from the PVC:
```
Events:
  Type     Reason                Age                  From                                                                                           Message
  ----     ------                ----                 ----                                                                                           -------
  Normal   WaitForFirstConsumer  5m5s                 persistentvolume-controller                                                                    waiting for first consumer to be created before binding
  Normal   Provisioning          113s (x9 over 5m4s)  rawfile.csi.openebs.io_ck-storage-rawfile-csi-node-j6p6v_bd68acdb-8251-4281-bf95-e331f948abf9  External provisioner is provisioning volume for claim "test-kubeflow/simple-pbt-pbt"
  Warning  ProvisioningFailed    113s (x9 over 5m4s)  rawfile.csi.openebs.io_ck-storage-rawfile-csi-node-j6p6v_bd68acdb-8251-4281-bf95-e331f948abf9  failed to provision volume with StorageClass "csi-rawfile-default": rpc error: code = InvalidArgument desc = Unsupported access mode: MULTI_NODE_MULTI_WRITER
  Normal   ExternalProvisioning  10s (x21 over 5m4s)  persistentvolume-controller                                                                    Waiting for a volume to be created either by the external provisioner 'rawfile.csi.openebs.io' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
```

The PVC itself has a `ReadWriteMany` accessmode, as [configured from the charm](https://github.com/canonical/katib-operators/blob/095f91892df507ff2d47118c62fee9bfdd6547a8/charms/katib-controller/src/templates/katib-config.yaml.j2#L51):
```
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    volume.beta.kubernetes.io/storage-provisioner: rawfile.csi.openebs.io
    volume.kubernetes.io/selected-node: dev
    volume.kubernetes.io/storage-provisioner: rawfile.csi.openebs.io
  creationTimestamp: "2025-09-02T11:05:45Z"
  finalizers:
  - kubernetes.io/pvc-protection
  name: simple-pbt-pbt
  namespace: test-kubeflow
  ownerReferences:
  - apiVersion: kubeflow.org/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: Suggestion
    name: simple-pbt
    uid: d4836487-513c-4cf5-85ef-147e5cf564d4
  resourceVersion: "210651"
  uid: 3397b096-9023-426f-bd60-9d2211c8cb3e
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  storageClassName: csi-rawfile-default
  volumeMode: Filesystem
status:
  phase: Pending
```

Logs from the `external-provisioner` container:
```
kubectl logs -n kube-system ck-storage-rawfile-csi-node-j6p6v -c external-provisioner
W0901 10:36:43.196596       1 feature_gate.go:246] Setting GA feature gate Topology=true. It will be removed in a future release.
I0901 10:36:43.196665       1 csi-provisioner.go:154] Version: v5.0.2
I0901 10:36:43.196670       1 csi-provisioner.go:177] Building kube configs for running in cluster...
I0901 10:36:43.197522       1 common.go:143] "Probing CSI driver for readiness"
I0901 10:36:43.200830       1 csi-provisioner.go:302] CSI driver does not support PUBLISH_UNPUBLISH_VOLUME, not watching VolumeAttachments
I0901 10:36:43.201319       1 csi-provisioner.go:351] using local topology with Node = &Node{ObjectMeta:{dev      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[hostname:dev] map[] [] [] []},Spec:NodeSpec{PodCIDR:,DoNotUseExternalID:,ProviderID:,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[],},Status:NodeStatus{Capacity:ResourceList{},Allocatable:ResourceList{},Phase:,Conditions:[]NodeCondition{},Addresses:[]NodeAddress{},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:0,},},NodeInfo:NodeSystemInfo{MachineID:,SystemUUID:,BootID:,KernelVersion:,OSImage:,ContainerRuntimeVersion:,KubeletVersion:,KubeProxyVersion:,OperatingSystem:,Architecture:,},Images:[]ContainerImage{},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,RuntimeHandlers:[]NodeRuntimeHandler{},},} and CSINode = &CSINode{ObjectMeta:{dev      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},Spec:CSINodeSpec{Drivers:[]CSINodeDriver{CSINodeDriver{Name:rawfile.csi.openebs.io,NodeID:dev,TopologyKeys:[hostname],Allocatable:nil,},},},}
I0901 10:36:43.207479       1 csi-provisioner.go:464] using apps/v1/DaemonSet ck-storage-rawfile-csi-node as owner of CSIStorageCapacity objects
I0901 10:36:43.207506       1 csi-provisioner.go:483] producing CSIStorageCapacity objects with fixed topology segment [hostname: dev]
I0901 10:36:43.208834       1 csi-provisioner.go:529] using the CSIStorageCapacity v1 API
I0901 10:36:43.310403       1 controller.go:824] "Starting provisioner controller" component="rawfile.csi.openebs.io_ck-storage-rawfile-csi-node-j6p6v_bd68acdb-8251-4281-bf95-e331f948abf9"
I0901 10:36:43.310419       1 capacity.go:243] Starting Capacity Controller
I0901 10:36:43.310453       1 volume_store.go:98] "Starting save volume queue"
I0901 10:36:43.310554       1 capacity.go:255] Started Capacity Controller
I0901 10:36:43.410925       1 event.go:389] "Event occurred" object="controller-concierge-k8s/storage-controller-0" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Normal" reason="Provisioning" message="External provisioner is provisioning volume for claim \"controller-concierge-k8s/storage-controller-0\""
I0901 10:36:43.410977       1 controller.go:873] "Started provisioner controller" component="rawfile.csi.openebs.io_ck-storage-rawfile-csi-node-j6p6v_bd68acdb-8251-4281-bf95-e331f948abf9"
I0901 10:36:43.421314       1 event.go:389] "Event occurred" object="controller-concierge-k8s/storage-controller-0" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Normal" reason="ProvisioningSucceeded" message="Successfully provisioned volume pvc-a431b0e7-fa8f-4aec-a983-c595f9fea711"
I0902 10:17:57.544443       1 event.go:389] "Event occurred" object="kubeflow/grafana-agent-k8s-data-11cfb2fa-grafana-agent-k8s-0" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Normal" reason="Provisioning" message="External provisioner is provisioning volume for claim \"kubeflow/grafana-agent-k8s-data-11cfb2fa-grafana-agent-k8s-0\""
I0902 10:17:57.563773       1 event.go:389] "Event occurred" object="kubeflow/grafana-agent-k8s-data-11cfb2fa-grafana-agent-k8s-0" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Normal" reason="ProvisioningSucceeded" message="Successfully provisioned volume pvc-6b7d93c7-46fd-4899-9aa2-a1c0b616ea91"
I0902 10:35:06.230183       1 event.go:389] "Event occurred" object="kubeflow/minio-minio-data-c98d9b31-minio-0" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Normal" reason="Provisioning" message="External provisioner is provisioning volume for claim \"kubeflow/minio-minio-data-c98d9b31-minio-0\""
I0902 10:35:06.248146       1 event.go:389] "Event occurred" object="kubeflow/minio-minio-data-c98d9b31-minio-0" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Normal" reason="ProvisioningSucceeded" message="Successfully provisioned volume pvc-5152757d-cc3a-4e1a-8579-54b3c371910f"
I0902 10:43:19.055457       1 controller.go:1569] "Failed to delete persistentvolume" PV="pvc-5152757d-cc3a-4e1a-8579-54b3c371910f" err="persistentvolumes \"pvc-5152757d-cc3a-4e1a-8579-54b3c371910f\" not found"
I0902 10:43:19.055487       1 controller.go:1007] "Retrying syncing volume" key="pvc-5152757d-cc3a-4e1a-8579-54b3c371910f" failures=0
E0902 10:43:19.055977       1 controller.go:1025] error syncing volume "pvc-5152757d-cc3a-4e1a-8579-54b3c371910f": persistentvolumes "pvc-5152757d-cc3a-4e1a-8579-54b3c371910f" not found
I0902 10:54:05.058912       1 event.go:389] "Event occurred" object="kubeflow/katib-db-database-b63f97be-katib-db-0" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Normal" reason="Provisioning" message="External provisioner is provisioning volume for claim \"kubeflow/katib-db-database-b63f97be-katib-db-0\""
I0902 10:54:05.113869       1 event.go:389] "Event occurred" object="kubeflow/katib-db-database-b63f97be-katib-db-0" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Normal" reason="ProvisioningSucceeded" message="Successfully provisioned volume pvc-4baea8cb-3b00-4669-b8d8-89a82f29e532"
I0902 11:05:46.004509       1 event.go:389] "Event occurred" object="test-kubeflow/simple-pbt-pbt" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Normal" reason="Provisioning" message="External provisioner is provisioning volume for claim \"test-kubeflow/simple-pbt-pbt\""
I0902 11:05:46.006526       1 controller.go:951] "Retrying syncing claim" key="3397b096-9023-426f-bd60-9d2211c8cb3e" failures=0
E0902 11:05:46.006561       1 controller.go:974] error syncing claim "3397b096-9023-426f-bd60-9d2211c8cb3e": failed to provision volume with StorageClass "csi-rawfile-default": rpc error: code = InvalidArgument desc = Unsupported access mode: MULTI_NODE_MULTI_WRITER
I0902 11:05:46.006601       1 event.go:389] "Event occurred" object="test-kubeflow/simple-pbt-pbt" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Warning" reason="ProvisioningFailed" message="failed to provision volume with StorageClass \"csi-rawfile-default\": rpc error: code = InvalidArgument desc = Unsupported access mode: MULTI_NODE_MULTI_WRITER"
I0902 11:05:47.007315       1 event.go:389] "Event occurred" object="test-kubeflow/simple-pbt-pbt" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Normal" reason="Provisioning" message="External provisioner is provisioning volume for claim \"test-kubeflow/simple-pbt-pbt\""
I0902 11:05:47.008159       1 controller.go:951] "Retrying syncing claim" key="3397b096-9023-426f-bd60-9d2211c8cb3e" failures=1
E0902 11:05:47.008186       1 controller.go:974] error syncing claim "3397b096-9023-426f-bd60-9d2211c8cb3e": failed to provision volume with StorageClass "csi-rawfile-default": rpc error: code = InvalidArgument desc = Unsupported access mode: MULTI_NODE_MULTI_WRITER
I0902 11:05:47.008224       1 event.go:389] "Event occurred" object="test-kubeflow/simple-pbt-pbt" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Warning" reason="ProvisioningFailed" message="failed to provision volume with StorageClass \"csi-rawfile-default\": rpc error: code = InvalidArgument desc = Unsupported access mode: MULTI_NODE_MULTI_WRITER"
I0902 11:05:49.009319       1 event.go:389] "Event occurred" object="test-kubeflow/simple-pbt-pbt" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Normal" reason="Provisioning" message="External provisioner is provisioning volume for claim \"test-kubeflow/simple-pbt-pbt\""
I0902 11:05:49.010031       1 controller.go:951] "Retrying syncing claim" key="3397b096-9023-426f-bd60-9d2211c8cb3e" failures=2
E0902 11:05:49.010051       1 controller.go:974] error syncing claim "3397b096-9023-426f-bd60-9d2211c8cb3e": failed to provision volume with StorageClass "csi-rawfile-default": rpc error: code = InvalidArgument desc = Unsupported access mode: MULTI_NODE_MULTI_WRITER
I0902 11:05:49.010122       1 event.go:389] "Event occurred" object="test-kubeflow/simple-pbt-pbt" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Warning" reason="ProvisioningFailed" message="failed to provision volume with StorageClass \"csi-rawfile-default\": rpc error: code = InvalidArgument desc = Unsupported access mode: MULTI_NODE_MULTI_WRITER"
I0902 11:05:53.011242       1 event.go:389] "Event occurred" object="test-kubeflow/simple-pbt-pbt" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Normal" reason="Provisioning" message="External provisioner is provisioning volume for claim \"test-kubeflow/simple-pbt-pbt\""
I0902 11:05:53.012057       1 controller.go:951] "Retrying syncing claim" key="3397b096-9023-426f-bd60-9d2211c8cb3e" failures=3
E0902 11:05:53.012084       1 controller.go:974] error syncing claim "3397b096-9023-426f-bd60-9d2211c8cb3e": failed to provision volume with StorageClass "csi-rawfile-default": rpc error: code = InvalidArgument desc = Unsupported access mode: MULTI_NODE_MULTI_WRITER
I0902 11:05:53.012104       1 event.go:389] "Event occurred" object="test-kubeflow/simple-pbt-pbt" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Warning" reason="ProvisioningFailed" message="failed to provision volume with StorageClass \"csi-rawfile-default\": rpc error: code = InvalidArgument desc = Unsupported access mode: MULTI_NODE_MULTI_WRITER"
I0902 11:06:01.013209       1 event.go:389] "Event occurred" object="test-kubeflow/simple-pbt-pbt" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Normal" reason="Provisioning" message="External provisioner is provisioning volume for claim \"test-kubeflow/simple-pbt-pbt\""
I0902 11:06:01.014003       1 controller.go:951] "Retrying syncing claim" key="3397b096-9023-426f-bd60-9d2211c8cb3e" failures=4
E0902 11:06:01.014024       1 controller.go:974] error syncing claim "3397b096-9023-426f-bd60-9d2211c8cb3e": failed to provision volume with StorageClass "csi-rawfile-default": rpc error: code = InvalidArgument desc = Unsupported access mode: MULTI_NODE_MULTI_WRITER
I0902 11:06:01.014042       1 event.go:389] "Event occurred" object="test-kubeflow/simple-pbt-pbt" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Warning" reason="ProvisioningFailed" message="failed to provision volume with StorageClass \"csi-rawfile-default\": rpc error: code = InvalidArgument desc = Unsupported access mode: MULTI_NODE_MULTI_WRITER"
I0902 11:06:17.014365       1 event.go:389] "Event occurred" object="test-kubeflow/simple-pbt-pbt" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Normal" reason="Provisioning" message="External provisioner is provisioning volume for claim \"test-kubeflow/simple-pbt-pbt\""
I0902 11:06:17.015155       1 controller.go:951] "Retrying syncing claim" key="3397b096-9023-426f-bd60-9d2211c8cb3e" failures=5
E0902 11:06:17.015187       1 controller.go:974] error syncing claim "3397b096-9023-426f-bd60-9d2211c8cb3e": failed to provision volume with StorageClass "csi-rawfile-default": rpc error: code = InvalidArgument desc = Unsupported access mode: MULTI_NODE_MULTI_WRITER
I0902 11:06:17.015237       1 event.go:389] "Event occurred" object="test-kubeflow/simple-pbt-pbt" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Warning" reason="ProvisioningFailed" message="failed to provision volume with StorageClass \"csi-rawfile-default\": rpc error: code = InvalidArgument desc = Unsupported access mode: MULTI_NODE_MULTI_WRITER"
I0902 11:06:43.269183       1 event.go:389] "Event occurred" object="test-kubeflow/simple-pbt-pbt" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Normal" reason="Provisioning" message="External provisioner is provisioning volume for claim \"test-kubeflow/simple-pbt-pbt\""
I0902 11:06:43.270429       1 controller.go:951] "Retrying syncing claim" key="3397b096-9023-426f-bd60-9d2211c8cb3e" failures=6
E0902 11:06:43.270485       1 controller.go:974] error syncing claim "3397b096-9023-426f-bd60-9d2211c8cb3e": failed to provision volume with StorageClass "csi-rawfile-default": rpc error: code = InvalidArgument desc = Unsupported access mode: MULTI_NODE_MULTI_WRITER
I0902 11:06:43.270514       1 event.go:389] "Event occurred" object="test-kubeflow/simple-pbt-pbt" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Warning" reason="ProvisioningFailed" message="failed to provision volume with StorageClass \"csi-rawfile-default\": rpc error: code = InvalidArgument desc = Unsupported access mode: MULTI_NODE_MULTI_WRITER"
I0902 11:06:49.016384       1 event.go:389] "Event occurred" object="test-kubeflow/simple-pbt-pbt" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Normal" reason="Provisioning" message="External provisioner is provisioning volume for claim \"test-kubeflow/simple-pbt-pbt\""
I0902 11:06:49.017010       1 controller.go:951] "Retrying syncing claim" key="3397b096-9023-426f-bd60-9d2211c8cb3e" failures=7
E0902 11:06:49.017028       1 controller.go:974] error syncing claim "3397b096-9023-426f-bd60-9d2211c8cb3e": failed to provision volume with StorageClass "csi-rawfile-default": rpc error: code = InvalidArgument desc = Unsupported access mode: MULTI_NODE_MULTI_WRITER
I0902 11:06:49.017043       1 event.go:389] "Event occurred" object="test-kubeflow/simple-pbt-pbt" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Warning" reason="ProvisioningFailed" message="failed to provision volume with StorageClass \"csi-rawfile-default\": rpc error: code = InvalidArgument desc = Unsupported access mode: MULTI_NODE_MULTI_WRITER"
I0902 11:08:57.018307       1 event.go:389] "Event occurred" object="test-kubeflow/simple-pbt-pbt" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Normal" reason="Provisioning" message="External provisioner is provisioning volume for claim \"test-kubeflow/simple-pbt-pbt\""
I0902 11:08:57.019045       1 controller.go:951] "Retrying syncing claim" key="3397b096-9023-426f-bd60-9d2211c8cb3e" failures=8
E0902 11:08:57.019063       1 controller.go:974] error syncing claim "3397b096-9023-426f-bd60-9d2211c8cb3e": failed to provision volume with StorageClass "csi-rawfile-default": rpc error: code = InvalidArgument desc = Unsupported access mode: MULTI_NODE_MULTI_WRITER
I0902 11:08:57.019123       1 event.go:389] "Event occurred" object="test-kubeflow/simple-pbt-pbt" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Warning" reason="ProvisioningFailed" message="failed to provision volume with StorageClass \"csi-rawfile-default\": rpc error: code = InvalidArgument desc = Unsupported access mode: MULTI_NODE_MULTI_WRITER"
I0902 11:13:13.020032       1 event.go:389] "Event occurred" object="test-kubeflow/simple-pbt-pbt" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Normal" reason="Provisioning" message="External provisioner is provisioning volume for claim \"test-kubeflow/simple-pbt-pbt\""
I0902 11:13:13.020661       1 controller.go:951] "Retrying syncing claim" key="3397b096-9023-426f-bd60-9d2211c8cb3e" failures=9
E0902 11:13:13.020680       1 controller.go:974] error syncing claim "3397b096-9023-426f-bd60-9d2211c8cb3e": failed to provision volume with StorageClass "csi-rawfile-default": rpc error: code = InvalidArgument desc = Unsupported access mode: MULTI_NODE_MULTI_WRITER
I0902 11:13:13.020736       1 event.go:389] "Event occurred" object="test-kubeflow/simple-pbt-pbt" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Warning" reason="ProvisioningFailed" message="failed to provision volume with StorageClass \"csi-rawfile-default\": rpc error: code = InvalidArgument desc = Unsupported access mode: MULTI_NODE_MULTI_WRITER"
I0902 11:18:13.021487       1 event.go:389] "Event occurred" object="test-kubeflow/simple-pbt-pbt" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Normal" reason="Provisioning" message="External provisioner is provisioning volume for claim \"test-kubeflow/simple-pbt-pbt\""
I0902 11:18:13.022210       1 controller.go:951] "Retrying syncing claim" key="3397b096-9023-426f-bd60-9d2211c8cb3e" failures=10
E0902 11:18:13.022227       1 controller.go:974] error syncing claim "3397b096-9023-426f-bd60-9d2211c8cb3e": failed to provision volume with StorageClass "csi-rawfile-default": rpc error: code = InvalidArgument desc = Unsupported access mode: MULTI_NODE_MULTI_WRITER
I0902 11:18:13.022282       1 event.go:389] "Event occurred" object="test-kubeflow/simple-pbt-pbt" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Warning" reason="ProvisioningFailed" message="failed to provision volume with StorageClass \"csi-rawfile-default\": rpc error: code = InvalidArgument desc = Unsupported access mode: MULTI_NODE_MULTI_WRITER"
I0902 11:21:43.270168       1 event.go:389] "Event occurred" object="test-kubeflow/simple-pbt-pbt" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Normal" reason="Provisioning" message="External provisioner is provisioning volume for claim \"test-kubeflow/simple-pbt-pbt\""
I0902 11:21:43.270883       1 controller.go:951] "Retrying syncing claim" key="3397b096-9023-426f-bd60-9d2211c8cb3e" failures=11
E0902 11:21:43.270911       1 controller.go:974] error syncing claim "3397b096-9023-426f-bd60-9d2211c8cb3e": failed to provision volume with StorageClass "csi-rawfile-default": rpc error: code = InvalidArgument desc = Unsupported access mode: MULTI_NODE_MULTI_WRITER
I0902 11:21:43.270961       1 event.go:389] "Event occurred" object="test-kubeflow/simple-pbt-pbt" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Warning" reason="ProvisioningFailed" message="failed to provision volume with StorageClass \"csi-rawfile-default\": rpc error: code = InvalidArgument desc = Unsupported access mode: MULTI_NODE_MULTI_WRITER"
I0902 11:23:13.023181       1 event.go:389] "Event occurred" object="test-kubeflow/simple-pbt-pbt" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Normal" reason="Provisioning" message="External provisioner is provisioning volume for claim \"test-kubeflow/simple-pbt-pbt\""
I0902 11:23:13.023843       1 controller.go:951] "Retrying syncing claim" key="3397b096-9023-426f-bd60-9d2211c8cb3e" failures=12
E0902 11:23:13.023862       1 controller.go:974] error syncing claim "3397b096-9023-426f-bd60-9d2211c8cb3e": failed to provision volume with StorageClass "csi-rawfile-default": rpc error: code = InvalidArgument desc = Unsupported access mode: MULTI_NODE_MULTI_WRITER
I0902 11:23:13.023926       1 event.go:389] "Event occurred" object="test-kubeflow/simple-pbt-pbt" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Warning" reason="ProvisioningFailed" message="failed to provision volume with StorageClass \"csi-rawfile-default\": rpc error: code = InvalidArgument desc = Unsupported access mode: MULTI_NODE_MULTI_WRITER"
I0902 11:28:13.024090       1 event.go:389] "Event occurred" object="test-kubeflow/simple-pbt-pbt" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Normal" reason="Provisioning" message="External provisioner is provisioning volume for claim \"test-kubeflow/simple-pbt-pbt\""
I0902 11:28:13.024703       1 controller.go:951] "Retrying syncing claim" key="3397b096-9023-426f-bd60-9d2211c8cb3e" failures=13
E0902 11:28:13.024720       1 controller.go:974] error syncing claim "3397b096-9023-426f-bd60-9d2211c8cb3e": failed to provision volume with StorageClass "csi-rawfile-default": rpc error: code = InvalidArgument desc = Unsupported access mode: MULTI_NODE_MULTI_WRITER
I0902 11:28:13.024736       1 event.go:389] "Event occurred" object="test-kubeflow/simple-pbt-pbt" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Warning" reason="ProvisioningFailed" message="failed to provision volume with StorageClass \"csi-rawfile-default\": rpc error: code = InvalidArgument desc = Unsupported access mode: MULTI_NODE_MULTI_WRITER"
I0902 11:33:13.025139       1 event.go:389] "Event occurred" object="test-kubeflow/simple-pbt-pbt" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Normal" reason="Provisioning" message="External provisioner is provisioning volume for claim \"test-kubeflow/simple-pbt-pbt\""
I0902 11:33:13.025751       1 controller.go:951] "Retrying syncing claim" key="3397b096-9023-426f-bd60-9d2211c8cb3e" failures=14
E0902 11:33:13.025772       1 controller.go:974] error syncing claim "3397b096-9023-426f-bd60-9d2211c8cb3e": failed to provision volume with StorageClass "csi-rawfile-default": rpc error: code = InvalidArgument desc = Unsupported access mode: MULTI_NODE_MULTI_WRITER
I0902 11:33:13.025846       1 event.go:389] "Event occurred" object="test-kubeflow/simple-pbt-pbt" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Warning" reason="ProvisioningFailed" message="failed to provision volume with StorageClass \"csi-rawfile-default\": rpc error: code = InvalidArgument desc = Unsupported access mode: MULTI_NODE_MULTI_WRITER"
I0902 11:36:43.270710       1 event.go:389] "Event occurred" object="test-kubeflow/simple-pbt-pbt" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Normal" reason="Provisioning" message="External provisioner is provisioning volume for claim \"test-kubeflow/simple-pbt-pbt\""
I0902 11:36:43.271405       1 controller.go:951] "Retrying syncing claim" key="3397b096-9023-426f-bd60-9d2211c8cb3e" failures=15
E0902 11:36:43.271422       1 controller.go:974] error syncing claim "3397b096-9023-426f-bd60-9d2211c8cb3e": failed to provision volume with StorageClass "csi-rawfile-default": rpc error: code = InvalidArgument desc = Unsupported access mode: MULTI_NODE_MULTI_WRITER
I0902 11:36:43.271437       1 event.go:389] "Event occurred" object="test-kubeflow/simple-pbt-pbt" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Warning" reason="ProvisioningFailed" message="failed to provision volume with StorageClass \"csi-rawfile-default\": rpc error: code = InvalidArgument desc = Unsupported access mode: MULTI_NODE_MULTI_WRITER"
I0902 11:38:13.026390       1 event.go:389] "Event occurred" object="test-kubeflow/simple-pbt-pbt" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Normal" reason="Provisioning" message="External provisioner is provisioning volume for claim \"test-kubeflow/simple-pbt-pbt\""
I0902 11:38:13.027050       1 controller.go:951] "Retrying syncing claim" key="3397b096-9023-426f-bd60-9d2211c8cb3e" failures=16
E0902 11:38:13.027067       1 controller.go:974] error syncing claim "3397b096-9023-426f-bd60-9d2211c8cb3e": failed to provision volume with StorageClass "csi-rawfile-default": rpc error: code = InvalidArgument desc = Unsupported access mode: MULTI_NODE_MULTI_WRITER
I0902 11:38:13.027123       1 event.go:389] "Event occurred" object="test-kubeflow/simple-pbt-pbt" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Warning" reason="ProvisioningFailed" message="failed to provision volume with StorageClass \"csi-rawfile-default\": rpc error: code = InvalidArgument desc = Unsupported access mode: MULTI_NODE_MULTI_WRITER"
```

### To Reproduce

.

### Environment

- Canonical K8s `1.32`

### Relevant Log Output

```shell
.
```

### Additional Context

The PVC is set to RWX also in upstream, which implies multiple Pods need to access this PVC
https://github.com/kubeflow/manifests/blame/253043e9c7745058eead4d92405dfaa9802b5c9e/applications/katib/upstream/installs/katib-standalone/katib-config.yaml#L50

Asking the K8s team about this AccessMode, I got this response:
> For both of these cases I believe we can not offer support out of the box as it goes against the "best practices". The main reason both of these don't work is that we mount an "disk image" through a loop device to ensure the storage limits and access. If you'd like to replicate the storage behavior of microk8s with mounting a directory you could look into adopting the [addon](https://github.com/canonical/microk8s-core-addons/tree/main/addons/hostpath-storage) to work on k8s-snap or try upstream https://github.com/kubernetes-csi/csi-driver-host-path

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

`test_katib_experiments` fails on Canonical K8s: Cannot provision PVC with `ReadWriteMany` Access Mode #347

Bug Description

To Reproduce

Environment

Relevant Log Output

Additional Context

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Uh oh!

test_katib_experiments fails on Canonical K8s: Cannot provision PVC with ReadWriteMany Access Mode #347

Description

Bug Description

To Reproduce

Environment

Relevant Log Output

Additional Context

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions

`test_katib_experiments` fails on Canonical K8s: Cannot provision PVC with `ReadWriteMany` Access Mode #347