PVC volume appears to mount correctly in pod but writes to mountpoint inside the pod go to the nodes local filesystem

[mattjwarren]

What happened:
A pvc used as a volume mount inside a test pod appears to be correctly mounted per kubelet logs, and csi-nfs-node logs, but writes the to the mountpoint are made to the node's local filesystem under the kubelet/..pod../.../mount directory.

What you expected to happen:
writes are made ultimately on the .../shares/pvc-..../ directory on the machine hosting the nfs shares

How to reproduce it:
create a dynamic pvc, create a simple pod yaml referencing the pvc as a volume mount in a container.

Anything else we need to know?:

kubelet logs on the node hosting the pod show the pvc apparently succeeding?

Sep 03 13:54:29 statler.ssd.hursley.ibm.com kubelet[3710]: I0903 13:54:29.157854    3710 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-3d5ca3d0-bec1-4266-af75-70c62d545286\" (UniqueName: \"kubernetes.io/csi/nfs.csi.k8s.io^host.dns.name#path/to/shares#pvc-3d5ca3d0-bec1-4266-af75-70c62d545286##\") pod \"user-info-san-mattsb1-cephbase6\" (UID: \"d665bb74-6a65-4b85-a58a-21a8b4349921\") " pod="sandboxer/user-info-san-mattsb1-cephbase6"
Sep 03 13:54:29 statler.ssd.hursley.ibm.com kubelet[3710]: I0903 13:54:29.262092    3710 csi_attacher.go:359] kubernetes.io/csi: attacher.MountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping MountDevice...
Sep 03 13:54:29 statler.ssd.hursley.ibm.com kubelet[3710]: I0903 13:54:29.262153    3710 operation_generator.go:658] "MountVolume.MountDevice succeeded for volume \"pvc-3d5ca3d0-bec1-4266-af75-70c62d545286\" (UniqueName: \"kubernetes.io/csi/nfs.csi.k8s.io^host.dns.name#path/to/shares#pvc-3d5ca3d0-bec1-4266-af75-70c62d545286##\") pod \"user-info-san-mattsb1-cephbase6\" (UID: \"d665bb74-6a65-4b85-a58a-21a8b4349921\") device mount path \"/kubelet/root/plugins/kubernetes.io/csi/nfs.csi.k8s.io/aa7abf5636712c06ecf961324a368786a69a6c251b7b78791a38a9384cb1d841/globalmount\"" pod="sandboxer/user-info-san-mattsb1-cephbase6"

csi-nfs-node logs on the node appear to show the mount succeeded

nfs I0903 12:54:29.264903       1 utils.go:109] GRPC call: /csi.v1.Node/NodePublishVolume                                                               

nfs I0903 12:54:29.264929       1 utils.go:110] GRPC request: {"target_path":"/kubelet/root/pods/d665bb74-6a65-4b85-a58a-21a8b4349921/volumes/kubernetes.io~csi/pvc-3d5ca3d0-bec1-4266-af75-70c62d545286/mount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["nfsvers=4.1"]}},"access_mode":{"mode":5}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3d5ca3d0-bec1-4266-af75-70c62d545286","csi.storage.k8s.io/pvc/name":"user-info-nfs-matthm1","csi.storage.k8s.io/pvc/namespace":"sandboxer","server":"host.dns.name","share":"/path/to/shares","storage.
kubernetes.io/csiProvisionerIdentity":"1724402785882-5888-nfs.csi.k8s.io","subdir":"pvc-3d5ca3d0-bec1-4266-af75-70c62d545286"},"volume_id":"host.dns.name#path/to/shares#pvc-3d5ca3d0-bec1-4266-af75-70c62d545286##"}

nfs I0903 12:54:29.265346       1 nodeserver.go:132] NodePublishVolume: volumeID(host.dns.name#path/to/shares#pvc-3d5ca3d0-bec1-4266-af75-70c62d545286##) source(host.dns.name:/path/to/shares/pvc-3d5ca3d0-bec1-4266-af75-70c62d545286) targetPath(/kubelet/root/pods/d665bb74-6a65-4b85-a58a-21a8b4349921/volumes/kubernetes.io~csi/pvc-3d5ca3d0-bec1-4266-af75-70c62d545286/mount) mountflags([nfsvers=4.1])

nfs I0903 12:54:29.265372       1 mount_linux.go:218] Mounting cmd (mount) with arguments (-t nfs -o nfsvers=4.1 host.dns.name:/path/to/shares/pvc-3d5ca3d0-bec1-4266-af75-70c62d545286 /kubelet/root/pods/d665bb74-6a65-4b85-a58a-21a8b4349921/volumes/kubernetes.io~csi/pvc-3d5ca3d0-bec1-4266-af75-70c62d545286/mount)
    
nfs I0903 12:54:29.629603       1 nodeserver.go:149] skip chmod on targetPath(/kubelet/root/pods/d665bb74-6a65-4b85-a58a-21a8b4349921/volumes/kubernetes.io~csi/pvc-3d5ca3d0-bec1-4266-af75-70c62d545286/mount) since mountPermissions is set as 0

nfs I0903 12:54:29.629642       1 nodeserver.go:151] volume(host.dns.name#path/to/shares#pvc-3d5ca3d0-bec1-4266-af75-70c62d545286##) mount host.dns.name:/path/to/shares/pvc-3d5ca3d0-bec1-4266-af75-70c62d545286 on /kubelet/root/pods/d665bb74-6a65-4b85-a58a-21a8b4349921/volumes/kubernetes.io~csi/pvc-3d5ca3d0-bec1-4266-af75-70c62d545286/mount succeeded                                 

nfs I0903 12:54:29.629664       1 utils.go:116] GRPC response: {}   

Manually making the nfs mount to the node hosting the test pod from the nfs server works as expected.

running 'mount | grep nfs' from inside the csi-nfs-node pod shows the mount as expected.

Environment:

• CSI Driver version: v4.8.0
• Kubernetes version (use kubectl version): 1.25.16
• OS (e.g. from /etc/os-release): RHEL 8.10
• Kernel (e.g. uname -a): 4.18.0-553.16.1.el8_10.x86_64
• Install tools:
• Others:

[andyzhangx]

seems your kubelet dir is not under /var/lib/kubelet, you need to specify --set kubeletDir="..." in helm chart install, follow guide here: https://github.com/kubernetes-csi/csi-driver-nfs/tree/master/charts#tips

[mattjwarren]

That turned out to be the issue. Thankyou. Using helm rather than local installing and setting kubeletDir fixed it.

Is it worth raising an issue for this situation to be flagged up ? - the current logging produced by the driver appears to imply the mount succeeded, whereas a message regarding the reuqirement to set kubeletDir , or indicating there is an issue with kubelet roots location would have been very helpful.