Fileset is not getting deleted in case of GUI HA when another GUI node is getting used #1061
Description
Describe the bug
When all GUI nodes are down , suppose we bring GUI node -1 and create PVC and we stop this GUI node and we delete the created PVC. Now bring another GUI up , in this case after some time pv is getting deleted but fileset is not getting deleted
How to Reproduce?
- Install GM build on GUI HA cluster (3 GUI Nodes):
[root@multiguiubu-master pr1050]# oc get pods
NAME READY STATUS RESTARTS AGE
ibm-spectrum-scale-csi-59sx7 3/3 Running 0 18s
ibm-spectrum-scale-csi-attacher-846bcfb84c-fx2mh 1/1 Running 1 (20s ago) 29s
ibm-spectrum-scale-csi-attacher-846bcfb84c-qrsd7 1/1 Running 0 28s
ibm-spectrum-scale-csi-operator-85f6545658-qr6z9 1/1 Running 0 3h23m
ibm-spectrum-scale-csi-provisioner-754c55ff57-lvxkf 1/1 Running 0 29s
ibm-spectrum-scale-csi-qn9ll 3/3 Running 0 26s
ibm-spectrum-scale-csi-resizer-6b966777fc-hntmf 1/1 Running 0 29s
ibm-spectrum-scale-csi-snapshotter-55448875d8-qvhp6 1/1 Running 0 29s
[root@multiguiubu-master pr1050]# oc describe pod | grep quay
Image: quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver:v2.10.0-GM
Image ID: quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver@sha256:57b4ee494ca48342d1ffaf22a166286202b0406b88316e4dcbe87212df6ca8f0
Normal Pulled 24s kubelet Container image "quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver:v2.10.0-GM" already present on machine
Image: quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-operator:v2.10.0-GM
Image ID: quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-operator@sha256:52fac65c6258a874a1fcd3a2feb45d21a2129b1d5b6914fd8699ee36f5e9d787
CSI_DRIVER_IMAGE: quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver:v2.10.0-GM
Image: quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver:v2.10.0-GM
Image ID: quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver@sha256:57b4ee494ca48342d1ffaf22a166286202b0406b88316e4dcbe87212df6ca8f0
Normal Pulled 31s kubelet Container image "quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver:v2.10.0-GM" already present on machine
[root@multiguiubu-master pr1050]# oc get cso
NAME VERSION SUCCESS
ibm-spectrum-scale-csi 2.10.0 True
- Make all GUI node down except one GUI node (GUI 1):
[root@multiguiubu-scalegui-1 ~]# systemctl status gpfsgui
● gpfsgui.service - IBM_Spectrum_Scale Administration GUI
Loaded: loaded (/usr/lib/systemd/system/gpfsgui.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2023-11-09 00:59:10 PST; 1h 31min ago
[root@multiguiubu-scalegui-2 ~]# systemctl stop gpfsgui
[root@multiguiubu-scalegui-2 ~]# systemctl status gpfsgui
● gpfsgui.service - IBM_Spectrum_Scale Administration GUI
Loaded: loaded (/usr/lib/systemd/system/gpfsgui.service; enabled; vendor preset: disabled)
Active: inactive (dead) since Thu 2023-11-09 02:31:26 PST; 8s ago
[root@multiguiubu-scalegui-3 ~]# systemctl stop gpfsgui
[root@multiguiubu-scalegui-3 ~]# systemctl status gpfsgui
● gpfsgui.service - IBM_Spectrum_Scale Administration GUI
Loaded: loaded (/usr/lib/systemd/system/gpfsgui.service; enabled; vendor preset: disabled)
Active: inactive (dead) since Thu 2023-11-09 02:32:49 PST; 27s ago
- Create PVC as following :
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: scale-advance-pvc-5
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
storageClassName: ibm-spectrum-scale-csi-advance
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ibm-spectrum-scale-csi-advance
provisioner: spectrumscale.csi.ibm.com
parameters:
volBackendFs: "fs1"
version: "2"
reclaimPolicy: Delete
[root@multiguiubu-master pr1050]# oc apply -f apply.yaml
persistentvolumeclaim/scale-advance-pvc-5 created
storageclass.storage.k8s.io/ibm-spectrum-scale-csi-advance created
[root@multiguiubu-master pr1050]# oc get pvc -w
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
scale-advance-pvc-5 Bound pvc-a1df5bd1-05a1-4f05-b7b5-d5075af376c7 1Gi RWX ibm-spectrum-scale-csi-advance 42s
- Stop gui node (GUI 1):
[root@multiguiubu-scalegui-1 ~]# systemctl stop gpfsgui
[root@multiguiubu-scalegui-1 ~]# systemctl status gpfsgui
● gpfsgui.service - IBM_Spectrum_Scale Administration GUI
Loaded: loaded (/usr/lib/systemd/system/gpfsgui.service; enabled; vendor preset: disabled)
Active: inactive (dead) since Thu 2023-11-09 02:35:28 PST; 2min 30s ago
- Delete the created PVC :
[root@multiguiubu-master pr1050]# oc delete -f apply.yaml
persistentvolumeclaim "scale-advance-pvc-5" deleted
storageclass.storage.k8s.io "ibm-spectrum-scale-csi-advance" deleted
[root@multiguiubu-master pr1050]# oc get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-a1df5bd1-05a1-4f05-b7b5-d5075af376c7 1Gi RWX Delete Released ibm-spectrum-scale-csi-driver/scale-advance-pvc-5 ibm-spectrum-scale-csi-advance 4m55s
- Now start GUI node 2 :
[root@multiguiubu-scalegui-2 ~]# systemctl start gpfsgui
[root@multiguiubu-scalegui-2 ~]# systemctl status gpfsgui
● gpfsgui.service - IBM_Spectrum_Scale Administration GUI
Loaded: loaded (/usr/lib/systemd/system/gpfsgui.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2023-11-09 02:39:53 PST; 1min 2s ago
- Check if pv is getting deleted :
[root@multiguiubu-master pr1050]# oc get pv
No resources found
- Check if fileset is present or not :
[root@multiguiubu-scalegui-2 pvc-a1df5bd1-05a1-4f05-b7b5-d5075af376c7]# ls
[root@multiguiubu-scalegui-2 pvc-a1df5bd1-05a1-4f05-b7b5-d5075af376c7]# pwd
/ibm/fs1/6505846e-5027-418c-8030-6fcf39da580b-ibm-spectrum-scale-csi-driver/pvc-a1df5bd1-05a1-4f05-b7b5-d5075af376c7
Expected behavior
Fileset should be deleted and CSI shouldn't delete PV if there is some issue from GUI
Logs:
/scale-csi/D.1061
csisnap.tar.gz