Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CSI Driver failed to mount PV to a Pod!!! #380

Open
arupdevops opened this issue Feb 28, 2024 · 21 comments
Open

CSI Driver failed to mount PV to a Pod!!! #380

arupdevops opened this issue Feb 28, 2024 · 21 comments

Comments

@arupdevops
Copy link

Hi Team,

While trying to mount a exsisting PV to a deployment, we are getting the below error.

Please help here where we are making mistake

================================================

17m Warning FailedAttachVolume pod/nginx-arup-7c7d84bf74-gwz5g AttachVolume.Attach failed for volume "pvc-ee5428bb-bae9-42d1-a739-e3d4e499503f" : rpc error: code = Internal desc = Failed to add ACL to volume pvc-ee5428bb-bae9-42d1-a739-e3d4e499503f for node &{ ocp-cp3.cluster.ocp.aspire.infra de78e75c-94e5-21de-7f15-cebdacc7d26f [0xc000920b80] [0xc000920bc0 0xc000920bd0 0xc000920be0 0xc000920bf0] [] } via CSP, err: Request failed with status code 500 and errors Error code (Internal Server Error) and message (VLUN creation error: failed to find any ready target port on array 192.168.236.192)

regards
Arup
@datamattsson
Copy link
Collaborator

VLUN creation error: failed to find any ready target port on array 192.168.236.192

This is usually a mismatch in protocols. Does the StorageClass have the correct accessProtocol set?

@arupdevops
Copy link
Author

Hi

Below is the YAML for the Storage class,
Access protocol is ISCSI

=============================

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: hpe-scod
uid: 66d582be-02c2-4c4a-8c2d-206b31a95d01
resourceVersion: '136206'
creationTimestamp: '2024-01-29T16:52:21Z'
managedFields:
- manager: Mozilla
operation: Update
apiVersion: storage.k8s.io/v1
time: '2024-01-29T16:52:21Z'
fieldsType: FieldsV1
fieldsV1:
'f:allowVolumeExpansion': {}
'f:parameters':
'f:csi.storage.k8s.io/controller-expand-secret-name': {}
'f:csi.storage.k8s.io/node-publish-secret-namespace': {}
'f:csi.storage.k8s.io/node-publish-secret-name': {}
'f:csi.storage.k8s.io/controller-expand-secret-namespace': {}
'f:description': {}
'f:csi.storage.k8s.io/controller-publish-secret-namespace': {}
'f:csi.storage.k8s.io/fstype': {}
.: {}
'f:csi.storage.k8s.io/provisioner-secret-namespace': {}
'f:accessProtocol': {}
'f:csi.storage.k8s.io/node-stage-secret-name': {}
'f:csi.storage.k8s.io/node-stage-secret-namespace': {}
'f:csi.storage.k8s.io/provisioner-secret-name': {}
'f:csi.storage.k8s.io/controller-publish-secret-name': {}
'f:provisioner': {}
'f:reclaimPolicy': {}
'f:volumeBindingMode': {}
provisioner: csi.hpe.com
parameters:
csi.storage.k8s.io/fstype: xfs
csi.storage.k8s.io/provisioner-secret-namespace: hpe-storage
csi.storage.k8s.io/provisioner-secret-name: hpe-backend
csi.storage.k8s.io/node-stage-secret-name: hpe-backend
csi.storage.k8s.io/controller-expand-secret-name: hpe-backend
csi.storage.k8s.io/node-publish-secret-namespace: hpe-storage
csi.storage.k8s.io/controller-publish-secret-name: hpe-backend
csi.storage.k8s.io/controller-publish-secret-namespace: hpe-storage
csi.storage.k8s.io/node-publish-secret-name: hpe-backend
csi.storage.k8s.io/controller-expand-secret-namespace: hpe-storage
accessProtocol: iscsi
description: Volume created by the HPE CSI Driver for Kubernetes
csi.storage.k8s.io/node-stage-secret-namespace: hpe-storage
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: Immediate

regards
Arup

@datamattsson
Copy link
Collaborator

If it's iSCSI then my best guess is that you don't have any iSCSI ports on the array. If you check the logs of the CSP, it may reveal more clues. k logs -nhpe-storage deploy/primera3par-csp

@arupdevops
Copy link
Author

Thanx, what is the exact command to check the logs "k logs -nhpe-storage deploy/primera3par-csp"?

@datamattsson
Copy link
Collaborator

kubectl logs -nhpe-storage deploy/primera3par-csp didn't work?

@arupdevops
Copy link
Author

==============================================================
time="2024-02-28T07:05:17Z" level=info msg="Response status is 200 OK" file="rest_client.go:72"
time="2024-02-28T07:05:17Z" level=info msg="Get ports response is : &{0xc000a4c870 0xc000317380}" file="port_client.go:23"
time="2024-02-28T07:05:17Z" level=info msg="=== GET Ports RESP FROM ARRAY: ===\n &{200 OK 200 HTTP/2.0 2 0 map[Cache-Control:[no-cache, no-store, private, max-age=-1, must-revalidate no-cache] Content-Language:[en-US] Content-Security-Policy:[default-src 'self' https://www.google-analytics.com/ ; script-src 'unsafe-eval' 'self' https://www.google-analytics.com/analytics.js 'sha256-v0Q8X6xSNmwWcfldSEWqW+3imFpJurfHYdywm3hw6+w=' ; style-src 'nonce-d29tZSBjb29sIHN0cmluZyB3aWxsIHBvcCB1cCAxMjM=' 'self' blob:; frame-src 'self'; object-src 'self'; media-src 'self' data: blob:; font-src 'self' data:; frame-ancestors 'self'] Content-Type:[application/json] Date:[Wed, 28 Feb 2024 06:37:34 GMT] Pragma:[no-cache no-cache] Server:[hp3par-wsapi] Strict-Transport-Security:[max-age=31536000; includeSubDomains;] Vary:[Accept-Encoding Origin] X-Content-Type-Options:[nosniff] X-Frame-Options:[SAMEORIGIN] X-Xss-Protection:[1; mode=block]] 0xc000327440 -1 [] false true map[] 0xc0006b2600 0xc0001fb600}" file="managed_client.go:214"
time="2024-02-28T07:05:17Z" level=info msg="RESULT of get: &{{ } {pvc-02119201-2f72-479b-b7bd-464144ffecf0 bf4e96d1-5464-4ffb-b646-994f0582f5c9 4933799 1 2024-02-09 05:30:43 +0000 UTC map[] map[] [] [] [{main Update storage.hpe.com/v2 2024-02-09 05:30:43 +0000 UTC FieldsV1 {"f:spec":{".":{},"f:record":{".":{},"f:Compression":{},"f:Description":{},"f:HostEncryption":{},"f:HostSeesVLUN":{},"f:Id":{},"f:Name":{},"f:ProvisioningType":{},"f:Published":{},"f:Size":{},"f:isVolumeReplicated":{}},"f:uuid":{}}} }]} {pvc-02119201-2f72-479b-b7bd-464144ffecf0 {pvc-02119201-2f72-479b-b7bd-464144ffecf0 pvc-02119201-2f72-479b-b7bd-464 tpvv false 1073741824 Volume created by the HPE CSI Driver for Kubernetes false false false false }}}" file="crd_utils.go:117"
time="2024-02-28T07:05:17Z" level=debug msg="[ REQUEST-ID 102101 ] -- CRD volume metadata found: map[Compression:false Description:Volume created by the HPE CSI Driver for Kubernetes HostEncryption:false HostSeesVLUN:false Id:pvc-02119201-2f72-479b-b7bd-464144ffecf0 Name:pvc-02119201-2f72-479b-b7bd-464 ProvisioningType:tpvv Published:false Size:1073741824 isVolumeReplicated:false]" file="create_vlun_cmd.go:190"
time="2024-02-28T07:05:17Z" level=info msg="Appending port 0:3:1 , no env set for fc ports" file="create_vlun_cmd.go:258"
time="2024-02-28T07:05:17Z" level=info msg="Appending port 0:3:2 , no env set for fc ports" file="create_vlun_cmd.go:258"
time="2024-02-28T07:05:17Z" level=info msg="Appending port 1:3:1 , no env set for fc ports" file="create_vlun_cmd.go:258"
time="2024-02-28T07:05:17Z" level=info msg="Appending port 1:3:2 , no env set for fc ports" file="create_vlun_cmd.go:258"
time="2024-02-28T07:05:17Z" level=info msg="[ REQUEST-ID 102101 ] -- [Array: 192.168.236.192] LIST OF PORTS []" file="create_vlun_cmd.go:327"
time="2024-02-28T07:05:17Z" level=info msg="[ REQUEST-ID 102101 ] -- [Array: 192.168.236.192] TARGET IQNS []" file="create_vlun_cmd.go:329"
time="2024-02-28T07:05:17Z" level=info msg="[ REQUEST-ID 102101 ] -- [Array: 192.168.236.192] DISCOVERY IPS []" file="create_vlun_cmd.go:330"
time="2024-02-28T07:05:17Z" level=info msg="[ REQUEST-ID 102101 ] -- [Array: 192.168.236.192] Checking If volume pvc-02119201-2f72-479b-b7bd-464 has already existing LUNs associated with host ocp-cp2" file="create_vlun_cmd.go:333"
time="2024-02-28T07:05:17Z" level=info msg="Get session returning existing session for user 3paradm on array 192.168.236.192" file="managed_client.go:169"
time="2024-02-28T07:05:17Z" level=info msg="Query luns string: /vluns?query=%22volumeName%20EQ%20pvc-02119201-2f72-479b-b7bd-464%22" file="vlun_client.go:82"
time="2024-02-28T07:05:17Z" level=trace msg="Header: {X-HP3PAR-WSAPI-SessionKey : *****}\n" file="utils.go:30"
time="2024-02-28T07:05:17Z" level=trace msg="Header: {Content-type : application/json}\n" file="utils.go:32"
time="2024-02-28T07:05:17Z" level=info msg="Request: action=GET path=https://192.168.236.192:443/api/v1//vluns?query=%22volumeName%20EQ%20pvc-02119201-2f72-479b-b7bd-464%22" file="http.go:65"
time="2024-02-28T07:05:17Z" level=info msg="HTTP Response: {\n "total": 0,\n "members": []\n }" file="utils.go:46"
time="2024-02-28T07:05:17Z" level=info msg="Response status is 200 OK" file="rest_client.go:72"
time="2024-02-28T07:05:17Z" level=info msg="=== GET HOST VLUNs RESP FROM ARRAY: ===\n &{200 OK 200 HTTP/2.0 2 0 map[Cache-Control:[no-cache, no-store, private, max-age=-1, must-revalidate no-cache] Content-Language:[en-US] Content-Length:[24] Content-Security-Policy:[default-src 'self' https://www.google-analytics.com/ ; script-src 'unsafe-eval' 'self' https://www.google-analytics.com/analytics.js 'sha256-v0Q8X6xSNmwWcfldSEWqW+3imFpJurfHYdywm3hw6+w=' ; style-src 'nonce-d29tZSBjb29sIHN0cmluZyB3aWxsIHBvcCB1cCAxMjM=' 'self' blob:; frame-src 'self'; object-src 'self'; media-src 'self' data: blob:; font-src 'self' data:; frame-ancestors 'self'] Content-Type:[application/json] Date:[Wed, 28 Feb 2024 06:37:34 GMT] Pragma:[no-cache no-cache] Server:[hp3par-wsapi] Strict-Transport-Security:[max-age=31536000; includeSubDomains;] Vary:[Accept-Encoding Origin] X-Content-Type-Options:[nosniff] X-Frame-Options:[SAMEORIGIN] X-Xss-Protection:[1; mode=block]] {0xc0006ec180} 24 [] false false map[] 0xc0006b2000 0xc0001fa580}" file="managed_client.go:638"
time="2024-02-28T07:05:17Z" level=error msg="[ REQUEST-ID 102101 ] -- VLUN creation error: failed to find any ready target port on array 192.168.236.192" file="create_vlun_cmd.go:364"
time="2024-02-28T07:05:17Z" level=info msg="Raising panic with following message: VLUN creation error: failed to find any ready target port on array 192.168.236.192" file="utils.go:166"
time="2024-02-28T07:05:17Z" level=info msg="[ REQUEST-ID 102101 ] -- <<<<< Create VLUN Cmd: [Array: 192.168.236.192] - Host ocp-cp2 and Volume pvc-02119201-2f72-479b-b7bd-464144ffecf0" file="panic.go:884"
time="2024-02-28T07:05:17Z" level=trace msg="Releasing mutex lock for PublishVolume:pvc-02119201-2f72-479b-b7bd-464144ffecf0:6bf9b0b6-3886-d3ac-e443-3286cd7302ab" file="concurrent.go:67"
time="2024-02-28T07:05:17Z" level=info msg="[ REQUEST-ID 102101 ] -- <<<<< Publish Volume Cmd for volume pvc-02119201-2f72-479b-b7bd-464144ffecf0 and host 6bf9b0b6-3886-d3ac-e443-3286cd7302ab" file="panic.go:884"
time="2024-02-28T07:05:17Z" level=info msg="CspManager: Recovering from panic..." file="csp_manager.go:42"
time="2024-02-28T07:05:17Z" level=info msg="Sending following message to CSI driver: VLUN creation error: failed to find any ready target port on array 192.168.236.192" file="utils.go:173"
time="2024-02-28T07:05:17Z" level=info msg="Error response to be sent is &models.ErrorsPayload{Errors:[]models.ErrorObject{models.ErrorObject{Code:"Internal Server Error", Message:"VLUN creation error: failed to find any ready target port on array 192.168.236.192"}}}" file="utils.go:181"
[root@ocp-svc ~]#

@datamattsson
Copy link
Collaborator 

time="2024-02-28T07:05:17Z" level=info msg="[ REQUEST-ID 102101 ] -- [Array: 192.168.236.192] LIST OF PORTS []" file="create_vlun_cmd.go:327"
time="2024-02-28T07:05:17Z" level=info msg="[ REQUEST-ID 102101 ] -- [Array: 192.168.236.192] TARGET IQNS []" file="create_vlun_cmd.go:329"
time="2024-02-28T07:05:17Z" level=info msg="[ REQUEST-ID 102101 ] -- [Array: 192.168.236.192] DISCOVERY IPS []" file="create_vlun_cmd.go:330"

Doesn't look like the CSP can find any iSCSI ports.

@arupdevops
Copy link
Author

just now deploy a PV with fc protocol but mounting again got failed. Attached yaml for fc

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: hpe-scod-fc
uid: d7ffeed6-bb09-4d00-8d8e-d228d1345600
resourceVersion: '14910417'
creationTimestamp: '2024-02-28T06:47:22Z'
managedFields:
- manager: Mozilla
operation: Update
apiVersion: storage.k8s.io/v1
time: '2024-02-28T06:47:22Z'
fieldsType: FieldsV1
fieldsV1:
'f:allowVolumeExpansion': {}
'f:parameters':
'f:csi.storage.k8s.io/controller-expand-secret-name': {}
'f:csi.storage.k8s.io/node-publish-secret-namespace': {}
'f:csi.storage.k8s.io/node-publish-secret-name': {}
'f:csi.storage.k8s.io/controller-expand-secret-namespace': {}
'f:description': {}
'f:csi.storage.k8s.io/controller-publish-secret-namespace': {}
'f:csi.storage.k8s.io/fstype': {}
.: {}
'f:csi.storage.k8s.io/provisioner-secret-namespace': {}
'f:accessProtocol': {}
'f:csi.storage.k8s.io/node-stage-secret-name': {}
'f:csi.storage.k8s.io/node-stage-secret-namespace': {}
'f:csi.storage.k8s.io/provisioner-secret-name': {}
'f:csi.storage.k8s.io/controller-publish-secret-name': {}
'f:provisioner': {}
'f:reclaimPolicy': {}
'f:volumeBindingMode': {}
provisioner: csi.hpe.com
parameters:
csi.storage.k8s.io/fstype: xfs
csi.storage.k8s.io/provisioner-secret-namespace: hpe-storage
csi.storage.k8s.io/provisioner-secret-name: hpe-backend
csi.storage.k8s.io/node-stage-secret-name: hpe-backend
csi.storage.k8s.io/controller-expand-secret-name: hpe-backend
csi.storage.k8s.io/node-publish-secret-namespace: hpe-storage
csi.storage.k8s.io/controller-publish-secret-name: hpe-backend
csi.storage.k8s.io/controller-publish-secret-namespace: hpe-storage
csi.storage.k8s.io/node-publish-secret-name: hpe-backend
csi.storage.k8s.io/controller-expand-secret-namespace: hpe-storage
accessProtocol: fc
description: Volume created by the HPE CSI Driver for Kubernetes
csi.storage.k8s.io/node-stage-secret-namespace: hpe-storage
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: Immediate

@arupdevops
Copy link
Author

pvc_mount_issue

@datamattsson
Copy link
Collaborator

datamattsson commented Feb 28, 2024 •
edited
Loading

Is the worker node zoned (since you're switching to FC) with the array?

@arupdevops
Copy link
Author

Yes...all are zoned!!!

Redhat OCP-----3 Controller & 2 Worker nodes(All VMs from the Ovirt Engine Infra).
Ovirts Physical hosts are properly zoned with Storage controller

@arupdevops
Copy link
Author

storage_pvc_mount_error

We have found something & checking on that, will update!!!

@datamattsson
Copy link
Collaborator

Redhat OCP-----3 Controller & 2 Worker nodes(All VMs from the Ovirt Engine Infra).
Ovirts Physical hosts are properly zoned with Storage controller

This won't work, you have to use the oVirt CSI driver for persistent storage if the workers are VMs. The HPE CSI Driver only works on virtual machines when using iSCSI.

@arupdevops
Copy link
Author

The HPE CSI Driver only works on virtual machines when using iSCSI--------While using SCSI, as you mentioned it is unable to find any scsi port in the array.

The backend storage is HPE Primera Storage.

@datamattsson
Copy link
Collaborator

The array needs iSCSI cards and have those configured. You also need two vNICs on the oVirt VMs in ideally two different subnets shared with your array iSCSI interfaces.

@arupdevops
Copy link
Author

@datamattsson : Thanks for the support. We are checking on the iSCSI part. As of now we don't see any iscsi port on the HPE Primera.
On the VLAN part, we have allowed traffic for inter VLANa already.

@arupdevops
Copy link
Author

Hi logged a case with HPE for Primera, so they checking the same on the iSCSI part,I have a confusion here no where in documents it is mentioned that to use access protocol iSCSI,the supported Primera storage should be iSCSI configured,also how to configure all these steps,please help with respect to CSI driver

@datamattsson
Copy link
Collaborator

The documentation for the 3PAR CSP (Alletra 9K/MP, Primera etc): https://scod.hpedev.io/container_storage_provider/hpe_alletra_9000/#storageclass_parameters

@arupdevops
Copy link
Author

FC should work as we have FC ports available in the physical storage & by default CSI driver will use the available FC ports.

@datamattsson
Copy link
Collaborator

The HPE CSI Driver does not support virtual FC HPAs (NPIV). Run OpenShift on bare-metal instead and FC will work.

@arupdevops
Copy link
Author

Can you please help us with the ovirt csi operator link, as we are unable to find any good resource on that. Appreciate for your extending support here....

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@datamattsson @arupdevops