DeadlineExceeded Error (failed to fetch secret)

[DrazThan]

TL;DR
Every few days the provider is unable to retrieve secrets from the Secret Manager

Expected behavior
Expecting secrets to be retrieved throughout

Observed behavior
Secrets are not retrieved and logs fill with DeadLine Exceeded error, only fixed by restarting the provider daemonset IE
kubectl -n kube-system rollout restart daemonset csi-secrets-store-provider-gcp

Reproduction
Normal operation, every few days I see new containers are stuck on containercreating status with this under the pod events -
{"ts":1754931810198.8691,"caller":"server/server.go:170","msg":"failed to fetch secret","resource_name":"projects/X/secrets/X/versions/latest","err":"rpc error: code = DeadlineExceeded desc = context deadline exceeded while waiting for connections to become ready"}

Environment
GKE Cluster using both provider and CSI driver installed via helm
I tried increasing the resources in helm for both

[k8s-ci-robot]

This issue is currently awaiting triage.

If secrets-store-csi-driver contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.