unable to obtain workload identity auth

[lalitc113]

Question

After setting up the CSI drivers I am getting below error

Error:
Unable to attach or mount volumes: unmounted volumes=[mysecret], unattached volumes=[mysecret kube-api-access-d]: timed out waiting for the condition | FailedMount |
-- | -- | -- | -- | -- | --
MountVolume.SetUp failed for volume "mysecret" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod app-ns/myapp-deployment-25lp23-svvcp, err: rpc error: code = PermissionDenied desc = unable to obtain auth for mount: unable to obtain workload identity auth: unable to fetch gcp service account token: rpc error: code = PermissionDenied desc = Permission 'iam.serviceAccounts.getAccessToken' denied on resource (or it may not exist). error details: name = ErrorInfo reason = IAM_PERMISSION_DENIED domain = iam.googleapis.com metadata = map[permission:iam.serviceAccounts.getAccessToken]

all permission are in place but still getting this error

Requesting you to please suggest any solution for this issue

[vitordeap]

Hello friend,
Make sure to:
1-create a service account with the annotation iam.gke.io/gcp-service-account
2-execute the binding according to this doc
3-Make sure the Service Account has the role roles/iam.serviceAccountTokenCreator it seems it could be permission
4-Assign the service account to the consuming pod or to the gcp-provider pod, depending on your config

[alff]

Hi there,
faced the same issue recently. Just check your SA workload identity value. I've found weird thing:
when I have bind for the first time from doc - i saw same error. Then when I checked Identity value I realized it was created incorrectly.
PrincipalSet ended with wildcard - ...global/workloadIdentityPools/my-pool/*

So I ran same binding command again and after that when I checked Identity i showed correct value - ..global/workloadIdentityPools/my-pool/attribute.repository/my_repo_name.

After that issue was gone.

Hope it helps someone else.

[clemenspeters]

@lalitc113 It took me a moment during the setup as well and I think the documentation should be improved.
In our case the issue was that we assumed the SA of the pod where the volume is mounted would be used to fetch the secret value, but (in hindsight obviously) it's actually the SA of the csi-secrets-store-provider-gcp workload which is secrets-store-csi-driver-provider-gcp if you installed via helm using the default values.

In our case this solved the issue:

1. Create a separate GCP SA: csi-driver-provider-gcp@<GCP_PROJECT_ID>.iam.gserviceaccount.com
2. Add the role roles/secretmanager.secretAccessor to that GCP SA
3. Enable Workload Identity is working by adding the policy binding in GCP:

gcloud iam service-accounts add-iam-policy-binding \
    csi-driver-provider-gcp@<GCP_PROJECT_ID>.iam.gserviceaccount.com \
    --role roles/iam.workloadIdentityUser \
    --member "serviceAccount:<GCP_PROJECT_ID>.svc.id.goog[kube-system/secrets-store-csi-driver-provider-gcp]"

and adding the required annotiation in the helm values.yaml file:

serviceAccount:
  annotations:
    {
      iam.gke.io/gcp-service-account: csi-driver-provider-gcp@<GCP_PROJECT_ID>.iam.gserviceaccount.com,
    }
  name: secrets-store-csi-driver-provider-gcp

I really hope this will save someone else some time setting this up.

[psandhu79]

@clemenspeters We have created a service account and attached it to our pods with the workload identity this works most of the time but sometimes times out trying to get the workload identity.

Is your method more efficient that only the csi-secrets-store-provider-gcp is making the call rather than all the pods that mount secrets

[rob-whittle]

@psandhu79 we are also experiencing timeout issues when have the service account attached to the pods. Most of the time it works fine but recently we had a bunch of cronjobs all starting at the same time, all using the same service account. We are seeing errors like unable to obtain workload identity auth: unable to fetch SA info: client rate limiter Wait returned an error: context canceled, in the csi-secrets-store-provider-gcp pods. Did you manage resolve the issue without switching over attaching the service account directly to gcp provider pod?