Support AWS EKS Pod Identity Associations

[jtele2]

Description

Feature Request: Enable Elastic Cloud on Kubernetes (ECK) to natively support AWS Pod Identity Association (PIA) for managing IAM roles and credentials when running Elasticsearch on Amazon EKS.

Background: Currently, ECK supports IAM Roles for Service Accounts (IRSA) to manage AWS permissions, especially for S3 Snapshots. However, AWS EKS introduced Pod Identity Association (PIA), which allows managing IAM role associations at the cluster level without requiring service account annotations. This is beneficial for simplifying permissions management and avoiding hard-coded annotations.

When using PIA, the AWS_CONTAINER_CREDENTIALS_FULL_URI environment variable set by ECK's init containers conflicts with the expected behavior of PIA, causing pods to fall back to ECS-style credential retrieval rather than using the IAM role associated via PIA.

Supporting PIA would allow smoother integration and avoid manual overrides or workarounds for environment variables.

Current Behavior

Platform: Amazon EKS
ECK Version: 2.14.0
Elasticsearch Version: 8.15.3
EKS Version: 1.30
eksctl Version: 0.194.0-dev+02ef28ee3.2024-10-22T18:42:34Z
kubectl Version

• Client Version: v1.31.2
• Kustomize Version: v5.4.2
• Server Version: v1.30.6-eks-7f9249a

When deploying an Elasticsearch cluster using ECK with PIA, the elastic-internal-init-filesystem init container sets the following environment variables:

AWS_CONTAINER_CREDENTIALS_FULL_URI=http://169.254.170.23/v1/credentials
AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE=/var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token

These variables cause the AWS SDK to attempt credential retrieval via the ECS metadata endpoint, which conflicts with the PIA mechanism and leads to permission issues. The expected behavior for PIA is to use the Web Identity Token mounted by the service account to assume the IAM role.

Expected Behavior

ECK should detect and respect Pod Identity Associations (PIA) when deployed on EKS.
The init containers should not set ECS-specific environment variables (AWS_CONTAINER_CREDENTIALS_FULL_URI) when PIA is being used.
Elasticsearch should seamlessly use the IAM role associated via PIA, leveraging the Web Identity Token.

Steps to Reproduce

Create an EKS cluster with Pod Identity Association enabled using eksctl:

eksctl create cluster -f cluster-config.yml
Create a Pod Identity Association for the aws-sa service account in the eck namespace:

eksctl create podidentityassociation \
  --cluster DevCluster02 \
  --namespace eck \
  --service-account aws-sa \
  --role-arn arn:aws:iam::YOUR_ACCOUNT_ID:role/eck-snapshots-role

Deploy Elasticsearch with ECK using the following configuration:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: es
  namespace: eck
spec:
  version: 8.15.3
  nodeSets:
    - name: master
      count: 3
      podTemplate:
        spec:
          serviceAccountName: aws-sa

Check the environment variables in the init container:

kubectl exec -n eck es-es-master-0 -- env | grep AWS

You will see:

AWS_CONTAINER_CREDENTIALS_FULL_URI=http://169.254.170.23/v1/credentials

Error Logs

Attempt to perform an S3 snapshot operation, which fails due to incorrect credential retrieval. Relevant error logs:

[YOUR_BUCKET] path  is not accessible on master node","error.stack_trace":"org.elasticsearch.repositories.RepositoryVerificationException: [YOUR_BUCKET] path  is not accessible on master node
Caused by: com.amazonaws.SdkClientException: The full URI (http://169.254.170.23/v1/credentials) contained withing environment variable AWS_CONTAINER_CREDENTIALS_FULL_URI has an invalid host. Host can only be one of [localhost, 127.0.0.1]

Click to Expand Error Logs

[YOUR_BUCKET] path is not accessible on master node","error.stack_trace":"org.elasticsearch.repositories.RepositoryVerificationException: [YOUR_BUCKET] path is not accessible on master node Caused by: java.io.IOException: Exception when listing blobs by prefix [index-] \tat org.elasticsearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:356) \tat [email protected]/org.elasticsearch.repositories.blobstore.BlobStoreRepository.listBlobsToGetLatestIndexId(BlobStoreRepository.java:3099) \tat [email protected]/org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:3069) \tat [email protected]/org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:2148) \tat [email protected]/org.elasticsearch.repositories.RepositoriesService.lambda$validatePutRepositoryRequest$11(RepositoriesService.java:361) \tat [email protected]/org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:36) \tat [email protected]/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984) \tat [email protected]/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) \tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) \tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) \tat java.base/java.lang.Thread.run(Thread.java:1570) Caused by: com.amazonaws.SdkClientException: The full URI (http://169.254.170.23/v1/credentials) contained withing environment variable AWS_CONTAINER_CREDENTIALS_FULL_URI has an invalid host. Host can only be one of [localhost, 127.0.0.1] \tat com.amazonaws.auth.ContainerCredentialsProvider$FullUriCredentialsEndpointProvider.getCredentialsEndpoint(ContainerCredentialsProvider.java:121) \tat com.amazonaws.auth.ContainerCredentialsFetcher.getCredentialsResponse(ContainerCredentialsFetcher.java:38) \tat com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:124) \tat com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:80) \tat com.amazonaws.auth.ContainerCredentialsProvider.getCredentials(ContainerCredentialsProvider.java:71) \tat com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75) \tat java.base/java.security.AccessController.doPrivileged(AccessController.java:319) \tat org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31) \tat org.elasticsearch.repositories.s3.S3Service$PrivilegedAWSCredentialsProvider.getCredentials(S3Service.java:313) \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1269) \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:845) \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:794) \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755) \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) \tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) \tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) \tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) \tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456) \tat com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:6431) \tat com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:6404) \tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5441) \tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403) \tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5397) \tat com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:928) \tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeListing$15(S3BlobContainer.java:403) \tat java.base/java.security.AccessController.doPrivileged(AccessController.java:319) \tat org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31) \tat org.elasticsearch.repositories.s3.S3BlobContainer.executeListing(S3BlobContainer.java:403) \tat org.elasticsearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:347) \t... 10 more

[pebrc]

ECK is not setting the AWS_CONTAINER_CREDENTIALS_FULL_URI environment variable.

[jtele2]

ECK is not setting the AWS_CONTAINER_CREDENTIALS_FULL_URI environment variable.

Is it set in the base elastic image then? Why doesn't PIA work OOB with ECK managed elastic clusters?

[barkbay]

Is it set in the base elastic image then?

Or by a webhook: https://github.com/aws/amazon-eks-pod-identity-webhook/blob/90b4c5247a1c73a5508f209a792e1efa67f2bf07/pkg/handler/handler.go#L234-L243