Unable to Use amazon-ecr-credential-helper with OIDC Token #581

lxbrd · 2023-08-08T15:03:58Z

Issue Description:
I am encountering an issue while attempting to use the amazon-ecr-credential-helper in combination with an OIDC token for authentication. The goal is to push a Docker image to an Amazon ECR registry using Kaniko within a specific context. However, the process is failing with authentication errors.

Steps to Reproduce:

Create the necessary directory structure and files:

$ mkdir -p /kaniko/.aws
$ echo "${MY_OIDC_TOKEN}" > /kaniko/web_identity_token
$ echo -e "[default]\nrole_arn=${AWS_ROLE_ARN}\nweb_identity_token_file=/kaniko/web_identity_token" > /kaniko/.aws/config
$ mkdir -p /kaniko/.docker
$ echo "{\"credsStore\":\"ecr-login\"}" > /kaniko/.docker/config.json

Execute Kaniko:

$ /kaniko/executor --context "${CI_PROJECT_DIR}" --dockerfile "${CI_PROJECT_DIR}/Dockerfile" --destination "${ECR_REGISTRY}:${DOCKER_IMAGE_TAG}"

Expected Behavior:
I expected the Kaniko process to authenticate successfully using the amazon-ecr-credential-helper with the provided OIDC token, and for the Docker image to be pushed to the specified Amazon ECR registry.

Actual Behavior:
The Kaniko process is failing with the following error messages:

SDK 2023/08/08 14:47:55 WARN falling back to IMDSv1: operation error ec2imds: getToken, http response error StatusCode: 405, request to EC2 IMDS failed
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "000000000.dkr.ecr.us-east-1.amazonaws.com/test:kaniko-test-455a9c73": POST https://000000000.dkr.ecr.us-east-1.amazonaws.com/v2/test/blobs/uploads/: unexpected status code 401 Unauthorized: Not Authorized

Additional Information:

AWS IAM

        {
            "Action": [
                "ecr:UploadLayerPart",
                "ecr:PutImage",
                "ecr:InitiateLayerUpload",
                "ecr:GetDownloadUrlForLayer",
                "ecr:CompleteLayerUpload",
                "ecr:BatchGetImage",
                "ecr:BatchCheckLayerAvailability"
            ],
            "Effect": "Allow",
            "Resource": "*",
            "Sid": "ECR"
        },
        {
            "Action": "ecr:GetAuthorizationToken",
            "Effect": "Allow",
            "Resource": "*",
            "Sid": "ECRGetAuthorizationToken"
        }

The provided OIDC token and Amazon ECR registry details are correct.
The amazon-ecr-credential-helper is expected to handle the authentication for ECR registry access.
The error messages suggest that there might be an issue with the authentication process or IAM permissions.
The use of IMDSv1 is also causing warnings, which might be related to the authentication failure.

Environment:

Kaniko Version: gcr.io/kaniko-project/executor:debug

Desired Solution:
I would appreciate assistance in resolving this issue. Specifically, I am looking for guidance on how to properly configure and use the amazon-ecr-credential-helper with an OIDC token to authenticate the Kaniko process for pushing Docker images to an Amazon ECR registry.

Thank you for your help!

The text was updated successfully, but these errors were encountered:

pauldthomson · 2023-08-10T07:13:28Z

(Just a drive-by observer, but) presumably you'd need to do an sts.assumeRoleWithWebIdentity i.e.:

aws sts assume-role-with-web-identity --role-arn <> --role-session-name <> --web-identity-token <>

first then then cred helper would work? And you'd need to set up your IAM role accordingly

lxbrd · 2023-08-14T08:29:41Z

Hey @pauldthomson, thank you for your response! Unfortunately, the image I'm using (kaniko) doesn't provide the option to install awscli. You could create a dedicated imag:

FROM alpine
RUN apk add --no-cache jq curl python3 py3-pip gettext libintl bash && pip install awscli
COPY --from=gcr.io/kaniko-project/executor:debug /kaniko/executor /kaniko/executor

But I was looking for that functionality in amazon-ecr-credential-helper, not in awscli.
Otherwise, that should be a viable workaround.

pauldthomson · 2023-08-14T08:45:37Z

Of course, I didn't think of the context of where Kaniko runs 🤦‍♂️

Are you running it in EKS? That would at least open the door to IRSA

lxbrd · 2023-08-14T08:48:23Z

Sadly I'm not running it in EKS. It's a Gitlab runner.

pauldthomson · 2023-08-14T09:01:38Z

Never mind me then 😛

kyleplant · 2023-11-09T23:13:51Z

You can use wget to call assume web identity. Although I found it was broken so patched it with an older version. You can use the example below in your gitlab jobs by using extend: .aws_login. This then exposes credentials via the standard ENV variables: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN

.aws_login:
 id_tokens:
   MY_OIDC_TOKEN:
     aud: https://gitlab.com
 before_script:
   - |
     echo "Script here"

The script to just get kaniko working...

#!/bin/sh

if [ -n "$ROLE_ARN" ]; then
    echo "Logging in with $ROLE_ARN"
else
    echo "Please set ROLE_ARN in CICD variables"
    exit 1
fi

# Regional endpoints eg. "https://sts.ap-southeast-2.amazonaws.com/"
sts_endpoint="https://sts.amazonaws.com/"

session_name="GitLabRunner-${CI_PROJECT_ID}-${CI_PIPELINE_ID}"
duration_seconds="3600"

action="Action=AssumeRoleWithWebIdentity"
duration="DurationSeconds=${duration_seconds}"
role_arn="RoleArn=${ROLE_ARN}"
role_session="RoleSessionName=${session_name}"
web_identity="WebIdentityToken=${MY_OIDC_TOKEN}"
version="Version=2011-06-15"

params="${action}&${duration}&${role_arn}&${role_session}&${web_identity}&${version}"
content_type="Content-Type: application/x-www-form-urlencoded"

# Known bug https://github.com/GoogleContainerTools/kaniko/pull/2765
# Download version of wget that works with sts endpoint
wget -q --no-check-certificate https://busybox.net/downloads/binaries/1.35.0-x86_64-linux-musl/busybox_WGET -O busybox_WGET
mv busybox_WGET /busybox/wget
chmod +x /busybox/wget
response=$(wget -q --no-check-certificate --header="$content_type" --post-data="$params" -O - "$sts_endpoint")

AWS_ACCESS_KEY_ID=$(echo "$response" | sed -n 's/.*<AccessKeyId>\(.*\)<\/AccessKeyId>.*/\1/p')
AWS_SECRET_ACCESS_KEY=$(echo "$response" | sed -n 's/.*<SecretAccessKey>\(.*\)<\/SecretAccessKey>.*/\1/p')
AWS_SESSION_TOKEN=$(echo "$response" | sed -n 's/.*<SessionToken>\(.*\)<\/SessionToken>.*/\1/p')

export AWS_ACCESS_KEY_ID
export AWS_SECRET_ACCESS_KEY
export AWS_SESSION_TOKEN

Alternatively, if you want multi container support, you could check for aws-cli, curl then wget.

#!/bin/sh

if [ -n "$ROLE_ARN" ]; then
    echo "Logging in with $ROLE_ARN"
else
    echo "Please set ROLE_ARN in CICD variables"
    exit 1
fi

# Regional endpoints eg. "https://sts.ap-southeast-2.amazonaws.com/"
sts_endpoint="https://sts.amazonaws.com/"

session_name="GitLabRunner-${CI_PROJECT_ID}-${CI_PIPELINE_ID}"
duration_seconds="3600"

action="Action=AssumeRoleWithWebIdentity"
duration="DurationSeconds=${duration_seconds}"
role_arn="RoleArn=${ROLE_ARN}"
role_session="RoleSessionName=${session_name}"
web_identity="WebIdentityToken=${MY_OIDC_TOKEN}"
version="Version=2011-06-15"

params="${action}&${duration}&${role_arn}&${role_session}&${web_identity}&${version}"
content_type="Content-Type: application/x-www-form-urlencoded"

if
    command -v aws >/dev/null 2>&1
then

    response=$(
        aws sts assume-role-with-web-identity \
            --role-arn "${ROLE_ARN}" \
            --role-session-name "${session_name}" \
            --web-identity-token "${MY_OIDC_TOKEN}" \
            --duration-seconds "${duration_seconds}" \
            --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' \
            --output text
    )

    # Set positional parameters
    # shellcheck disable=SC2086
    set -- $response

    # Assign the credentials to environment variables
    AWS_ACCESS_KEY_ID=$1
    AWS_SECRET_ACCESS_KEY=$2
    AWS_SESSION_TOKEN=$3

elif
    command -v curl >/dev/null 2>&1
then

    response=$(curl -s -X POST "$sts_endpoint" -H "$content_type" -d "$params")

    AWS_ACCESS_KEY_ID=$(echo "$response" | sed -n 's/.*<AccessKeyId>\(.*\)<\/AccessKeyId>.*/\1/p')
    AWS_SECRET_ACCESS_KEY=$(echo "$response" | sed -n 's/.*<SecretAccessKey>\(.*\)<\/SecretAccessKey>.*/\1/p')
    AWS_SESSION_TOKEN=$(echo "$response" | sed -n 's/.*<SessionToken>\(.*\)<\/SessionToken>.*/\1/p')

elif
    command -v wget >/dev/null 2>&1
then
    wget_path=$(which wget)

    if [ "$wget_path" = "/busybox/wget" ]; then
        # Known bug https://github.com/GoogleContainerTools/kaniko/pull/2765
        wget -q --no-check-certificate https://busybox.net/downloads/binaries/1.35.0-x86_64-linux-musl/busybox_WGET -O busybox_WGET
        mv busybox_WGET /busybox/wget
        chmod +x /busybox/wget
        response=$(wget -q --no-check-certificate --header="$content_type" --post-data="$params" -O - "$sts_endpoint")
    else
        response=$(
            wget --quiet --method=POST --header="$content_type" --body-data="$params" --output-document - "$sts_endpoint"
        )
    fi

    AWS_ACCESS_KEY_ID=$(echo "$response" | sed -n 's/.*<AccessKeyId>\(.*\)<\/AccessKeyId>.*/\1/p')
    AWS_SECRET_ACCESS_KEY=$(echo "$response" | sed -n 's/.*<SecretAccessKey>\(.*\)<\/SecretAccessKey>.*/\1/p')
    AWS_SESSION_TOKEN=$(echo "$response" | sed -n 's/.*<SessionToken>\(.*\)<\/SessionToken>.*/\1/p')

    echo "AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID"

else
    echo "Neither AWS CLI, curl, nor wget is available to call STS assume role. 
          Ensure your pipeline image contains one of these binaries."
    exit 1
fi

export AWS_ACCESS_KEY_ID
export AWS_SECRET_ACCESS_KEY
export AWS_SESSION_TOKEN

troyswanson · 2023-11-14T16:15:58Z

I have solved this in GitLab using environment variables. The credential helper is able to automatically assume a role given an OIDC token and a role ARN.

Example job in GitLab:

deploy_to_ecr:
  stage: deploy
  image:
    name: gcr.io/kaniko-project/executor:v1.16.0-debug
    entrypoint: [""]
  id_tokens:
    GITLAB_OIDC_TOKEN:
      aud: https://gitlab.com
  variables:
    AWS_WEB_IDENTITY_TOKEN_FILE: /kaniko/gitlab-oidc-token
    AWS_EC2_METADATA_DISABLED: "true"
    AWS_SDK_LOAD_CONFIG: "true"
  before_script:
    - mkdir -p /kaniko/.docker
    - echo "{\"credsStore\":\"ecr-login\"}" > /kaniko/.docker/config.json
    - echo $GITLAB_OIDC_TOKEN > $AWS_WEB_IDENTITY_TOKEN_FILE
  script:
    - /kaniko/executor
      --context "${CI_PROJECT_DIR}"
      --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
      --target "lambda-ctx"
      --destination "${ECR_REPO_URI}:${CI_COMMIT_SHORT_SHA}"
      --skip-unused-stages=true
      --snapshot-mode=redo
  after_script:
    - cat ~/.ecr/log/ecr-login.log

The important part of this setup involves saving the value of the GITLAB_OIDC_TOKEN environment variable to the path that is defined in the AWS_WEB_IDENTITY_TOKEN_FILE environment variable. Additionally, you need to set the AWS_ROLE_ARN environment variable somewhere (I used a variable via the CI/CD Settings for that one).

I would be curious to know if this works for anyone else, so please respond one way or another. The hoops that a lot of folks seem to be jumping through to make this work seem arduous, and hopefully with this solution it will be cleaner.

DaniWS · 2024-04-02T03:38:09Z

I have solved this in GitLab using environment variables. The credential helper is able to automatically assume a role given an OIDC token and a role ARN.

Example job in GitLab:
deploy_to_ecr:
  stage: deploy
  image:
    name: gcr.io/kaniko-project/executor:v1.16.0-debug
    entrypoint: [""]
  id_tokens:
    GITLAB_OIDC_TOKEN:
      aud: https://gitlab.com
  variables:
    AWS_WEB_IDENTITY_TOKEN_FILE: /kaniko/gitlab-oidc-token
    AWS_EC2_METADATA_DISABLED: "true"
    AWS_SDK_LOAD_CONFIG: "true"
  before_script:
    - mkdir -p /kaniko/.docker
    - echo "{\"credsStore\":\"ecr-login\"}" > /kaniko/.docker/config.json
    - echo $GITLAB_OIDC_TOKEN > $AWS_WEB_IDENTITY_TOKEN_FILE
  script:
    - /kaniko/executor
      --context "${CI_PROJECT_DIR}"
      --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
      --target "lambda-ctx"
      --destination "${ECR_REPO_URI}:${CI_COMMIT_SHORT_SHA}"
      --skip-unused-stages=true
      --snapshot-mode=redo
  after_script:
    - cat ~/.ecr/log/ecr-login.log
The important part of this setup involves saving the value of the GITLAB_OIDC_TOKEN environment variable to the path that is defined in the AWS_WEB_IDENTITY_TOKEN_FILE environment variable. Additionally, you need to set the AWS_ROLE_ARN environment variable somewhere (I used a variable via the CI/CD Settings for that one).

I would be curious to know if this works for anyone else, so please respond one way or another. The hoops that a lot of folks seem to be jumping through to make this work seem arduous, and hopefully with this solution it will be cleaner.

After busting my head for a whole day following the docs, using this EXACT configuration (same variable names, same paths) solved it for me.

Thanks for sharing! <3 !

troyswanson · 2024-04-18T14:25:39Z

After busting my head for a whole day following the docs, using this EXACT configuration (same variable names, same paths) solved it for me.

Thanks for sharing! <3 !

@DaniWS I'm so glad it worked for you! Happy to help ❤️

moosilauke18-reify mentioned this issue Apr 16, 2024

no basic auth credentials on push_image step CircleCI-Public/aws-ecr-orb#335

Closed

plaffitt mentioned this issue May 7, 2024

Error when pulling image from private ECR enix/kube-image-keeper#303

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to Use amazon-ecr-credential-helper with OIDC Token #581

Unable to Use amazon-ecr-credential-helper with OIDC Token #581

lxbrd commented Aug 8, 2023

pauldthomson commented Aug 10, 2023

lxbrd commented Aug 14, 2023

pauldthomson commented Aug 14, 2023

lxbrd commented Aug 14, 2023

pauldthomson commented Aug 14, 2023

kyleplant commented Nov 9, 2023

troyswanson commented Nov 14, 2023

DaniWS commented Apr 2, 2024

troyswanson commented Apr 18, 2024 •

edited

Loading

Unable to Use amazon-ecr-credential-helper with OIDC Token #581

Unable to Use amazon-ecr-credential-helper with OIDC Token #581

Comments

lxbrd commented Aug 8, 2023

pauldthomson commented Aug 10, 2023

lxbrd commented Aug 14, 2023

pauldthomson commented Aug 14, 2023

lxbrd commented Aug 14, 2023

pauldthomson commented Aug 14, 2023

kyleplant commented Nov 9, 2023

troyswanson commented Nov 14, 2023

DaniWS commented Apr 2, 2024

troyswanson commented Apr 18, 2024 • edited Loading

troyswanson commented Apr 18, 2024 •

edited

Loading