Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retry behavior causes failure to take too long with invalid parameters, without ability to configure retry behavior #683

Closed
retzero opened this issue Mar 4, 2023 · 9 comments
Labels
effort/small This issue will take less than a day of effort to fix feature-request A feature should be added or improved. p2

Comments

@retzero
Copy link

retzero commented Mar 4, 2023

Describe the bug

Hello.
It takes too long time to fail in case of invalid parameters.
Took 1 minute 20 seconds to fail...
Wrong audience also have the same issue.

How to configure to fail immediately?

Expected Behavior

It should fail immediately.

Current Behavior

Took 1 minute 20 seconds to fail.

Sat, 04 Mar 2023 12:03:55 GMT ##[debug]Evaluating condition for step: 'Authenticate to AWS'
Sat, 04 Mar 2023 12:03:55 GMT##[debug]Evaluating: success()
Sat, 04 Mar 2023 12:03:55 GMT ##[debug]Evaluating success:
Sat, 04 Mar 2023 12:03:55 GMT ##[debug]=> true
Sat, 04 Mar 2023 12:03:55 GMT ##[debug]Result: true
Sat, 04 Mar 2023 12:03:55 GMT ##[debug]Starting: Authenticate to AWS
Sat, 04 Mar 2023 12:03:55 GMT ##[debug]Register post job cleanup for action: aws-actions/configure-aws-credentials@v1
Sat, 04 Mar 2023 12:03:55 GMT ##[debug]Loading inputs
Sat, 04 Mar 2023 12:03:55 GMT ##[debug]Evaluating: secrets.AWS_ROLE_TO_ASSUME
Sat, 04 Mar 2023 12:03:55 GMT ##[debug]Evaluating Index:
Sat, 04 Mar 2023 12:03:55 GMT ##[debug]..Evaluating secrets:
Sat, 04 Mar 2023 12:03:55 GMT ##[debug]..=> Object
Sat, 04 Mar 2023 12:03:55 GMT ##[debug]..Evaluating String:
Sat, 04 Mar 2023 12:03:55 GMT ##[debug]..=> 'AWS_ROLE_TO_ASSUME'
Sat, 04 Mar 2023 12:03:55 GMT ##[debug]=> '***'
Sat, 04 Mar 2023 12:03:55 GMT ##[debug]Result: '***'
Sat, 04 Mar 2023 12:03:55 GMT ##[debug]Loading env
Sat, 04 Mar 2023 12:03:55 GMT Run aws-actions/configure-aws-credentials@v1
Sat, 04 Mar 2023 12:03:55 GMT   with:
Sat, 04 Mar 2023 12:03:55 GMT     aws-region: ap-northeast-2
Sat, 04 Mar 2023 12:03:55 GMT     role-to-assume: ***
Sat, 04 Mar 2023 12:03:55 GMT     audience: sts.amazonaws.com
Sat, 04 Mar 2023 12:03:55 GMT     role-duration-seconds: 900
Sat, 04 Mar 2023 12:03:55 GMT ##[debug]ID token url is https://pipelines.actions.githubusercontent.com/rlDZnPtejP0IgUinrtutIZjnLWJ1A[6](https://github.com/retzero/oidc-test/actions/runs/4330883414/jobs/7562381435#step:2:6)g5kR2DtXy3FzTnYSWfEP/00000000-0000-0000-0000-000000000000/_apis/distributedtask/hubs/Actions/plans/6d05abca-e7a9-4546-814f-a2b2eff543ba/jobs/b9acd266-12b9-52e4-499f-13f4a944bdac/idtoken?api-version=2.0&audience=sts.amazonaws.com
Sat, 04 Mar 2023 12:03:55 GMT ::add-mask::***
Sat, 04 Mar 2023 12:03:55 GMT ##[debug]7 role session tags are being used.
Sat, 04 Mar 2023 12:03:56 GMT ##[debug]7 role session tags are being used.
Sat, 04 Mar 2023 12:03:57 GMT ##[debug]7 role session tags are being used.
Sat, 04 Mar 2023 12:03:58 GMT ##[debug]7 role session tags are being used.
Sat, 04 Mar 2023 12:04:00 GMT ##[debug]7 role session tags are being used.
Sat, 04 Mar 2023 12:04:02 GMT ##[debug]7 role session tags are being used.
Sat, 04 Mar 2023 12:04:05 GMT ##[debug]7 role session tags are being used.
Sat, 04 Mar 2023 12:04:09 GMT ##[debug]7 role session tags are being used.
Sat, 04 Mar 2023 12:04:11 GMT ##[debug]7 role session tags are being used.
Sat, 04 Mar 2023 12:04:33 GMT ##[debug]7 role session tags are being used.
Sat, 04 Mar 2023 12:04:38 GMT ##[debug]7 role session tags are being used.
Sat, 04 Mar 2023 12:05:15 GMT Error: No OpenIDConnect provider found in your account for https://token.actions.githubusercontent.com
Sat, 04 Mar 2023 12:05:15 GMT ##[debug]Node Action run completed with exit code 1
Sat, 04 Mar 2023 12:05:15 GMT ##[debug]AWS_DEFAULT_REGION='ap-northeast-2'
Sat, 04 Mar 2023 12:05:15 GMT ##[debug]AWS_REGION='ap-northeast-2'
Sat, 04 Mar 2023 12:05:15 GMT ##[debug]Finishing: Authenticate to AWS

Reproduction Steps

Ex) Workflow logs in no provider is configured: https://github.com/retzero/oidc-test/actions/runs/4330883414/jobs/7562381435

Possible Solution

No response

Additional Information/Context

No response

@retzero retzero added bug Something isn't working needs-triage This issue still needs to be triaged labels Mar 4, 2023
@razor54
Copy link

razor54 commented Mar 7, 2023

facing the same issue. while it takes a long time to fail, it doesn't state anything other than Error: Not authorized to perform sts:AssumeRoleWithWebIdentity on my case. no debug info that is useful or any other helpful messages are displayed

@peterwoodworth
Copy link
Contributor

We implemented an exponential backoff method because multiple workflows attempting to assume a role at the same time (at least with OIDC) can result in errors. We currently don't have a way to opt out of, or modify this behavior in any way.

The approach the original PR took to solve this was to implement a blanket exponential backoff method regardless of authentication method. Given this was only reported for OIDC, and that it throws a certain error in those cases, there seems to be potential for optimizing when we use a retry or not, and for how long when we do. However, this could be risky and result in side effects

Would people want this exponential backoff behavior to be configurable? If so, in what ways? The easiest thing to do would be to allow for an input which configures the isRetryable prop - Does anyone think there is merit to being able to configure any other behavior related to this method?

Function defined here

const retryAndBackoff = async (fn, isRetryable, retries = 0, maxRetries = 12, base = 50) => {

Called here

const roleCredentials = await retryAndBackoff(

@peterwoodworth peterwoodworth added feature-request A feature should be added or improved. p2 effort/small This issue will take less than a day of effort to fix and removed bug Something isn't working needs-triage This issue still needs to be triaged labels Mar 21, 2023
@peterwoodworth peterwoodworth changed the title The action takes long time to fail Retry behavior causes failure to take too long with invalid parameters Mar 21, 2023
@peterwoodworth peterwoodworth changed the title Retry behavior causes failure to take too long with invalid parameters Retry behavior causes failure to take too long with invalid parameters, without ability to configure retry behavior Mar 21, 2023
@pauldraper
Copy link

pauldraper commented Mar 27, 2023

If so, in what ways?

Allow max retries to be configurable. If you try something and it fails the first time, and the second time, and the third time, and the forth time, and the fifth time, and the sixth time, and the seventh time, and the eighth time....etc you might want to reconsider what you're doing with your short time on this earth.

@sjoshi-jpl
Copy link

sjoshi-jpl commented May 26, 2023

facing the same issue. while it takes a long time to fail, it doesn't state anything other than Error: Not authorized to perform sts:AssumeRoleWithWebIdentity on my case. no debug info that is useful or any other helpful messages are displayed

+1 having this exact same issue.

I also noticed that this might have something to do with aws-region. When I passed it as a secret ${{ secrets.AWS_REGION }} it didn't work, but when I hardcoded the value it worked.

Alternatively, after a while with the hardcoded value it started failing and when I reverted back to pass aws-region as a secret it worked fine.

aws-region passed as a secret :

    - name: Configure AWS Credentials
      uses: aws-actions/configure-aws-credentials@v2
      with:
        aws-region: ${{ secrets.AWS_REGION }}
        role-to-assume: ${{ secrets.AWS_ASSUME_ROLE }}

aws-region value hardcoded :

    - name: Configure AWS Credentials
      uses: aws-actions/configure-aws-credentials@v2
      with:
        aws-region: us-east-1
        role-to-assume: ${{ secrets.AWS_ASSUME_ROLE }}

@sjoshi-jpl
Copy link

sjoshi-jpl commented Jun 22, 2023

Any update on this issue? I am consistently getting this failure msg now. I've tried experimenting with different versions of aws-actions/configure-aws-credentials but it hasn't helped.

Workflow setup :

    - name: Configure AWS Credentials
      uses: aws-actions/configure-aws-credentials@v2
      with:
        aws-region: "${{ secrets.AWS_REGION }}"
        role-to-assume: ${{ secrets.AWS_ASSUME_ROLE }}
        role-session-name: GitHub_OIDC_TERRAFORM

Trust Policy configuration :

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::[ACCOUNT-NUMBER]:oidc-provider/token.actions.githubusercontent.com"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
                },
                "StringLike": {
                    "token.actions.githubusercontent.com:sub": [
                        "repo:<username>/<repo-name>:ref:refs/heads/*",
                        "repo:<username>/<repo-name>:ref:refs/heads/*"
                    ]
                }
            }
        }
    ]
}

Note: I am using this role to authenticate multiple repos.

Error:

Error: Not authorized to perform sts:AssumeRoleWithWebIdentity

cloud303-cholden added a commit to cloud303-cholden/configure-aws-credentials that referenced this issue Jul 22, 2023
cloud303-cholden added a commit to cloud303-cholden/configure-aws-credentials that referenced this issue Jul 22, 2023
@cloud303-cholden
Copy link

I made this option configurable. Hopefully #777 gets merged soon. In the meantime, feel free to test this with cloud303-cholden/configure-aws-credentials@main. Instructions are in the PR.

@cloud303-cholden
Copy link

cloud303-cholden commented Jul 22, 2023

A couple things I forgot to mention. I changed as little as I had to to implement this functionality, so take note that max-retries: 1 actually means to try to assume the role once, not to retry once on failure. Maybe it would be better to change the configuration label to max-attempts and to throw an error if max-attempts <= retries. Also, I'm guessing most people would want the default to be 1 and for exponentially increasing their action duration to be an opt-in feature. Let me know if this is reasonable and I can make the change.

@peterwoodworth
Copy link
Contributor

This is now configurable in v3 https://github.com/aws-actions/configure-aws-credentials#adjust-the-retry-mechanism

@github-actions
Copy link

** Note **
Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
effort/small This issue will take less than a day of effort to fix feature-request A feature should be added or improved. p2
Projects
None yet
Development

No branches or pull requests

6 participants