SSO Credentials Provider fails to load AWS credentials if the AWS IAM Identity Center access token associated with an SSO session has expired or is about to expire

[sjakthol]

Describe the bug

When using AWS IAM Identity Center authentication with the sso-session based configuration, the AWS SDK SSO Credential Provider fails to load AWS credentials if the AWS IAM Identity Center access token cached to disk requires a refresh.

Expected Behavior

AWS SDK SSO Credential Provider refreshes the AWS IAM Identity Center access token associated with the SSO session and uses the fresh access token to fetch AWS credentials from AWS IAM Identity Center.

Current Behavior

AWS SDK SSO Credential Provider fails to obtain a fresh AWS IAM Identity Center access token if the previous token requires refresh (has expired or is expiring within next 5 minutes).

Reproduction Steps

1. Create file called config with following content (fill in your AWS IAM Identity Center details):

[sso-session mysso]
sso_start_url = https://mysso.awsapps.com/start
sso_region = eu-west-1

[profile profile-1]
sso_session = mysso
sso_account_id = 000000000000
sso_role_name = MyRole

2. Login to AWS IAM Identity Center: aws sso login --session-name mysso
3. Update the expiresAt field value to a timestamp that is in the past in the token cache file Step 2 created under ~/.aws/sso/cache/ (you can also wait for an hour for the access token on disk to expire)
4. Create file called index.js with following content:

const AWS = require("aws-sdk");
const sts = new AWS.STS();
(async () => {console.log(await sts.getCallerIdentity().promise())})();

5. Execute following command:

env AWS_CONFIG_FILE=./config AWS_REGION=eu-west-1 AWS_PROFILE=profile-1 node index.js

This should fail with the following error which indicates SSO Credential Provider failed to resolve credentials and SDK fell through to latter providers in the chain (SSO credential provider errors are not logged):

Error [CredentialsError]: Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1
    at Timeout.connectTimeout [as _onTimeout] (/sdkbug/node_modules/aws-sdk/lib/http/node.js:69:15)
    at listOnTimeout (node:internal/timers:569:17)
    at process.processTimers (node:internal/timers:512:7) {
  code: 'CredentialsError',
  time: 2023-06-08T04:16:52.349Z,
  retryable: true,
  originalError: {
    message: 'Could not load credentials from any providers',
    code: 'CredentialsError',
    time: 2023-06-08T04:16:52.349Z,
    retryable: true,
    originalError: {
      message: 'EC2 Metadata roleName request returned error',
      code: 'TimeoutError',
      time: 2023-06-08T04:16:52.348Z,
      retryable: true,
      originalError: {
        message: 'Socket timed out without establishing a connection',
        code: 'TimeoutError',
        time: 2023-06-08T04:16:52.347Z,
        retryable: true
      }
    }
  }
}

Possible Solution

Here's the possible cause for this issue.

SsoCredentials.load() calls SsoCredentials.getToken():

aws-sdk-js/lib/credentials/sso_credentials.js

Line 114 in cae1321

this.getToken(this.profile, profile, function (err, token) {

SsoCredentials.getToken() creates a new SSOTokenProvider:

aws-sdk-js/lib/credentials/sso_credentials.js

Lines 176 to 178 in cae1321

	var ssoTokenProvider = new AWS.SSOTokenProvider({
	profile: profileName,
	});

SSOTokenProvider constructor invokes SSOTokenProvider.get() to load/refresh the token:

aws-sdk-js/lib/token/sso_token_provider.js

Line 99 in cae1321

this.get(options.callback || AWS.util.fn.noop);

Call to get() triggers a call to SSOTokenProvider.load(). This reads the the expired token from disk, notices it has expired and starts to refresh it in the background (async call). load() also sets the lastRefreshAttemptTime to current time:

aws-sdk-js/lib/token/sso_token_provider.js

Line 202 in cae1321

lastRefreshAttemptTime = AWS.util.date.getDate().getTime();

SSOTokenProvider constructor returns as load() made an async call. SsoCredentials.getToken() continues and calls SSOTokenProvider.load() (this is the second call to SSOTokenProvider.load()):

aws-sdk-js/lib/credentials/sso_credentials.js

Lines 179 to 184 in cae1321

	ssoTokenProvider.load(function (err) {
	if (err) {
	return callback(err);
	}
	return callback(null, ssoTokenProvider.token);
	});

SSOTokenProvider notices that the token on disk is still expired (refresh process is still running in the background). But when it checks the lastRefreshAttemptTime, it seems that a refresh was just started and it triggers the SsoCredentials callback without a valid token:

aws-sdk-js/lib/token/sso_token_provider.js

Lines 180 to 184 in cae1321

	// Skip new refresh, if last refresh was done within 30 seconds.
	if (currentTime - lastRefreshAttemptTime < 30 * 1000) {
	refreshUnsuccessful(currentTime, tokenExpireTime, callback);
	return;
	}

SsoCredentials.getToken() assumes a token is now available and returns that to SsoCredentials.load():

aws-sdk-js/lib/credentials/sso_credentials.js

Line 183 in cae1321

return callback(null, ssoTokenProvider.token);

However, since the refresh had not yet completed, ssoTokenProvider.token is undefined and SsoCredentials.load() is unable to fetch AWS credentials with it.

Because of this AWS SDK discards the SsoCredentials provider and moves forward to try other providers in the chain. During this time the initial AWS IAM Identity Center access token refresh might complete and a fresh access token might be written to disk. So the next time SsoCredentials provider tries to load AWS credentials, it might find a valid token from disk and the provider is able to fetch AWS credentials from AWS IAM Identity Center.

If I'm reading the code correctly, SsoCredentials.getToken() should call the SSOTokenProvider.get() instead of SSOTokenProvider.load(). The get method seems to fold multiple token load attempts into one load as per logic in:

aws-sdk-js/lib/token.js

Line 175 in cae1321

coalesceRefresh: function coalesceRefresh(callback, sync) {

Additional Information/Context

No response

SDK version used

2.1393.0

Environment details (OS name and version, etc.)

[spoco2]

Just to confirm that any fix for these issues will also fix the issue where it seems aws-cdk doesn't engage the token refreshing at all? Because aws/aws-cdk#24782 has been said to be tracked by the changes on ☝️. And so I hope that carries over to this too!

[sjakthol]

It's not entirely clear if this is the issue that causes the CDK issue you linked to. Currently CDK uses both SDK v2 and v3 internally and I'm not sure which SDK is used for the credential resolution over there.

However, it's very likely that the CDK problem is caused by either this or aws/aws-sdk-js-v3#4798 depending on which SDK version is used there. I guess we'll see that when fix to this and aws/aws-sdk-js-v3#4798 is merged and released, and CDK is updated to take newer version of SDKs into use.

[Ishmaello]

I'm still trying actions in maticnetwork/polygon-edge but nightly build #76 run fails in https://github.com/Ishmaello/polygo-edge/actions/runs/5779793298/job/15662572723 with response "Run aws-actions/configure-aws-credentials@v2
with:
audience: sts.amazonaws.com
Error: Input required and not supplied: aws-region
(node:1703) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.

Please migrate your code to use AWS SDK for JavaScript (v3).
For more information, check the migration guide at https://a.co/[7](https://github.com/Ishmaello/polygon-edge/actions/runs/5779793298/job/15662572723#step:3:8)PzMCcy
(Use node --trace-warnings ... to show where the warning was created"

[rangerthegood]

I believe this is also related, see: aws/aws-cdk#27265 (comment)

@marcusirgens found that removing the sso_session from his ~/.aws/config file fixed his issues. I was able to verify that removing sso_session also fixed my cdk cli issues.

Here is a diff I made which triggers the bug. The inclusion of sso_session does not break using the aws cli, or boto3 session using the same sso profile I've configured. just cdk, which uses aws-sdk-js.

master...rangerthegood:aws-sdk-js:sso_bug

[arturenault]

Hi, what's holding back this issue's PR from getting merged?

My team has to log in every hour despite my session length being 12 hours. This is quite frustrating

[kstromeiraos]

Hi, could you get this issue's PR in? We are also being affected by this issue and having to log in every hour is a pain.

[ConnorHeggie]

+1 ^

[Ishmaello]

I hope my aws account has been deleted I'll start afresh