CloudFront KeyValueStore client API fails with assumed role/STS credentials

[pitkley]

Describe the bug

Using the cloudfront-keyvaluestore client, trying to invoke the describe_key_value_store function when using an assumed role fails with an Authentication failed error. Using an access-key-ID and secret-access-key pair of the target-account works without issues. Both the role tested and the direct access key have the same full AdministratorAccess permissions.

Expected Behavior

Just like for all other AWS services used with boto3 (at least the ones I have ever used), assuming a role should work with the cloudfront-keyvaluestore client and operations like DescribeKeyValueStore should not fail.

Current Behavior

The describe_key_value_store operation fails with this error when using an assumed role:

Traceback (most recent call last):
  File "...", line 34, in <module>
    boto3.client("cloudfront-keyvaluestore").describe_key_value_store(KvsARN=kvs_arn)
  File ".../python3.12/site-packages/botocore/client.py", line 553, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../python3.12/site-packages/botocore/client.py", line 1009, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.AccessDeniedException: An error occurred (AccessDeniedException) when calling the DescribeKeyValueStore operation: Authentication failed

Reproduction Steps

The following is a minimal script that reproduces the error. It expects to be invoked with an environment variable ASSUMED_IAM_ROLE set to the ARN of the IAM role that should be assumed.

import os
import boto3

ASSUMED_IAM_ROLE = os.environ["ASSUMED_IAM_ROLE"]

session_base = boto3.Session()

assumed_role = session_base.client("sts").assume_role(
    RoleArn=ASSUMED_IAM_ROLE,
    RoleSessionName="cfkvs_403_reproducer",
)
print(f"++ Assumed role: {assumed_role['AssumedRoleUser']['Arn']}")
assumed_credentials = assumed_role["Credentials"]
session_assumed = boto3.Session(
    aws_access_key_id=assumed_credentials["AccessKeyId"],
    aws_secret_access_key=assumed_credentials["SecretAccessKey"],
    aws_session_token=assumed_credentials["SessionToken"],
)

# Get a KVS ARN via the automatically assumed client, showing that role-assumption works
kvs_arn = session_assumed.client("cloudfront").list_key_value_stores()["KeyValueStoreList"]["Items"][0]["ARN"]
print(f"++ Retrieved KVS ARN with assumed session: {kvs_arn}")

# Try to describe the KVS using the automatically assumed client, which fails
try:
    response = session_assumed.client("cloudfront-keyvaluestore").describe_key_value_store(KvsARN=kvs_arn)
    print(f"++ Describe KVS with assumed session: KvsARN={response['KvsARN']}, ETag={response['ETag']}")
except Exception as e:
    print("!! Assumed session: CloudFrontKeyValueStore.describe_key_value_store failed")
    raise e

The script will print output like this:

++ Assumed role: arn:aws:sts::123412341234:assumed-role/RoleName/cfkvs_403_reproducer
++ Retrieved KVS ARN with assumed session: arn:aws:cloudfront::123412341234:key-value-store/abcd1234-abcd-1234-abcd-1234abcd1234
!! Assumed session: CloudFrontKeyValueStore.describe_key_value_store failed
Traceback (most recent call last):
  File ".../reproducer.py", line 30, in <module>
    raise e
  File ".../reproducer.py", line 26, in <module>
    response = session_assumed.client("cloudfront-keyvaluestore").describe_key_value_store(KvsARN=kvs_arn)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../python3.12/site-packages/botocore/client.py", line 553, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../python3.12/site-packages/botocore/client.py", line 1009, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.AccessDeniedException: An error occurred (AccessDeniedException) when calling the DescribeKeyValueStore operation: Authentication failed

If it were to succeed, the output should look like the following. It has never succeeded in any of my tests.

++ Assumed role: arn:aws:sts::123412341234:assumed-role/RoleName/cfkvs_403_reproducer
++ Retrieved KVS ARN with assumed session: arn:aws:cloudfront::123412341234:key-value-store/abcd1234-abcd-1234-abcd-1234abcd1234
++ Describe KVS with assumed session: KvsARN=arn:aws:cloudfront::123412341234:key-value-store/abcd1234-abcd-1234-abcd-1234abcd1234, ETag=KV1A23B4C5DEF6G

Possible Solution

No response

Additional Information/Context

Interestingly enough, using the aws-cli (aws-cli/2.15.2 Python/3.11.6 Darwin/22.6.0 exe/x86_64 prompt/off) works without issues:

$ aws --profile <temporary-profile> cloudfront-keyvaluestore describe-key-value-store --kvs-arn arn:aws:cloudfront::123412341234:key-value-store/abcd1234-abcd-1234-abcd-1234abcd1234
{
    "ETag": "KV1A23B4C5DEF6G",
    "ItemCount": 30,
    "TotalSizeInBytes": 792,
    "KvsARN": "arn:aws:cloudfront::123412341234:key-value-store/abcd1234-abcd-1234-abcd-1234abcd1234",
    "Created": "...",
    "LastModified": "..."
}

Here, the temporary-profile profile is configured through the ~/.aws/credentials file after invoking aws sts assume-role --role-arn "$ASSUMED_IAM_ROLE" --role-session-name aws-cli, looking roughly like this:

[temporary-profile]
aws_access_key_id = AKIA...
aws_secret_access_key = ...
aws_session_token = ...

I have looked at debug-output of both boto3 (through boto3.set_stream_logger("botocore")) and the aws-cli (using the --debug CLI flag). One difference I did notice in the output (which might not be the only one, just one I did notice), is that the aws-cli will have output like this:

...
2023-12-21 17:19:09,635 - MainThread - botocore.hooks - DEBUG - Event request-created.cloudfront-keyvaluestore.DescribeKeyValueStore: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x1104eb4d0>>
2023-12-21 17:19:09,635 - MainThread - botocore.hooks - DEBUG - Event choose-signer.cloudfront-keyvaluestore.DescribeKeyValueStore: calling handler <function set_operation_specific_signer at 0x10e6087c0>
[INFO] [2023-12-21T16:19:09Z] [0000000202e7e280] [AuthCredentialsProvider] - (id=0x600000423b70) Static credentials provider successfully sourced credentials
[INFO] [2023-12-21T16:19:09Z] [0000000202e7e280] [AuthSigning] - (id=0x600002930230) Signing successfully built canonical request for algorithm SigV4Asymmetric, with contents
...
[INFO] [2023-12-21T16:19:09Z] [0000000202e7e280] [AuthSigning] - (id=0x600002930230) Signing successfully built string-to-sign via algorithm SigV4Asymmetric, with contents
...
[INFO] [2023-12-21T16:19:09Z] [0000000202e7e280] [AuthSigning] - (id=0x600002930230) Http request successfully built final authorization value via algorithm SigV4Asymmetric, with contents
...
2023-12-21 17:19:09,638 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest ...>

When running the reproducer-script above with debug output enabled, will have similar botocore.hooks log-statements, but showing none of the AuthCredentialsProvider or AuthSigning output. (Maybe this is expected, since it does not show for the CloudFront ListKeyValueStores API-call either.)

2023-12-21 17:21:41,894 botocore.hooks [DEBUG] Event request-created.cloudfront-keyvaluestore.DescribeKeyValueStore: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x107c935c0>>
2023-12-21 17:21:41,894 botocore.hooks [DEBUG] Event choose-signer.cloudfront-keyvaluestore.DescribeKeyValueStore: calling handler <function set_operation_specific_signer at 0x105c0e7a0>
2023-12-21 17:21:41,895 botocore.hooks [DEBUG] Event request-created.cloudfront-keyvaluestore.DescribeKeyValueStore: calling handler <function add_retry_headers at 0x105c30b80>
2023-12-21 17:21:41,895 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest ...>

SDK version used

boto3 1.34.4 (botocore 1.34.4)

Environment details (OS name and version, etc.)

macOS 13.6.3, Python 3.12.1