Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aws rds generate-db-auth-token generating invalid token when using .aws/credentials to authenticate #8234

Closed
scott-vandevoorde opened this issue Oct 10, 2023 · 5 comments
Assignees
Labels
closing-soon This issue will automatically close in 4 days unless further comments are made. p3 This is a minor priority issue rds

Comments

@scott-vandevoorde
Copy link

Describe the bug

When using credentials files (.aws/credentials & .aws/config) to authenticate with aws, aws rds generate-db-auth-token generates an invalid token.

The invalid token is missing the field X-Amz-Security-Token

If I set environment variables AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN, the token is generated successfully and is usable.

Expected Behavior

I expect the cli to generate a working token

Current Behavior

an invalid token is generated. The db says password authentication failed for user "rdsproxyuser"

Reproduction Steps

  1. create an iam user
  2. add unrestricted access to the user { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "*", "Resource": "*" } ] }
  3. run aws configure
  4. enter your access key, secret acccess key and region
  5. test your cli access to aws. For example, run aws s3 ls and ensure it connects ok
  6. stand up an aurora serverless v2 postgres RDS instance
  7. stand up an rds proxy and configure it for IAM authentication
  8. verify you can connect and login to the RDS proxy
  9. on the command line run aws rds generate-db-auth-token --hostname adhoc3-amber-engine.proxy-cfds9ixmayu4.us-east-1.rds.amazonaws.com --port 5432 --region us-east-1 --username rdsproxyuser (replace rds host name, and user name with the appropriate info)
  10. copy the generated token into an RDS connection and it will fail
  11. note that the token does not containt the field X-Amz-Security-Token
  12. Create a role with the following policy: { "Action": [ "rds-db:connect" ], "Resource": "arn:aws:rds-db:us-east-1:*:dbuser:*/*", "Effect": "Allow" }
  13. Update the trust policy so you can assume this role: { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::793478358630:user/scott" }, "Action": "sts:AssumeRole" }
  14. run the following command on the command line to assume the role and set your environment variablesexport $(printf "AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s AWS_SESSION_TOKEN=%s" \ $(aws sts assume-role \ --role-arn arn:aws:iam::793478358630:role/Adhoc3BastionStack-BastionHostRole-JDL6Y8KJ69VB \ --role-session-name Adhoc3BastionSession \ --query "Credentials.[AccessKeyId,SecretAccessKey,SessionToken]" \ --output text))
  15. run generate-db-auth-token again: aws rds generate-db-auth-token --hostname adhoc3-amber-engine.proxy-cfds9ixmayu4.us-east-1.rds.amazonaws.com --port 5432 --region us-east-1 --username rdsproxyuser
  16. copy the generated token and paste it to your rds connection.
  17. this time the token will work.
  18. Note that the token now contains X-Amz-Security-Token.

Possible Solution

trace the cli workflows between the two authentication mechanisms and figure out why the X-Amz-Security-Token is not being included in the token. Also recommend generating an error message instead of a bad tokens.

Additional Information/Context

Two co-workers validated the same behavior. One on windows and one on mac.

CLI version used

aws-cli/2.10.3 Python/3.9.11 Windows/10 exe/AMD64 prompt/off

Environment details (OS name and version, etc.)

tested on windows 10

@scott-vandevoorde scott-vandevoorde added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Oct 10, 2023
@aBurmeseDev aBurmeseDev self-assigned this Oct 11, 2023
@aBurmeseDev
Copy link
Member

Hi @scott-vandevoorde - thanks for reaching out and for well-detailed post.

I followed steps to reproduce the issue but was unable to and I'd like verify a couple things:

  • on step 7, you mentioned to stand up RDS proxy and configure it for IAM auth. Could you share how your IAM auth setup is like? Here's how I enabled during my setup: (reference guide)
aws rds create-db-instance \
    --db-instance-identifier mydbinstance \
    --db-instance-class db.m3.medium \
    --engine MySQL \
    --allocated-storage 20 \
    --master-username masterawsuser \
    --manage-master-user-password \
    --enable-iam-database-authentication
  • Step 12 and 13 is also mentioned here in these docs: here and here
  • I'd also like to clarify that when you start RDS connection with generated token, how are you using the token to connect? Here's how I generate the token and connect:
#  save the token to an environment variable, and use that variable to connect
export RDSHOST="mypostgres-cluster.cluster-abcdefg222hq.us-east-1.rds.amazonaws.com"

export PGPASSWORD="$(aws rds generate-db-auth-token \
--hostname $RDSHOST \
--port 5432 \
--region us-east-1 \
--username mydbuser)"

psql "host=$RDSHOST port=5432 sslmode=verify-full sslrootcert=/sample_dir/rds-combined-ca-bundle.pem dbname=dbName user= mydbuser"
  • I came across similar issue where a user commented the culprit was "found the issue it was due to an empty extra aws_session_token = and the end of ~/.aws/credentials" and another user suggested a "workaround" here.
  • Found another similar issue where one of our members suggested running aws sts get-caller-identity before token generation could help.

If the issue persists, could you share full debug logs from generate-db-auth-token command by adding --debug to the command, as that would give us more insight into this behavior? Please make sure to redact any sensitive info before sharing.

Hope that helps,
John

@aBurmeseDev aBurmeseDev added rds response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. p3 This is a minor priority issue and removed bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Oct 11, 2023
@scott-vandevoorde
Copy link
Author

Hi @aBurmeseDev , thanks for looking into this issue!

I think the main difference in our setups is I'm connecting to an IAM enabled RDS proxy. My RDS instance doesn't have IAM configured as the proxy handles it.

Here's more detail around my setup. Our instances are complex, so I stood up a slimmed down stack and I was able to reproduce with this configuration:

  1. Create a db instance. Update the db-subnet-group-name to choose which VPC you provision into.
aws rds create-db-instance \
    --db-instance-identifier temp-db \
    --db-instance-class db.m5d.large \
    --engine postgres \
    --allocated-storage 100 \
    --master-username masterawsuser \
    --manage-master-user-password \
	--db-subnet-group-name adhoc3-amber-engine-rds-adhoc3rdssubnets93ef7f60-zkd5nnppgsot
  1. Create a policy to allow RDS proxy to fetch secrets from secrets manager:
aws iam create-policy \
--policy-name my-rds-proxy-sample-policy \
--policy-document '{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"secretsmanager:GetResourcePolicy",
"secretsmanager:GetSecretValue",
"secretsmanager:DescribeSecret",
"secretsmanager:ListSecretVersionIds"
],
"Resource": [
"*"
]
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": [
"secretsmanager:GetRandomPassword",
"secretsmanager:ListSecrets"
],
"Resource": "*"
}
]
}'
  1. Create a role to allow the rds proxy to assume role:
aws iam create-role --role-name my-rds-proxy-sample-role --assume-role-policy-document '{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "rds.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}'
  1. attach the policy to the role: (update your AWS account # in the arn)
aws iam attach-role-policy --role-name my-rds-proxy-sample-role --policy-arn arn:aws:iam::123456789:policy/my-rds-proxy-sample-policy
  1. Create the RDS proxy. (update the secret arn to your RDS secret, your role arn and vpc subnet ids)
aws rds create-db-proxy \
--db-proxy-name temp-db-proxy \
--engine-family POSTGRESQL \
--auth '{
"AuthScheme": "SECRETS",
"SecretArn": "arn:aws:secretsmanager:us-east-1:123456789:secret:rds!db-d7bb886b-b548-4d21-b15d-694d34c3c9b1-MbGtmT",
"IAMAuth": "REQUIRED"
}' \
--role-arn arn:aws:iam::123456789:role/my-rds-proxy-sample-role \
--vpc-subnet-ids subnet-09c83b23370eeb245 subnet-04dff361ae0758434  \
--require-tls
  1. Register proxy targets to point your proxy to the rds instance
aws rds register-db-proxy-targets --db-proxy-name temp-db-proxy --db-instance-identifiers temp-db
  1. Go to your RDS proxy in the AWS console and find it's security group. Allow inbound traffic from wherever you'd like to connect from.

------ testing the newly provisioned setup ------

We have an ec2 bastion running in the VPC. When I run the following. it can connect to the RDS proxy via IAM token:

export RDSHOST="temp-db-proxy.proxy-cfds9ixmayu4.us-east-1.rds.amazonaws.com"
export PGPASSWORD="$(aws rds generate-db-auth-token --hostname $RDSHOST --port 5432 --region us-east-1 --username masterawsuser)"
psql -h $RDSHOST -p 5432 "sslmode=require dbname=postgres user=masterawsuser"

Tokens generated on my local workbench via the same commands don't work.

once I set the following environment variables, aws rds generate-db-auth-token starts to generate working tokens

export $(printf "AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s AWS_SESSION_TOKEN=%s" \
 $(aws sts assume-role \
 --role-arn arn:aws:iam::123456789:role/Adhoc3BastionStack-BastionHostRole-JDL6Y8KJ69VB \
 --role-session-name Adhoc3BastionSession \
 --query "Credentials.[AccessKeyId,SecretAccessKey,SessionToken]" \
 --output text))

Note: Adhoc3BastionStack-BastionHostRole-JDL6Y8KJ69VB is the role on the working bastion instance. My stored credentials have the following permissions, so it should be an issue with privilege.

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "*", "Resource": "*" } ] }

I checked ~/.aws/credentials to see if anything was out of place. no dangling aws_session_token =. that would have been a great catch though.

aws sts get-caller-identity is quite interesting. I unset my environment variables:

unset AWS_ACCESS_KEY_ID; 
unset AWS_SECRET_ACCESS_KEY; 
unset AWS_SESSION_TOKEN

then I called aws sts get-caller-identity and the token generation started working properly! The tokens are still missing the field X-Amz-Security-Token but still seem to work fine without it. Do you have more info on what that does? is it refreshing the client session? I read the link you provided, and it looks like my session might expire again, so I'm going to keep an eye on it to see if things break again.

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Oct 13, 2023
@aBurmeseDev
Copy link
Member

Glad to hear that it's working as expected now after running aws sts get-caller-identity. The command is to confirm and validate the identity of the caller and IAM role. Here's more about it for reference: https://docs.aws.amazon.com/STS/latest/APIReference/API_GetCallerIdentity.html

@aBurmeseDev aBurmeseDev added the closing-soon This issue will automatically close in 4 days unless further comments are made. label Oct 23, 2023
@scott-vandevoorde
Copy link
Author

thanks for the detail!

@github-actions
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
closing-soon This issue will automatically close in 4 days unless further comments are made. p3 This is a minor priority issue rds
Projects
None yet
Development

No branches or pull requests

2 participants