Ability to specify endpoint-url in profile

[StFS]

Currently I don't seem to be able to specify an endpoint URL in my profile. I always have to specify the --endpoint-url option on the command line.

I would like to be able to do something like the following (in my ~/.aws/config file):

[default]
region = us-east-1
output = json

[profile nextcode]
region = myregion-1a
output = json
endpoint-url = http://c.my.other.aws.compatable.service.com

[kyleknap]

@StFS
If you do not mind me asking, what is the specific use case do you need it for (like which service or services would you plan to use it for)? The thing with --endpoint-url is that the value set will most likely not apply across differing services. So if you configure a global endpoint url, it may work for one service, but you will run into errors if you try to make requests to other services.

I could see having a per-service configuration for endpoint-url, but I am having trouble imagining a global endpoint-url across all AWS services.

[StFS]

Thanks for your reply.

Basically, we use AWS for some things but we also have our own EC2/S3 compatable private cloud setup (we're using https://qstack.com/).

Maybe I'm misunderstanding something with this. I know that the endpoints seem to differ between whether you're "talking" S3 or EC2. Basically I think I want to be able to point one of my profiles to a different "set" of endpoints. The aws cli tool works fine for our AWS account, but when I want to use it for our private cloud setup I always have to specify both --profile (to get the credentials right) and --endpoint-url (so that aws contacts our private cloud endpoint instead of the AWS ones).

[StFS]

@kyleknap Am I totally out there with this?

[kyleknap]

@StFS
Your use case makes sense to me. I am not sure how many users have a similar use case. I think if we were to add it as a feature we would be more inclined to do it per service, instead of setting a global value for endpoint url. The good news is that if we were to add this feature you would be able to configure the endpoint url for each service per profile (so you can still just use --profile and not have to include --endpoint-url).

Marking as feature request.

[StFS]

Ok thanks for the response again.

I'm a bit confused about these endpoints. It seems that for AWS the endpoint may change depending on what region you're using and also what service you're using.

For us, we would only need to specify ec2_endpoint and s3_endpoint, is that what you mean by "per service"? Are there other services than EC2 and S3? I think I've seen something involving user management as well but I'm not quite sure.

[jerrycattell]

We have a similar use case (S3 alternative in Canada), and we would love to be able to configure the endpoint-url in the config file. Even if that means having a separate profile per service (currently it is only used for object storage), it would allow us to use the same commands in all of our environments.

[kennu]

I think this would also be useful for configuring a "local" profile for accessing DynamoDB Local. Currently you have to write something like

aws --profile local --endpoint-url http://localhost:8000 dynamodb list-tables

[viniciussbs]

Same case here, @kennu. I have to set endpoint-url everytime I need to run some command on dynamodb-local.

I would love to set http://localhost:8000 as my default endpoint-url for dynamodb and set a blank endpoint-url to an production profile. It's more error-proof.

[mathurasri]

How to set dyanmodb local endpoint-url ? I am using ubuntu OS.

[shravanpn7]

+1 for feature-request.

I am currently using aws cli to access minio.io server and specifying --endpoint-url every time is a pain.

[slejworks]

I need this also, connecting to local dnyamodb and I don't want to specify the endpoint for each cmd, simple entry into the config will work for me so I can concentrate on the command. thanks.

[christians3]

+1 for this feature of allowing endpoint-url in the configuration file.

[philloooo]

if accepting endpoint-url on the top level doesn't make sense, what about accepting it on service level?
eg:

[profile development]
aws_access_key_id=foo
aws_secret_access_key=bar
s3 =
  endpoint-url = test.org

[jhalterman]

Similar use case, pointing to a Eucalyptus cloud. Would be nice to have a per-service config for endpoint-url.

[lincolnthomas]

How about also for the convenience of not having to type the --endpoint on the cmdline?

[wrsuarez]

+1

[jmarshall]

We have a third-party library that (as a minor aspect of its remote data access facilities) itself parses ~/.aws/credentials in order to construct the HTTP headers to access data in S3 buckets. Similarly to this issue, our users want to access their own S3-compatible data stores (cf samtools/htslib#436) and it would be useful if there were a standard well-known configuration file setting name for this purpose.

[jedi4ever]

👍

[tomviner]

We also have this use case, except our S3-like-service has a different authorisation scheme (a simple authorization: <token> header). How are people using aws-cli to authorise against their S3-like-services?

[wbingli]

Check out this awscli plugin to set endpoint on profile: https://github.com/wbingli/awscli-plugin-endpoint

Once you install(pip install awscli-plugin-endpoint) and config plugin(aws configure set plugins.endpoint awscli_plugin_endpoint), you can set endpoint in your profile per service as following:

[profile local]
dynamodb =
    endpoint_url = http://localhost:8000

Now you can run command with this endpoint for this service with only profile name:

aws dynamodb list-tables --profile local

See more details on project homepage(https://github.com/wbingli/awscli-plugin-endpoint).

Let me know your feedback, :)

[RichardBronosky]

Just to be clear, this is not only a benefit for people who are running AWS-compatible competitive products. It would also benefit those of us who are trying to use Amazon tools for local/offline development. http://stackoverflow.com/a/32260680/117471 is an example with DynamoDB. Using different profiles allows us to run our code with different configs for different environments. Not allowing endpoint_url to be specified in ~/.aws/config means that we have to build logic into our apps to follow a different (although small) path depending on the environment. That is something that should be avoided.

[davidparks21]

We are investigating using the AWS Snowball Edge devices, and as such we want to use the AWS CLI with the AWS Snowball Device and having to specify --end-point every time is really quite tiresome.

There's an environment variable option for specifying the endpoint, I typically create a script called aws3 and put the following in:

#!/bin/bash
aws --endpoint YOUR_ENDPOINT_URL --profile YOUR_PROFILE s3 $@

Usage:
aws3 ls s3://bucket/voila/that/looks/better

I avoid using alias because you can't use it as easily with piped commands like xargs.

Of course, specified in the profile would be better. But until that days comes, there can still be joy in the world with these little shell-life-hacks.

[artgromov]

that is why i haven't migrate to aws-cli v2 yet

[ctavplay]

I have a slightly different use-case similar to @flickerfly's. Curious to hear if folks think I should file a separate issue.

This comes up when:

• calling an API in one region from within a VPC that's in another, and
• assuming a role before-hand by specifying the role_arn, role_session_name and credentials_source, in ~/.aws/config, and
• the role being assumed has a condition requiring the use of the STS VPC endpont in your VPC

There appears to be no way for me to specify the specific STS endpoint to use in the config file.

Here's some sample code:

~/.aws/config

[default]
region = us-east-1
sts_regional_endpoints = regional
credential_source = EcsContainer
role_arn = arn:aws:iam::11111111:role/my_role
role_session_name = my_session

my_role's trust relationship:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::22222222:role/my-ecs-task"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "aws:SourceVpce": "vpce-01234deadbeef"
        }
      }
    }
  ]
}

Code running in the VPC containing the required STS VPC endpoint

$ aws --region us-west-2 s3 ls

The above errors out because the role can't be assumed.

[ctavplay]

Filed a new issue with a more targeted ask: #6754

[kellertk]

Thank you for posting your feedback here, and our apologies that we’ve been thinking this over for a long time without much forward motion. There are similar requests to implement this feature in a few of the AWS SDKs and the AWS CLI, so in order to coordinate those teams - and hopefully make the discussions a little easier to follow - we’ve created a new issue in aws/aws-sdk here: aws/aws-sdk#229

[kdaily]

Hi all,

We recently added a pull request (aws/aws-sdk#230) that contains a proposal based on community comments and suggestions and our own discussions. This document proposes to extend the options for configuring the endpoint to allow users to provide an endpoint URL independently for each AWS service via an environment variable or a profile subsection in the shared configuration file.

You can read the proposal here.

For more information on how to give feedback, please see this comment on the aws/aws-sdk repository:

aws/aws-sdk#229 (comment)

Thanks!

[kdaily]

I'm happy to announce that the ability to configure the endpoint URL via the shared configuration file and environment variables is now available in the AWS CLI v1 and v2! You can now specify the endpoint to use for all service requests through the shared configuration file and environment variables, as well as specify the endpoint URL for individual AWS services.

To start using this feature, install the AWS CLI >=1.29.0 or >=2.13.0.

To read more about this feature, see the documentation page "Service-specific Endpoints" in the AWS SDKs and Tools Reference Guide:

https://docs.aws.amazon.com/sdkref/latest/guide/feature-ss-endpoints.html

Look forward to a blog post demonstrating the use of this feature with the AWS CLI on the AWS Developer Tools Blog!

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

[kdaily]

I'm excited to announce that we have published an AWS Developer Blog post about this feature. Let us know in the feedback links on the post what you think!

https://aws.amazon.com/blogs/developer/new-improved-flexibility-when-configuring-endpoint-urls-with-the-aws-sdks-and-tools/