boto3 set endpoint_url from environment variables

[fatihtekin]

Could you please advise how to set endpoint_url outside of the code by setting an environment variable or a ~/.aws/config file?

[fatihtekin]

I found a workaround to do that as below for boto3. But it would be better if it was as above

from boto3 import Session
Session.client.defaults = (None, None, False, None, 'http://localhost:4575', None, None, None, None)

[stealthycoin]

You can set this during client creation as an argument. Documented here.

There is no environment variable for this but there is nothing stopping you from looking in os.environ yourself and passing that value along to the endpoint_url argument of the client if that is the behavior you want.

[jamesls]

Closing as dupe of aws/aws-cli#1270

[ozbillwang]

@jamesls

Could you please reopen this issue? This feature isn't supported currently in boto3.

The dup of aws/aws-cli#1270 has been fixed by an awscli plugin

https://github.com/wbingli/awscli-plugin-endpoint

aws/aws-cli#1270 (comment)

But it only works with aws cli.

It seems the same feature has been added in php sdk as well

https://github.com/aws/aws-sdk-php/pull/1243/files/aeb25899897fe746580ae6c80e0df6ad37931126

If it can work with aws config profile file directly, better than implemented with environment variables, because one python functions can use different endpoint for different aws services.

[profile localstack]
s3 =
    endpoint_url = http://localhost:4572
kms =
    endpoint_url = http://localhost:4599
lambda =
    endpoint_url = http://localhost:4574
iam =
    endpoint_url = http://localhost:4593
kinesis =
    endpoint_url = http://localhost:4568
logs =
    endpoint_url = http://localhost:4586
sts =
    endpoint_url = http://localhost:4592
ec2 =
    endpoint_url = http://localhost:4597

[jaklinger]

For those interested in a workaround until this feature is implemented, see aws/aws-cli#1270 (comment)

[rwillmer]

I've added a PR to provide this #2746.
If it isn't accepted, or needs more work, my fork might give you what you need in the meantime.
https://github.com/rwillmer/boto3

[bradenkinard]

Can we please get this re-opened? Its a widely requested feature, with a simple, two line implementation that has been languishing in a ready to merge state for nearly a year here: #2746

[jaimergp]

If you need to temporarily patch something in a test, this how you can do it with unittest.mock.patch. The example shows how to override the defaults to connect to a local Minio instance.

from unittest.mock import patch
import boto3
from botocore.client import Config

patched_defaults = (
    "us-east-1",                       # region_name
    None,                              # api_version
    True,                              # use_ssl
    None,                              # verify
    "http://localhost:9000",           # endpoint_url
    "minioadmin",                      # aws_access_key_id
    "minioadmin",                      # aws_secret_access_key
    None,                              # aws_session_token
    Config(signature_version="s3v4"),  # config
)
with patch.object(boto3.session.Session.resource, "__defaults__", patched_defaults):
    session = Session()
    s3 = session.resource("s3")
    # proceed as needed

You can also patch boto3.session.Session.client in the same way.

[ljmc-github]

I also faced this problem, and since #2746 seems to be in limbo, and aws/aws-sdk/issues/229 is only a proposal, I went ahead an build a little SHIM to provide the proposal's base functionality (specific env vars and nested config).

EDIT (2022-12-22)

I have implemented the proposal in a workaround package: boto-endpoint-url-shim.

Base usage for service-specific environment variable and service subsection in shared configuration file:

import boto_endpoint_url_shim
import boto3

boto_endpoint_url_shim.proposed_endpoint_url_resolution()

s3 = boto3.resource("s3")  # uses the custom endpoint

This is my first public python package and it is in early stages, but it implements all the proposed endpoint resolution methods. If you have any feedback, feel free to drop me a message or open an issue on gitlab.

[yankevn]

FYI, #2746 was closed due to boto3 directly implementing this feature. With boto3>=1.28.0, you can refer here for using environment variables/a shared configuration file: https://docs.aws.amazon.com/sdkref/latest/guide/feature-ss-endpoints.html