Use custom url for s3 using AWS_ENDPOINT_URL

[Kotwic4]

AWS_ENDPOINT_URL is now supported by the AWS for custom url (for example localhost).
More info about it docs and original github issue. It was merged into botocore in this pr.

What I can do with boto:

import os
import boto3

os.environ["AWS_ACCESS_KEY_ID"] = "ACCESS_KEY"
os.environ["AWS_SECRET_ACCESS_KEY"] = "SECRET_KEY"
os.environ["AWS_ENDPOINT_URL"] = "http://localhost:9000"

session = boto3.session.Session()
s3_client = session.client(
    service_name="s3",
)
print(s3_client.list_buckets()["Buckets"])

What I have to do in s3fs:

import os
import s3fs

os.environ["AWS_ACCESS_KEY_ID"] = "ACCESS_KEY"
os.environ["AWS_SECRET_ACCESS_KEY"] = "SECRET_KEY"
os.environ["AWS_ENDPOINT_URL"] = "http://localhost:9000"

s3 = s3fs.S3FileSystem(endpoint_url=os.environ["AWS_ENDPOINT_URL"])
print(s3.ls(""))

What I would like to do in s3fs:

import os
import s3fs

os.environ["AWS_ACCESS_KEY_ID"] = "ACCESS_KEY"
os.environ["AWS_SECRET_ACCESS_KEY"] = "SECRET_KEY"
os.environ["AWS_ENDPOINT_URL"] = "http://localhost:9000"

s3 = s3fs.S3FileSystem()
print(s3.ls(""))

[martindurant]

Assume s3fs will also get this automatically without doing anything as soon as aiobotocore bumps its dependency on botocore. Maybe ask them when they plan to do that. botocore is at 1.31.5, but the dep is 1.29.161 .

[Kotwic4]

ok, created issue there as well

[dpprdan]

botocore has been bumped to 1.31.17 in aiobotocore, aio-libs/aiobotocore#1029

[martindurant]

s3fs 2023.9.0 uses aiobotocore 2.5.4, which is after the PR linked above. However, 2.6.0 is also out, but we don't have that as our dependency due to conda-forge/aiobotocore-feedstock#68

[dpprdan]

It would be great if this would include support for configuring endpoints via a shared config file as well.

Or should I open a separate issue for that?

[martindurant]

s3fs does accept profile=, please test if this works now for you.

[dpprdan]

The following examples all work for me (with s3fs version 2023.9.0 and play.min.io config):

import os
import s3fs

os.environ["AWS_ACCESS_KEY_ID"] = "Q3AM3UQ867SPQQA43P2F"
os.environ["AWS_SECRET_ACCESS_KEY"] = "zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG"
os.environ["AWS_ENDPOINT_URL"] = "https://play.min.io:9000"

s3 = s3fs.S3FileSystem()
s3.ls("asiatrip")

import os
import s3fs

os.environ["AWS_PROFILE"] = 'play'

s3 = s3fs.S3FileSystem()
s3.ls('asiatrip')

import s3fs

s3 = s3fs.S3FileSystem(profile='play')
s3.ls('asiatrip')

The latter two with ~/aws/config

# https://docs.aws.amazon.com/sdkref/latest/guide/file-format.html
# https://docs.aws.amazon.com/sdkref/latest/guide/feature-ss-endpoints.html#ss-endpoints-config

[default]
# "Explicit is better than implicit."

[profile play]
region = us-east-1
services = play-s3

[services play-s3]
S3 =
  endpoint_url = https://play.min.io:9000
  signature_version = s3v4

s3api =
  endpoint_url = https://play.min.io:9000

and ~/aws/credentials

# https://docs.aws.amazon.com/sdkref/latest/guide/file-format.html

[default]

[play]
aws_access_key_id = Q3AM3UQ867SPQQA43P2F
aws_secret_access_key = zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG

This can be closed IMHO.

[martindurant]

Thanks for testing