Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add S3 source against minio S3 object storage #1967

Closed
mskyttner opened this issue Mar 22, 2023 · 2 comments · Fixed by #3519
Closed

Add S3 source against minio S3 object storage #1967

mskyttner opened this issue Mar 22, 2023 · 2 comments · Fixed by #3519
Labels
community Type:Feature New feature request

Comments

@mskyttner
Copy link

When setting up a connection to an S3 source and not using AWS but for example minio, the change in #1691 allows specifying the "endpoint-url" in source.yaml using a line that says "endpoint: https://my.s3.server.com". This setting cannot be made in the UI, as the S3 settings there does not provide a field for specifying the "endpoint-url".

It would neat to be able to make this setting from the UI and maybe directly from the CLI too, without having to manually edit the source.yaml-file.

Alternatives that I considered was to install and use "aws configure" to set the endpoint URL to use, but due to this long standing bug in the aws CLI, it doesn't work: aws/aws-cli#1270

I would like the "Amazon S3" settings to support also minio (and thereby also other compatible S3 sources), and ideally be renamed "S3" and allow configuration of the endpoint-url.

It would be nice if in addition (ie if not specified) the endpoint URL could be picked up from an environment variable. I also think the "minio client" is nicer to configure rather than the aws CLI if one needs to access data from various different S3 sources (other than just AWS).

@mskyttner mskyttner added the Type:Feature New feature request label Mar 22, 2023
@mskyttner
Copy link
Author

Related to #1161. "Object storage" which is not distantly located in the outer clouds provide faster latency due to smaller network lags and minio is a component often used to "bring data closer to computations", recently with support also for "legacy" protocols like ftp/sftp.

@mskyttner
Copy link
Author

S3 connections against non-AWS servers might be possible with the aws cli if allowing the AWS_ENDPOINT_URL environment variable to be used, see aws/aws-cli#1270 (comment) where this feature recently after 8 years since inception is now available as documented in https://aws.amazon.com/blogs/developer/new-improved-flexibility-when-configuring-endpoint-urls-with-the-aws-sdks-and-tools/.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
community Type:Feature New feature request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants