[DNM] Add inherit_aws_config_from_environment function so user can set s3 config values using environment variables
#211
+295
−51
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Tmonster
changed the title
inherit_aws_config_from_environment function so user can set s3 config values using environment variablesinherit_aws_config_from_environment function so user can set s3 config values using environment variables
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
DO NOT MERGE, this is a breaking change and we need to discuss if this is a break we want to make.
Follow up for https://github.com/duckdblabs/duckdb-internal/issues/6620
In v1.4.4 duckdb-httpfs will automatically read environment variables like
AWS_SECRET_ACCESS_KEY, andAWS_SECRET_KEY_IDand populate the config valuess3_secret_access_keyandDBConfigOptionswith those environment variables. These config values would then be used for any remote reads on a blob storage like s3/r2. #182 introduced the ability to disable the automatic use of these values during remote reads, it did not disable automatically reading and setting those values.With this PR for v1.5, we will disable auto loading environment values not prefixed with DUCKDB into the config options, and only do when the user requests it. To allow a user to set config values from environment variables, we introduce the function
inherit_aws_config_from_environment()that a user can call which will set the config valuess3_region,s3_secret_access_key,s3_session_token,s3_endpoint,s3_use_ssl,s3_kms_key_id, ands3_requester_pays. These config options are only set locally, we can set these config options globally if needed.