feat: Support loading Parquet files from the s3 compatible storage directly #3378
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
as51340
added
CI -build=community -test=core
as51340
changed the base branch from
master
to
feat/queue-based-batching-thread-allocators-v2
as51340
force-pushed
the
feat/s3-arrow
branch
2 times, most recently
from
2905676 to
9dbc24c
Compare
as51340
force-pushed
the
feat/s3-arrow
branch
from
8174ab4 to
011b700
Compare
Draft
10 tasks
as51340
force-pushed
the
feat/s3-arrow
branch
from
fb4b59e to
be1f15f
Compare
Base automatically changed from
feat/queue-based-batching-thread-allocators-v2
to
master
October 27, 2025 08:58
as51340
force-pushed
the
feat/s3-arrow
branch
from
080d673 to
08b8247
Compare
andrejtonev
reviewed
Oct 28, 2025
andrejtonev
approved these changes
Oct 28, 2025
as51340
removed
CI -build=community -test=core
|
This was referenced Oct 30, 2025
28 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds AWS S3 integration for loading Parquet files from S3. AWS SDK C++ library is brought transitively through arrow library. For the authentication, we use region, access_key and secret_key. Users can also use aws_endpoint_url for pointing to some other S3 compatible storage. Settings can be set through env variables, query settings and runtime settings.
Tested using
localstackwhich mocks S3 locally.