Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor LogicalIO for better dependency management #93

Open
ahmarsuhail opened this issue Sep 4, 2024 · 2 comments
Open

Refactor LogicalIO for better dependency management #93

ahmarsuhail opened this issue Sep 4, 2024 · 2 comments
Labels
enhancement New feature or request logical IO

Comments

@ahmarsuhail
Copy link
Contributor

ahmarsuhail commented Sep 4, 2024

Currently when instantiating S3SeekableInputStream and LogicalIoFactory we create ParquetColumnPrefetchStore, even though this may not be required if we are not further reading parquet files.

We need to look at how to improve this.

@oleg-lvovitch-aws
Copy link
Collaborator

Thank you for opening this! Wasn't there a PR floating around that separated LogicalIO into format specific and generic one?

I think this could be a good place to start to tease this apart - can we do this first? The generic one most definitely won't need to take anything Parquet specific.

@oleg-lvovitch-aws
Copy link
Collaborator

At! I think this PR is it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request logical IO
Projects
None yet
Development

No branches or pull requests

2 participants