Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

High Memory usage in gosnowflake #11

Closed
rompetroll opened this issue Aug 23, 2023 · 1 comment
Closed

High Memory usage in gosnowflake #11

rompetroll opened this issue Aug 23, 2023 · 1 comment

Comments

@rompetroll
Copy link
Contributor

When streaming data to a snowflake stage using PUT, the driver allocates excessive amounts of memory. This is a known limitation snowflakedb/gosnowflake#536

To enable as many parallel uploads as possible, we should go back to using file writers instead of streaming. We should also gzip the files on disk though, so that we do not run out of disk as easily either.

@rompetroll
Copy link
Contributor Author

Addressed in #10

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant