-
Notifications
You must be signed in to change notification settings - Fork 184
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow PARQUET format for uploading data. #609
Conversation
…age depends on R > 4.0.0
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for working on this!
Co-authored-by: Hadley Wickham <[email protected]>
Co-authored-by: Hadley Wickham <[email protected]>
Hi @hadley, Thank you for your review. I have taken into account all of your comments, and I hope I haven't missed any. Regards, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Two more small points. I really appreciate you working on this and your responsiveness to feedback 😄
NEWS.md
Outdated
@@ -1,5 +1,14 @@ | |||
# bigrquery (development version) | |||
|
|||
## Significant improvements |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you leave these headings off? We add them as part of the final release process?
DESCRIPTION
Outdated
@@ -29,7 +29,8 @@ Imports: | |||
prettyunits, | |||
rlang (>= 1.1.0), | |||
tibble | |||
Suggests: | |||
Suggests: | |||
arrow, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would you consider trying nanoparquet instead? It's very new but has no dependencies, so we could use it from imports.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree that nanoparquet is a good alternative because it has no dependencies. The only downside I see is that you would have to write to disk to read the raw data, as it lacks an output stream buffer implementation.
If you think the advantages outweigh the disadvantages, I could start testing and adapting the code.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah I didn't think about that, but I suspect it's still worthwhile given the lighter dependencies. Do you mind filing a nanoparquet issue to add a stream buffer output?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I will begin implementing nanoparquet. If BufferedOutputStream is added in the future, an update will be necessary.
Hi @hadley, The implementation of Nanoparquet to replace Arrow has been completed. After several data loading tests, I believe it works very well. I look forward to your comments. Regards, |
Thanks so much for working on this! |
Hi @hadley,
This pull request contains the implementation for allowing the user to decide in which format they want to transmit the data (JSON or PARQUET) to BigQuery (For large amounts of data, loading data in JSON format is very time-consuming due to the size of the data that needs to be transmitted. To address this problem, BigQuery accepts other file formats, including Parquet). The most significant change enabling PARQUET data transmission is that the uploadType is no longer multipart; it is now resumable.
https://cloud.google.com/bigquery/docs/reference/api-uploads
Regards,