Add logic for handling large files in MultipartWriter uploads to s3#796
Merged
mpenkov merged 11 commits intopiskvorky:developfrom Feb 22, 2024
Merged
Add logic for handling large files in MultipartWriter uploads to s3#796mpenkov merged 11 commits intopiskvorky:developfrom
mpenkov merged 11 commits intopiskvorky:developfrom
Conversation
Contributor
Author
|
Ah oops, totally missed there was no pre-commit with black configured. Ran my sections through black and copy-pasted the output and it's passing flake8 locally now |
Contributor
Author
|
Any possibility of getting this reviewed? :) |
Collaborator
|
Hi, thank you for the PR. I'm busy with other parts of my life at the moment, but I'll try to review your work within the next couple of weeks. |
jakkdl
commented
Feb 20, 2024
mpenkov
reviewed
Feb 20, 2024
Collaborator
|
Thank you for your work on this! |
ddelange
added a commit
to ddelange/smart_open
that referenced
this pull request
Feb 22, 2024
…open into patch-2 * 'develop' of https://github.com/RaRe-Technologies/smart_open: fix test, for real this time update integration test Add advanced usage sections to README.rst (piskvorky#741) Add logic for handling large files in MultipartWriter uploads to s3 (piskvorky#796) Fix __str__ method in SinglepartWriter (piskvorky#791)
ethanjyx
reviewed
Mar 7, 2024
| # from the MIT License (MIT). | ||
| # | ||
| """Implements file-like objects for reading and writing from/to AWS S3.""" | ||
| from __future__ import annotations |
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fixes #380
I'll likely go through the PR later and add a couple more comments, but otherwise it's done.
It'd be great if botocore allowed
memoryviews to be directly passed to it, which would mean we wouldn't need to write a copy of the data in the buffer. But we probably don't want to wait for if/when boto/botocore#3107 is merged+releasedI added a couple types as I was working to understand the current code, and given #518 I thought I might as well keep them. But can remove them if you don't want a couple random types hanging around.