Remove S3FS dependency #332
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
matteofigus
commented
Jul 28, 2022
| with s3.open(object_path, "rb") as f: | ||
| source_version = f.version_id | ||
| with s3.open_input_stream( | ||
| "{}/{}".format(input_bucket, input_key), buffer_size=FIVE_MB |
There was a problem hiding this comment.
Buffer size is same as s3fs: https://github.com/fsspec/s3fs/blob/main/s3fs/core.py#L246
matteofigus
commented
Jul 28, 2022
| protection against automated library upgrade PRs that may silently introduce | ||
| issues. | ||
| """ | ||
| assert s3transfer.__version__ == "0.6.0" |
There was a problem hiding this comment.
I discussed this with @HieronymusLex - may seem a bit over cautious but I think it's worth it given Dependabot may make very easy to silently upgrade and leave dangerous things unnoticed.
HieronymusLex
previously approved these changes
Jul 28, 2022
HieronymusLex
approved these changes
Aug 2, 2022
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of changes:
This PR removes s3fs to simplify handling communications with S3.
It uses pyarrow's S3FileSystem to read, and boto3 to write back.
The reason I am using pyarrow to read is that it allows us to get rid of s3fs and I haven't noticed differences in terms of performance.
I tried to use pyarrow to write back to s3 too, but unfortunately I don't have an API that gives me back the VersionId of the write. If I would read the latest version using boto3, I would lose confidence that the version I'm reading is the one I just wrote, and therefore I wouldn't be able to perform consistency checks.
In order to remove s3fs, I investigated boto3 and found a (new?) convenient method called upload_fileobj that is smart enough to figure out if doing multipart or not, and I noticed an increase in performance compared to s3fs. The issue is that this method doesn't return the versionID as well, but I managed to find a way to monkey patch the code while we wait for this issue to be resolved. I found some Pull Requests too so perhaps we can see this soon landing in boto.
I ran the acceptance tests as well as some manual tests with very large files (8GB snappy compressed parquet), for which I didn't see any difference in reads, and some performance improvement (3x) in writing.
PR Checklist:
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.