-
Notifications
You must be signed in to change notification settings - Fork 354
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
bugfix: await on size, assuming it can be an async function #1281
base: master
Are you sure you want to change the base?
Conversation
@martindurant what do you think? |
I think you are right in your argument. However, I have a feeling that f.size is never async actually, so it won't matter. open_async/AsyncStreamedFiles could use work! |
fsspec/s3fs#742 makes it async in order to call the async _info needed for _cp_file |
Funny, because S3 is the only one where for sure we should know the size before starting the download: the content-size header is always populated and, possibly outside compressive transcoding, will be correct. |
Perhaps I should change https://github.com/fsspec/filesystem_spec/blob/master/fsspec/generic.py#L277 to just remove the assumption that |
maybe_await should never be harmful and not add any overhead |
Checking
size
may require a call to the async_info
function. On line 277,_cp_file
checks iff1.size
is async (callback.set_size(await maybe_await(f1.size))
), but then later it simply referencesf1.size
which fails if this function is async. As such, this PR corrects the code to ensuref1.size
references are wrapped inawait maybe_await()
.