-
Notifications
You must be signed in to change notification settings - Fork 133
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
s3_client.copy() drops metadata for small files #246
Comments
adding a TransferConfig of multipart_threshold=1 will work around this, in all except the pathological 0 byte file scenario.
|
Thanks for the tip, I'll try that promptly, Any insight into what's going on? |
I chased it all the way down to a function called |
I've been spending hours digging through libraries trying to get to a similar point, thanks much! |
I think I figured it out. Once I added that setting I received an exception related to a missing permission on a kms key needed to encrypt the object at rest. Now that I've fixed the permission, I no longer receive the EOF issue, even without the transfer config. This makes me think it was somehow masking the underlying error. |
@cariaso <https://github.com/cariaso> just to clarify is the metadata lost
when using the default multipart_threshold value?
yes, metadata is lost with the default multipart_threshold value.
…On Wed, Aug 3, 2022 at 8:49 AM Tim Finnigan ***@***.***> wrote:
@cariaso <https://github.com/cariaso> just to clarify is the metadata
lost when using the default multipart_threshold value? If there is an issue
with s3transfer then we would consider transferring this issue here
<https://github.com/boto/s3transfer/issues>.
—
Reply to this email directly, view it on GitHub
<#246>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAA6TES7BLQIYVADP5ELOSDVXJTEHANCNFSM54W3ZOFQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
--
Mike Cariaso
http://www.cariaso.com
|
Thanks @cariaso for following up. Have you also tried using the copy_from method to copy the metadata over as described here? |
A similar issue was opened recently and another user confirmed that I also tested the copy_object command and it copies the metadata by default. There may be slight performance implications/trade-offs between these three copy commands. But I think this issue specifically relates to s3transfer and so I'm going to move it to that repository for further review and clarification from the team. |
For more background on |
Describe the bug
s3_client.copy() stores metadata for files large enough to trigger multipart handling, but smaller files trigger a different code path, and the metadata is lost.
Expected Behavior
metadata should be copied regardless of the file size
Current Behavior
metadata is missing
Reproduction Steps
I'm using this to emulate variable sized files, but you can replace it with a static file.
Possible Solution
the size determination is made, at
s3transfer/s3transfer/copies.py
Line 74 in 7a0d980
and results in either calling
_submit_copy_request
or_submit_multipart_request
Additional Information/Context
No response
SDK version used
boto3 1.24.37
Environment details (OS name and version, etc.)
Ubuntu 22.04 LTS
The text was updated successfully, but these errors were encountered: