unclear / misleading FAQ for s3 re: checksum

[abroy77]

Describe the issue

The FAQ page says:

The AWS CLI will calculate and auto-populate the Content-MD5 header for both standard and multipart uploads. If the checksum that S3 calculates does not match the Content-MD5 provided, S3 will not store the object and instead will return an error message back the AWS CLI

I don't think this is accurate. When I upload a file using aws s3 cp file.txt s3://bucket/file.txt I can't see a checksum for the object on the AWS console. Perhaps I have misunderstood the documentation?

Links

https://awscli.amazonaws.com/v2/documentation/api/latest/topic/s3-faq.html?highlight=checksum

[tim-finnigan]

Thanks for reaching out. If you add --debug to your command, then you can see what the request looks like. For example when I run aws s3 cp there is a Content-MD5 header logged in my request.

The S3 console has a table for additional checksums (which you can find more info on here in the S3 User Guide: https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html). The additional checksums are not supported via high-level s3 commands but there is a feature request for that: #6750. As an alternative you can use aws s3api put-object.

[abroy77]

Ah! I see, my misunderstanding then. I thought the page meant that the checksum will be stored.

Am I correct in understanding that this Content-MD5 header is used to ensure data integrity during the s3 cp command? but does not store an additional checksum with the data on s3 permanently?

Thank you for all your help!

[tim-finnigan]

Thanks for following up. Yes the Content-MD5 header is used to ensure data integrity, and the ETag of the object might be an MD5 digest of the object data, this section in the S3 User Guide has more details on that: https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html#checking-object-integrity-etag-and-md5. But I recommend applying one of the additional checksum algorithms if you want to store that on S3.

[abroy77]

thanks! very helpful!

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.