GetObjectAsync varying ResponseStream Type, MD5Stream causes exception

[RiversJohn]

Having an issue where some images we try to get with
AWSSDK.S3" Version="3.5.10.2" AmazonS3Client.GetObjectAsync
Have a return type of MD5Stream and when the stream disposes an exception is thrown:

  Message: 
    Test method Namespace.S3GetObjectAsyncHashTest.Test threw exception: 
    Amazon.Runtime.AmazonClientException: Expected hash not equal to calculated hash

  Stack Trace: 
    HashStream.CalculateHash()
    HashStream.Dispose(Boolean disposing)
    Stream.Close()
    StreamResponse.Dispose(Boolean disposing)
    StreamResponse.Dispose()
    S3FileService.ReadImageStreamAsync(String relativePath, String filename) line 82
    S3FileService.ReadOriginalFileAsync(String ean, String filename, FileTypeData typeData) line 41
    S3GetObjectAsyncHashTest.<Test>g__BrokenFileRead|0_1(S3FileService s3Service, FileTypeData fileTypeData) line 34
    S3GetObjectAsyncHashTest.Test() line 22

All the objects were trying to get have been uploaded with the same method:

        using (var client = await CreateClientAsync())
        {
            fileStream.Seek(0, SeekOrigin.Begin);
            await client.PutObjectAsync(new PutObjectRequest
            {
                BucketName = relativePath,
                Key = filename,
                InputStream = fileStream
            });
        }

This makes me wonder why some calls return a stream of type: {Amazon.Runtime.Internal.Util.CachingWrapperStream} with underlying typeof: {System.Net.Http.HttpConnection.ContentLengthReadStream}

While other calls return stream of type: {Amazon.Runtime.Internal.Util.MD5Stream} with underlying typeof: {Amazon.Runtime.Internal.Util.CachingWrapperStream} (and the ContentLengthReadStream under that ofc)

Currently any calls that return an instance of MD5Stream cause the exception when the stream disposes, our calls do not specify an algorithm.
Calling code:

        // Create request object
        var request = new GetObjectRequest()
        {
            BucketName = relativePath,
            Key = filename,
        };
        var ms = new MemoryStream();
        // Read the object from S3
        using var client = await CreateClientAsync();
        using var response = await client.GetObjectAsync(request);
        await response.ResponseStream.CopyToAsync(ms);
        // going out of scope after this will throw the exception

I'm left wondering if this is a bug or user error, either way i kind of need to know.

I could work around by simply catching the exception and reading the data but doing so blindly without knowing why this occurs doesn't seem terribly smart in the environment this code is running in.

Any advice would be greatly appreciated!

[ashishdhingra]

@RiversJohn Good morning. Please share the following:

• Did you try upgrading to latest version of AWSSDK.S3 and see if the error goes away?
• Could you please share the minimal code snippet for Namespace.S3GetObjectAsyncHashTest.Test for us to reproduce the issue?

Thanks,
Ashish

[RiversJohn]

3.7.x didn't run at all due to signature changes and would prefer to defer the work but if it's required to resolve i will ofcourse.

The files are on my work computer, I'll give you a test file, the code ran and json the response object sometime on monday when i get to office.

[RiversJohn]

Apologies for my tardiness.

Attached is a reproduction MSTest class with relevant code, example files and also screenshots of the response data during debug for both the expected Ok result and for the borked one. This should be a pretty comprehensive reproduction.

I'd imagine the files themselves aren't the issue as even the borked file can be opened and interpreted fully with image processing libraries, but rather some stateful data regarding the borked file when it was uploaded maybe? But perhaps i'm wrong 🤷‍♂️

Thank you for handling this, my interest in this issue is not academic and the data i need to retrieve that has this issue is not insignificant in quantity.

3283_Reproduction.zip

[RiversJohn]

@ashishdhingra Apologies for pinging, my initial response was a bit slow, would still appreciate if any guidance is at all possible here.

If that isn't possible, i would greatly appreciate any information on what determines whether a given file is returned with an MD5Stream or just a regular ContentLenghtStream?

[ashishdhingra]

@RiversJohn Looking at your repro code, looks like you are using Minio service. Could you confirm if you are able to reproduce the issue using actual AWS S3 service?

Also, could you please clarify the previous comment 3.7.x didn't run at all due to signature changes and would prefer to defer the work but if it's required to resolve i will ofcourse..

Thanks,
Ashish

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.