Getting error trying to upload to S3 compatible bucket (Netapp)

[tannerrs]

After configuring the Mimir helm chart to use S3, the ingester logs show this error:

level=error ts=2022-08-22T12:58:07.232034936Z caller=shipper.go:160 user=anonymous msg="shipping failed" block=01GB2KBY3AR8EA6DAB6DRPNJ61 err="upload index: upload file /data/tsdb/anonymous/01GB2KBY3AR8EA6DAB6DRPNJ61/index as 01GB2KBY3AR8EA6DAB6DRPNJ61/index: upload s3 object: The Content-MD5 you specified is not valid."

When trying the mimirtools bucket validation, I get the exact same message.

My config for storage is:

    blocks_storage:
      backend: s3
      bucket_store:
        chunks_cache:
          backend: memcached
          memcached:
            addresses: <memcache address>
            timeout: 450ms
        index_cache:
          backend: memcached
          memcached:
            addresses: <memcach address>
        metadata_cache:
          backend: memcached
          memcached:
            addresses: <memcache address>
        sync_dir: /data/tsdb-sync
      s3:
        endpoint: <endpoint>
        bucket_name: dev-mimir-tsdb
        access_key_id: < access ID>
        secret_access_key: <access secret>
        insecure: true        
      tsdb:
        dir: /data/tsdb

Any help appreciated.

[pstibrany]

Could you try to run mimirtool bucket-validation command (try --help and --bucket-config-help for list of parameters) to check compatibility of your bucket?

[pracucci]

Which S3-compatible storage are you using? I'm wondering if it's something I can try myself to debug it.

[tannerrs]

@pracucci The S3 implementation is from Netapp. I can try a proxy, but will take a few days to setup.

[pracucci]

The S3 implementation is from Netapp

Ok, unfortunately not something I can try myself.

[pracucci]

I checked the Minio client code we use for S3 and it doesn't send the MD5 on upload by default (but it's a feature that can be enabled). If the MD5 is required but not sent in the request then the S3 backend should return a different error than what you get (The Content-MD5 you specified is not valid).

I'm wondering if Netapp actually requires MD5 for uploads. Is it something you can check in the Netapp doc?

[tannerrs]

Apologies for the delay. I've engaged the vendor and will let you know what they find. Thanks.