Unable to successfully connect to s3 block storage bucket: ACCESS DENIED

[yebolenko]

Describe the bug

Mimir fails to access S3 bucket with ACCESS DENIED error
It's an error
ts=2024-06-26T19:23:25.609453896Z caller=sanity_check.go:115 level=warn msg="Unable to successfully connect to configured object storage (will retry)" err="2 errors: blocks storage: unable to successfully send a request to object storage: Get "http://mimir-blocks.s3.cn-north-1.amazonaws.com.cn/sanity-check-at-startup": context deadline exceeded; ruler storage: unable to successfully send a request to object storage: Get "[http://mimir-ruler.s3.cn-north-1.amazonaws.com.cn/sanity-check-at-startup](http://mimir-ruler.s3.cn-north-1.amazonaws.com.cn/sanity-check-at-startup%5C)": context deadline exceeded"

To Reproduce

Steps to reproduce the behavior:
Just deploy mimir
I used build which supports a new config feature dualstack_enabled to disable dual stack endpoint,
Mimir release is grafana/mimir:r296-3410422

Expected behavior

Mimir can successfully connect to block storage and ruler buckets

A clear and concise description of what you expected to happen.

Environment

Deployment is done on k8s using helm on closed AWS VPC (without NAT)
it's my setup for mimir-blocks:
blocks_storage:
backend: s3
s3:
bucket_name: "mimir-blocks"
endpoint: "s3.cn-north-1.amazonaws.com.cn"
region: "cn-north-1"
insecure: true
dualstack_enabled: false
Is my endpoint set as expected?

Additional Context

[narqo]

Mimir fails to access S3 bucket with ACCESS DENIED error

From your description, it's not clear if dualstack_enabled: false or something in the latest version of mimir (e.g. the update in the vendored s3 client) caused the issue. The "access denied" suggests there might be something wrong with the AWS credentials or IAM policies, that Mimir uses to access the bucket.

1. Do the credentials you provide work when you tested them outside Mimir's context?
2. Did it work before you introduced dualstack_enabled: false?

Also, I'm not sure why do you need insecure: true. This causes s3 client to use HTTP instead of HTTPS, and I can't tell AWS allows that.

[Kozatoo]

Where has it been explicitly said that you have an Access Denied message?

[yebolenko]

I'm sorry for misunderstanding. insecure is false
We have an issue with China when dualstack is enabled, so I needed to disable it.
I was not able to cause it to work on china.
regarding credentials: we use mimir service account which has trusted identity on AWS IAM (IRSA)
Exactly the same config works ok on global.

[yebolenko]

by the way, grafana and loki works ok on china, only mimir fails
I have designated that on all loki pods standard AWS Env vars appear, e.g. AWS_DEFAULT_REGION, AWS_REGION, AWS_WEB_IDENTITY_TOKEN_FILE, etc.
On mimir pods - no AWS ENV vars
Do I miss something?

[yebolenko]

any progress?