Ingesters cannot connect to S3 storage - Compactor can

[Andysimcoe]

I'm using 2.1.0, running in AWS ECS cluster, the EC2 instances have an IAM role attached which allows both listing of the bucket and all actions on any objects.

If I upload a file from the host to the bucket, the compactor discovers this as a user and proceeds to create a directory with that name and uploads the file bucket-index.json.gz. This at least confirms the IAM role is fine and in theory the config is too.

The exact same configuration (Mimir conf is on a shared EFS and mounted as a volume for all containers) doesn't work for ingesters, the logs for the ingester:

level=warn ts=2022-06-19T18:58:27.064578882Z caller=sanity_check.go:116 msg="Unable to successfully connect to configured object storage (will retry)" err="blocks storage: unable to successfully send a request to object storage: Get \"https://mimir-bucket.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded"

The storage config:

blocks_storage:
  backend: s3
  s3:
    endpoint: s3.us-east-1.amazonaws.com
    region: us-east-1
    bucket_name: mimir-bucket
    insecure: false
    signature_version: v4
    http:
      insecure_skip_verify: true
    sse:
      type: "SSE-S3"

It seems strange to me the compactor doesn't have a problem but the ingesters never pass the sanity check. Is there a step I should be doing? From what I can tell from other posts no, that should be it.

[pracucci]

The issue you're reporting made me realise that we're not running the sanity check on the compactor:

mimir/pkg/mimir/sanity_check.go

Line 131 in b13d2df

if cfg.isAnyModuleEnabled(All, Ingester, Querier, Ruler, StoreGateway) {

This explains why the compactor starts, while ingesters don't, but still doesn't explain why the sanity check fails.

level=warn ts=2022-06-19T18:58:27.064578882Z caller=sanity_check.go:116 msg="Unable to successfully connect to configured object storage (will retry)" err="blocks storage: unable to successfully send a request to object storage: Get \"https://mimir-bucket.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded"

The "context deadline exceeded" error is a timeout error. It means the sanity check hasn't been able to GET https://mimir-bucket.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup within 10 seconds (this timeout is hardcoded). Unfortunately, this log is not telling us anything more useful on the "why".

If I upload a file from the host to the bucket, the compactor discovers this as a user and proceeds to create a directory with that name and uploads the file bucket-index.json.gz. This at least confirms the IAM role is fine and in theory the config is too.

I agree this confirms config for compactor looks good.

Question: The same exact sanity check of ingesters is also run by queriers, rulers and store-gateways. Do they also fail the sanity check?

[pracucci]

So reading the code you link to the check is for a file it expects not to be there. Is it expecting the same error if connecting directly to a bucket as opposed to a service like MinIO?
[...]
So it makes me think the compactor works because it never does this check, but then, how is it working for others? The only thing I can think of is they're using MinIO and that spits out a different error or... Something is different about my environment.

MinIO exposes an S3-compatible API and behavior is expected to be the same. At Grafana Labs, we run Mimir (and so this sanity check) on AWS S3, Google Cloud Storage (GCS) and Azure Blob Storage, and it works for all the three (other than we're aware of several Mimir community users running it on S3), so I would exclude this check doesn't work at all on S3.

I double checked the code, and the S3 client we use in the sanity check is initialised the same exact way as in other Mimir components, including the compactor. I can't see any obvious reason why compactor should run fine with regards to object storage, but sanity check doesn't.

I'm assuming the S3 latency is not terribly high, and 10 seconds are enough to run a simple authenticated GET operation on S3 (sanity check has an hardcoded timeout of 10s). As a test, I've built the Mimir docker image out of main with the timeout increased from 10s to 60s (changes). Docker image is at: docker.io/grafana/mimir:increase-sanity-check-bucket-client-timeout-17857c122

Could you give it a try?

Keep in mind earlier today we merged a PR #2144 to run the sanity check in the compactor too, so compactor is expected to run sanity check too in the custom docker image I built for you.

[Andysimcoe]

Yeah compactor running the same check now and failing for compactor:

level=info ts=2022-06-20T17:08:08.882172766Z caller=main.go:202 msg="Starting application" version="(version=2.1.0, branch=increase-sanity-check-bucket-client-timeout, revision=17857c122)"
level=info ts=2022-06-20T17:08:08.883162493Z caller=server.go:285 http=[::]:9009 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2022-06-20T17:08:08.887343509Z caller=module_service.go:82 msg=initialising module=sanity-check
level=info ts=2022-06-20T17:08:08.88742067Z caller=sanity_check.go:33 msg="Checking directories read/write access"
level=info ts=2022-06-20T17:08:08.887495511Z caller=module_service.go:82 msg=initialising module=activity-tracker
level=info ts=2022-06-20T17:08:08.904544489Z caller=sanity_check.go:38 msg="Directories read/write access successfully checked"
level=info ts=2022-06-20T17:08:08.904587539Z caller=sanity_check.go:40 msg="Checking object storage config"
level=warn ts=2022-06-20T17:10:12.999505442Z caller=sanity_check.go:116 msg="Unable to successfully connect to configured object storage (will retry)" err="blocks storage: unable to successfully send a request to object storage: Get \"https://mimir-bucket.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded"

I tried debug log level but nothing extra. Latency shouldn't be an issue, it's all in us-east-1 region, I'm out of ideas. The buckets have encryption forced by config rule but can't think of anything else special about them as as said, without the sanity check the compactor was working.

[Andysimcoe]

@pracucci for what it's worth, it looks like each of our tenants is below ~10TBs of storage (we only want 1 year retention), we can use EFS and set the ingesters to a short retention period and then the compactor for the 1 year. As there's no benefit to going to the ingesters rather than store-gateways which are ultimately retrieving from the same type of storage (so it limits the duplicates of data).

So we can still move forward with it. But... I can't stop thinking about why this doesn't work, I'm tempted to build my own version and remove the sanity check just to see if it works. Then I might be able to workout what it doesn't like about our environment.

[pracucci]

@Andysimcoe Have you figured out the root cause here? Could you share your learnings, please?

[Andysimcoe]

No, just as above I've been using cross-AZ EFS for storage while working on other issues, like how the ingesters can replay a WAL while running in ECS without creating multiple separate services, which I think I have working solution for - it's not quite as easy as a K8s cluster. The other issue is we're running all components in Fargate apart from the ingesters, so the ingesters are running in bridge network type for Docker in EC2 instances (I don't want to use awsvpc mode to cut down on the IP addresses consumed).
Meaning they can resolve the DNS to join the memberlist cluster but they'd advertise a port and IP only local components can connect to.

The short solution to that was to write my own entrypoint, I wrote a small Go binary which retrieves the correct port (as it's dynamic ports) and host IP accessing the metadata URI environment variable in ECS containers. It then supplies these values as CLI flags to the mimir binary. Again it seems to be working fine.

I may come back to this issue if I can get some time but mostly been trying to get this running to see if there's any blockers preventing us from going to production with it. We can get away with not using S3 at all though so it got relegated as far as priorities go.