Bug: install failed

[daddyauden]

What is the bug?

I just follow the steps in https://grafana.com/docs/helm-charts/mimir-distributed/latest/get-started-helm-charts/

steps:

1. kubectl create namespace monitoring
2. helm repo add grafana https://grafana.github.io/helm-charts
3. helm repo update
4. helm -n monitoring install mimir grafana/mimir-distributed
5. kubectl -n monitoring get pods

NAME                                        READY   STATUS             RESTARTS        AGE
mimir-alertmanager-0                        0/1     CrashLoopBackOff   4 (35s ago)    5m57s
mimir-compactor-0                           0/1     Running            3 (3m3s ago)   5m57s
mimir-distributor-7664846b46-5txn7          0/1     CrashLoopBackOff   6 (9s ago)     5m58s
mimir-ingester-zone-a-0                     0/1     Error              5              5m58s
mimir-ingester-zone-b-0                     0/1     CrashLoopBackOff   4 (28s ago)    5m57s
mimir-ingester-zone-c-0                     0/1     CrashLoopBackOff   4 (26s ago)    5m58s
mimir-make-minio-buckets-5.4.0-xfdfl        1/1     Running            4 (55s ago)    5m58s
mimir-minio-5477c4c7b4-z2hzc                1/1     Running            0              5m59s
mimir-minio-post-job-5f28c                  0/2     CrashLoopBackOff   4 (13s ago)    5m57s
mimir-nginx-7c65ff9dfd-k7ct2                1/1     Running            0              5m59s
mimir-overrides-exporter-84fcc88767-9nbhj   1/1     Running            0              5m59s
mimir-querier-56c7f8864b-8dnpx              0/1     CrashLoopBackOff   5 (65s ago)    5m58s
mimir-querier-56c7f8864b-8pcdb              0/1     Running            4 (60s ago)    5m58s
mimir-query-frontend-5995d7b875-wfzdl       0/1     CrashLoopBackOff   6 (7s ago)     5m57s
mimir-query-scheduler-5fdfc8d578-6gztr      1/1     Running            0              5m58s
mimir-query-scheduler-5fdfc8d578-98dhr      1/1     Running            0              5m58s
mimir-rollout-operator-5bc66dbb95-6wvw7     1/1     Running            0              5m59s
mimir-ruler-67596574dc-2svbg                0/1     Running            0              5m59s
mimir-store-gateway-zone-a-0                0/1     Error              4 (66s ago)    5m58s
mimir-store-gateway-zone-b-0                0/1     Running            4 (59s ago)    5m58s
mimir-store-gateway-zone-c-0                0/1     CrashLoopBackOff   4 (37s ago)    5m58s

logs:

1. kubectl -n monitoring logs mimir-alertmanager-0

ts=2025-07-21T03:13:19.762905996Z caller=main.go:225 level=info msg="Starting application" version="(version=2.16.0, branch=HEAD, revision=b4f36da)"
ts=2025-07-21T03:13:19.766667772Z caller=server.go:368 level=info msg="server listening on addresses" http=[::]:8080 grpc=[::]:9095
ts=2025-07-21T03:13:19.783621373Z caller=modules.go:1000 level=info msg="Starting Alertmanager in classic mode"
ts=2025-07-21T03:13:19.783642123Z caller=featurecontrol.go:97 level=warn msg="Classic mode enabled"
ts=2025-07-21T03:13:19.784511799Z caller=memberlist_client.go:463 level=info msg="Using memberlist cluster label and node name" cluster_label= node=mimir-alertmanager-0-193eec75
ts=2025-07-21T03:13:19.78575232Z caller=module_service.go:82 level=info msg=starting module=activity-tracker
ts=2025-07-21T03:13:19.785792886Z caller=module_service.go:82 level=info msg=starting module=sanity-check
ts=2025-07-21T03:13:19.785860859Z caller=sanity_check.go:32 level=info msg="Checking directories read/write access"
ts=2025-07-21T03:13:19.786067798Z caller=sanity_check.go:37 level=info msg="Directories read/write access successfully checked"
ts=2025-07-21T03:13:19.786082278Z caller=sanity_check.go:39 level=info msg="Checking object storage config"
ts=2025-07-21T03:13:29.786788045Z caller=sanity_check.go:115 level=warn msg="Unable to successfully connect to configured object storage (will retry)" err="alertmanager storage: unable to successfully send a request to object storage: Get \"http://mimir-minio.monitoring.svc:9000/mimir-ruler/?location=\": context deadline exceeded"
ts=2025-07-21T03:13:41.246381743Z caller=sanity_check.go:115 level=warn msg="Unable to successfully connect to configured object storage (will retry)" err="alertmanager storage: unable to successfully send a request to object storage: Get \"http://mimir-minio.monitoring.svc:9000/mimir-ruler/?location=\": context deadline exceeded"
ts=2025-07-21T03:13:49.79284971Z caller=sanity_check.go:115 level=warn msg="Unable to successfully connect to configured object storage (will retry)" err="alertmanager storage: unable to successfully send a request to object storage: Get \"http://mimir-minio.monitoring.svc:9000/mimir-ruler/?location=\": dial tcp: lookup mimir-minio.monitoring.svc on 30.30.0.10:53: no such host"
ts=2025-07-21T03:14:04.695803963Z caller=sanity_check.go:115 level=warn msg="Unable to successfully connect to configured object storage (will retry)" err="alertmanager storage: unable to successfully send a request to object storage: Get \"http://mimir-minio.monitoring.svc:9000/mimir-ruler/?location=\": context deadline exceeded"
ts=2025-07-21T03:14:19.330218856Z caller=sanity_check.go:115 level=warn msg="Unable to successfully connect to configured object storage (will retry)" err="alertmanager storage: unable to successfully send a request to object storage: Get \"http://mimir-minio.monitoring.svc:9000/mimir-ruler/?location=\": context deadline exceeded"
ts=2025-07-21T03:14:34.055491168Z caller=sanity_check.go:115 level=warn msg="Unable to successfully connect to configured object storage (will retry)" err="alertmanager storage: unable to successfully send a request to object storage: Get \"http://mimir-minio.monitoring.svc:9000/mimir-ruler/?location=\": context deadline exceeded"
ts=2025-07-21T03:14:39.061363228Z caller=sanity_check.go:44 level=info msg="Object storage config successfully checked"
ts=2025-07-21T03:14:39.061467763Z caller=module_service.go:82 level=info msg=starting module=server
ts=2025-07-21T03:14:39.061591111Z caller=module_service.go:82 level=info msg=starting module=runtime-config
ts=2025-07-21T03:14:39.061610662Z caller=module_service.go:82 level=info msg=starting module=memberlist-kv
ts=2025-07-21T03:14:39.061786526Z caller=mimir.go:958 level=error msg="module failed" module=memberlist-kv err="starting module memberlist-kv: invalid service state: Stopping, expected: Running"
ts=2025-07-21T03:14:39.061896168Z caller=mimir.go:958 level=error msg="module failed" module=runtime-config err="starting module runtime-config: context canceled"
ts=2025-07-21T03:14:39.06206575Z caller=mimir.go:958 level=error msg="module failed" module=alertmanager err="failed to start alertmanager, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: starting module memberlist-kv: invalid service state: Stopping, expected: Running"
ts=2025-07-21T03:14:39.063594844Z caller=server_service.go:55 level=info msg="server stopped"
ts=2025-07-21T03:14:39.063616416Z caller=module_service.go:120 level=info msg="module stopped" module=server
ts=2025-07-21T03:14:39.063653666Z caller=module_service.go:120 level=info msg="module stopped" module=sanity-check
ts=2025-07-21T03:14:39.066895458Z caller=module_service.go:120 level=info msg="module stopped" module=activity-tracker
ts=2025-07-21T03:14:39.066923725Z caller=mimir.go:945 level=info msg="Application stopped"
ts=2025-07-21T03:14:39.067003213Z caller=log.go:134 level=error msg="error running application" err="failed services\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/__w/mimir/mimir/pkg/mimir/mimir.go:1001\nmain.main\n\t/__w/mimir/mimir/cmd/mimir/main.go:227\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:272\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"

2.kubectl -n monitoring logs mimir-minio-5477c4c7b4-z2hzc

INFO: Formatting 1st pool, 1 set(s), 1 drives per set.
INFO: WARNING: Host local has more than 0 drives of set. A host failure will result in data becoming unavailable.
MinIO Object Storage Server
Copyright: 2015-2025 MinIO, Inc.
License: GNU AGPLv3 - https://www.gnu.org/licenses/agpl-3.0.html
Version: RELEASE.2024-12-18T13-15-44Z (go1.23.4 linux/amd64)

API: http://20.20.225.30:9000  http://127.0.0.1:9000 
WebUI: http://20.20.225.30:9001 http://127.0.0.1:9001   

Docs: https://docs.min.io

3. kubectl -n monitoring logs mimir-make-minio-buckets-5.4.0-xfdfl

Connecting to MinIO server: http://mimir-minio:9000
mc: <ERROR> Unable to initialize new alias from the provided credentials. Get "http://mimir-minio:9000/probe-bsign-3m0atuog0wywsmf5ej3wuv41bggas1/?location=": dial tcp: lookup mimir-minio: i/o timeout.
"Failed attempts: 1"
Added `myminio` successfully.
Suspending versioning for 'mimir-tsdb'
mc: <ERROR> Unable to suspend versioning. Get "http://mimir-minio:9000/mimir-tsdb/?location=": dial tcp: lookup mimir-minio: i/o timeout.

4. kubectl -n monitoring logs mimir-querier-56c7f8864b-8dnpx

ts=2025-07-21T03:16:53.026595612Z caller=main.go:225 level=info msg="Starting application" version="(version=2.16.0, branch=HEAD, revision=b4f36da)"
ts=2025-07-21T03:16:53.050443092Z caller=server.go:368 level=info msg="server listening on addresses" http=[::]:8080 grpc=[::]:9095
ts=2025-07-21T03:16:53.070106087Z caller=memberlist_client.go:463 level=info msg="Using memberlist cluster label and node name" cluster_label= node=mimir-querier-56c7f8864b-8dnpx-02d390ec
ts=2025-07-21T03:16:53.078929902Z caller=worker.go:144 level=info msg="Starting querier worker connected to query-scheduler" scheduler=mimir-query-scheduler-headless.monitoring.svc:9095
ts=2025-07-21T03:16:53.080013274Z caller=module_service.go:82 level=info msg=starting module=sanity-check
ts=2025-07-21T03:16:53.080036421Z caller=module_service.go:82 level=info msg=starting module=usage-stats
ts=2025-07-21T03:16:53.080093472Z caller=sanity_check.go:32 level=info msg="Checking directories read/write access"
ts=2025-07-21T03:16:53.080123897Z caller=sanity_check.go:37 level=info msg="Directories read/write access successfully checked"
ts=2025-07-21T03:16:53.08013223Z caller=sanity_check.go:39 level=info msg="Checking object storage config"
ts=2025-07-21T03:16:53.080093738Z caller=module_service.go:82 level=info msg=starting module=activity-tracker
ts=2025-07-21T03:17:03.081199509Z caller=sanity_check.go:115 level=warn msg="Unable to successfully connect to configured object storage (will retry)" err="blocks storage: unable to successfully send a request to object storage: Get \"http://mimir-minio.monitoring.svc:9000/mimir-tsdb/?location=\": context deadline exceeded"
ts=2025-07-21T03:17:13.089521997Z caller=sanity_check.go:44 level=info msg="Object storage config successfully checked"
ts=2025-07-21T03:17:13.089578056Z caller=module_service.go:82 level=info msg=starting module=server
ts=2025-07-21T03:17:13.089628975Z caller=module_service.go:82 level=info msg=starting module=runtime-config
ts=2025-07-21T03:17:13.089658129Z caller=module_service.go:82 level=info msg=starting module=memberlist-kv
ts=2025-07-21T03:17:13.089846552Z caller=mimir.go:958 level=error msg="module failed" module=memberlist-kv err="starting module memberlist-kv: invalid service state: Stopping, expected: Running"
ts=2025-07-21T03:17:13.089893213Z caller=mimir.go:958 level=error msg="module failed" module=ingester-ring err="failed to start ingester-ring, because it depends on module runtime-config, which has failed: context canceled"
ts=2025-07-21T03:17:13.089929974Z caller=mimir.go:958 level=error msg="module failed" module=store-queryable err="failed to start store-queryable, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: starting module memberlist-kv: invalid service state: Stopping, expected: Running"
ts=2025-07-21T03:17:13.089973558Z caller=mimir.go:958 level=error msg="module failed" module=querier err="failed to start querier, because it depends on module distributor-service, which has failed: context canceled"
ts=2025-07-21T03:17:13.089992391Z caller=mimir.go:958 level=error msg="module failed" module=distributor-service err="failed to start distributor-service, because it depends on module ingester-ring, which has failed: context canceled"
ts=2025-07-21T03:17:13.090010802Z caller=mimir.go:958 level=error msg="module failed" module=runtime-config err="starting module runtime-config: context canceled"
ts=2025-07-21T03:17:13.090201408Z caller=server_service.go:55 level=info msg="server stopped"
ts=2025-07-21T03:17:13.090224761Z caller=module_service.go:120 level=info msg="module stopped" module=server
ts=2025-07-21T03:17:13.09027625Z caller=module_service.go:120 level=info msg="module stopped" module=sanity-check
ts=2025-07-21T03:17:13.090494417Z caller=seed.go:127 level=warn msg="failed to read cluster seed file from object storage" err="Get \"http://mimir-minio.monitoring.svc:9000/mimir-tsdb/__mimir_cluster/mimir_cluster_seed.json\": context canceled"
ts=2025-07-21T03:17:13.090546476Z caller=module_service.go:120 level=info msg="module stopped" module=usage-stats
ts=2025-07-21T03:17:13.094391915Z caller=module_service.go:120 level=info msg="module stopped" module=activity-tracker
ts=2025-07-21T03:17:13.094422928Z caller=mimir.go:945 level=info msg="Application stopped"
ts=2025-07-21T03:17:13.094517696Z caller=log.go:134 level=error msg="error running application" err="failed services\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/__w/mimir/mimir/pkg/mimir/mimir.go:1001\nmain.main\n\t/__w/mimir/mimir/cmd/mimir/main.go:227\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:272\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"

I can not provide all the issue, in general I can not deploy.

How to reproduce it?

just do it with steps in https://grafana.com/docs/helm-charts/mimir-distributed/latest/get-started-helm-charts/

What did you think would happen?

deploy succeed.

What was your environment?

mimir-helm(5.7.0)
helm(3.18.4)
k8s(1.33.2)

Any additional context to share?

No response

[karlskewes]

Did the cluster come up?

Were there any more logs for kubectl -n monitoring logs mimir-make-minio-buckets-5.4.0-xfdfl?
There are additional buckets that get created after myminio and in the output above it looks like the job's pod is still running.

How much free cpu & memory does your Kubernetes node have?

I have just successfully installed the helm chart on MacOS using per the linked instructions using:

• kind 0.29.0
• kubernetes server/node 1.33.1
• kubectl 1.33.4
• 8GB VM created with podman

$ kubectl get pods -n mimir-test 
NAME                                        READY   STATUS      RESTARTS      AGE
mimir-alertmanager-0                        1/1     Running     0             13m
mimir-compactor-0                           1/1     Running     0             13m
mimir-distributor-687dc5d85d-4sn5l          1/1     Running     0             13m
mimir-ingester-zone-a-0                     1/1     Running     0             13m
mimir-ingester-zone-b-0                     1/1     Running     1 (12m ago)   13m
mimir-ingester-zone-c-0                     1/1     Running     0             13m
mimir-make-minio-buckets-5.4.0-cjmxf        0/1     Completed   0             13m
mimir-minio-5477c4c7b4-mc7dw                1/1     Running     0             13m
mimir-nginx-5d84ff87d-27qsn                 1/1     Running     0             13m
mimir-overrides-exporter-5dfd8b9988-8drqc   1/1     Running     0             13m
mimir-querier-58df98b6db-dpn8j              1/1     Running     0             13m
mimir-querier-58df98b6db-tt4vq              1/1     Running     0             13m
mimir-query-frontend-8656d84cc4-zn8df       1/1     Running     0             13m
mimir-query-scheduler-7b97f4d7bb-8zlg4      1/1     Running     0             13m
mimir-query-scheduler-7b97f4d7bb-bzdth      1/1     Running     0             13m
mimir-rollout-operator-654f6cfcf5-sbnnw     1/1     Running     0             13m
mimir-ruler-784cdd9b96-9ptr5                1/1     Running     0             13m
mimir-store-gateway-zone-a-0                1/1     Running     0             13m
mimir-store-gateway-zone-b-0                1/1     Running     0             13m
mimir-store-gateway-zone-c-0                1/1     Running     0             13m

[aknuds1]

Converting this into a discussion, as I'm currently not seeing any evidence of a bug.