Mimir fails to start with memberlist-kv module error

[denniseng88]

Hello,

I am deploying Mimir v2.12.0 as a single-instance application within Azure Container Apps. While the same configuration successfully runs in one Azure environment, it consistently fails to start in another Azure Container App environment.

Error Logs:

2025-08-04T10:34:28.909281786Z ts=2025-08-04T10:34:28.906820693Z caller=sanity_check.go:44 level=info msg="Object storage config successfully checked"
2025-08-04T10:34:28.909284086Z ts=2025-08-04T10:34:28.906865393Z caller=module_service.go:82 level=info msg=starting module=server
2025-08-04T10:34:28.909286286Z ts=2025-08-04T10:34:28.906928793Z caller=module_service.go:82 level=info msg=starting module=memberlist-kv
2025-08-04T10:34:28.909293186Z ts=2025-08-04T10:34:28.907031692Z caller=mimir.go:880 level=error msg="module failed" module=memberlist-kv err="starting module memberlist-kv: invalid service state: Stopping, expected: Running"
2025-08-04T10:34:28.909295686Z ts=2025-08-04T10:34:28.907061392Z caller=mimir.go:880 level=error msg="module failed" module=compactor err="failed to start compactor, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: starting module memberlist-kv: invalid service state: Stopping, expected: Running"
2025-08-04T10:34:28.909297986Z ts=2025-08-04T10:34:28.907075392Z caller=mimir.go:880 level=error msg="module failed" module=ingester-ring err="failed to start ingester-ring, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: starting module memberlist-kv: invalid service state: Stopping, expected: Running"
2025-08-04T10:34:28.909309986Z ts=2025-08-04T10:34:28.907085292Z caller=mimir.go:880 level=error msg="module failed" module=store-gateway err="failed to start store-gateway, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: starting module memberlist-kv: invalid service state: Stopping, expected: Running"
2025-08-04T10:34:28.909313386Z ts=2025-08-04T10:34:28.907095292Z caller=mimir.go:880 level=error msg="module failed" module=query-frontend err="failed to start query-frontend, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: starting module memberlist-kv: invalid service state: Stopping, expected: Running"
2025-08-04T10:34:28.909316886Z ts=2025-08-04T10:34:28.907104292Z caller=mimir.go:880 level=error msg="module failed" module=ruler err="failed to start ruler, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: starting module memberlist-kv: invalid service state: Stopping, expected: Running"
2025-08-04T10:34:28.909320986Z ts=2025-08-04T10:34:28.907103992Z caller=module_service.go:120 level=info msg="module stopped" module=active-groups-cleanup-service
2025-08-04T10:34:28.909324486Z ts=2025-08-04T10:34:28.907113792Z caller=mimir.go:880 level=error msg="module failed" module=ingester-service err="failed to start ingester-service, because it depends on module ingester-ring, which has failed: context canceled"
2025-08-04T10:34:28.909331386Z ts=2025-08-04T10:34:28.907122392Z caller=mimir.go:880 level=error msg="module failed" module=querier err="failed to start querier, because it depends on module distributor-service, which has failed: context canceled"
2025-08-04T10:34:28.909334786Z ts=2025-08-04T10:34:28.907128292Z caller=mimir.go:880 level=error msg="module failed" module=store-queryable err="failed to start store-queryable, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: starting module memberlist-kv: invalid service state: Stopping, expected: Running"
2025-08-04T10:34:28.909337886Z ts=2025-08-04T10:34:28.907134192Z caller=mimir.go:880 level=error msg="module failed" module=distributor-service err="failed to start distributor-service, because it depends on module ingester-ring, which has failed: context canceled"
2025-08-04T10:34:28.909340986Z ts=2025-08-04T10:34:28.907306892Z caller=server_service.go:55 level=info msg="server stopped"
2025-08-04T10:34:28.909344086Z ts=2025-08-04T10:34:28.907318392Z caller=module_service.go:120 level=info msg="module stopped" module=server
2025-08-04T10:34:28.909347686Z ts=2025-08-04T10:34:28.907363291Z caller=module_service.go:120 level=info msg="module stopped" module=sanity-check
2025-08-04T10:34:28.909352586Z ts=2025-08-04T10:34:28.908995687Z caller=module_service.go:120 level=info msg="module stopped" module=activity-tracker
2025-08-04T10:34:28.909354986Z ts=2025-08-04T10:34:28.909018987Z caller=mimir.go:869 level=info msg="Application stopped"
2025-08-04T10:34:28.909357686Z ts=2025-08-04T10:34:28.909079687Z caller=log.go:103 level=error msg="error running application" err="failed services\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/go/src/github.com/grafana/mimir/pkg/mimir/mimir.go:923\nmain.main\n\t/go/src/github.com/grafana/mimir/cmd/mimir/main.go:228\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"

This is the Dockerfile:

FROM ubuntu:22.04

ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update && apt-get install -y \
    curl \
    && rm -rf /var/lib/apt/lists/*

RUN curl -fLo /usr/local/bin/mimir https://github.com/grafana/mimir/releases/download/mimir-2.12.0/mimir-linux-amd64 && \
    chmod +x /usr/local/bin/mimir

EXPOSE 9009

CMD ["/usr/local/bin/mimir", "-config.file=/etc/mimir/mimir.yaml"]

and this is the mimir.yaml:

multitenancy_enabled: false

activity_tracker: {}

alertmanager: {}

alertmanager_storage:
  backend: local

server:
  http_listen_port: 9009

  # Configure the server to allow messages up to 100MB.
  grpc_server_max_recv_msg_size: 104857600
  grpc_server_max_send_msg_size: 104857600
  grpc_server_max_concurrent_streams: 1000

distributor:
  pool:
    health_check_ingesters: true

ingester_client:
  grpc_client_config:
    grpc_compression: gzip
    max_recv_msg_size: 104857600
    max_send_msg_size: 104857600

ingester:
  ring:
    final_sleep: 0s
    kvstore:
      store: inmemory
    min_ready_duration: 0s
    num_tokens: 512
    replication_factor: 1

blocks_storage:
  backend: azure
  azure:
    container_name: <MIMIR_CONTAINER>
    account_name: <STORAGE_ACCOUNT_NAME>
    account_key: <STORAGE_ACCOUNT_KEY>
    endpoint_suffix: "blob.core.windows.net"
  tsdb:
    dir: /tmp/tsdb

compactor:
  sharding_ring:
    kvstore:
      store: inmemory

ruler:
  enable_api: true

ruler_storage:
  backend: filesystem
  local:
    directory: /tmp/mimir/rules


limits:
  ingestion_burst_size: 500000
  ingestion_rate: 250000
  compactor_blocks_retention_period: 1y
  
usage_stats:
  enabled: false

Given that this configuration works in another Azure Container App environment, the issue might be related to specific network or resource configurations within the problematic Azure environment that affect the memberlist-kv module's startup sequence.

Even though I am running Mimir in a monolithic configuration (single instance), the logs indicate a dependency on memberlist-kv. Could you clarify why this module is required in a monolithic setup and if there are specific configurations to mitigate issues related to its startup in such environments?

Any insights or suggestions for debugging this would be highly appreciated.

[denniseng88]

We are still experiencing this issue. We would be very grateful if you could help us.

[denniseng88]

I solve the problem by adding the following configuration in the mimir.yaml file. advertise_addr should be the outbounded ip of the container app.

memberlist:
advertise_addr: {CONTAINER_OUTBOUNDED_IP}