Connecting Grafana Mimir with etcd

[ponsakorn30214]

I've been trying to use deduplication feature in Grafana Mimir to deduplicate 2 Prometheus input as documented in : this documentation
I've put 2 Prometheus, Grafana Mimir, and, Etcd on one docker-compose.
My setting for Grafana Mimir are as followed:

distributor:
  ha_tracker: 
    enable_ha_tracker: true
    kvstore:
      store: etcd
      etcd:
        endpoints: ["host.docker.internal:2379"]

But after running Mimir, the app throws:

{"level":"warn","ts":"2022-07-20T02:41:18.054Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0002c3c00/#initially=[host.docker.internal:2379]","attempt":0,"error":"rpc error: code = Canceled desc = context canceled"}

I've also tried using localhost/container_name as endpoints in the configuration.
Are there any resources that I can read to resolve this issue am I doing the configuration wrong?

[pracucci]

Can you confirm that etcd is listening on host.docker.internal:2739 when resolved inside the Mimir container? I just would like to double check it, before we deep dive into it.

[ponsakorn30214]

I've tested both host.docker.internal:2739 and localhost:2379 from inside the container using netcat, The connection seems to be working.
Could the issue be about docker networking?

Here is the docker-comtainer setting for both etcd and mimir

etcd:
    image: quay.io/coreos/etcd:v2.3.8
    container_name: etcd
    networks:
      - kong-net
    ports:
    - 2379:2379
    - 2380:2380
    - 4001:4001
    volumes:
      - ./keys:/etc/ssl/certs
    command: "-name etcd0 -advertise-client-urls http://etcd:2379,http://etcd:4001 -listen-client-urls http://0.0.0.0:2379,http://0.0.0.0:4001 -initial-advertise-peer-urls http://etcd:2380 -listen-peer-urls http://0.0.0.0:2380 -initial-cluster-token etcd-cluster-1 -initial-cluster etcd0=http://etcd:2380 -initial-cluster-state new"

  mimir:
    image: grafana/mimir:latest
    container_name: grafana-mimir
    networks:
      - kong-net
    ports:
    - 9009:9009
    volumes:
      - ./cfg/demo.yaml:/etc/mimir/demo.yaml
      - ./mimir:/tmp/mimir/
    command: "--config.file /etc/mimir/demo.yaml"
    depends_on:
      - etcd

[pracucci]

The error you get is:

rpc error: code = Canceled desc = context canceled

This means the request has been canceled, but it doesn't tell us why or by who. Could you share more Mimir logs and your Mimir configuration file too (redacted of any sensitive information), please? I would like to check if anything stands out.