push error: the sample has been rejected because another sample with the same timestamp, but a different value, has already been ingested (err-mimir-sample-duplicate-timestamp)

[dbrennand]

Describe the bug

Hi all,

I'm not sure if this is an issue specific to Mimir or it's an issue with my setup, so I apologise in advance if that's the case.

I've recently started using Mimir and I've got it up and running with container image tag: r194-61b589d

I've noticed in the logs that I get many errors like:

level=error ts=2022-07-13T20:07:50.151240215Z caller=push.go:96 user=anonymous msg="push error" err="rpc error: code = Code(400) desc = user=anonymous: the sample has been rejected because another sample with the same timestamp, but a different value, has already been ingested (err-mimir-sample-duplicate-timestamp). The affected sample has timestamp 2022-07-13T20:07:44.457Z and is from series {__name__=\"container_spec_cpu_period\", id=\"/system.slice/systemd-journal-flush.service\", instance=\"cadvisor:8080\", job=\"cadvisor\"}"
level=error ts=2022-07-13T20:10:45.024019274Z caller=push.go:96 user=anonymous msg="push error" err="rpc error: code = Code(400) desc = user=anonymous: the sample has been rejected because another sample with the same timestamp, but a different value, has already been ingested (err-mimir-sample-duplicate-timestamp). The affected sample has timestamp 2022-07-13T20:10:44.457Z and is from series {__name__=\"container_spec_cpu_period\", id=\"/system.slice/collectd.service\", instance=\"cadvisor:8080\", job=\"cadvisor\"}"
level=error ts=2022-07-13T20:10:50.037245415Z caller=push.go:96 user=anonymous msg="push error" err="rpc error: code = Code(400) desc = user=anonymous: the sample has been rejected because another sample with the same timestamp, but a different value, has already been ingested (err-mimir-sample-duplicate-timestamp). The affected sample has timestamp 2022-07-13T20:10:44.457Z and is from series {__name__=\"container_start_time_seconds\", id=\"/system.slice/systemd-tmpfiles-setup.service\", instance=\"cadvisor:8080\", job=\"cadvisor\"}"
level=error ts=2022-07-13T20:15:45.063296879Z caller=push.go:96 user=anonymous msg="push error" err="rpc error: code = Code(400) desc = user=anonymous: the sample has been rejected because another sample with the same timestamp, but a different value, has already been ingested (err-mimir-sample-duplicate-timestamp). The affected sample has timestamp 2022-07-13T20:15:44.457Z and is from series {__name__=\"container_spec_memory_limit_bytes\", id=\"/system.slice/systemd-remount-fs.service\", instance=\"cadvisor:8080\", job=\"cadvisor\"}"
level=error ts=2022-07-13T20:15:50.072482279Z caller=push.go:96 user=anonymous msg="push error" err="rpc error: code = Code(400) desc = user=anonymous: the sample has been rejected because another sample with the same timestamp, but a different value, has already been ingested (err-mimir-sample-duplicate-timestamp). The affected sample has timestamp 2022-07-13T20:15:44.457Z and is from series {__name__=\"container_spec_cpu_shares\", id=\"/system.slice/avahi-daemon.service\", instance=\"cadvisor:8080\", job=\"cadvisor\"}"

I have Prometheus configured to push to Mimir using remote_write and I see the culprit instance is cadvisor but I haven't got any special cadvisor configuration. Just started with all defaults:

cadvisor_image: gcr.io/cadvisor/cadvisor
cadvisor_image_tag: v0.39.3
prometheus_image: quay.io/prometheus/prometheus
prometheus_image_tag: v2.36.2
mimir_image: grafana/mimir
mimir_image_tag: r194-61b589d

- name: Create cadvisor Docker container
  community.docker.docker_container:
    name: cadvisor
    image: "{{ cadvisor_image }}:{{ cadvisor_image_tag }}"
    restart_policy: unless-stopped
    networks:
      - name: metrics
    exposed_ports:
      - 8080
    privileged: yes
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
      - /etc/machine-id:/etc/machine-id:ro
    devices:
      - /dev/kmsg
    state: started

- name: Create Prometheus Docker container
  community.docker.docker_container:
    name: prometheus
    image: "{{ prometheus_image }}:{{ prometheus_image_tag }}"
    restart_policy: unless-stopped
    networks:
      - name: metrics
    exposed_ports:
      - 9090
    volumes:
      - ~/prometheus/config/prometheus.yml:/etc/prometheus/prometheus.yml
    state: started

- name: Create Mimir Docker container
  community.docker.docker_container:
    name: mimir
    image: "{{ mimir_image }}:{{ mimir_image_tag }}"
    restart_policy: unless-stopped
    networks:
      - name: mimir
      - name: metrics
    exposed_ports:
      - 9009
    command: --config.file=/etc/mimir/mimir.yml
    volumes:
      - ~/mimir/config/mimir.yml:/etc/mimir/mimir.yml
      - mimir:/data
    state: started

Are these error logs anything to be concerned about?

To Reproduce

1. Start cAdvisor container
2. Start Prometheus container
3. Start Mimir container

Log errors shown above begin to occur.

Expected behaviour

No errors in the Mimir logs.

Environment

• Infrastructure: Docker version 20.10.17, build 100c701
• Deployment tool: Ansible

Additional Context

Prometheus config:

# https://prometheus.io/docs/prometheus/latest/configuration/configuration/
remote_write:
  - url: https://mimir.domain/api/v1/push

scrape_configs:
  # https://github.com/google/cadvisor/blob/master/docs/runtime_options.md#http
  - job_name: cadvisor
    static_configs:
    - targets: ['cadvisor:8080']

Mimir config:

# https://github.com/grafana/mimir/tree/main/docs/sources/tutorials/play-with-grafana-mimir
# https://grafana.com/docs/mimir/latest/operators-guide/configuring/reference-configuration-parameters/
# https://grafana.com/docs/mimir/latest/operators-guide/getting-started/
multitenancy_enabled: false

server:
  http_listen_port: 9009
  log_level: error

common:
  storage:
      backend: s3
      s3:
        endpoint: 
        bucket_name: 
        access_key_id: 
        secret_access_key: 
        # https://grafana.com/docs/mimir/latest/operators-guide/securing/encrypting-data-at-rest/
        sse:
          type: "SSE-S3"

# Blocks storage requires a prefix when using a common object storage bucket
blocks_storage:
  storage_prefix: blocks
  tsdb:
    dir: /data/ingester

store_gateway:
  sharding_ring:
    replication_factor: 1

compactor:
  sharding_ring:
    kvstore:
      store: memberlist

distributor:
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: memberlist

ingester:
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: memberlist
    replication_factor: 1

[pracucci]

This is an issue we've already seen in the past for the cAdvisor metrics because of how they're exported. See:
#2131 (comment)

[dbrennand]

Hi @pracucci Thank you for your response. Modifying the cAdvisor housekeeping_interval as suggested in #2131 (comment) resolved the issue. Thank you!