Unable to push data from prometheus to AWS s3

[vijaybiradar]

Unable to push data from prometheus to AWS s3

this is my mimir yaml file

metaMonitoring:
  serviceMonitor:
    enabled: true
    labels:
      release: kube-prometheus-stack
mimir:
  structuredConfig:
    alertmanager_storage:
      s3:
        access_key_id: access_key
        bucket_name: mimir-alert
        endpoint: s3.eu-central-1.amazonaws.com
        secret_access_key: secret_access
    blocks_storage:
      backend: s3
      bucket_store:
        sync_dir: /data/tsdb-sync
      s3:
        access_key_id: access_key
        bucket_name: mimir-tsdb
        endpoint: s3.eu-central-1.amazonaws.com
        secret_access_key: secret_access
      tsdb:
        dir: /data/tsdb
    compactor:
      data_dir: /data
    frontend:
      align_queries_with_step: true
      log_queries_longer_than: 10s
    ingester:
      instance_limits:
        max_ingestion_rate: 0
      ring:
        final_sleep: 0s
        num_tokens: 512
    ingester_client:
      grpc_client_config:
        max_recv_msg_size: 104857600
        max_send_msg_size: 104857600
    limits:
      ingestion_rate: 40000
      max_global_series_per_metric: 0
      max_global_series_per_user: 0
    memberlist:
      abort_if_cluster_join_fails: false
      compression_enabled: false
    ruler:
      alertmanager_url: dnssrvnoa+http://_http-metrics._tcp.{{ template "mimir.fullname"
        . }}-alertmanager-headless.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain
        }}/alertmanager
      enable_api: true
      rule_path: /data
    ruler_storage:
      s3:
        access_key_id: access_key
        bucket_name: mimir-ruler-test
        endpoint: s3.eu-central-1.amazonaws.com
        secret_access_key: secret_access
    runtime_config:
      file: /var/{{ include "mimir.name" . }}/runtime.yaml
    server:
      grpc_server_max_concurrent_streams: 1000
      grpc_server_max_recv_msg_size: 104857600
      grpc_server_max_send_msg_size: 104857600
minio:
  enabled: false
querier:
  replicas: 3

this is my prometheus yaml file

server:
  remoteWrite:
    - url: http://mimir-distributor:8080/api/v1/push
  remoteRead:
    - url:  http://mimir-test-nginx.mimir-test.svc:80/prometheus

[pstibrany]

How to troubleshoot this:

1. Make sure Prometheus is successfully sending data to Mimir.

• There should be no errors in Prometheus logs
• You can check /distributor/all_user_stats endpoint on distributors component to see if ingesters are receiving expected data
• Check ingester logs to see if ingesters create TSDB blocks. They will have similar (but not exact same) names to block names in your screenshot (eg. 01GBSRXTM4P3P8VCHYP9QAA48G). Message looks like this:

level=info ts=2022-09-05T07:00:43.182077973Z caller=compact.go:709 user=132829 msg="write block" mint=1662350400000 maxt=1662357600000 ulid=01GC677358RXNBB74RSAY3XP1P duration=389.381088ms

2. If ingesters are receiving data and creating blocks, then they should upload them soon after creation. Ingester will log a message indicating that block was uploaded, and it looks like this:

level=info ts=2022-09-05T07:01:42.588664143Z caller=shipper.go:186 user=132829 msg="upload new block" id=01GC677358RXNBB74RSAY3XP1P

Try to search for block ID found in previous step. Instead of "upload new block" you may find an error saying why ingester failed to upload the block.