Gaps when displaying metrics in Grafana

[andreimiclea99]

I am trying Grafana Mimir as a POC for my current project.

I encountered a gap in the metrics, (on every grafana dashboards) when trying to display the metrics for the last 12 hours, the metrics for around 2 hours are missing.
The gap is continuously appearing only when I try to query the metrics for the last 12 hours, see the pictures below:

another time, but the same issue:

Even when querying the metrics for the last 7 days i still see the same issue:

As you can see the problem is always there, the metrics are not lost but for some reason are not displaying properly.

I deployed the mimir-distributed helm chart and here are the configs:

   blocks_storage:
      backend: gcs
      bucket_store:
        sync_dir: /data/tsdb-sync
      gcs:
        bucket_name: ""
        service_account: ""
      tsdb:
        retention_period: 6h
        dir: /data/tsdb
    compactor:
      max_closing_blocks_concurrency: 2
      max_opening_blocks_concurrency: 4
      symbols_flushers_concurrency: 4
      data_dir: "/data"
      sharding_ring:
        wait_stability_min_duration: 1m
    frontend:
      parallelize_shardable_queries: false
      align_queries_with_step: true
      log_queries_longer_than: 10s
      max_outstanding_per_tenant: 4096
      split_queries_by_interval: 0
      scheduler_address: grafana-mimir-query-scheduler-headless.monitoring.svc:9095
    frontend_worker:
      grpc_client_config:
        max_send_msg_size: 419430400
      scheduler_address: grafana-mimir-query-scheduler-headless.monitoring.svc:9095
    distributor:
      ha_tracker:
        enable_ha_tracker: true
        kvstore:
          store: consul
          consul:
            host: http://consul-consul-expose-servers.monitoring.svc.cluster.local:8500
    ingester:
      ring:
        final_sleep: 0s
        num_tokens: 512
        tokens_file_path: /data/tokens
        unregister_on_shutdown: true
    ingester_client:
      grpc_client_config:
        max_recv_msg_size: 104857600
        max_send_msg_size: 104857600
    limits:
      accept_ha_samples: true
      ha_cluster_label: lh_cluster
      ha_replica_label: replica
      max_total_query_length: 1200h
      max_query_parallelism: 240
      max_cache_freshness: 10m
      max_label_names_per_series: 120
      max_global_series_per_user: 12000000
      ingestion_rate: 500000
      ingestion_burst_size: 10000000
      out_of_order_time_window: 480m
      compactor_blocks_retention_period: 336h
    memberlist:
      abort_if_cluster_join_fails: false
      compression_enabled: false
      join_members:
      - dns+grafana-mimir-gossip-ring.monitoring.svc.cluster.local:7946
    querier:
      max_concurrent: 2048
    query_scheduler:
      max_outstanding_requests_per_tenant: 4096
    server:
      grpc_server_max_concurrent_streams: 1000
      grpc_server_max_connection_age: 2m
      grpc_server_max_connection_age_grace: 5m
      grpc_server_max_connection_idle: 1m
      http_server_read_timeout: 3m
      http_server_write_timeout: 3m

There are memory issues in any of the components nor clear error logs in the mimir components.

Other meaningful information:

I have tried to adjust the configs in different ways with no luck, in the begging I tried it without the query-scheduler, but the behaviour was the same.

Thank you.

[dimitarvdimitrov]

can you try increasing the blocks_storage.tsdb.retention_period to 12h? I suspect the ingesters are deleting the blocks locally 6h after uploading them (so around 9h after the metrics in the block were ingested) but the store-gateways aren't discovering these yet.

Another option is to decrease the store_gateway.bucket_store.ignore_blocks_within and querier. query_store_after and to maybe 6h. This will trade the load on ingesters with load on the store-gateways. I think this is a bit less preferable unless you're trying to reduce disk usage on the ingesters.

[andreimiclea99]

@dimitarvdimitrov increasing blocks_storage.tsdb.retention_period to 12h it did the trick, thank you.

[codechirag123]

Does the blocks_storage.tsdb.retention_period config means that the data will be stored locally for more amount of time?
We have stopped local retention for mimir components by disabling persistent volumes, as we store them in S3.
Is there any way by which we can increase the efficiency of store_gateway to discover the latest data from S3??

[dimitarvdimitrov]

@jhalterman and @56quarters have been working on this theme recently. Jonathan, Nick, can you share some advice here?

[jhalterman]

Does the blocks_storage.tsdb.retention_period config means that the data will be stored locally for more amount of time?

Yes, this influences how long data is stored locally on ingesters. While we haven't run with a lower retention_period, we have run with more recent data being queried from store gateways rather than ingesters, by setting, for example, on queriers:

'querier.query-store-after': '5h',
'querier.query-ingesters-within': '6h',

And on store gateways:

'blocks-storage.bucket-store.ignore-blocks-within': '4h',  

To help new blocks be discovered and compacted more quickly with the above settings, we've used these settings on compactors:

'compactor.compaction-interval': '15m',
'compactor.first-level-compaction-wait-period': '20m',
'compactor.cleanup-interval': '10m',

And these on queriers and store-gateways:

'blocks-storage.bucket-store.ignore-deletion-marks-delay': '30m',
'blocks-storage.bucket-store.metadata-cache.bucket-index-content-ttl': '3m',
'blocks-storage.bucket-store.sync-interval': '5m',

The main caveat with shifting query load from ingesters to store gateways is that out of order data/blocks in particular may end up being queried from multiple store gateways before it's compacted, and querying this redundant data has a cost. In general you want to be sure that compaction is happening in a timely manner. The current Mimir has a cortex_bucket_store_series_blocks_queried_sum that you can use to keep an eye on whether non-compacted (level=1, source=receive) data is being queried via store gateways.

[felipefrx]

Does the blocks_storage.tsdb.retention_period config means that the data will be stored locally for more amount of time? We have stopped local retention for mimir components by disabling persistent volumes, as we store them in S3. Is there any way by which we can increase the efficiency of store_gateway to discover the latest data from S3??

In my case I noticed that the ingester was restarting. I also disabled local storage, I'm only using s3. It could be the case...

[56quarters]

@andreimiclea99 had you modified that setting previously? Meaning it wasn't the default that we include with the Helm chart, right?

[andreimiclea99]

@56quarters in the values file isn't mentioned, but according to the grafana mimir documentation the default is [retention_period: <duration> | default = 24h] . I modified it to have smaller local volumes.

[macrokernel]

The shorter blocks_storage.tsdb.retention_period is likely originated from the official example config: https://github.com/grafana/mimir/blob/main/development/mimir-monolithic-mode/config/mimir.yaml.

[pstibrany]

Hello. That is not "official example config". It's a configuration to run Mimir locally in monolithic mode for development purposes.

[pstibrany]

Example configuration files can be found at https://github.com/grafana/mimir/tree/main/docs/configurations

[macrokernel]

True, thanks for the correction. I meant that it is published in the official GitHub repo and is referenced in official documentation.