Mimir losing cached blocks after node restart #2553
Answered
by
pstibrany
mari-arondeus
asked this question in
Help and support
-
|
Hello all, We're running a bit of a non-traditional setup: Docker Swarm Mode with Mimir services deployed as single replicas on 4 different nodes. Here's the Docker deploy file: version: "3.8"
x-mimir-common: &mimir-common
image: grafana/mimir:2.2.0
networks:
- database
volumes:
- tsdb:/tsdb:rw
environment:
- S3_SECRET=$SECRET
configs:
- source: mimir.conf
target: /etc/mimir/mimir.conf
mode: 0640
uid: "10001"
gid: "10001"
command: -config.file=/etc/mimir/mimir.conf -config.expand-env -auth.multitenancy-enabled=false
dns: 127.0.0.11
logging:
driver: loki
options:
loki-url: "https://172.18.0.1:3100/loki/api/v1/push"
loki-tls-insecure-skip-verify: "true"
services:
mimir-1:
<<: *mimir-common
hostname: mimir-1
deploy:
endpoint_mode: dnsrr
placement:
constraints:
- node.hostname == $NODE1
mimir-2:
<<: *mimir-common
hostname: mimir-2
deploy:
endpoint_mode: dnsrr
placement:
constraints:
- node.hostname == $NODE2
mimir-3:
<<: *mimir-common
hostname: mimir-3
deploy:
endpoint_mode: dnsrr
placement:
constraints:
- node.hostname == $NODE3
mimir-4:
<<: *mimir-common
hostname: mimir-4
deploy:
endpoint_mode: dnsrr
placement:
constraints:
- node.hostname == $NODE4
networks:
database:
external: true
volumes:
tsdb:
configs:
mimir.conf:
external: trueAnd here's our mimir.conf file: server:
http_listen_port: 3200
distributor:
ring:
kvstore:
store: memberlist
memberlist:
abort_if_cluster_join_fails: false
bind_port: 7946
max_join_backoff: 1m
max_join_retries: 10
min_join_backoff: 1s
join_members:
- mimir-1:7946
- mimir-2:7946
- mimir-3:7946
- mimir-4:7946
frontend:
address: ${HOSTNAME}
align_queries_with_step: true
split_queries_by_interval: 48h
parallelize_shardable_queries: true
querier:
max_concurrent: 2048
query_scheduler:
max_outstanding_requests_per_tenant: 2048
ingester:
ring:
kvstore:
store: memberlist
replication_factor: 4
final_sleep: 0s
blocks_storage:
backend: s3
s3:
endpoint: ingress:9000
bucket_name: mimir
access_key_id: mimir
secret_access_key: ${S3_SECRET}
insecure: false
http:
idle_conn_timeout: 90s
response_header_timeout: 0s
insecure_skip_verify: true
tsdb:
dir: /tsdb/mimir-tsdb-ingester
ship_interval: 1m
block_ranges_period: [ 2h ]
retention_period: 3h
bucket_store:
sync_dir: /tsdb/mimir-tsdb-querier
limits:
ingestion_rate: 100000
ingestion_burst_size: 2000000
ingestion_tenant_shard_size: 3
accept_ha_samples: true
max_query_length: 0
max_query_parallelism: 90
compactor_blocks_retention_period: 365d
compactor_split_and_merge_shards: 8
compactor_split_groups: 4
compactor_tenant_shard_size: 3
store_gateway:
sharding_ring:
replication_factor: 4
kvstore:
store: memberlist
compactor:
sharding_ring:
kvstore:
store: memberlistOccasionally a host node will reboot for updates (as expected), and this causes Mimir to lose a few hours of logs prior to the reboot time. So, my question is this: am I missing something obvious? I'm admittedly very new to Mimir and not particularly well-practiced with Prometheus or other related metrics software like Graphite, so I'm very much going off of my experience configuring Loki (which is absolutely FANTASTIC and a breeze to run at scale) and a whole lot of RTFM+trial and error. If anyone has some insight into why this happening and/or how to fix it, I would love to hear. Here's an example of the metrics being dropped: Based on the errors I'm seeing, it looks like the other Mimir nodes are crashing or otherwise rebooting after a single node goes down. Sorry for the long post, but I figured it was best to include as much info as possible! It really seems like Mimir is considering all nodes "failed" if even one fails, but I'm not certain. Any advice is much appreciated! |
Beta Was this translation helpful? Give feedback.
Answered by
pstibrany
Jul 27, 2022
Replies: 1 comment 9 replies
-
|
Hello. Your ingesters use quite short retention period of 3h. However queriers are configured to query data from store only after 12h ( |
Beta Was this translation helpful? Give feedback.
9 replies
-
|
It looks like everything is working great after making the changes suggested above. Thank you all again for your help on this! One last thing: I noticed a lot of log lines every time compactions are happening:
Roughly 3,000 of these log lines are generated at 15-minute intervals. I'm assuming this is the same issues as #1677. |
Beta Was this translation helpful? Give feedback.
-
I don't think it's caused by #1677. I think you have 3,000 partial blocks in the object storage that will never recover. Maybe some leftover from previous Mimir unstable cluster, while experimenting with it? My recommended would be delete all these partial blocks and start clean (and then see if they happen again, they shouldn't). We also recently introduced a new configuration option |
Beta Was this translation helpful? Give feedback.
-
|
Hmm. I'd hate do delete usable data, even if it's stored in partial blocks - do you think there would be any nasty repercussions from simply leaving the partial blocks where they are for a couple of months, at least until that data is no longer wanted? I'm assuming deleting partial blocks would involve a loss of some amount of data. |
Beta Was this translation helpful? Give feedback.
-
|
To better investigate why you're having partial blocks, I would suggest to focus on 1 single block, for example "01G8WXXVKGC9G1GAP19CS9X8TP". I don't know if you're collecting your logs, but could you show me all logs containing "01G8WXXVKGC9G1GAP19CS9X8TP" since when that block was created (either in the ingester or compactor)? |
Beta Was this translation helpful? Give feedback.
-
|
Absolutely! Loki's great for that :) That last warning at the top is deduped, and the timestamps are in EST. The logfmt 'ts' label is in UTC. Please let me know if I can provide any additional info - we store all logs in Grafana Loki + MinIO so I can pull from any time period. Thanks again for your help so far! |
Beta Was this translation helpful? Give feedback.
Answer selected by
mari-arondeus
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hello. Your ingesters use quite short retention period of 3h. However queriers are configured to query data from store only after 12h (
-querier.query-store-after), and query for more recent data from ingesters only. I would suggest to either use longer retention period in ingesters, or decrease-querier.query-store-afterto 3h.