store-gateway > chunks-cache bandwidth usage

[hobbsh]

Hi,

I have noticed that our store-gateways are transmitting/receiving a lot of data from chunks-cache (maybe other places) with very low query load and seemingly non-stop. We have ~2 million active series and bandwidth usage across these two components is 150MBs roughly for Tx/Rx. First off, I have not been able to find if any compression settings are available here, which would help with our bandwidth usage, so I'm wondering if there's a configuration setting for that (given that store-gateway is interacting with S3/Memcached, it seems maybe not). Additionally, could we be missing a configuration setting that would limit network behavior?

I am unsure what additional things to investigate at this point so would appreciate any help. Thanks!

[56quarters]

I have noticed that our store-gateways are transmitting/receiving a lot of data from chunks-cache (maybe other places) with very low query load and seemingly non-stop

Note that recording rules can also result in store-gateways having to do work (anything that falls outside of the range ingesters are responsible for, 12h by default). The chunks cache will definitely be the highest bandwidth thing that store-gateways talk to in most cases.

First off, I have not been able to find if any compression settings are available here, which would help with our bandwidth usage, so I'm wondering if there's a configuration setting for that (given that store-gateway is interacting with S3/Memcached, it seems maybe not)

There's no tuning for the format or encoding of data stored in the chunks cache. The store-gateways fetch TSDB samples from the chunks cache so compression wouldn't really help very much here since the samples are already pretty well compressed.

I am unsure what additional things to investigate at this point so would appreciate any help. Thanks!

Hopefully you're running the caches in the same AZ as the store-gateways (or within your own DC) so this traffic is "free"? I'd make sure that the caches are scaled enough so that they usually aren't evicting anything and causing data to need to be fetched from object storage and stored to the cache over and over. TSDB data is basically immutable once it's in object storage so it caches very well. We usually see close to 99% hit ratio on the chunks cache internally. You can confirm the hit ratio from Mimir's perspective using the Reads dashboard from the Mimir mixin and from Memcached's perspective using the Memcached mixin.

[dimitarvdimitrov]

one last thing you can configure is -blocks-storage.bucket-store.partitioner-max-gap-bytes (default 524288). The store-gateway extends the object ranges it needs from object store (aka data touched) and tries to merge them if they are at most partitioner-max-gap-bytes apart, so that it does fewer requests to the object store. Reducing partitioner-max-gap-bytes should reduce the volume of bytes fetched from object store (and cache) at the cost of increasing the number of requests.

[hobbsh]

I appreciate the detailed reply @56quarters! What you said makes sense and confirmed my suspicions. We are unfortunately deploying the store-gateway (and other Mimir) with zone awareness so it looks like we should look into changing this. Mimir bandwidth usage is definitely one of the bigger costs associated as you know and I'm not sure the ROI is there when deploying with zone awareness. At least the ingesters can use compression which helps. The cache evictions is an interesting theory and I can see how under-resourcing could cause an issue like this, where the cache is populated endlessly. I see our chunks-cache instances are using a pretty small amount of memory (anywhere from 110MB to 250MB). I will see if increasing memory and memcache memory settings will help.

@dimitarvdimitrov I will look into that setting, thank you!