Can a download rate from long-term s3 object storage be managed when compacting data?

[bobykus31]

We use sefl-hosted s3 object storage for long-stored metrics. Recently we found the storage overloaded when the download rate exceeds 200MB/sec. A higher download rate leads to maxing out CPU usage.

Unfortunately reducing parallelism did not help much. Or at least the options

--wait --wait-interval=1m --block-viewer.global.sync-block-timeout=30m --consistency-delay=2h --log.level=debug --block-files-concurrency=48 --block-meta-fetch-concurrency=48 --compact.blocks-fetch-concurrency=2 --compact.concurrency=8 --downsample.concurrency=8 --hash-func=SHA256 --compact.enable-vertical-compaction

Is there a proper way to limit thanos compactors download rate with its settings and still have data compacted/downsampled?

[yeya24]

Have you tried lowering --compact.concurrency=8 and --downsample.concurrency=8?

[bobykus31]

Oh, I thought this settings for processing data already downloaded to the --data-dir, thank you for the tip, I'll try it definitely.

[bobykus31]

So, running with --compact.concurrency=4 and --downsample.concurrency=4 does not make any difference. When concurrency and other related settings are 1, the rate is acceptable (up to 200Mb/s). Now the question is if it can catch up with incoming data. But of course, this is beyond this topic. Thank you for your help, @yeya24! You are the best, as always ;)

[yeya24]

Other than this one, I see you already tuned block-files-concurrency and compact.blocks-fetch-concurrency. Lowering these two should help as well.