Using one single bucket for different clusters (each with one thanos instance)

[localleon]

Dear all,

we are currently experiencing some issues with our current Thanos setup that consists of the following:

• One global S3 bucket per cluster tier (e.g. all test clusters go into the test bucket)
• Each K8s cluster has it's own kube-prometheus-stack installed + the Bitnami Thanos chart
• Each K8s cluster has a Thanos installation (query, compactor and store) + local thanos sidecar in Prometheus.
• Each K8s cluster tier has the same objectstore config for Thanos, pointing to the tier's s3 bucket

We are experiencing an issue where metrics of different clusters are getting merged e.g. we are seeing different thanos-query instances returning the same long-term metrics. :

• thanos-query.test-cluster1 -> backed by bucket "test" -> same long-term metrics as cluster2
• thanos-query.test-cluster2 -> backed by bucket "test" -> same long-term metrics as cluster1

Installation is done with Thanos 0.38.0 Bitnami Helm Chart. From different discussions (example 1) we always assumed that putting multiple clusters into one single bucket may not be best practice, but should work and metrics should stay unique per cluster. Each cluster has unique "External Labels" and therefore all metrics containing that cluster's label should be available with Thanos query and only these cluster's metrics. Query each unique thanos-instance should result in:

• thanos-query.test-cluster1 -> backed by bucket "test" -> long-term metrics unique to cluster1
• thanos-query.test-cluster2 -> backed by bucket "test" -> long-term metrics unique to cluster2

We are quite unsure on where to search for the issue e.g. where in the configuration there may be an error why the metrics get merged? Would appreciate if anyone could point us into the right direction! If we are taking some false assumptions with the multi-cluster > single bucket approach here, i would also love to get some feedback!

Cheers!

[localleon]

This has been solved by using a "prefix" with an unique clustername in the s3-config