💡 [Feature] monitoring: Add a second Prometheus to scrap the first Prometheus to keep stats for long term at a lower granularity

[tlvu]

Description

Currently component ./components/monitoring scrap every 5 minutes and keep the stats for 90 days.

If we want longer stats for high level trend keeping (ex: 5 years), we could use a second Prometheus to scrap the first Prometheus daily (averaging the values) so we can keep longer stats without consuming too much disk space.

This second Prometheus should be on a different machine than PAVICS so when PAVICS is down, we can still access those longterm stats.

To explore the federation feature of Prometheus whether it is simpler to implement/deploy or gives better longterm stats results.

[huard]

I suspect that it is possible to simply add new rules that aggregate metrics at a lower frequency. That is, I don't think we need a second instance.

https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/

[fmigneault]

I would also prefer a lower frequency logging approach than having a duplicate instance.

[tlvu]

The reason I suggested a second Prometheus is because the retention policy is instance wide and not per metric, see

birdhouse-deploy/birdhouse/components/monitoring/docker-compose-extra.yml

Line 46 in 749c230

- --storage.tsdb.retention.time=90d

However, this has been a while ago. Maybe newer version of Prometheus allow for per metric data retention. To explore.

So yes we can lower the polling frequency, but if we can not increase the retention duration, then we still do not have long term stats, which is our ultimate goal.

Also, I suggested a second instance of Prometheus on a different machine. So it's not really "duplicated" because it's not the same role:

• it can still provide stats even if the real PAVICS host is down (hardware failure, data corruption, ...)
• it can aggregate all other PAVICS hosts (staging, tests, ...)

[huard]

No, you're right, there is one retention policy per instance: https://stackoverflow.com/questions/69630832/how-to-store-data-in-prometheus-with-different-retention-time-per-job-or-targets#:~:text=Prometheus%20doesn't%20support%20multiple,configs%20and%20distinct%20retention%20periods.

[fmigneault]

Ok. If it is a limitation of Prometheus, then let's try with a second one.

[huard]

Post on solutions to this problem, which in the jargon seems to be known as "downsampling":
https://last9.io/blog/downsampling-aggregating-metrics-in-prometheus-practical-strategies-to-manage-cardinality-and-query-performance/

[mishaschwartz]

Instead of a second prometheus to scrape the first we could also use one of the other technologies they recommend for longterm storage (https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage). Both Thanos and M3 seem to be recommended elsewhere.

We could create recording rules for the metrics we care about storing in prometheus and then use an external tool to store those specific metrics over a longer term and query just those metrics over larger time-scales as needed.

[mishaschwartz]

I've been playing around with Thanos. One issue with Thanos is that is stores all metrics from a prometheus instance which can end up being a very large amount of data (even if the data is compacted somewhat: https://thanos.io/tip/thanos/quick-tutorial.md/#compactor).

As discussed here there are potential ways to store only those metrics that we care about which would reduce the amount of additional disk space needed to store the data.

It looks like we're likely going to need to introduce a second prometheus instance even if we also use Thanos so that we can select which metrics we store long term.