Grafana managed alerts vs Mimir alerts

[kochetov-dmitrij]

Hello, I want my alert rules to be provisioned, like dashboards, and manage silences and notifications in Grafana.

There are 2 options:

1. Use "Grafana managed alerts" and the integrated Alertmanager. But I must create all the alerts via UI, there is no way to provision them using ConfigMaps. Are there any plans to allow it?

2. Use Prometheus/Mimir ruler with provisioning. Can I still use the grafana integrated Alertmanager for sending notifications and managing silences instead of the Prometheus Alertmanager pod coming with Mimir? It looks not so convenient to have 2 Alertmanagers, would prefer to disable the Mimir Alertmanager and use only Grafana integrated.

"Grafana managed alerts" look better overall (except for lack of provisioning). What are the benefits of using Prometheus/Mimir/Loki alerts? A separated ruler for offloading grafana pod, that's all?

Am I missing something, what would you suggest? Thanks

[pracucci]

Hi!

Use "Grafana managed alerts" and the integrated Alertmanager. But I must create all the alerts via UI, there is no way to provision them using ConfigMaps. Are there any plans to allow it?

I don't know. This is more a question for people working on Grafana Alerting. Let me ping @gotjosh for this question.

Use Prometheus/Mimir ruler with provisioning. Can I still use the grafana integrated Alertmanager for sending notifications and managing silences instead of the Prometheus Alertmanager pod coming with Mimir?

@gotjosh can you follow up on this, please?

What are the benefits of using Prometheus/Mimir/Loki alerts? A separated ruler for offloading grafana pod, that's all?

Mimir ruler is horizontally scalable and can take advanced of the query acceleration techniques implemented by Mimir. For example, when the Mimir ruler is configured to use the remote evaluation mode (doc), the rule evaluations can leverage on query sharding too.

[gotjosh]

Use "Grafana managed alerts" and the integrated Alertmanager. But I must create all the alerts via UI, there is no way to provision them using ConfigMaps. Are there any plans to allow it?

Grafana has a concept of "provisioning". As a first step, we built a provisioning API (doc) that allows you to use HTTP calls for provisioning.

This same API is being used to power file-based (this is where mounted config maps come into play) provisioning that is now coming in Grafana 9.1 some of the PRs are already merged (in particular, the one about alerts) in master.

So, in the next few weeks, you should have no problem provisioning everything related to grafana-managed alerting via files.

Can I still use the grafana integrated Alertmanager for sending notifications and managing silences instead of the Prometheus Alertmanager pod coming with Mimir?

I'm not sure I fully understand this question but my assumption here is that you want to use grafana as alert receiver (effectively replacing the prometheus Alertmanager). I have written an extensive explanation on alert generators vs alert receivers in this comment that covers my rationale for not allowing Grafana to receive Alerts.

tl;dr: alerts would be compatible, configuration would be slightly different, but grafana cannot receive alerts - so that answer is no.

"Grafana managed alerts" look better overall (except for lack of provisioning). What are the benefits of using Prometheus/Mimir/Loki alerts? A separated ruler for offloading grafana pod, that's all?

The internals are completely different thus, the reliability guarantees and speed of query execution are completely different as @pracucci mentioned. Grafana alerting is nowhere near as optimised as the Mimir ruler / alertmanager in these factors.

I would always recommend sticking with the Prometheus/Mimir ecosystem if you plan to alert on Prometheus-based data. The true power of Grafana Alerting is being able to alert on non-Prometheus data (or mix and match with prom and non-prom), but reliability is lost in the process as the data being queried (ingesters, queriers) no longer lives next to the thing that queries the data (ruler).

I'm more than happy to suggest an approach but I'll need more information about your setup and how much you care about multi-tenancy.

If you care about multi-tenancy use both the Mimir ruler / alertmanager.

If you don't care about multi-tenancy I would probably start with the Mimir ruler and the vanilla alertmanager.

Provisioning everything via GitOps to the ruler / alertmanager using mimir-tool

Finally, you can view everything within Grafana and play around with things but the configuration is immutable as it gets overridden with GitOps.

[kochetov-dmitrij]

Thank you for your quick and comprehensive answers. Would be great to have a summary of it somewhere in the docs.

Having a centralized alerting system in Grafana would be great, still not clear why you are against making Grafana an "alert receiver". Pls elaborate about losing reliability using Grafana Alerting, is it related to scrape intervals?

My point is that it's not convenient to use Mimir Alertmanager. Notifications in Slack will have a link to the Alertmanager (with a mediocre UI honestly). Then I need to open Grafana, find the dashboard that triggered the alert. Or try to generate links to grafana dashboards using dashboardUid and panelId in the Alertmanager config...

Grafana Alerting just feels better with State history, panel annotations, having a silence button (instead of manually typing labels for new silences). But at the cost of performance.

For now I will stick to Prometheus/Mimir managed alerts. Will check out whether it's worth switching to Grafana Alerting in Grafana 9.1 (with alerts provisioning)