migrating alertmanager crds from local prometheus?

[clux]

Currently, we have a ton of PrometheusRule objects to control the behaviour of Alertmanager.

If we were to migrate to grafana cloud, is there a recommended way to config manage our alerts?

(The grafana cloud alerting doc and api docs are pretty sparse.)

I feel like ideally the CRDs could be supported without having a prometheus/alertmanager installation and instead be picked up by a small kubernetes controller that reconciles against the grafana api (perhaps embedded in one of the agents). This way people can migrate seamlessly without having to rewrite their existing alerts.

[gotjosh]

While the concept of a CRD sounds reasonable. We (at Grafana) have a preference to do it at a version control level. We've built cortextool and a corresponding github action as a way to interact with Grafana Cloud Rules and Alerts programmatically.

We feel that these primitives provide a good foundation for others to self-serve but as always feedback is pretty much welcome.

Our docs cover some example interaction but again feedback is very much appreciated!

[rfratto]

Long term I'd really like to be able to support adding Prometheus rules to the Agent's config file in the same way they're defined for Prometheus. This would pair really nicely alongside reading PrometheusRule CRDs from an Operator, which we're planning on building in the next few months.

[rfratto]

I'd still like this but don't have time to implement it myself. I've opened #523 for now and labeled it as help wanted.

(cc @gotjosh, I remember you expressing interest in having this)