Operator Level 4 Capability Level - Deep Insights: Monitoring and Alerting

[rm3l]

See https://sdk.operatorframework.io/docs/overview/operator-capabilities/#level-4---deep-insights

Goal
Setup full monitoring and alerting for your operand. All resources such as Prometheus rules (alerts) and Grafana dashboards should be created by the operator when the operand CR is instantiated.

TODO

• Add ability in CRD to create a ServiceMonitor resource (follow-up to Integration with OpenShift logging and Monitoring #180)
• Implement Prometheus metrics in RHDH Operator for Backstage CR reconciliation failure/success
• Implement Grafana dashboards to monitor a) Whether the operator is up and running as well as how long it has been running, b) memory & CPU consumption by the operator
• Implement alerts so that when the operator is down, certain actions get triggered, eg, a notification gets sent to the user's slack channel, a Jira ticket is created, etc.

[gazarenkov]

Do we also consider upstream (vanilla K8s) here or Openshift only?

[rm3l]

Do we also consider upstream (vanilla K8s) here or Openshift only?

Not only OpenShift, I think. It should work on both.