Setup MCOA dashboards and scrape configs

[thibaultmg]

This PR refactors the Grafana dashboards organisation to improve its maintenance and adds configuration for MCOA metrics collection. More precisely:

• It encapsulates coherent groups of dashboards in a common directory.
• Each directory contains the list of metrics to be collected for making them work as well as the prometheus rules to be deployed on the spokes.
• Some CI scripts to check alignment between the metrics needed for the dashboards and the collected ones.
• For MCOA, the whole list of collected metrics has been reviewed and optimised. This required changing some queries in dashboards. When this is the case, I have duplicated the whole directory for MCOA.
• Specific resources for MCOA (scrapeConfigs and prometheus rules) are only deployed when MCOA is activated.
• Dashboards that are not supported by MCOA have their title being suffixed with the string "DEPRECATED" when MCOA is active. MCOA specific dashboards are always deployed with the directories being suffixed with the string "MCOA".

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: thibaultmg

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [thibaultmg]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[thibaultmg]

/retest

[berenss]

OCP 3.11 dashboards can be deprecated and removed. The documentation indicated deprecate with ACM 2.9, over 3 releases ago.

[jacobbaungard]

This looks pretty good to me.

Each directory contains the list of metrics to be collected for making them work as well as the prometheus rules to be deployed on the spokes.

Where do I find this metrics list? I don't seem to see it, but might be I've just missed it somehow.

[thibaultmg]

Where do I find this metrics list? I don't seem to see it, but might be I've just missed it somehow.

I mean that each directory encapsulates: the dashboards, the scrapeConfigs (containing the metrics list) and the prometheusRules. So this is encapsulated for MCOA. But for the current setup, nothing changes. All metrics to be collected are defined in a single allowList configmap.
I am setting up a demo env for the review.

[sonarqubecloud]

Quality Gate failed

Failed conditions
27.6% Coverage on New Code (required ≥ 70%)
C Security Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

[openshift-ci]

@thibaultmg: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/sonarcloud	7c5d62e	link	false	/test sonarcloud
ci/prow/test-e2e	7c5d62e	link	true	/test test-e2e

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.