Operator attempting to apply namepace scoped GrafanaNotificationPolicy to all Grafanas

[marpears]

Describe the bug
When multiple Grafana instances exist in the same cluster and a GrafanaNotificationPolicy is created, the operator attempts to apply it to all Grafana instances

Version
5.14.0

To Reproduce
Steps to reproduce the behavior:

1. Create Grafana instances in different namespaces in the same cluster
2. Create a GrafanaNotificationPolicy
3. Run oc describe GrafanaNotificationPolicy <> and observe the attempts to apply the GrafanaNotificationPolicy to other Grafana instances. Note in the example below, there are 8 Grafanas in different namespaces. The operator is attempting to apply the
   GrafanaNotificationPolicy to all 8 of them and is only succeeding in one.

Status:
  Conditions:
    Last Transition Time:  2024-10-30T16:00:19Z
    Message:               Notification Policy failed to be applied for 7 out of 8 instances. Errors:
- hcp-monitoring/grafana: applying notification policy: [PUT /v1/provisioning/policies][400] putPolicyTreeBadRequest {"message":"invalid object specification: receiver 'test' does not exist"}
- il-core-monitoring/grafana: applying notification policy: [PUT /v1/provisioning/policies][400] putPolicyTreeBadRequest {"message":"invalid object specification: receiver 'test' does not exist"}
- il-vest-monitoring/grafana: applying notification policy: [PUT /v1/provisioning/policies][400] putPolicyTreeBadRequest {"message":"invalid object specification: receiver 'test' does not exist"}
- testapp-01-monitoring/grafana: applying notification policy: [PUT /v1/provisioning/policies][400] putPolicyTreeBadRequest {"message":"invalid object specification: receiver 'test' does not exist"}
- mytestapp-02-monitoring/grafana: applying notification policy: [PUT /v1/provisioning/policies][400] putPolicyTreeBadRequest {"message":"invalid object specification: receiver 'test' does not exist"}
- testapp06-monitoring/grafana: applying notification policy: [PUT /v1/provisioning/policies][400] putPolicyTreeBadRequest {"message":"invalid object specification: receiver 'test' does not exist"}
- testval0-monitoring/grafana: applying notification policy: [PUT /v1/provisioning/policies][400] putPolicyTreeBadRequest {"message":"invalid object specification: receiver 'test' does not exist"}
    Observed Generation:  2
    Reason:               ApplyFailed
    Status:               False
    Type:                 NotificationPolicySynchronized
Events:                   <none>

Expected behavior
The GrafanaNotificationPolicy should only be applied to the Grafana in the same namespace

Suspect component/Location where the bug might be occurring
unknown

Runtime (please complete the following information):

• OS:Linux
• Grafana Operator Version 5.14.0
• Environment: Openshift 4.16.17
• Deployment type: Openshift OLM

Additional information
Example GrafanaNotificationPolicy

---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaNotificationPolicy
metadata:
  name: test
  namespace: testapp-monitoring
spec:
  instanceSelector:
    matchLabels:
      dashboards: "grafana"
  route:
    receiver: grafana-default-email
    routes:
      - receiver: test
        object_matchers:
          - - type
            - =
            - test

[theSuess]

Thanks for the report! I'll try to reproduce the issue this week and provide a fix!

[marpears]

It seems this issue is not limited to GrafanaNotificationPolicies. For example, I have 2 Grafana instances deployed by the operator, each in different namespaces. Both Grafana instances have the same label.

In one of those namespaces I deploy a GrafanaDashboard which matches the label of the Grafana in the instanceSelector. Although this label matches both Grafana instances, I would only expect the dashboard to be applied to the Grafana which exists in the same namespace.

The .status field of the GrafanaDashboard says the "Dashboard was successfully applied to 2 instances" but it should only have applied to the Grafana that exists in the same namespace.

status:
  conditions:
    - lastTransitionTime: '2024-11-07T17:19:37Z'
      message: Dashboard was successfully applied to 2 instances
      observedGeneration: 1
      reason: ApplySuccessful
      status: 'True'
      type: DashboardSynchronized
  hash: fa829f4b4f5657900961b0f5952f40319de9673388858c48e69af293f9853ca6
  lastResync: '2024-11-07T17:24:38Z'
  uid: f90b4fa2a99d1861baa28c7fb96abb2ecb61be88

The Grafana operator pod log confirms it has identified 2 matching instances

2024-11-07T17:29:39Z    INFO    GrafanaDashboardReconciler      found matching Grafana instances for dashboard  {"controller": "grafanadashboard", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDashboard", "GrafanaDashboard": {"name":"my-dashboard","namespace":"test-monitoring"}, "namespace": "test-monitoring", "name": "my-dashboard", "reconcileID": "5209bbd8-5fd2-48dd-adf0-6bccff1ab60a", "count": 2

If I check the Grafana instance that exists in the other namespace, the dashboard has not been created.