[victoria-metrics-k8s-stack] bug: victoriametrics dashboards dont apply the multicluster setting

[bmack94]

Chart name and version
chart: victoria-metrics-k8s-stack
version: v0.55.0

Describe the bug
All kubernetes clusters respect the setting but the victoriametrics dashboards dont:

grafana:
  sidecar:
    dashboards:
      searchNamespace: ALL
      multicluster: true

The value is looked up but will never applied. Example can be found here:
https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-k8s-stack/files/dashboards/generated/victoriametrics-vmalert.yaml#L3

[bmack94]

I just checked the source code of the k8s dashboards and victoriametrics dashboards.
Looks like cluster is a default variable and setting in the k8s dashboards which just gets hidden when multicluster is false.
So i think the upstream dashboards of victoriametrics need to be adjusted if this feature is wanted

[AndrewChubatiuk]

right, feel free to create pull request

[hagen1778]

Hello @bmack94 @AndrewChubatiuk!

The upstream dashboards don't expect cluster label to be present. They rely only on other required labels like job and instance, or labels that exposed by processes itself.

cluster label is environment specific and will be useless for installations which don't use official helm charts.

Workarounds are the following:

1. Use ad-hoc variable for extra filtering. This variable is already available in most of dashboards.
2. Inject the variable and the filter on the dashboards generation stage in helm-chart.

Ideal solution would be to have a mixin or any other form of a dashboard definition that could be parametrized and used for generating the end dashboard. But I am not aware of reliable engine that is compatible with recent Grafana versions to do this. If you have suggestions how to do it reliably - I will be happy to migrate official dashboards to such an engine and generate them on demand.