Write timeouts panel is not filtered by the scheduling group

[piodul]

scylla-version=2024.1
monitoring-version=4.6.2,4.8.3
dashboard=detailed-2024-1

The "Write Timeouts/Seconds" panel's Prometheus query is:

$func(rate(scylla_storage_proxy_coordinator_write_timeouts{instance=~"[[node]]",cluster=~"$cluster|$^", dc=~"$dc", shard=~"[[shard]]"}[1m])) by ([[by]])

It does not filter by the scheduling group name. Some writes are asynchronous and Scylla-initiated (e.g. hinted handoff writes) and do not have a direct impact on the user workload; the current query includes them all together, which might confuse the user and give them a perception that their writes are timing out.

I believe that this panel should filter by scheduling group name, considering that many panels on the same dashboard do filter by it.

[mykaul]

monitoring-version=4.6.2

This is an old version.

[piodul]

monitoring-version=4.6.2

This is an old version.

I didn't see any issues about it being fixed, so I created an issue. I can check if the problem is also present on the newest version.

[piodul]

I confirm that the issue is also present on 4.8.3. Although the Prometheus query is different:

topk([[topk]], $func(rate(scylla_storage_proxy_coordinator_write_timeouts{instance=~"[[node]]",cluster="$cluster", dc=~"$dc", shard=~"[[shard]]"}[$__rate_interval])) by ([[by]])) or on ([[by]]) bottomk([[bottomk]], $func(rate(scylla_storage_proxy_coordinator_write_timeouts{instance=~"[[node]]",cluster="$cluster", dc=~"$dc", shard=~"[[shard]]"}[$__rate_interval])) by ([[by]]))

it also doesn't seem to filter by the scheduling group.