Update Spegel Grafana dashboard with Spegel-specific metrics

[AhmedTremo]

Why updated Grafana dashboard?

• The old dashboard included general Kubernetes metrics such as node statuses, running containers, pods, Kubelet operations, which was making it hard to follow Spegel-specific metrics.
• The old dashboard wasn't function properly and didn't include time-series panels to monitor change over time.

By concentrating on Spegel-specific metrics, users can more effectively monitor and troubleshoot Spegel without clutter from other Kubernetes metrics (I think they should have their own dashboard).

This is a view of the new dashboard in action. Let me know your thoughts

[phillebaba]

Thanks for the contribution @AhmedTremo.

@onedr0p just wanted to check to see if you have any opinions about these changes.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

see 16 files with indirect coverage changes

[onedr0p]

@AhmedTremo

I don't see spegel_mirror_requests_total or spegel_resolve_duration_seconds_bucket being exported by spegel. This might be on my end, but I do have some spegel metrics being ingested so not sure where those are.

I also don't have any data graphed in the http_request_duration_seconds_bucket or http_response_size_bytes_bucket panels.

[AhmedTremo]

@onedr0p

I re-checked pkg/metrics/metrics.go and these metrics are implemented and should be exportable, are you sure Prometheus is correctly configured?

The images I shared were for Spegel running v0.0.23 but I don't think there were any metrics changes since then, (correct me if I'm wrong @phillebaba)

[onedr0p]

As I said it's probably on my end, but I do have other spegel metrics being ingested by prometheus so I'm stumped as to why those two are missing.

[AhmedTremo]

Yes, I think it's worth investigating. one thing to mention is I set 30-second window for calculating rate which might be shorter than scraping interval for some folks, and I do it for the metrics you mentioned

spegel_mirror_requests_total
spegel_resolve_duration_seconds_bucket
http_request_duration_seconds_bucket
http_response_size_bytes_bucket

So, I will update it to 5m just to be safe. Can you check if updating the window shows the metrics for you? @onedr0p

[onedr0p]

Now I am getting data in the http_request_duration_seconds_bucket and http_response_size_bytes_bucket panels. Still no dice for the other two.

[AhmedTremo]

I think we would need someone else to try it and validate to be sure