[FEATURE] Enrich the Operational Metrics in Query Insights

[ansjcy]

Is your feature request related to a problem?

Currently, we have error logs for Query Insights, but we need to introduce more operational metrics so we can create more alarms (e.g., memory usage, queue sizes) for Query Insights features.

What solution would you like?

Introduce additional metrics, potentially using the OpenTelemetry metric framework. The metrics should comprehensively cover:

• Size information for top queries priority queues
• Size information for multiple blocking queues
• Exported data size
• Errors and exceptions

[dblock]

[Catch All Triage, attendees 1, 2, 3, 4, 5, 6, 7]

[ansjcy]

We need to pick this up for 2.18.

[ansjcy]

We want to provide a health_stats API to expose the important metrics in query insights like queue sizes. Also, we need to emit metrics when error happens (possibly through oTel).