*: add concurrently processing for individual metrics and introduce some flags

[zimulala]

What problem does this PR solve?

The current bottleneck in diag data pulling lies in the long export time or direct failure of metrics with the largest storage footprint.

What is changed and how it works?

• To address potential failures in exporting metrics with large data volumes:
  • When a request retrieves too many samples or encounters Prometheus OOM issues, mitigation strategies include:
    • Reducing the time range for each query request. The current minimum value is 1 minute (this setting works for metrics with less than 8 million series).
  • To address the issue of excessive concurrency (currently default is 5):
    • Set an upper limit, not exceeding Prometheus’s default value of 20.
  • Optimized connection pool management (use default http.Client)
    • By customizing the connection pool size and layered timeouts, blocked connections are prevented from affecting requests, improving concurrency stability.
• When a single metric has a large data volume, it can cause slow execution:
  • Enable parallel processing for pulling data of individual metrics, although this may lead to OOM issues.
• Regarding OOM problems caused by pulling top-heavy metrics:
  • When the metrics-low-priority flag is set, separate the pulling of top-heavy metrics from normal metrics. This ensures that normal metric pulling is unaffected, and attempts can still be made to retrieve top-heavy metrics.
    • If pulling top-heavy metrics fails, consider reducing concurrency and decreasing the time range of each request.
    • This may result in very slow data retrieval.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)

• No code

Code changes

• Has exported function/method change
• Has exported variable/fields change
• Has interface methods change
• Has persistent data change

Side effects

• Possible performance regression
• Increased code complexity
• Breaking backward compatibility

Related changes

• Need to cherry-pick to the release branch
• Need to update the documentation

[CLAassistant]

All committers have signed the CLA.

[zimulala]

This needs to be considered: whether to separate the data pulling of high-volume metrics from regular metrics by default.

[zimulala]

PTAL @XuHuaiyu

[zimulala]

PTAL @XuHuaiyu

[zimulala]

PTAL @XuHuaiyu

[zimulala]

The static-tests issue will be fixed in #485

[zimulala]

PTAL @XuHuaiyu