[Feature] Improve Concurrency of Merging Operations in Data and Index Storage

[hanahmily]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

The data and index storage components currently have independent merge scheduling strategies. This approach can lead to many concurrent merging operations—sometimes exceeding 10 simultaneous merges.

When heavy merging tasks occur among these operations, CPU and memory usage can spike significantly, potentially impacting system stability and performance.

To address this issue, I propose introducing a concurrency barrier as a global controller for all merging schedulers. This barrier will limit the number of concurrent merging operations across both data and index storage, effectively reducing the likelihood of resource contention and performance degradation.

Use case

No response

Related issues

No response

Are you willing to submit a pull request to implement this on your own?

• Yes I am willing to submit a pull request on my own!

Code of Conduct

• I agree to follow this project's Code of Conduct